Questions and Answers

1. What was the major objective of your project and what was your plan to achieve it?

Through my project, I aimed to develop a novel method for codon optimization.

Codon optimization is a strategy for improving the expression of recombinant proteins, which are proteins created in and extracted from host organisms for biomedical use. Recombinant proteins are crucial to modern biology and medicine, and they include vaccines and pharmaceuticals that combat disease and save lives amid disease outbreaks. To manufacture a recombinant protein, scientists must design a recombinant gene encoding the protein. In particular, for each amino acid within the protein, scientists must select one codon encoding that amino acid.

Research demonstrates that codons used frequently in a genome increase translation speeds for individual genes (since cells produce more corresponding tRNA molecules for ribosomes to use). Therefore, common codon optimization techniques remove all rare codons from recombinant genes and replace them with more frequent codons, with the intent of increasing translational efficiency. However, this strategy causes tRNA imbalance and even affects protein folding, since rare codons allow polypeptide chains to fold by momentarily slowing translation at specific moments. In host cells, tRNA imbalance causes toxic metabolic stress; in patients, misfolded therapeutic proteins cause the generation of anti-drug antibodies, which can inhibit patients' natural protein pools and cause dangerous side effects.

As I reviewed the challenges of common codon-optimization techniques, I realized that natural high-expression genes have been tuned by evolution for both efficient translation and safe protein production without misfolding. Deep learning models could learn evolutionary patterns from these genes to understand where rare codons must be used to preserve protein stability. Therefore, in my project, I sought to develop deep-learning models that could emulate natural high-expression genes to improve upon traditional codon optimization techniques.

I achieved this goal by first curating a dataset of highly expressed genes for three common heterologous hosts. Then, I trained multiple, sequence-to-sequence neural networks to predict the codon sequences for these genes from the corresponding amino acid sequences. I tested eight distinct model architectures, performing a systematic comparison of their performance and laying the foundation for future research extending these models. Having trained these models, I can apply them to protein production by inputting the amino-acid sequence for a recombinant protein and generating an optimized codon sequence that emulates the highly expressed genes of a host organism.

       a. Was that goal the result of any specific situation, experience, or problem you encountered?

During the COVID-19 pandemic, I learned about the manufacturing of various vaccines, including mRNA-based and subunit-based ones, and how codon optimization drives the efficient production of such vaccines and other synthetic proteins. However, many optimization algorithms are relatively simple, so they can induce host-cell stress and protein misfolding by failing to consider the significance of individual codons. Based on my experience with deep learning, I understood that a neural network could learn hard-to-detect patterns and structures embedded in the genomes of heterologous hosts. By revising sequences to resemble the high-expression genes of these genomes, a neural network could enable efficient protein production without inducing harmful side effects. Thus, I researched how neural networks could optimize genetic sequences for heterologous expression by learning and emulating the codon-usage patterns embedded in hosts' genomes.

       b. Were you trying to solve a problem, answer a question, or test a hypothesis?

Through my project, I worked to solve the problem of the limitations of commonly used codon optimization tools. Typical strategies for codon optimization replace all rare codons with frequent codons, since rare codons can slow down translation and thus hinder protein production. However, slowdowns in translation are necessary at points where a growing polypeptide chain begins folding into its final protein form. Therefore, removing all rare codons alters protein-folding dynamics, resulting in the production of misfolded proteins that can endanger patients. Furthermore, using frequent codons alone for recombinant expression can exhaust the corresponding tRNA pools. This depletion slows the production of all proteins, causing metabolic stress that damages and can kill host cells. With my research, I sought to solve these limitations in common codon-optimization techniques.

Through my research, I sought to answer three questions about the capabilities of deep learning for codon optimization: can deep learning achieve a balance between efficiency and safety in codon optimization and surpass the protein expression levels achieved by common optimization techniques? Can deep learning extract abstract evolutionary phenomena related to rare-codon usage from natural high-expression genes and reveal factors related to codon usage that researchers have not previously considered? Can a web application provide efficient optimization tools based on deep learning to researchers around the globe to accelerate the design and production of crucial vaccines and pharmaceuticals?

I sought to test multiple hypotheses regarding these questions. I believed that deep learning could successfully learn evolutionary patterns by training on large genomic datasets. I believed that experimentally verification with a real-world recombinant protein could demonstrate that deep learning can significantly improve protein expression.

 

2. What were the major tasks you had to perform in order to complete your project?

After my literature review, I identified deep learning as a solution to the limitations of conventional codon-optimization tools. After training with examples of input and output data, neural networks can comprehend abstract features embedded in large datasets. I recognized that, with this potential, deep learning models could learn the evolutionary phenomena that underlie contextual codon usage in natural genes. Since evolution has tuned natural genes for both translational efficiency and protein stability, I trained my neural networks to predict the codon sequences of highly expressed genes from the corresponding amino-acid sequences.
    
Since different heterologous hosts exhibit different patterns of codon bias, individual deep-learning models must be trained for individual host species. In my research, I created a host-independent pipeline, so researchers can utilize my algorithms to train models for specialized or less common species. To test and demonstrate my pipeline, I chose to train models for three common heterologous hosts: Escherichia coli, yeast (Saccharomyces cerevisiae), and Chinese hamster ovary cells (Cricetulus griseus).
    
To begin building my neural networks, I downloaded genetic sequencing data from the NCBI Assembly database. For each heterologous host, I downloaded all available genomes from the Assembly database in the FASTA file format (a genomic data format). For popular hosts such as E. coli, thousands of such genomes with millions of genes altogether are available. To reduce redundancy in the downloaded data, I performed distance-based greedy clustering to collect sequences with pairwise similarities above 90% together. I stored the centroid sequences, along with their associated metadata, in a new FASTA file. Finally, I validated the remaining sequences to prune sequences that contained ambiguous bases, had an invalid start or stop codons, or were pseudogenes.

To construct my final training dataset, I filtered the remaining sequences for those with high protein expression levels. To predict expression, I used the global codon adaptation index (gCAI) algorithm. I recursively traversed the validated sequences to calculate a weight for every codon according to the gCAI algorithm. Then, to calculate the gCAI for each codon sequence, I calculated the geometric mean of the weights for the sequence's codons. Research demonstrates that gCAI correlates well with protein expression, so I sorted the remaining sequences by their gCAI values. To curate a dataset of standardized, highly expressed genes, I collected the sequences with gCAI values above the 80th percentile.

After curating my training dataset, I considered multiple neural network architecture for translating amino-acid sequences to codon sequences. From my literature review, I identified multiple architectures specialized for sequential data processing, including a one-dimensional convolutional neural network with strides of one, multiple recurrent neural networks (including simple RNNs, GRU networks, and LSTM networks), and a transformer with multiple attention heads.

For each heterologous host, I trained each model using my dataset of highly expressed genes. During training, I offered amino-acid sequences as input and used the corresponding codon sequences as the expected output. The training sequences had varying lengths, ranging from fifty amino acids to thousands, so the training batches contained one sequence each. I used the categorical cross-entropy loss function to evaluate model performance during training, and I trained the models until the categorical accuracy increased negligibly between epochs.

As I trained the models, I tuned all available hyperparameters for each model architecture, experimenting with hundreds of specific neural networks. For each architecture, I tuned the size and number of hidden layers; larger hidden layers offered better performance but sometimes caused overfitting according to the validation metrics. I also experimented with multiple encoding methods for the input amino acids, including one-hot encoding and linear embedding. Finally, I experimented with the optimizer and learning rate for training, and I ultimately chose the Adam optimizer with a learning rate between 0.0001 and 0.00001 for all model architectures.

After training and tuning all the model architectures, I saved the final model for each architecture in the Open Neural Network Exchange (ONNX) file format. I compared the final models according to the average gCAI of the sequences optimized by each model. Since gCAI correlates closely with protein expression, this comparison distinguishes the models by their ability to improve protein production. I found that my stacked LSTM model achieved the highest average gCAI (0.949) on the testing dataset. With a one-sided Wilcoxon signed-rank test, I confirmed that this average was significantly greater than the average of 0.600 for the unoptimized sequences (p < 0.0001).

To confirm the efficacy of my models, I evaluated the GC content of the optimized sequences. The GC content of a genetic sequence equals the proportion of its bases that are either guanine or cytosine. Values for GC content below 30% or above 70% can cause the formation of mRNA secondary structures that hinder translation, hurting protein production. The GC content of all optimized sequences fell within the desired range of 30% to 60%, so my models did not drastically change codon frequencies to affect GC content.

After tuning the models and determining the most performant architecture, I used feature analysis to confirm that my models could recognize evolutionary phenomena that underlie rare-codon usage in highly expressed genes. The most performant model, the stacked LSTM, contained five bidirectional recurrent cells with 128 units each. During training, each unit of each cell learns to identify specific features and sequence properties. As the stacked LSTM model generated predictions for the testing sequences, I captured the output values from every recurrent unit and created output heatmaps. The heatmaps demonstrated that the stacked LSTM had learned multiple, discernible evolutionary phenomena that affect rare-codon usage, such as how many genes begin with rare-codon clusters that aid translation initiation.

To validate the CodOpt models experimentally, I worked with my mentor to express a recombinant protein in Escherichia coli cells, using both an original DNA sequence and a sequence optimized with CodOpt. The protein expressed was pA-Tn5, a fusion protein containing protein A and Tn5 transposase. pA-Tn5 is crucial to CUT&Tag, a high-resolution method for studying protein–DNA interactions.The original DNA sequence for pA-Tn5 was sourced from the plasmid repository Addgene, and the optimized sequence was created with the stacked LSTM. Both the original and optimized sequences were cloned into the same plasmid, and the plasmids were introduced into separate E. coli colonies. After expression and lysing, the pA-Tn5 proteins were isolated with gel electrophoresis and visualized with staining. Visibly, the optimized plasmid achieved significantly greater expression than the unoptimized plasmid. Therefore, the codon sequences optimized with CodOpt can significantly enhance protein expression.

Finally, I built a web application to provide researchers around the globe with the functionality of the CodOpt models. Using the app, researchers developing vaccines and pharmaceuticals can accelerate their work, broadening the impact of these crucial treatments. Researchers can log in to the application and select a host species, such as E. coli or yeast. When a researcher submits an input sequence, my application determines the amino acids for the corresponding protein using the genetic code. Then, my application loads the saved stacked LSTM model with the ONNX Runtime for Python and predicts a codon probability distribution for each amino acid in the sequence. Finally, the application returns the codons with the highest probabilities as the output DNA sequence. Some researchers or organizations may seek to integrate the models into their own pipelines. For this scenario, I built a JSON API, through which developers can request optimized sequences programmatically and forward the output to other applications or databases.

       a. For teams, describe what each member worked on.

My project was not a team project.

 

3. What is new or novel about your project?

My project's application of artificial intelligence, host-independent data pipeline, systematic architecture comparison, and feature analysis are all novel developments not found in previous studies. Only one previous study from my literature review applied deep learning to codon optimization, and that study investigated a single model architecture rather than comparing multiple architectures in parallel. Furthermore, while other studies analyzed only E. coli, I created a host-independent data pipeline that researchers can extend for host species of their choice. Finally, I utilized feature analysis to demonstrate and confirm that neural networks can comprehend evolutionary phenomena that underlie rare codon usage in natural high-expression genes. These novel developments are explained in detail below.

       a. Is there some aspect of your project's objective, or how you achieved it that you haven't done before?

To achieve my project's objective, I built and compared multiple neural networks inspired by natural-language processing.

In previous research projects, I constructed machine-learning models for analyzing tabular data and computer-vision models for analyzing medical imagery. However, I never constructed models based on natural-language processing architectures. Through this project, I learned about multiple new neural-network architectures and their applications for analyzing sequential data, including the capabilities of one-dimensional convolutional networks and the underpinnings of recurrent neural networks and transformers.

Furthermore, in previous projects, I built and tuned individual models to address particular tasks. Although my models' architectures evolved over time, I never created and compared multiple models with distinct architectures in parallel. Through this project, I conducted a systematic comparison of architectures to lay the foundation for future research that builds upon the most successful model.

Finally, although I have trained many neural networks in the past, I never used feature analysis to understand my models' decision-making and the features that my models recognize. Through this project, I used feature analysis to identify discernible patterns learned and recognized by my stacked LSTM model.

       b. Is your project's objective, or the way you implemented it, different from anything you have seen?

During my literature review before beginning my project, I found only one published study that applied artificial intelligence to codon optimization. All other studies and online tools used standard techniques such as removing all rare codons and selecting codons randomly to match genomic codon frequencies.

The host-independent pipeline that I developed is novel. In published studies regarding codon optimization, data processing and analysis is performed almost exclusively for Escherichia coli. Although E. coli is a popular heterologous host, many researchers use other, more specialized species. For example, the Novavax COVID-19 vaccine is manufactured using moth cells, a relatively uncommon host. In my research, I developed a host-independent pipeline, through which researchers can develop optimization models for any host by using genomic sequencing data. Since I implemented the gCAI algorithm (an extension of the commonly used CAI algorithm), my pipeline does not require expression data that may not be available for uncommon host species.

Furthermore, my systematic comparison of neural network architectures for codon optimization is novel. The only published study from my literature review that used deep learning for codon optimization tested a single deep-learning architecture. In my research, I identified eight distinct architectures commonly used for sequence analysis, including the revolutionary transformer architecture that underlies ChatGPT and other modern breakthroughs in AI. I built models based on each architecture, and I tuned all eight models to compare their final performance. With this systematic analysis, I identified the stacked LSTM as the most successful architecture for codon optimization.

After comparing multiple model architectures, I novelly used feature analysis to understand the patterns used by my model when generating codon sequences. Although neural networks have transformed data science, their abilities to learn abstract features remains notoriously difficult for researchers to decipher. Feature analysis, however, can reveal the patterns that a neural network recognizes and utilizes to generate output. My most successful architecture was the stacked LSTM, which contains 5 successive LSTM cells, each with 128 units. By generating heatmaps to visualize the output of each unit, I identified multiple, discernible evolutionary details related to codon usage that my stacked LSTM learned during training, such as the clustering of rare codons at the beginning and ending of many genes.

       c. If you believe your work to be unique in some way, what research have you done to confirm that it is?

I conducted an extensive literature review into studies that apply neural networks to genetic sequence analysis and, specifically, to codon optimization. Although neural networks have gained popularity, I found almost no research employing deep learning for codon optimization, and I found no research that attempted to construct a host-independent pipeline for practical usage by researchers. Furthermore, I found no study that systematically compared multiple architectures with different strengths for sequence analysis. Finally, I found no other studies that deployed such a system through a web application to demonstrate and enable practical application of the models.

 

4. What was the most challenging part of completing your project?

In the course of building my computational pipeline and training my models, I had some challenges with my dataset's size and my models' many configurations. The sequencing data I downloaded for Escherichia coli totalled ten million genes and 12 gigabytes, so I spent time considering numerous methods to reduce the dataset's size for training. Later, as I trained my models, I encountered some difficulties with tuning when, at first, I changed multiple hyperparameters at the same time, obscuring which hyperparameter changes improved my models' performance. These challenges are explained in detail below.

      a. What problems did you encounter, and how did you overcome them?

When programming and testing my data pipeline, my primary hurdle was the vast amount of genetic sequencing data available for curating my training dataset. For Escherichia coli, the most extensively studied model organism, the Assembly database contained over 190,000 entries. Due to the size of this data, I downloaded only the complete genomes available in RefSeq, a database curated by the NCBI to have less redundancy than the general Assembly database. Still, this download consisted of over two thousand genomes with over ten million sequences, totalling almost 12 gigabytes. This dataset was far too large for training a neural network given timing constraints, so I considered techniques for reducing dataset size, such as random sampling.

I sought to filter my dataset of ten million genes to around fifty thousand genes. However, I feared that random sampling would produce a biased sample due to the small sample size. I realized that clustering would allow me to filter the dataset while verifiably extracting a representative sample. I researched multiple sequence analysis tools that supported clustering, including CD-HIT-EST, USEARCH, and VSEARCH, and I finally chose to use VSEARCH for its ease of installation and efficient clustering implementation. Clustering the ten million sequences required almost thirty-five hours of running VSEARCH, but I ultimately acquired a representative sample that was small enough to train my models.

Alongside some challenges when preprocessing my data, I had some difficulty tuning all my model architectures to evaluate and compare them fairly. Since many neural networks exist for sequence analysis, each with specific strengths and weaknesses, I sought to establish which architecture would best succeed at codon optimization. Thus, I constructed eight distinct networks and individually tuned their hyperparameters over multiple rounds of training. Difficulties arose because I initially chose to change multiple hyperparameters at the same time, preventing me from determining which hyperparameter had changed my model's performance. I then chose to change the hyperparameters one by one. Although this approach required more time, I successfully determined the best value for each hyperparameter to create the most performant models overall.

      b. What did you learn from overcoming these problems?

In overcoming my difficulties with data preprocessing, I learned the value of continuously reviewing literature while conducting a project. I conducted a thorough literature review before beginning my project, but I did not consider challenges such as the magnitude of the NCBI data available. By returning to the literature after encountering this challenge, I identified clustering with VSEARCH as a potential solution. With VSEARCH, I attained a small but representative sample of the data I had downloaded for Escherichia coli. Therefore, continuing to review the literature while conducting a project can reveal new and effective solutions to novel challenges.

In overcoming my difficulties with hyperparameter tuning, I learned the importance of documenting and versioning changes when working on a computational project. By consistently documenting the performance of different models as I tuned them and continuously versioning my code, I found ease in organizing changes and determining which exact model architectures yielded the best performance. Therefore, consistent documentation and versioning can help organize and streamline a computational project.

 

5. If you were going to do this project again, are there any things you would you do differently the next time?

If I were beginning my project once more, I would spend more time reviewing literature related to large genomic sequencing datasets. My initial literature review focused on the biological underpinnings of codon optimization and the computational underpinnings of natural-language processing, but I assumed that I would not encounter any hurdles while preprocessing my data. By spending more time reviewing the tools available for processing and filtering large genomic datasets, I could avoid challenges related to the magnitude of the NCBI data available. Furthermore, I would invest earlier in computational resources available through Amazon Web Services (AWS). Through an AWS instance, I was able to run longer algorithms and train larger models than possible on my personal computer.

 

6. Did working on this project give you any ideas for other projects? 

Working on this project has encouraged me to consider many options for future work.

In my research, I sought to optimize codon sequences that encode recombinant proteins. However, genes contain other regions that contribute to translation. For example, all genes begin with promoter sequences that encourage the initiation of translation. In future research, I could build deep-learning models to create optimal promoter sequences by emulating those before natural high-expression genes.

In my research, I created and trained all my neural networks from scratch. To improve performance in future iterations of CodOpt, I could use transfer learning and fine tuning to utilize the knowledge learned by other deep learning systems. For example, the innovative AlphaFold model, created by Google's DeepMind team, can accurately predict protein structure from amino-acid sequences better than any computational system in history. By using initial layers and pretrained weights from the AlphaFold model, my codon-optimization models could utilize the features and patterns that AlphaFold recognizes in amino-acid sequences. Alternatively, I could use output from AlphaFold as another input to my models, giving my models knowledge of where an amino-acid sequence will likely fold.
    
In my research, I successfully built a host-independent pipeline that enables researchers to construct models for uncommon heterologous hosts using only sequencing data. However, in some scenarios, full-genome sequencing may not be not available or obtainable. Transfer learning, which fine-tunes neural networks for new tasks, could solve this data disparity. Future research could use transfer learning to build networks for less common hosts by fine-tuning the ones trained for popular hosts.
    
In my research, I compared a sequence optimized with CodOpt to an unoptimized sequence by collaborating with my mentor to express both in E. coli. The evidence collected from the experiment demonstrates that CodOpt significantly increases recombinant protein expression. Further lab trials could verify this increase for multiple proteins and compare CodOpt to commercial optimization tools on metrics such as expression, protein folding, and protein solubility.

 

7. How did COVID-19 affect the completion of your project?

During the COVID-19 pandemic, I saw researchers around the world collaborate to develop vaccines and other therapeutics that have now saved millions of lives. Seeing the success of these efforts spurred my interest in vaccine and pharmaceutical development. In particular, I focused on the efficiency of vaccine manufacturing, as rapid vaccine design is crucial when responding to epidemics and pandemics such as COVID-19. As I researched the techniques used to produce vaccines, such as recombinant expression in heterologous hosts. I learned that codon optimization can drive efficient vaccine production by improving translational efficiency. I also learned, however, about the drawbacks of common optimization techniques and especially the dangerous consequences of generating misfolded proteins. As I considered how codon optimization (translating an amino-acid sequence to an optimal codon sequence) resembles the task of translating sentences between languages, I recognized that I could apply my skills in deep learning to address the consequences of common codon-optimization algorithms.