Questions and Answers

1. What was the major objective of your project and what was your plan to achieve it?

The major objective of my project was to develop a less computationally expensive model that
can predict the long-term stability of planets in exoplanetary systems. To achieve it, I used early
dynamical features from N-body simulations of diverse planet populations. I used this data to
train, test, and refine a gradient boosting random forest classifier.

a. Was that goal the result of any specific situation, experience, or problem you
encountered?

Due to the nature of orbital dynamics problems, there exists a sort of computational bottleneck; it
is a brutally complicated problem that scientists have worked on for centuries. In the past, such
work would have taken thousands of hours, but current literature has made notable
breakthroughs in accelerating the process, and I believe there is room for research to further
optimize it.

b. Were you trying to solve a problem, answer a question, or test a hypothesis?

Since most models still use a significant amount of data, I wanted to see if I could develop a
high-accuracy classifier with less computational effort. Beyond just the development of this
classifier, this research could tell us more about the signs of orbital instability, system
architectures, and dynamics of planetary systems.

2. What were the major tasks you had to perform in order to complete your project?

To complete this research, I first had to process all of the data. With N-body simulations from
100 different systems, with 10 planets each, integrated to 10 million years, there was nearly 75
Gb of raw data. After I did so, I used current literature and my own undertakings with the data to
determine which dynamical features may be indicative of a planet’s long-term fate. After doing
so, I trained several classifiers on derivations of these features for various timesteps, to gain
insight as to how different time integrations affect accuracy. With this, I was able to make an
informed decision for which timestep to use in my final classifier, to maximize efficiency and
minimize computational power. I then refined the hyperparameters and optimized training
features for my classifier, keeping in mind specific metrics that are helpful in evaluating a
machine learning method (e.g. class probabilities). With the classifier trained and tested, I
analyzed the results for causes for misclassifications, feature importance and correlations,
distributions of probability, and signs of orbital instability.

a. For teams, describe what each member worked on.

N/A

3. What is new or novel about your project?

a. Is there some aspect of your project's objective, or how you achieved it that you haven't
done before?

This entire research project was new to me. I had not previously worked with exoplanets, or
orbital dynamics, or N-body simulations. I just began looking into these things before settling on
this project; the objective and how I achieved it was something I was learning along the way.

b. Is your project's objective, or the way you implemented it, different from anything you
have seen?

There are a few things about my research that are different from previous literature I am familiar
with. For one, the planet populations I used in my research are far more uncommon. Most similar
work uses systems with three or five planets, whereas my systems have ten planets. Additionally,
the dynamical feature distributions (inclination, eccentricity, etc.) of my systems are relatively
more diverse. Furthermore, my classifier uses short-term integration data from the first 41.1
years (≤500 orbits) of the systems, which is significantly less data than most other models using,
despite achieving similar overall accuracies.

c. If you believe your work to be unique in some way, what research have you done to
confirm that it is?

I have thoroughly explored the academic literature on the subject. Additionally, my research
mentor (who works in the field) helped me find resources to contextualize my own work with the
broader research field.

4. What was the most challenging part of completing your project?

The most challenging part of my project was some of the conceptual aspects. A conceptual
framework was difficult for several reasons. For one, research in orbital dynamics of exoplanets,
especially with machine learning, is quite a new, and niche, field. Many of the existing literature
is from a fairly small subset of researchers who have worked on such problems, so it was not
very clear what I should expect from my research and how it might be similar or different from
other models. Furthermore, while my research suggested ideas that were supported by other
literature, it wasn’t quite what the previous researchers had found in developing their own
models. And since the nature of orbital dynamics problems are so complicated, there are many
things one could examine in the research and analysis processes; I ended up looking at things
that are fairly unexplored in the machine learning for orbital dynamics niche, so understanding
the underlying physics in my work, and drawing conclusions from these, was quite new and
something I had to navigate. I had to be really considerate in my work, and not overlook the
small things that could impact the physics behind the systems and my understanding/analysis of
them.

a. What problems did you encounter, and how did you overcome them?

In addition to the above, there were some problems with the data and programming. Because of
the large amount of data I needed to process, my personal computer was less than ideal.
However, due to Los Alamos National Laboratory’s policies, I was unable to get access to their
supercomputers. This also meant I couldn’t generate my own simulations (which is so
computationally expensive, it would be impossible on my computer), which is part of the reason
why I opted to use my mentor’s Mordasini population. Another problem I faced was during the
development of my classifier; existing literature has quite different machine learning practices,
and reconciling the metrics of a ‘good’ classifier (e.g. low learning rate, high probability for
correct classifications, low probability for misclassified planets) with accuracy was an act of
balancing the two, a matter of further research in machine learning and numerous rounds of
refinement.

b. What did you learn from overcoming these problems?

Throughout my research, I learned a lot about the physics behind my work and the field of
exoplanets and orbital dynamics, in addition to data analytics and machine learning. Because
these were all fairly new to me, there was a bit of a learning curve at first. But overcoming the
problems I faced throughout the process is what ultimately taught me the most about these. Of
course, I had support from my mentor, but I often worked to figure out things myself- what went
wrong, how to fix it, what to look at next, etc. It taught me a lot about the experimental, curious,
and resilient process that is research itself, and improved my critical thinking and problem
solving skills. It has made me much more thoughtful and meticulous in the way I approach and
solve problems, and has given me a greater sense of research intuition.

5. If you were going to do this project again, are there any things you would you do differently
the next time?

While I did use a fairly diverse planetary population, training and testing my classifier on
populations with different numbers of planets could have been a more comprehensive approach,
and more representative of the diversity of true exoplanetary systems. Furthermore, I would have
also used additional metrics to evaluate my classifier’s accuracy, such as accuracy, precision, and
recall rates. These are often useful metrics for machine learning methods, and using them would
have made it easier to contextualize the performance of my own classifier relative to existing
methods.

6. Did working on this project give you any ideas for other projects?

The progression of this research project has naturally led to more inquiries that offer the potential
for other work. Notably, I could apply similar concepts to develop a classifier for circumbinary
systems, to better understand their dynamics and drivers of instability. This is an emerging field
of research, and is fairly unexplored so far; it offers a lot of potential to learn about a wider range
of exoplanetary systems and system architectures, as well as how planetary motion and orbital
dynamics vary between single and binary star systems.

7. How did COVID-19 affect the completion of your project?

Due to COVID-19, my entire research experience has been virtual. While this did enable me to
work with my mentor (who is based out of Los Alamos National Laboratory in New Mexico), it
did make completing the project a little more challenging, and limit my research to something
computational so it could be done virtually.