Genetic Engineering Attribution Challenge

We solicited inventive solutions to a crucial problem in genetic engineering: where was this engineered?

BACKGROUND

Every day, genetic engineering techniques are used to solve critical challenges in agriculture, manufacturing, and medicine. However, as the power of genetic engineering increases, so too does the potential for serious negative consequences if the technology is misused. It is often difficult to trace the origins of a genetically engineered product, making it difficult to ensure due credit for its creators – or to hold them accountable.

At altLabs, we’ve developed tools to identify the original source of engineered DNA – a process known as genetic engineering attribution. Better attribution technology would increase transparency and accountability in synthetic biology, while still promoting and rewarding innovation. To further advance the state of the art in attribution technology, we sponsored the Genetic Engineering Attribution Challenge, a data science competition on the DrivenData competition platform.

THE COMPETITION

The competition consisted of two tracks: the Prediction Track and the Innovation Track. Each track had a total prize pool of $30,000. The Prediction Track ran from August 18 to October 19, 2020, while the Innovation Track ran from October 20 to November 1, 2020.

In the Prediction Track, teams competed to identify the lab-of-origin of engineered DNA sequences with the highest possible accuracy. Prizes were awarded to the teams who could achieve the highest accuracy score.

In the Innovation Track, high-performing teams from the Prediction Track were invited to showcase their approaches to a multidisciplinary panel of expert judges. Prizes were awarded to teams who exhibited novel and creative approaches to the problem, or who demonstrated that their algorithms possessed useful properties other than raw accuracy.

THE RESULTS

More than 300 teams from around the world competed in the Challenge. Top accuracy scores quickly exceeded the previous state of the art: given 10 guesses for each sequence, the best teams were able to predict the source lab of an unfamiliar plasmid DNA sequence almost 95% of the time, compared to 85% in the best published model.

In total, prizes were awarded to six winning teams. Two teams won prizes in both tracks, demonstrating both exceptional accuracy and a creative and compelling approach to the problem. Click here to find out more about the winning teams and their models.

Winning teams adopted a variety of different technical approaches, demonstrating the diversity of methods that can be applied to genetic engineering attribution, as well as the potential for new machine learning approaches to further improve on existing tools.

TIMELINE

2020-08-18: Prediction Track Opens
Participants accessed competition data and began making submissions.
2020-10-19: Prediction Track Closes
Top participants invited to submit code for review.
High-scoring teams invited to submit reports to Innovation Track.
2020-11-01: Judging Begins
Innovation track submissions pre-screened for quality, then passed to judging panel.
Judges assessed submissions for originality, technical quality, and engagement with real-world attribution questions.
2020-11-25: Judging Closes
Judges submitted assessments to altLabs.
Innovation Track winners determined based on assessment scores.
2020-12 to 2021-01: Verification
Code from winning teams checked and prepared for release.
Performance of winning submissions checked against an out-of-sample verification set.
Prizes awarded.
2020-01-26: Results Announced
Competition results announced publicly by altLabs, DrivenData, and partners.

PARTICIPATING ORGANIZATIONS

Host:
DrivenData

Partners:
The Johns Hopkins Center for Health Security
The Johns Hopkins University Applied Physics Laboratory
iGEM Safety and Security Program

Frequently Asked Questions

How is this even possible?

There are many different ways of doing genetic engineering, and many different choices the designer of a genetic system can make at each stage. Together, these choices produce a genetic fingerprint that allows sophisticated algorithms to identify the designer. Even in cases where a single lab-of-origin cannot be identified, GEA tools can substantially narrow the field for human decision-makers.

What's a Data Science Challenge?

A data science challenge is a competition to find better solutions to a data-rich problem. Competitions are run through a host organization, which maintains a community of “solvers” looking to hone their technical and analytical problem-solving skills. Through the host’s website, solvers can view challenge information, enter to participate, access the datasets, submit solutions, and receive awards and recognition! Our host for this challenge was DrivenData.

Who could participate?

The competition was open to anyone over the age of 18 at the time of entry, except for residents of certain countries sanctioned by the US Government. Teams from over 70 countries signed up for the competition, and prizes were awarded to competitors from five different countries. Winners included academic researchers, professional data scientists, and machine learning enthusiasts from a variety of backgrounds.

Why did altLabs sponsor this competition?

altLabs is dedicated to the safe and beneficial development and use of emerging technologies. Bioengineering is a rapidly expanding field of technology, with the potential for both major benefits and serious unintended consequences. Genetic engineering attribution technology has the potential to play a key role in making the future of synthetic biology safer and more accountable, without stifling innovation. We sponsored the Genetic Engineering Attribution Challenge because we believe in this vision.

Could you use this technology to distinguish engineered from natural sequences?

While the top algorithms from the competition are highly effective at identifying where genetically engineered sequences were designed, they cannot identify whether a given sequence – or the microbe that contains it – was engineered in the first place. Identifying engineered sequences is a separate technical challenge, known as genetic engineering detection. Genetic engineering detection was not part of the Genetic Engineering Attribution Challenge.

What happens to the models after competition?

To further promote the development and use of attribution technology, all code from winning submissions will be made fully open-source.

Who funded this?

As with altLabs’ other early projects, funding for the Genetic Engineering Attribution Challenge was provided by Open Philanthropy.