One thing the Personal Genome Project prides itself on is its ethics: the methods of "open consent" stem from seeking an ethical solution to the dilemmas presented by human genome research. Making your personal genome and health data public is still a radical departure from traditional research, and so it is important to us that participants truly understand the expectations and risks of participation. Indeed, the first ten participants (the PGP-10) were selected from individuals who had a master's level degree or equivalent training in genetics. Since then, participation has been opened to almost anyone1 -- provided they demonstrate an understanding of the expectations and risks involved in participation. As part of demonstrating this, prospective participants are asked to complete an "enrollment exam" testing their understanding of basic genetics concepts, human subjects research, PGP protocols, and some of the open-ended risks associated with participation.

Issues with the old exam

The exam experience has been unfortunately difficult for many, it is the single largest hurdle to participation. An analysis of user logs showed me that for every person that fully enrolled in the project, there was one that had given up at the enrollment exam. It wasn't meant to be exclusive, we welcome autodidacts: anyone meeting our minimum requirements1 should be able to join, provided they are willing to learn. Seeking to improve the experience, a couple months ago I decided to look at the user logs for the enrollment exam. What questions were problematic? Why was the exam so painful?

[caption id="attachment_478" align="aligncenter" width="498"] Histogram of PGP enrollment progress, logs from 12/2010 to 12/2011. Nearly as many people gave up at the enrollment exam as made it through to enrollment application submission & participation.[/caption]

One problem was the design. Each module of the exam requires all questions to be answered correctly, and the user is not informed which was incorrect. Because one module contained nine questions, that module was a combinatorial nightmare! 10% of participants retook the module over a dozen times (and note: I was only looking at data from enrolled participants -- people who made it through the whole thing!).

[caption id="" align="aligncenter" width="500"] One in ten participants had to take the "Risks and Benefits" module at least a dozen times to pass the exam! This module had the most questions of any (nine), which was probably part of the problem.[/caption]

So one thing I changed in our updated exam was to reduce the number of questions in a single module: now the highest is six, and most modules have four. Another issue was a little less obvious, but seemed to be an issue based on user behavior and communications: it was unclear for many users how they could learn the correct answers to the questions. An old study guide was linked, but the link wasn't obvious (it appeared in the lower right on exam pages, while taking the test) and the site was difficult for some to navigate.

An updated exam with integrated study guide

To figure out which questions were problematic (or just plain out-of-date), I asked the PGP staff to go through the old exam and rate each question on relevance and clarity. Using this survey data, we hashed out which questions we wanted at one of our weekly meetings. Based on this, I went through and made a new exam: one with an integrated study guide for each module. All questions on this new exam are directly related to the study material, and the study guides can be revisited during the test. Hopefully the new design avoids the frustrating user experience of numerous repeated attempts, trying to find the correct answer combinations.

I hope the new exam is both easier and more educational. The PGP staff reviewed and tweaked the new exam and study material, then we submitted our updated exam for approval with our institutional review board at the start of April, it was approved April 30th, and Ward implemented the changes on the enrollment website last week. A document containing the study guides is available on the Harvard PGP website. It's not just for new recruits: if you're a current participant -- or even a curious onlooker -- reading this study material is a great way to learn more about the PGP!

1PGP minimum requirements include an age requirement (21 years or older) and US citizenship or permanent residency. This second requirement is because legal issues associated with human genome data are changing rapidly; our project is only prepared to handle the regulatory issues for the United States in particular. We hope PGP projects in other countries will grow and provide additional options internationally.