We hire 61 contractors through Upwork to write and validate the dataset. We require that they have completed or are currently in a PhD program in their field of expertise and indicate that they are proficient or fluent in English. We preferentially select individuals with high ratings on Upwork.
We estimate an average hourly payment of approximately $95 per hour. The maximum possible hourly rate given our time estimates is $150 per hour.
Question writers receive a $10 base payment for each question, a $20 bonus for each of two expert validators that answers correctly (to incentivize objective questions), a $15 bonus for each of three non-expert validators that answers incorrectly (to incentivize hard questions), and a $30 extra bonus if both expert validators answer correctly and at least 2/3 non-expert validators answer incorrectly (to especially incentivize questions that are both objective and hard).
Non-experts receive a base payment of $10 for attempting to answer the question, and a $30 bonus for each question they answer correctly.