← Forum

Making Human Studies as Reproducible as Open-Source Code

In machine learning, we’re used to picking up where someone else left off. Download a model and a dataset, run the public code, change something, and try an idea.

But when I read an interesting social science experiment and want to change the wording or a condition, I often have to reconstruct the materials, conditions, procedure, and analysis first. Running it with people again also involves ethics review and recruitment. A small change becomes a whole new project.

My advisor recommended David Donoho’s Data Science at the Singularity, which discusses frictionless reproducibility. My takeaway: a field can move quickly when one person’s work becomes the next person’s starting point. I want that for human studies too.

I want an experiment to be something we can share and run like a program, with the materials participants see, the tasks they complete, the procedure, and the statistical analysis all built in. Others can rerun it, change a condition for a pilot, or plug in different agents as participants. Each attempt keeps its materials, procedure, and analysis together, ready for the next person to build on.

That’s why I started building HumanStudy-Hub: working with a community to make human studies reusable and runnable.

Researchers can also build a benchmark [1] within the Hub: keep the experiment fixed, change the model, prompts, or participant settings, then compare with human results. See where the behavior matches, where it doesn’t, and keep adjusting. [1] presents preliminary experiments exploring how these settings affect whether agents reproduce human behavioral effects.

HumanStudy-Hub ↗

References

  1. HumanStudy-Bench: Towards AI Agent Design for Participant Simulation.
  2. David Donoho. Data Science at the Singularity.
  3. HumanStudy-Hub.