My research on AI in social science formally began in 2024. Thanks to Gordon Pennycook and his coauthors' A Practical Guide to Doing Behavioral Research on Fake News and Misinformation [1], and to the Qualitative and Mixed-Methods Research course at HEC Lausanne, I gradually learned how to conduct human studies in a systematic way.
I learned to begin with research questions and theory, choose a research design, formulate testable hypotheses for confirmatory research, and use a priori power analysis to plan the quantitative sample. I learned to design questionnaires and interview guides, to ask questions and follow up, and to pilot materials with the necessary ethics approval and informed consent. After refining the protocol, I learned to preregister before collecting the main confirmatory data, then recruit participants, obtain informed consent, conduct the study, and integrate statistical and qualitative analyses. What began as terms in a methods course gradually became part of my everyday research.
Through my research in 2024 and 2025, I began building my own experience. That work eventually became three papers: on recommendation explanations and AI education [2][3], and cognitive homophily in creation work [4]. In particular, combining experiments with follow-up interviews gave me a more concrete sense of how real human beings feel about AI.
Whether people were learning about AI and sharing their feedback in our workshops, or receiving AI recommendations and assistance on the recommendation or writing platforms I had built, I felt I was in conversation with them. We paid them for their time, and perhaps they approached the experience simply as a task to complete. Still, their trust and distrust, satisfaction and dissatisfaction, felt very real to me.
What saddens me is that even when our procedures meet the relevant requirements and our materials are complete, preserving the data does not preserve the encounter. We can rerun an analysis with the original data and code. We can recruit new participants to test whether a finding holds. Yet neither can fully reconstruct the people and circumstances of that particular moment. Sometimes, the traces they leave end up fixed in rows of data in a GitHub repository.
The GPT-4o that participants once praised as "like a friend" in our experiments is also subject to version changes and service retirement. The same Hello may receive a different response. Fixed prompts and random seeds help control variation, but they cannot guarantee identical results across model versions and execution environments. Even a saved conversation cannot fully bring back the experience of having it. Statistical analysis helps us estimate effects and uncertainty. Yet why a particular person trusted or hesitated, and how the AI came to meet their expectations, can still become blurred when we aggregate their experiences.
In my model evaluation work, we asked agents to complete auctions and other decision-making tasks under different behavioral specifications. I saw their ability to adjust decisions in response to instructions and examples. I also saw how stable their strategies could be: even with interventions, they showed less variation from round to round than humans did. [5] That seeming composure made me reconsider what we call human noise. Those fluctuations may themselves contain human experiences we need to understand.
As these agents become silicon subjects in social simulations, likeness has ceased to be my only concern. I care more about which behaviors and processes we want to align with human behavior, and why. Perhaps insisting on a complete digital twin of a person, or treating human likeness as the sole objective, is a little like looking for fish in a tree. Perhaps we sometimes resemble Narcissus, too intent on finding our own reflection in what we create. But I also think of Nüwa, the goddess who shaped humans from clay in Chinese mythology. Part of the beauty of creation may be that it brings into being something different from its creator.
For all these reasons, I am happy to contribute to HumanStudy-Hub. Our team shares a wish: to help social science researchers preserve human traces from the past and the present, and to retain as much as we can of the rational and emotional traces of each experiment in this AI era. We hope to preserve shareable experimental materials, research procedures, and the thinking behind their design in a structured form, so that other scholars can find, understand, and continue to use them.
The thoughts and experiences of researchers and participants alike deserve to be seen. At a time when AI slop keeps multiplying, this feels like one thing I can still do as an interdisciplinary researcher.
And finally, thank you to everyone who has read this far. Thank you for seeing our work.
References
[1] A Practical Guide to Doing Behavioral Research on Fake News and Misinformation (https://doi.org/10.1525/collabra.25293).
[2] Contextualizing Recommendation Explanations with LLMs: A User Study (https://doi.org/10.1609/icwsm.v20i1.42667).
[3] The Effect of Education in Prompt Engineering: Evidence from Journalists (https://doi.org/10.1609/icwsm.v20i1.42634).
[4] Cognitive Homophily in Human-AI ensembles: Explaining the Quality-Satisfaction Trade-off (https://doi.org/10.2139/ssrn.6924659).
[5] Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making (https://aclanthology.org/2025.emnlp-main.391/).