Framing Effects

Tal August

Online experimentation with volunteers relies on participants’ non-financial motivations to complete a study, such as to altruistically support science or to compare oneself to others. Researchers rely on these motivations to attract study participants and often use incentives, like performance comparisons, to encourage participation. Often, these study incentives are advertised using a slogan (e.g., “What is your thinking style?”). Research on framing effects suggests that advertisement slogans attract people with varying demographics and motivations. Could the slogan advertisements for studies risk attracting only specific users?

We investigated the existence of potential sample biases by measuring how different slogan frames affected which participants self-selected into studies. We collected preferences for different slogan frames and motivations for volunteering from 59 participants on LabintheWild, a volunteer-based online experimentation site. We then explored how these preferences could affect sample size and composition in online studies by deploying these slogans on five LabintheWild studies, measuring how each affected subject recruitment with 2,670 participants.

We found that slogan frames impact recruitment significantly; changing the slogan frame from a ‘supporting science’ frame to a ‘comparing oneself to others’ frame lead to a 9% increase in recruitment for some studies. Additionally, slogans framed as learning more about oneself attract participants significantly more motivated by boredom compared to other slogan frames. We discuss design implications for using frames to improve recruitment and mitigate sources of sample bias in online research with volunteers.