The Trouble with Bots
The Trouble with Bots
Dealing with Bots in Online Survey Research
Lynda Lahti Anderson, PhD, and Sandra Pettingell, PhD
Note: This is the accessible version of a poster session presented at the AAIDD Annual Meeting, June 10-12, 2024, in Louisville, Kentucky.
Background
Online survey research is a convenient way to reach study participants. However, the increase in bots and Internet bad actors can quickly contaminate your data making it useless. This poster summarizes the efforts made to prevent and detect bot survey responses in a national survey developed for caregivers. The survey was distributed via national partners, social media, and professional networks.
Survey Security
- Qualtrics survey security tools were used, including: screening for duplicate responses, CAPTCHA, and bot detection tools.
- Three inclusion questions were used. Participants had to respond positively to one and negatively to one to be included.
- Eyes on data checks for unusual response patterns and illogical responses to open-ended questions.
Results
Category | Number of Responses |
|---|---|
Total responses | 7,032 |
Failed inclusion criteria | 1,384 |
Didn't complete survey | 966 |
-- | |
Screened in survey participants | 4,682 |
Duplicates | 34 |
Failed CAPTCHA | 49 |
Ballot stuffing | 109 |
Bots | 1,403 |
Final study sample | 3,115 |
One-third of survey responses that passed screening questions were likely bots or bad actors.
Six Ways to Foil Bots
- Understand and use the tools in your survey software. (Beware — soft bot detection software isn’t perfect.)
- If you have a defined population, send unique email links.
- Use honeypot and trap questions.
- Use more than one inclusion question.
- If you offer payments, include a link to a separate survey to complete contact information for payment.
- Eyes on data checks – the same response choice, illogical responses to open-ended questions, etc.
Lynda Lahti Anderson, PhD lla@umn.edu
Sandra Pettingell, PhD erick100@umn.edu
The University of Minnesota is an equal opportunity educator and employer. This document is available in alternative formats upon request.
The University of Minnesota stands on Miní Sóta Makhóčhe, the rightful homelands of the Dakhóta Oyáte. We recognize the U.S. did not uphold its end of these land treaties. It is the current and continued displacement of the Dakhóta Oyáte that allows the University to remain today.
Ongoing oppression and discrimination in the United States has led to significant trauma for many people of color, immigrants, people with disabilities and other oppressed persons. At ICI, we affirm our commitment to address systemic racism, ableism and all other inequalities and forms of oppression to ensure inclusive communities.