Panel data quality: what 300 panelists say about disqualification and pay

We asked 300 active online panelists what they actually think about taking surveys: why they do it, what frustrates them, and what they consider fair pay. The answers are useful. But one fact about this dataset matters more than any of them.

Everyone in it is still here.

These are 300 people who are still taking surveys. The ones who burned out, got screened out one time too many, or decided the math wasn't worth it are not in the file. They already left, or never joined. So what we have is not a picture of "respondents." It's a picture of the people the current system manages to retain — the ones who tolerate it. Read that way, the data stops being a satisfaction survey and starts explaining why sample quality is hard to buy.

1. The sample is a survivor pool

Start with frequency. 79% of these panelists take surveys every day — 63% "a few times a day," another 16% "about once a day." Add the weekly takers and you're at 93%. This is not the general public giving an occasional opinion. It's a population that has organized part of its day around surveys.

Now the tell. The more often someone takes surveys, the more they enjoy them. Average enjoyment (1–5) runs 4.19 for the multiple-times-a-day group, down to 3.6–3.8 for the occasional takers. The heaviest users are the happiest. That is exactly what survivorship looks like: the people who find the experience tolerable take more of it and stay; everyone else drops out of the sample (and the panel) entirely.

It would be easy to conclude they're all just here for the money, and most are — 82% list "earn money / rewards" among their reasons. But not only that: 45% also want to share their opinion, 36% say they enjoy the topics or learn something, 16% want to have an impact. There is real intrinsic motivation in this pool. Hold onto that, because it's the thing low incentives quietly destroy.

2. Two frustrations, and they are not the same problem

We asked which issues frustrate them most (multiple choice). The top of the list:

Frustration	Share
Getting screened out / disqualified after starting	63%
Surveys taking longer than promised	56%
Rewards that feel too low for the time spent	45%
Technical issues (errors, crashes, slow loading)	40%
Being asked the same questions repeatedly	21%

Separately, on a direct question about pay, 57% rate the rewards they usually receive as "low" or "very low."

Pay and disqualification look like one complaint — "this isn't worth it" — but they damage data in completely different ways, and conflating them is how the industry keeps mis-diagnosing the problem.

Low pay doesn't make people leave. It makes them care less. The 57% who feel underpaid are still here, still completing this very questionnaire. Underpayment doesn't show up as exit; it shows up inside the dataset as rushing, straight-lining, and thin open-ends. A panelist put it plainly:

"be respectful of peoples time and you will get better quality data"

Disqualification is what makes people leave. The median panelist here finishes 4–5 of every 10 surveys they start — which means they're screened out of the other 5–6, usually after sinking time in. Nearly a third (28%) finish 3 or fewer out of 10.

Most panelists finish only half the surveys they start Share of 300 panelists, by surveys finished per 10 started

28%

0–3n=85

34%

4–5n=103

23%

6–7n=69

14%

8–10n=43

The open-ends about it don't read like a feature request. They read like a complaint:

"I will get almost to the end of the survey, and then it will kick me out and say that I didn't match."

"Don't show them to me if I am ineligible"

"…they use you until they got their information…"

One makes the data worse. The other shrinks the pool it's drawn from. The industry treats both as "we would need to pay more," and so fixes neither.

3. The two problems compound

Here's something the headline numbers hide. Pay dissatisfaction is not evenly spread — it tracks how often you get screened out.

The more they're screened out, the more underpaid they feel Share who rate their pay "low" or "very low", by surveys finished per 10 started

74%

0–3n=85

61%

4–5n=103

49%

6–7n=69

23%

8–10n=43

Group panelists by how many surveys they actually finish, and the share who feel underpaid falls in a straight line: 74% among those who finish 3 or fewer out of 10, down to 23% among those who finish 8 or more. Same reward levels, more than three times the grievance.

Disqualification doesn't just waste time in the moment. It poisons how fairly paid people feel across everything else they do. The incentive problem and the targeting problem aren't two separate line items you can budget for independently — fix the screen-outs and a chunk of the pay grievance dissolves with it.

4. The people hurt most by screen-outs want one thing

We asked one more question, the one that gets closest to what people actually want fixed: if a platform could keep only one promise to you, which matters most?

The one promise panelists most want kept Single most important promise — share of first choices

Fair pay

42%

You'll qualify

29%

Accurate time

18%

Clear questions

7%

Data transparency

4%

Fair pay and "you'll likely qualify" together account for 71% of first choices. And the second one isn't wanted evenly: among panelists who flagged disqualification as a top frustration, 35% pick "you'll likely qualify" as their single most important promise — versus 18% of everyone else. The people the screen-out machine burns most are, unsurprisingly, the ones who most want it gone.

This isn't only a comfort issue. We asked what would most improve the quality of their own answers, and "fewer screen-outs / better targeting before starting" came second only to higher pay (50% vs 63%), ahead of shorter surveys. Respondents are telling you, directly, that matching them better would make their data better. Some of them describe the fix in almost exactly the terms a sample provider would:

"focus more on matching surveys to my profile that i already set up"

"a more intelligent routing system that matches surveys with the right professionals from the start. Respecting our time by providing high quality."

"Find a way to prequalify people for your surveys so they dont waste time on something they won't qualify for."

5. "Fair" has been quietly redefined downward

Then we asked what a fair reward for a 10-minute survey would be. The median answer was $1.25 — about $7.50 an hour. Nearly half (49%) named a dollar or less, which is six dollars an hour or under. Only one in ten said more than $5.

Sit with that against the 57% who feel underpaid. They feel that way relative to a bar they've already set at roughly minimum wage or below. The anchor itself has drifted down to meet a depressed market. That's the second face of survivorship: it's not just who stays, it's what staying does to your sense of what's normal. Anyone who still thinks ten minutes of their attention is worth $15 isn't in this pool to tell you otherwise — they priced themselves out and left.

For contrast, I ran a quick LinkedIn poll (58 votes, illustrative only) asking a professional, employed audience whether, for a 10-minute survey, no incentive or $0.50 felt more respectful. A narrow majority — 51.7% — chose no incentive. To people whose time is genuinely worth something, a token $0.50 doesn't read as a small payment; it reads as a small insult, and a well-known one: a tiny reward can crowd out the willingness to help that a sincere "no charge" can preserve.

So the two ends of the market point at the same mechanism. Insiders have adapted down to "$1.25 is fair." Outsiders won't engage at $0.50 at all. Both are telling you the current price band selects a narrow, non-representative slice of people — and filters out almost everyone else.

"we are already getting paid very little (Penny's, nickels, etc) the least you could do is just take the answers"

6. Who's missing — and why weighting won't save you

Put the pieces together. The pool over-represents people who take surveys daily, who tolerate a 50–60% screen-out rate, and who have made peace with roughly $1–2 for ten minutes of their time. It under-represents, or entirely excludes, anyone with a higher opportunity cost of time — which is to say a large share of the professionals, higher earners, and B2B decision-makers that research finds hardest to reach.

The standard answer is statistical weighting. But weighting reweights the people you have. It can do nothing about the people who were never in the frame. If your target population systematically declines to join panels at the prices on offer, no weight corrects for their absence — you're scaling up the opinions of whoever was willing to show up.

And there's a quieter cost, which one panelist put more sharply than any methodologist would:

"Stop with the surveys aimed at doctors or people that spends thousands on luxury items. These surveys are done by people that need the money and those two aren't going to do them. You get liars and cheats."

When the people a study actually wants aren't in the pool, the screener doesn't return nothing — it returns whoever was willing to claim the profile to qualify. Misrepresentation isn't a fraud problem layered on top of a quality problem. It's the predictable result of asking a survivor pool to stand in for a population it doesn't contain.

7. What a healthier version looks like

The two problems are connected (Section 3 showed it in the data), so the fix has an order.

Fix the match first. When people are invited mainly to studies they're genuinely likely to qualify for, screen-out rates fall, the experience stops feeling adversarial, and — per these same respondents — answer quality rises. A provider can then promise something more valuable than a few extra cents: if you start, you'll most likely finish. Twenty-nine percent of this pool ranked that promise above everything else, and the people most scarred by screen-outs ranked it highest of all.

Then price the matched session up. Not a blanket CPI hike — a premium for profiled respondents completing studies they were actually selected for. That's what re-engages people whose time is worth more than $1.25 for ten minutes, and it widens the pool back toward the populations weighting can't reach. The economics only work in this order: you can't afford to pay more while you're burning half your invites on screen-outs, and you can't attract better respondents while you're paying for rushed ones.

It's worth saying plainly, because a panelist already did:

"Don't take the participants for granted. The better I'm treated/compensated, the better effort/higher quality my responses will be."

Fixing disqualification is not a respondent-experience nicety. It's a representativeness intervention — and representativeness is the thing your data is supposed to deliver.