Skip to content

Good survey questions

Statistics and data · level 2, lesson 6 of 6

A question to start

Sam wants a longer recess. He asks everyone in his class:

"Don't you agree that recess is much too short and should be longer?"

Twenty of the 24 children say yes. Sam runs to the principal: "The data proves that almost everyone wants a longer recess!"

Does it? Imagine being asked that question. "Don't you agree ...?" It sounds as if the right answer is yes, and that you would be strange to say no. Some children who were happy with recess may have said yes just to be friendly.

Now imagine Sam's friend asks the same class: "Is recess too short, about right, or too long?" She might get very different answers. The children are the same. The question changed the data.

In this lesson you learn what makes a survey question fair, so that the answers tell you what people really think.

The idea

A survey question has jobs to do

In level 1 you learned to give choices and to make sure everyone answers once. Now we look more closely. A good survey question:

  1. asks about one thing at a time;
  2. is clear, so everyone understands it the same way;
  3. has choices that do not overlap;
  4. has a choice for everyone;
  5. does not push people towards one answer.

Let us look at each one.

One thing at a time

"Do you like apples and bananas?" What does a child who likes apples but not bananas say? Yes? No? There is no right answer, because the question is really two questions stuck together.

Fix: ask two questions. "Do you like apples?" and "Do you like bananas?" Or ask one: "Which do you like better: apples or bananas?"

Clear words

"Do you read a lot?" What is "a lot"? To one child it is a book a week. To another it is a book a day. They might both answer yes, while reading very different amounts.

Fix: ask about something you can count. "How many books did you read last week: 0, 1, 2, 3, or more than 3?" Now everyone means the same thing.

Choices that do not overlap

Here is a question about how long children spend getting to school.

How many minutes does it take you to get to school? 0-5, 5-10, 10-15, 15-20

What does a child who takes exactly 5 minutes choose? 5 is in "0-5" and in "5-10". The choices overlap.

0 5 10 15 20 0-5 5-10 10-15 15-20 5, 10 and 15 are in two choices each

The child might pick either one, so the counts are muddled. Some children who take 10 minutes will be in one group, some in the next.

Fix: make sure each number belongs to only one choice.

0 to 4 minutes, 5 to 9 minutes, 10 to 14 minutes, 15 minutes or more

Now 5 is only in "5 to 9", and 10 is only in "10 to 14". Every child fits in exactly one choice. This is the same rule as for sorting in level 1: every thing in exactly one group.

A choice for everyone

The question above used to end at 20 minutes. What about a child who takes 25 minutes? There was no choice for them. The fix was the last choice: "15 minutes or more". It catches everyone who is left.

For choices that are not numbers, the catch-all choice is "other" or "something else". And sometimes people need a choice like "I don't know" or "I don't have one". A child with no pet should not be forced to pick a favourite pet.

Do not push

A question that pushes people towards one answer is called a leading question. It leads them where the asker wants them to go. Sam's question was leading in two ways:

  • "Don't you agree ...?" makes yes sound like the expected answer.
  • "much too short" tells you what Sam thinks before you answer.

Other kinds of pushing:

  • Loaded words: "Do you like the boring old library or the fun new gym?" The words "boring" and "fun" do the pushing.
  • Only some choices: "Is our school great or amazing?" There is no way to say anything else.

Fix: use plain words and give every side a fair choice. "Is recess too short, about right, or too long?"

Who you ask matters too

A fair question is not enough. Sam could also ask only his friends, who all love soccer at recess. They are not like the whole class. If you want to know what the whole class thinks, ask the whole class, or at least people from every part of it, not just the people who are easiest to ask. You will study this carefully in later levels, where it is called sampling.

And people answer more honestly when nobody is watching, so for a question people might feel shy about, let them answer privately, with a slip of paper or a block in a cup.

The whole survey, start to finish

Now you can run a complete survey with everything from levels 1 and 2:

  1. Question: one thing, clear words, no pushing.
  2. Choices: no overlaps, one for everyone.
  3. Collect: everyone answers once; keep a tally chart.
  4. Table: turn the tallies into numbers, with a total that matches the number of people.
  5. Graph: a bar graph, a pictograph or (for numbers) a line plot.
  6. Say what it shows, and what it does not.

Worked examples

Example 1. Fixing overlapping choices. A survey asks "How old are you? 5-7, 7-9, 9-11." What is wrong, and how do you fix it?

  1. A child who is 7 fits in "5-7" and "7-9". A child who is 9 fits in "7-9" and "9-11". The choices overlap at 7 and at 9.
  2. Also, a child who is 4 or 12 has no choice.
  3. Fix: "How old are you? 6 or younger, 7 or 8, 9 or 10, 11 or older."
  4. Check: 7 is only in "7 or 8". 9 is only in "9 or 10". Every age fits exactly one choice.

Example 2. Fixing a leading question. "Wouldn't it be great to have pizza for the class party?"

  1. "Wouldn't it be great" pushes people to say yes.
  2. It also gives only one food to think about. Other ideas never get a chance.
  3. Fix: "Which food would you like for the class party: pizza, tacos, sandwiches, or something else?"
  4. Now each food gets a fair chance, and nobody is told which answer is expected.

Example 3. Two questions in one, and a missing choice. "What is your favourite sport and team?" with choices "hockey, soccer, basketball".

  1. Two questions in one: sport and team. Split them into "What is your favourite sport?" and (if needed) "What is your favourite team?"
  2. Missing choices: a child who loves swimming has nowhere to go. A child who does not like sports has nowhere to go. Add "other sport" and "I don't have a favourite".
  3. Fixed: "Which sport do you like best: hockey, soccer, basketball, another sport, or I don't have a favourite?"

Try it

  1. What is wrong with this question? "Do you like cats and dogs?"
  2. What is wrong with these choices? "How many hours do you sleep? 6-8, 8-10, 10-12." Write better choices.
  3. Rewrite this leading question: "You don't eat junk food every day, do you?"
  4. Which word pushes people in this question? "Should we keep the messy old art room or get a shiny new one?"
  5. "How much TV do you watch: a little, some, or a lot?" Why is this hard to compare between children? Write a better version.
  6. Mia wants to know the favourite subject of everyone in her school. She asks the 6 children at her lunch table. What is the problem?
  7. Plan a full survey for this question: "What should our class grow in the garden?" Write the question and the choices, and say how you would collect and show the answers.
Answers
  1. It is two questions in one. A child who likes cats but not dogs cannot answer. Ask "Do you like cats?" and "Do you like dogs?" separately, or "Which do you like better: cats or dogs?"
  2. They overlap at 8 and 10: a child who sleeps 8 hours fits two choices. And a child who sleeps 5 or 13 hours has no choice. Better: "fewer than 8 hours, 8 or 9 hours, 10 or 11 hours, 12 hours or more."
  3. "How many days last week did you eat junk food: 0, 1 or 2, 3 or 4, 5 or more?" (The original makes "no" sound like the expected answer, so people who do eat junk food may not admit it.)
  4. "Messy old" and "shiny new" are loaded words. They make the new room sound better before anyone answers. Fairer: "Should we keep the art room as it is, or change it?"
  5. "A little" means different things to different people. Better: "How many hours of TV did you watch yesterday: none, less than 1, 1 to less than 2, 2 or more?" Hours can be compared fairly.
  6. Six friends at one table are not like the whole school. They might all be the same age and like the same things. She should ask children from every class, or everyone.
  7. One plan. Question: "Which would you most like to grow in our class garden: tomatoes, carrots, sunflowers, beans, or something else? Pick one." Collect: each child writes their choice on a slip of paper, so nobody copies; tick each name on the class list so everyone answers once. Tally and table: read the slips out and tally them, then make a table with a total, and check the total equals the number of children. Graph: a bar graph. Say what it shows, for example "beans got the most votes, 9 of 24".

Common mistake

Choices that overlap at the edges, like 5-10 and 10-15. Every number must belong to exactly one choice. Write "5 to 9" and "10 to 14", or "under 10" and "10 or more".

Leading questions. "Don't you agree ...?" and loaded words push people towards one answer, and then the survey measures the question, not what people think. Ask in plain words with fair choices for every side.

Where it leads