A guide to leading questions, with advice on how to be mindful of wording that can cloud a UX test.
In social science research, a leading question encourages a person to give a particular answer. Leading questions are not ideal for research because they introduce bias and influence the way participants respond.
Leading questions are also relevant to courtroom eyewitness testimonies. For example, if an examiner asks a witness whether he was home on the night of the murder, that’s a leading question. The phrasing assumes a murder indeed took place, and leads the witness to answer in a way that directly relates to his home. Hmm, sneaky.
Your job as a UX researcher is to uncover truth and honesty. Your job is to gather user feedback that isn’t coloured by your own hopes or expectations. Your job is to listen, and to be deliberate with your words and actions.
People often tell you things they think you want to hear. It’s driven by natural kindness, which is theoretically very nice! But if your in-depth interview questions lead participants to give inaccurate answers, you might miss important UX insights.
Imagine the following scenario. You meet a research participant, Sally, and introduce yourself as a UX researcher. You say, “Thank you so much, Sally, for agreeing to participate in this research. I’m so excited to get feedback on this app. The team has worked very hard on it, and we know you’re going to love it.” Then, imagine you conclude your research session by asking Sally if there was anything she did not like about the app. She answers, “No, not really!”
Notice a problem? Sally went into this UX test already thinking a few things. First, you’ve probably given her high expectations for your app, and second, she doesn’t want to disappoint you or hurt your feelings!
Although it’s great to be polite and personable, it’s not a good idea to introduce too much emotion and personal expectations. Even worse, leading questions, like the one that gave Sally a hunch that you wanted her to respond favorably, led to responses that are not particularly true or useful. Strive to remain neutral and ask open questions.
It’s easy enough to understand how leading questions are bad for UX tests, but leading questions can be tricky beasts. Here is a list of questions. Which ones do you think are leading?
Now, let’s break them down.
They may not always be leading, but avoid questions that can have single-word answers, like yes/no questions. Many participants give the shortest answer possible, especially for written surveys. Remember that you really want the juicy stuff: stories, explanations, and examples!
Whenever I talk about yes/no questions, I think about a scene from the Beatles movie A Hard Day’s Night in which the fab four respond cheekily to a barrage of reporter questions during peak Beatlemania. One reporter, obviously expecting a newsworthy response, asks George, “How has success changed your life?” and George responds with a curt “Yes.”
Consider how to word questions to get richer, non-leading feedback. Rephrase direct questions into open questions, like “Please describe an example of…” or “Tell me about a time you…” People are good at speaking on their own experiences, and not very good at elaborating on direct questions.
It takes practice for UX tests and interviews to come easily, and we all make mistakes along the way. But you can plan ahead to minimize these pitfalls. While prepping for user interviews, write out the questions you intend to ask beforehand, and group similar questions together. Get feedback from team members or other UXers. Just be flexible about when the questions might naturally come up during the tests.
If you’re writing a survey, have others look it over and double check your wording. And if you have the resources, consider a pilot test of the survey questions—give it out to a handful of people before deploying it to a large audience. Essentially, there’s no reason why you can’t usability test your test itself.
Finally, beyond your words, your physical behavior can also be leading. Be encouraging and grateful for user feedback, but avoid responding too positively when participants say things you are hoping to hear, and avoid getting defensive in the face of product criticisms. Your poker face should be fully engaged.
Neutral behavior is easier said than done, so be mindful! During UX tests, I’ve seen research observers smile and nod when a participant behaves as expected, or point to features the participant didn’t see, or intervene to explain things the participant finds confusing. That’s all well and good during normal human interactions, but during a UX test? Please, don’t.
Of course, once the research session is over and you’ve collected your unbiased insights, feel free to open up, shake the seriousness off, and become human again with your participants. I actually really encourage that part!