It’s time to talk about OEs—what’s going on with them?

By: James Rogers, Managing Director APAC, PureSpectrum

Love them or hate them, OEs (open-ended questions) are here to stay—and rightfully so.  Regardless of how market research methodologies have moved from face-to-face, to pen and paper, to online, and now progressively (at speed) to mobile, OEs provide rich context and nuances that fixed-ended responses simply cannot.  

OEs, by their nature, give respondents the chance to answer how they want, in their language, to describe how they feel about a new or existing product. Given this level of freedom there have always been challenges around what makes a good (or a bad) response.

If the question being asked is not adequately specific, or guidance parameters have not been provided, then it is easy for the respondent to go rogue.  That, however, does not necessarily mean it’s a bad response, but perhaps simply a poorly expressed request for information. Does the question ask for a length of response?  Does the question include the right context for the answer?  Does the question provide simple, yet explicit, directions about what is acceptable, or not acceptable?  

Obviously, gibberish, copy/paste, or out of context profanity are not going to help with understanding what the respondent is thinking—but does this also provide insight into what the actual survey experience was like for those undertaking it?  How many OEs were included?  How many are actually acceptable, and importantly—where are they placed within the survey itself?  If you are asking eight OEs in the survey, can you really expect all of them to pass with flying colours?  What percentage is an acceptable pass/fail for the entire survey response?  With the volume of surveys being presented to a finite number of respondents, these are all key criteria that need to be thought about thoroughly before committing design to script.

Assuming a respondent-centric approach to design, OEs also pose considerable challenges post-field.  With end-to-end timing pressures very evident (along with pricing), being able to review, clean, accept / reject OEs becomes a very daunting task.  Whilst basic tools can be used to assist the process, there are no corners that can be cut—it remains very much a manual task.  Looking specifically at complex methodologies such as Conjoint, MaxDiff or Distinctive Asset testing, reviewers can easily be overwhelmed by 10s of 1,000s of OEs on a single project.  Handling this review can take days or weeks, depending on the DP team in charge.  No wonder there is so much pressure on fieldwork timings.

To add to what is already a time and work intensive task, we now welcome new challenges that are only coming to fruition in 2023.  These include (but are not limited to) bots and human / bot hybrids, developer tool windows, Google translate (or other tool) and of course, the new favourite public enemy—ChatGPT (or any other flavour of generative AI).

For the most part, those reviewing OE responses and general quality only had to worry about lazy responders or a handful of those looking to ‘game’ the survey in order to increase their incentive bank balances.  Now they have to be able to marshal bots answering choice-based questions, auto-filling OEs, web pages being translated into other languages (and back again) and OEs being too good.  This last one is of course tied to generative AI and it has caused reviewers to now find themselves in a ‘goldilocks’ conundrum. Can’t be too good, can’t be too bad, must be just right.

The good news is that there are already some everyday tools available to those scripting and hosting the questionnaire itself.  Note that the onus is right here, it’s not about the respondent source, which can only do so much when they can’t see the whole picture.  As well as the traditional OE guidance around expectations, nearly all of the mainstream survey scripting tools come with the ability to:

  • Ensure a min / max word count per OE response
  • Demand appropriate response format (i.e., numerical or alphabetical)
  • Detect translation widgets / developer windows
  • Detect VPNs (this is a minority but remains contestable to some extent)

No longer is this just an offline manual task, but one where the reviewer must work hand in hand with the tools available to achieve optimum results.

Enter PureText and the PureText API.  Part of the PureScore product offering, PureText enhances the relationship between the technology and reviewer. Regardless of whether the survey is hosted by PureSpectrum (although the sample must originate from our marketplace) the PureText API checks for the open-ended (OE) survey responses to ensure data quality. It performs language comparison, AI-generated answer detection, exclusion list checks, and PureText™ deduplication.  

  • Language Compare—compares the language of the OE answer with the language in which the survey is being conducted.
  • AI Detection—uses semantic analysis to look for certain linguistic patterns commonly associated with AI-generated text.
  • Excluded Text—checks if the answer is in an exclusion list of undesirable/inappropriate responses, contains mostly gibberish, contains contextually inappropriate profanity, etc.
  • PureText™ Deduplication—ensures the answer’s uniqueness, both within and across sections of a survey.

Now, to be clear, PureText is not a silver bullet; OEs are unique because of context and nuance, so whilst real time natural language processing tools such as this can be trained, they are not a one and done solution.  What they can do, however, is provide further guidance to aid in processing OEs. Again, not to replace that process entirely, but rather to move some measurable percentage of cleaning into a real-time, automated function.  This makes fielding more predictable, and final cleaning simpler, which means time and money saved.

PureText can help with improving the quality of data collected by reducing bad responses.  It can be used to either flag a response for easier cleaning or terminate during the survey (at clients´ request) saving post-field review.  In conjunction with PureScore™ (respondent level quality scoring by PureSpectrum) it can assist in attaining more reliable data by ‘weeding’ out low-quality behaviour and high pitch noise, and in part it can be used as a marker (of which there are multiple) in identifying and flagging click farm/bot activity.

Based on our ongoing review (hundreds of surveys since inception) we have good evidence to believe that around 30% of the ‘reviewers’ time can be better used when introducing tools such as PureText™.  Note again that this is not a replacement for line-by-line review, but advances in this space can help guide those checking to look in the best places.

Despite nearly 80% of surveys run via our Marketplace being ‘mobile’-compatible OEs are and should be here to stay to provide sentiment and unique responses.  But as they will continue to be asked a couple of recommendations:

  1. Are all of your OEs required?  Could they be partial (i.e. if not these, then choose ´Other´ and specify) or closed?
  2. Make sure you are using all the available tools at your disposal to get the desired outcome

This article was first published in the Q4 2023 edition of Asia Research Media.

Leave a Comment

Your email address will not be published. Required fields are marked *