Step 4: Verifying Success & Compensating Participants

Welcome to the fourth part of our series on how to create a remote unmoderated user testing flow that won’t break the bank. This helps you keep costs low for your clients or company, freeing up resources for more and better user research.

In Part 3 of our series, we gave you low-cost options for sharing tasks and recording participants after recruiting and managing them yourself to avoid having to hire a recruiter (Part 1 and Part 2).

Most of the time, everything has gone well, and each participant was able to successfully complete the test and upload their video. All that’s left is to compensate them. Seems easy enough–and most of the time it is. We’ll describe some of our preferred methods of compensation later in this article.

But what do you do when a participant’s test cannot be used for some reason?

Continue reading for the fourth installment in this series, as we walk you through some options for compensating participants, as well as making sure that you are only paying for sessions that are actually usable for your research. Managing your participants at this stage requires fewer software tools and more soft skills: great communication, an ethical approach, and creative problem-solving.

Verifying Success

No matter what tools you’ve selected to perform your remote unmoderated user testing sessions (see Part 3), the first thing you need to do is make sure the tests were completed successfully.

1. First, quickly review each video. Did the video upload? Is the audio working? Did the participant share their screen? Was the video capture successful?

2. If they successfully completed the test, mark each participant as “completed” in your recruiting tools such as Respondent.io or userinterviews.com. If you recruited on your own, you won’t need to mark them completed, but of course, you still need to make sure to compensate everyone who is successful (more on that below).

3. If one of your participants was not successful, however, you’ll need to manage that situation actively – and on a case-by-case basis.

What to do if a user test is unsuccessful

When a test is unsuccessful, you’ll first need to determine whether things can be salvaged. If the participant didn’t upload their video, for example, you can contact them and try to correct it.

If the test is not salvageable, then you’ll have to decide who is responsible for the failure–and compensate (or not) accordingly. With some tools, like UserTesting.com, you can contact support and get a credit back for the test if the quality is low or the participant misrepresented themselves in the screener questions.

So take a deep breath and search your soul (and your test setup). It’s frustrating when participants don’t complete their tests successfully, and it hurts to pay for those, but sometimes it’s your fault. Be man (or woman) enough to admit it to yourself! For example, if your screening protocol missed something that caused the participant to be unable to complete the test, that’s on you. You need to compensate the participant.

Other times, it’s no one’s fault. For example, if the participant made a good effort and completed all the tasks but inadvertently messed something up or experienced a technical breakdown (e.g., if there’s a full-length video with no audio). In such cases, it’s best to compensate them anyway. Sometimes it’s possible to negotiate with such participants to participate in another study you’re running (for which they qualify) as a reasonable compromise to get them paid, while still giving you something in return.

Other times, however, the participant is clearly at fault. Perhaps they ignored the instructions you sent. Or maybe their screener responses misrepresented the hardware they’d be using or indicated they would share their screen when actually they were unwilling to do so. In these cases, the best approach is to courteously and respectfully remind them of the documented information that they agreed to. Explain that their effort is not usable to you, build rapport, and explain your position. Ask them to understand you cannot compensate participants who fail to read instructions or misrepresent their situation in the screener.

In our experience, a participant who is diplomatically shown they are at fault will accept responsibility and waive the payment. However, if you cannot reach an agreement, and things are getting contentious, then the best policy is still to pay. It’s just not worth the bad karma and possible damage to your reputation as a researcher.

Compensating Participants

And now we come to the fun part (depending on how you look at it)–sending out the compensation. It’s a big advantage to use an electronic form of compensation that can be sent shortly after a successful test. Most participants don’t like to wait for payment, and some won’t want to share their physical mailing address with a stranger.

There are several options for quick, electronic compensation.

1. Most recruiting tools, like Userinterviews or Respondent have built-in payment options. These are straightforward and need no other introduction.

2. Alternatively, there are a number of e-payment options you may want to consider, including Stripe, Paypal, Venmo, Google Wallet, and more. Make sure the transactions are set as “private,” and you include a note referencing the study so participants are clear on what the payment is.

3. If you’re not doing cash compensation, Amazon gift cards are a great choice. It’s fast and easy to send an e-gift card in any amount with no minimum. We’ve also compensated with gift cards from other brands but be sure to be transparent when not paying in cash.

Experiment Zone’s Tips

• When sending compensation, it’s always nice to include a note thanking the participant for their help, particularly if that’s the only contact they have with you post-test.
• When in doubt, err on the side of compensating participants, even if their session is not usable.
• Resolving disputes amicably is important, both for your reputation and peace of mind.
• Warm, sincere, and personal communication goes a long way toward resolving conflicts when they occur.
• Setting expectations early about the consequences of failed tests can help resolve issues later. See Part 2 for more details.
• Any time you experience a failed test, use it as an opportunity for learning and iterate your workflows to eliminate any sources of error on your end.

So there you have it, with this final step, you’ve completed a round of testing at rock bottom cost. Congratulations on building your own low-cost remote unmoderated testing setup that will help you to help your clients and company while saving money.

We’d love to hear what’s working for you to hack your user testing workflows! Email us at info@experimentzone.com.