When to do user testing?

User testing is vital ingredient for creating and optimising your customer experience on your website, app or page. To be clear from the outset (and yes, there is still some confusion) user testing, as a process, tests how users react to your website, whether they can complete key tasks or goals as well as how they find the overall experience.

No matter how much of a UX expert you are, there really is no telling how ‘real’ users will interact with your site until you see them doing it first-hand. It’s quite common for seemingly obvious blockers, that have remained unnoticed or even considered, to only be uncovered once a site or new piece of functionality, has been tested by real users.

When to do user testing?

I get asked this a lot. Should you user test once your site or app is live and finished to give the users the full and complete experience, or should user testing be baked into the design and development process so that learnings can be applied? The short answer is, all the above. However, where you really don’t want to find yourself is with user testing being an afterthought or scheduled as one of the last stages of your development roadmap. Imagine for a second; you have been working tirelessly on your new site, with design, functionality and development approved by the multitude of stakeholders and, as you are in the last leg and about to go live … the results of your user testing unearth several key blockers that delay your launch date as it goes back into development. Less than ideal.

Likewise, to undertake user testing too early in the development process could skew your results as any broken functionality or incomplete components the users are testing could become blockers in themselves, leading to frustration plenty of mouse wiggling and spacebar tapping.

User testing recruitment

It seems obvious, but when thinking about user testing it’s important to recruit participants who are as close as possible to your actual users. This means, as a minimum, they should have similar demographics to your users – age, social grade, location etc. However, for more accurate and real results from your testing, they should also have the same needs as your users. For example, if you’re testing a university website then you’ll want to test with students, or people looking to study, to make the process as ‘real’ as possible. It can take a bit more time to recruit the right participants, however, you will reap the benefits when it comes to the learnings and insights you will glean.

• User testing can be done in person, often in some kind of ‘lab’ - not really a laboratory but think of it as more a controlled environment.
• Testing can also be done remotely, where the user completes the test in the comfort of their own home.
• User testing can be moderated, where the participants are taken through tasks, or run in an unmoderated format, where users are given a list of tasks to complete in their own time with no moderator present.

There are lots of variables when it comes to user testing, and some have a clear preference for others. However, whatever your preference, your approach will broadly sit within 3 camps.

Remote unmoderated testing

There are some advantages to remote unmoderated user testing. The main advantage is that it is much quicker to recruit participants and turn tests around. Online testing companies have huge databases of users, normally broken down by demographics, who are ready to begin testing almost immediately. As participants can be selected that match your target audience, remote testing will normally give you a more diverse range of users than if you are recruiting locally, which can also be a little more time-consuming. Also, with a large number of users waiting to start it means you can run tests concurrently and outside of office hours.

Remote users will also be comfortable in their own environment, sitting in familiar surroundings and using their own computers or mobile.

Remote unmoderated testing can be very cost effective. Companies run bespoke online user tests for around £50 per participant, which is a small price to pay for the insights that you get.

However, the main disadvantage of remote unmoderated testing is that you have no interaction with the participant. You will receive a screen recording of their test, clearly showing any blockers or issues they came up against, but that will only tell you so much. As we can all appreciate, so much of our communication is unspoken, with body language providing a richer, and more rounded, source of insight. This is why some will always prefer to conduct their tests with a moderator present, which is where lab-based testing comes in.

Another issue is the likelihood of, inadvertently, having semi-professional testers in your panel of participants (yes, it is a thing). From my own experience, I sometimes get the impression that users recruited through these types of websites take part in several tests a day, which makes them somewhat different from your average or real users.

Moderated lab testing

In lab-based testing, a user is invited in to run through the test in a moderated environment. Testing in a controlled environment means that you can dictate how the test is run by preparing your 'lab' to suit your needs. This could be a simple desk setup or more scientific lab style environment, whereas some will prefer a lounge style lab that is set up to create a more relaxed and real-life atmosphere for the participant. Lab based testing also makes it easier to test on different devices and to test on prototypes or WIP websites where users may require more direction.

The test will be moderated by a UX professional who is there to give the user instructions and interact with them during the test. This is one of the main advantages of moderated lab testing. It gives the moderator the opportunity to engage with the participant to find out more about their needs, giving you a richer source of insight and understanding from the test. As a facilitator, you can really observe and interact with each participant. While you don’t want to lead users in any way you can make them feel at ease and encourage them to act naturally. A good facilitator will react to different situations to get the most insight from each test.

However, when compared to remote unmoderated testing, getting moderated lab testing up and running is a bit more work. Remembering the number one rule is to use participants that match your real users, finding the right number of participants that tick all the right boxes, and are local to where you are running your test, means you will need to be a bit more patient. There are plenty of companies who can assist you in finding your participants for user testing. However, be mindful of the professional testers and ask how many tests each participant has previously taken part in so not to skew your results (yes, there are people who make a decent living taking part in research groups, user testing and panels).

Remote moderated testing (the best of both worlds?)

The wonders of modern technology mean there is a solution that gives you the best of both worlds. Remote moderated testing uses tools such as Lookback to allow you to moderate tests with users no matter where they are located. 

On paper, remote moderated testing should be the answer to all our user testing prayers.

• As it is facilitated online you can recruit your participants quickly.
• All the user needs to participate is an internet connection and their computer or device.
• Webcams and built in cameras mean you can capture the unspoken signals and expressions from your participants.
• Your multiple stakeholders can observe the tests in real time.
• There are no additional costs with lab hire and travel expenses.

However, there are a few downsides. Technology is brilliant and facilitates all the above … when it works. However, as a moderator myself, I can tell you from first-hand experience, just some of the issues I have had when facilitating remote moderate testing.

• The live stream and sharing of user to moderator requires a fast, and reliable, internet connection. Any break in connection will be incredibly disruptive. It will often go unnoticed by the participant, who can be midway through a test, and when connection is restored, they need to repeat the process and, therefore, will skew the results as they will have already been exposed to the task. Yes, this happens frequently, remembering not all areas of the UK have fast internet speeds, and, for some, the cost of a super-fast and reliable internet connection could make it prohibitive.
• Often, participants will have not read through their pre-test instructions and will have failed to download a required app or plug in. This means the first 10 minutes of the test is spent waiting for them to do just that and ensure the connection is working correctly.
• Although one of the big positives is that participants and taking the tests in the comfort of their own home; they are really in the comfort of their own home. Babies crying, the doorbell ringing, dogs barking, or even teenagers arguing are all distractions I have encountered when facilitating remote moderated testing.  Users might not actually be in their own home either.  I've had users taking tests in coffee shops, which can be quite disruptive to say the least.

Summary

In short, moderated testing, when done in-person or remotely, can be more thorough and in-depth than its unmoderated counterpart. The key difference, and one that I think this blog has made clear, is that moderated testing can help you to get a better idea of ‘why’ users are doing what they’re doing. It gives you the chance to ask follow-up questions during the test that remove the guesswork from the process.

Getting to know the users can be an additional benefit of in-person testing. This may give you something to relate to when it comes to persona development or devising user journeys. You can learn a lot by chatting with users in person that will be useful for other stages in the development process.