User test —observe, understand and empathize with real users!

9 min readOct 23, 2018

Hey, I’m Rajesh, Product designer and project manager at BizLee — The Global Business Network, Bengaluru.

Today, I’m gonna share my user test experience which I did for a business mobile app.

I had been reading and planning for running a user test for months. I know that without real user feedback, it’s not gonna be OK to deliver app features.

When I first landed on conducting real-time user testing, I had moments that I didn’t know what to do.I started feeling less confident about my design solution and flows which I designed for each user problem, even though I have done a couple of Design sprints ( big thanks to Jake Knapp) and AJ&Smart!

Here is the main glimpse of the user test session.

“The goal of a designer is to listen, observe, understand, sympathize, empathize, synthesize, and glean insights that enable him or her to ‘make the invisible visible.’”– Hillman Curtis

Yeah, it is always a best practice to get real user (*selected users from beta user group) feedback at every stage of your app design and development.

Let’s dive into some of the major advantages of running user tests.

By running a successful user test, you can get real user feedback on everything including wireframe sketches, High-fidelity prototypes, unreleased features, unreleased versions.

Why do we love user testing? The key benefits are :

Real-time feedback and human interaction.
User’s input can be helpful at every stage of the mobile app.
Finding bugs quickly by exploring real scenarios.
Low cost, high return.
Interacting with end-users in their working environment.
A designer can get real feedback from the user by showing the upcoming feature’s prototype.

Main things to follow:

Invite each one of the users by sending a warm welcome user test emailer. let them feel like they are going to be a member of your product family. But never mention testing word in emailer !!

Create one community group for users who attended the user test. We have created a WhatsApp group with a Bi -Beta name. (*why WhatsApp means, Users belongs to different domains and professions, so it’s better to use a commonly used easy platform for collaboration )
Inform while testing that We are testing the product, not you and be comfortable ! and watch users’ actions for the tasks you assigned, and record their gestures while interacting with your app.

“The mantra of usability testing is, “We are testing the product, not you.”
― Carol Barnum, Usability Testing Essentials

Roles

Main roles required to run a successful user test :

1. Facilitator: Guiding the test and ensuring everything runs smoothly
2. Data Logger: measure task completion rates, analyze questionnaire data and note down findings (1 person )
3. Test Observers: You can observe a lot just by watching (2 people)
4. Video Camera Recording: 1 person

* you can increase the number of users and Observers.

Please sit out of the participants’ line of sight (behind them) and try to be completely silent. Your main goal is to gather as much feedback as you can. This helps you to identify any design issues and flow changes before you get to the expensive part of the development process.

Smile in a friendly way and watch carefully while taking notes.
Observing what people do
During and before the sessions, don’t engage participants in the conversation, offer advice or answer their questions, because introducing new information or giving more clarity can bias the research results and even invalidate that session’s data. It’s difficult not to respond when someone asks you questions because it seems impolite, but this is important. Say something like, “Sure I'll be talking about that soon” and look to the facilitator.
Be careful not to distract anyone. Your fidgeting, sighing, posture, facial expressions, noise from phones or clothing, and so on, can easily change the participant’s behavior and comfort level. If you need to, please leave the room quietly. No in and out.
Typing noise is fine if data Logger is taking notes, but try to type constantly so participants will ignore it.
It’s okay to laugh if the participant makes a joke, but not at any other time.

content courtesy @nngroup

Another big advantage of running a user test is that it clarifies so many wrong flows and ambiguities. especially when a small startup team making a product with the vision and idea of the product owner ( Usually CEO in startup), there will be a chance to blindly believing the product owner’s idea without validating it with real users. it will eventually make more problems and delays.

Traditional user testing is conducted with a single participant. The participant is given specific scenarios to carry out on a product, while their actions and behavior are observed by an observer.

User testing can help a lot in the mobile app design process. The product designer can decide by analyzing quality data derived from user tests in each phase of design. A designer can get real-time feedback on design variants and decide the best one (A/B test).

A user can tell you more information from a successful user test session:

They can tell you what they like about the app. it can be your main feature.
What feature or flow they find frustrating and irrelevant.
Compares with competitor apps and standout features
Ambiguities on user flow and their suggestions.
What value they have received from the app.

“If the product doesn’t add value to the way in which the user currently performs tasks or needs to learn to perform tasks, then the user will have no use for the product.”
― Carol Barnum, Usability Testing Essentials

Required Tools

Mobile devices with an installed latest development version of the app or InVision /marvel /adobe XD Prototype.
Flexible 360 degrees Snake Style Stand — Mobile Holder ( Most of the startups can’t afford high-end usability lab setup ).
Mobile devices for recording user activity. ensure that the device got enough storage and battery charge before starting the test.
Get more paper and pens.

Once you have done with the pre-test arrangements, it’s better to revisit all the checklist which you have already completed.

Answer only the questions that the facilitator asks you directly. It’s very common for facilitators to ask participants questions that they already know the answers to, questions to help the user relax, and questions to find out what the participant understands. Don’t answer those questions either. It’s important to understand what the participant believes to be true, even when that’s not technically correct.

Screening Question

Tasks (usually 60 minutes)

Tasks are typically small sentences, it should cover one scenario and set of interactions that need to be performed by the users.

Few examples:

Create a new account
Complete 50% of your profile information
Activate Hire me
Change the profile image
Apply for job
Accept a friend request

(sample question which I used for mobile app testing)

Instead of asking straight tasks it is always better to reframe the sentence to real task scenarios.

People tend to perform more naturally if you provide them with scenarios (real tasks from BizLee user test) :

You just got a message from your close friend asking to create one account in th BizLee app. You decide to give it a look. Open the app and create your account.
You got to know that your follower “Rajesh Raghavan” has written an article. Read and Save it. If you like then comment and share to WhatsApp contacts.
Since it’s your first-time visit on the BizLee app and you have got some great stories to tell. Write your article and publish it.
You live in the city of Bangalore, and you want to get some professional contacts. Find them and start following.
You are in a search of finding the perfect job, But It seems like your profile completeness is low to apply for any job. Try to reach 50% of profile completeness.
Perfect! Now start to find your interesting jobs and try to save 3 jobs for future reference.
You badly need one job now. To Activate the “Hire me” alert for your account to get more visibility from recruiters.
You need to clean-out your saved stories and jobs now.
I know you are in a hurry to read some stories now, forgot your password? Try to get a new one.
You have got some new thoughts in your mind and you are gonna start write it as a BizLee story. Sounds great huh? You decided to write and keep it as a draft for publishing later.
You have got one notification about the new opening — senior interior designer at empathizers. apply and wait for notifications and recruiter confirmation.
Cool. now it’s time to share your genuine feedback about the overall app experience. Use Bizlee to share your feedback.

Post-test questions

1. Did you happen to stumble upon a feature you didn’t expect? Which are they?

2. If the app is free do you get enough features to make it worth the download?

3. After checking this app, what you understood? Bizlee gave more importance to anything or less importance to good features?

4. What’s your opinion on Profile? Which acts as your complete digital resume.

5. Do you think this app missing any features, that a candidate will look at while searching for a job?

6. This app contains Stories, jobs, products, tenders and more related to business and people. Any idea that how much extent it captures the market? And how you felt like a general user?

7. What did you felt about UI and UX flow, any suggestions regarding usability?

8. Did you come across any app which has these features, which means a complete solution for business in one place?

And finally ask about overall experience:

List the most negative aspect(s):

List the most positive aspect(s):

Voting

Moments Captured :

Present your findings

User testing can take your app to a whole new level, but it can’t just be something you’re checking off your list to make investors happy. It’s important to invest the time into analyzing the data and then implementing changes based on the user feedback. If you do this properly, you might just find it to be one of the most helpful steps in the mobile app prototyping process.

Always remember this “Usability testing is not market research.” one more thing is that usability test results won’t be accurate if you think entire target demographic perspective. Because needs and requirements from person to person will be different depends on various scenarios and a much bigger user base needed to reach reliable conclusions.

Don’t forget to send some love and goodies to those users who attended the user test.