Let us image… you have found your spark, you have explored
the market space and found a problem worth solving, you now even have part of
the product that may solve that problem. Your objective is to make the product
the best thing for solving that problem. You have been working on this for
months maybe even a year or more. The product passes all of your automated test
but how do you know customers will actually be able to use it to solve their
problem? When you think about how your product works you view it as a clear
path to success, similar to the image below.
You enter some information, tweak this, change that, press a
button and taa-dah, the problem is solved! Unfortunately, we are often blinded
by our closeness to the product. What our users often see is similar to the
image below. A bewildering array of choices, with no clear path forward.
How can we show them the path? This is where Observational Testing
comes in. Observational Testing allows us to understand the pains of our user
allowing us to remove those pains and improve our product.
On Metacritic.com Half life 2 is the highest rated PC game
of all time; Half life 1 comes in at #4. Both games are made by Valve
corporation. One of the key practices that Valve used to take their games from
mediocre to great is Observational Testing. They call it Play Testing. Valve
would get in volunteers to sit and play their partially finished game, while
members of the team would observe them and take notes. The team was not allowed
to say anything to the player.
Quoting from Ken Birdwell a senior designer there: “Nothing
is quite so humbling as being forced to watch in silence as some poor
play-tester stumbles around your level for 20 minutes, unable to figure out the
"obvious" answer that you now realize is completely arbitrary and
impossible to figure out.”
A two-hour play test would result in 100 or so "action
items" — things that needed to be fixed, changed, added, or deleted from
the game. That is a phenomenal amount of feedback.
I personally ran many observational tests when developing
prototype games “Planty”, “Bargain Variety Store” & “Siege Breakers”, at
Halfbrick Studios. I can tell you that observational tests are easy to run,
horribly painful and immensely beneficial all at once. That hair pulling
frustration of the user seeing a forest of trees while you see a clear path
really pushes you to improve your product.
Running an Observational Test is straight forward:
- Bring in a customer or potential customer. This bit is hard.
- Provide them an objective to achieve in the test, either verbally or written out. This could be a hypothesis you want to test.
- While they attempt to achieve the objective, video record over their shoulder (a smart phone will do just fine).
- Observe what they do/don’t do; while not saying anything or offering any guidance. This is the hard part.
- Afterwards ask what they were thinking at key steps (i.e. when they got stuck, when they achieved success).
Observational Testing is how you can dramatically improve
your product. It brings three key benefits:
- Challenge your design approach. Are we tackling this problem in the right way?
- Validate hypothesis. As mentioned the objective you provide at the start could determine if they will use the product in the way you anticipated. Can they understand the information provided? Etc.
- Dramatically increase usability. This is moving them from the forest to the path, and is the most evident benefit when people start to use Observational Testing.
Halfbrick Studios maintains full Copyright over Siege
Breakers, Planty and Bargain Variety Store.
Reference: Play Testing process described
Reference: Keep play testing for HL2
Photo Reference: https://www.flickr.com/photos/eggrole/7524458398
