The walls in room 3328, the observation room at Sun Microsystem’s usability lab in Prague, are painted a subdued blue. It swallows all the light, ensuring the testing scenario is not interrupted by curious guests like us, the Kiwi-project team members who were granted the privilege of a tour of the inner sanctum of Sun’s developer den. Through the one-way mirror, we can see a rosy-cheeked developer, talking to himself in Czech, interrupted by little sips from a coke bottle. He does not see us. The fact that very few of us understand Czech gives the situation an even more experimental appeal.

The new usability lab at Sun Microsystems, Prague

Jakub Franc, the cognitive psychologists in charge of the design of the study, explains to us that Sun rely on the Think Aloud method and observation in most of their test cases, rather than analyzing data from biofeedback sensors or eye-tracking devices. “Eye-tracking is good for testing the usability of web sites,” he says, “but for our purposes, the think aloud method, where the test person describes what he does and thinks, has greater benefits to offer.” The authenticity of the tasks to be performed in the study is a key: The developer behind the sound-proof glass wall is currently busy importing his own PHP application into NetBeans, Sun’s open source development environment, while the interaction designers and developers who created the tested module observe. A typical testing scenario lasts about 90 minutes, with the final 20 minutes consisting of an interview. “I always tell the testers that it’s not their fault if they fail to perform a task,” says Jakub. “If they fail, it’s the product’s fault. After all, that’s why we’re testing it.”

Before a software product is tested in its design or redesign phase, the ideal candidates are identified based on the results of questionnaires that are sent out to people in the tester database. The database includes both users of open source software as well as of competitive products, with the ideal test sample consisting of people who represent the whole spectrum of the target group, ranging from expert to newbie – and they must not necessarily be open source enthusiasts: “We offer a relatively high reward of 1000 CZK* as we want testers from all levels and backgrounds and not just the volunteering enthusiasts.”

Until Sun Microsystems moved into their new building in 2006, they collaborated with the Department of Computer Science at Czech Technical University (CTU), where they set up the very first usability lab in the Czech Republic in 2004. The deal was that Sun would supply the equipment and know-how, and CTU would supply the space and construction. Both institutions shared the facility until, after three years, all usage rights and equipment were transferred to CTU. One of the features of the new lab is the one-way mirror – the previous one relied on video observation: “From our experience, despite the fact that some participants feel less comfortable in this set-up, it makes a difference to observers”, writes Jiri Mzourek on his, i.e. one of the many Sun blogs, “they feel more connected to the participants”.

Jakub Franc, cognitive psychologist and usability researcher

Even though there is now an in-house usability lab at Sun, the collaboration between Sun and CTU continues, in particular in research and design projects. Students participate in projects led by Sun that focus on Sun products, learning about research methodology as well as gaining experience in project management in a real business environment. Jakub Franc also gives seminars in cognitive psychology and research methodology to CTU students, and is himself pursuing a PhD in environmental psychology – a relatively new discipline which, according to Jakub, deals with questions such as: “How should buildings be designed so that people are not getting lost in them? What recreational areas help people to recover from daily stress? What kinds of front gardens discourage burglars from invading the place?” In other words: Jakub studies the cognitive parameters of the usability of real objects.

Once the KiWi/Sun usecase enters the evaluation stage, the KiWi team will again be given access to the lab – but this time not as visitors, but as observers, witnessing how usable the KiWi-Wiki system really is to the inclined user. We are looking forward to the experience – and thank the designers of the lab for implementing a sound-proof wall, just in case the KiWis get emotional!

*) worth about 2 monthly passes for the metro in Prague, or 40 beers in a good pub