Failing at focus groups

Back in March, I was part of a focus group for an online shopping site. I’m not a big fan of focus groups - there are too many interpersonal dynamics at work and I think you get better, more authentic responses in an interview. But it was $100, and an opportunity to be on the other side of the table for a change, so I was legitimately psyched walking in to the session.

Unfortunately, it wasn’t the good experience I was hoping for. I’ve sat on my post about it for a while, unsure whether I wanted to say anything publicly, but I think it’s worth it if it helps anyone stumbling across this post run a better user research session in the future.

The first thing I noticed, after checking in at the door, was that there was no release or mutual NDA for me to sign. Then, they immediately wanted to take my picture. I said I wasn’t comfortable with that, and they didn’t pressure me, but the same photographer was snapping away throughout the two hour session, and I am sure there were plenty of photos of me in the mix.

I know that the NDA is an easy piece of paper to forget or handwave away as unnecessary; I’ve done it myself in the past. That was a mistake. Don’t ever skip the NDA, especially if you’re taking pictures or video. I was surprised to discover how much having my picture taken during the session threw me out of the experience and put me on my guard, all because I didn’t have confidence that my picture wouldn’t end up on some corporate promotional site somewhere.

(And for those of you who say “oh, we’d never do that; we’re really careful with our confidential research material!” let me tell you a story I heard from someone who worked in-house: one of her designers took all the pictures they’d taken during a usability session and proudly posted them to the corporate Flickr feed. Only being reminded of the mutual NDA that all the testing participants had signed got the designer to take them down. Now you can say, and rightfully so, that this story proves NDAs aren’t guarantees someone’s privacy will be respected. But at least they give the interviewee some confidence and some recourse if it isn’t. Don’t skip this step.)

The second thing I noticed were the set of cafe tables on the far side of the room, and the group of people who all seemed to know each other sitting there and chatting. Oh, that can’t be the clients, can it? I thought. Yes it can. The moderator, from a small SF-based brand research company, tried to explain that it was more “authentic” for the team from the online shopping site to be in the room with us, that you can’t really grasp all of the nuances of body language and the subtlety of interpersonal communication while eating M&Ms behind a one-way mirror.* If anyone ever tries to sell you, as a client, on this being the best way to do research, please show them the door, because they are talking nonsense. Don’t be in the room observing your focus group.

Photograph © AMCThere is a huge body of research that shows that people’s behavior, perceptions, and attitudes are changed by their environment, and part of their environment is the people who around them. Doing user research, one of your biggest challenges is getting people comfortable with an extremely artificial situation, one in which you are paying them for their opinions on something that the typically believe (rightly or wrongly) that you have some personal investment in, so that you can get past polite awkwardness and into something like truth. When you feel yourself being watched by the people who want your opinions, it’s going to change what you say, and make it more what you think they want to hear. 

What’s more, the rationale that the clients could see us better out from behind a one-way mirror was ridiculous on the face of it, because they were sitting behind the table, which meant they were staring at the back of two focus group members’ heads. Lots of ability to see the nuance of what was going on there, indeed.

I’ve spent some time trying to figure out what led the brand consultancy to use this weird setup, and I’ve only come up with two possible answers. One is that they had us get up and look at images pinned to the wall, and in a typical focus group room, that would have been harder. The other is that, since they were specifically targeting people who live in Brooklyn, they wanted to make sure the sessions were in Brooklyn, and they couldn’t find a proper research facility outside of Manhattan. These are both understandable concerns, but neither of them are insurmountable, and neither one is worth giving up whatever measure of critical distance that not having the people paying for the research in the room during the research can provide.

The final thing I noticed was the composition of the group. Out of the ten women at the table, two were in marketing, one worked in branding, and three (including one of the marketers and me) had experience running focus groups. I can almost make an argument that if you want Brooklyn tastemakers, you want to include a few people whose job involves branding and marketing, but even so, you need diversity, and more people who don’t know how the sausage is made. Screen your focus group members effectively.

Portigal Consulting, one of the top user research consultancies on the West Coast, talks about the complexity of writing a good screener on their blog, with links to a sample screener. If you go to that screener (which is a great example of a script for a recruitment firm), you’ll see that a lot of the early questions are looking for, and getting rid of, people just like the ones who were around that table. Why? Because most people don’t work in marketing and user research, and don’t think like them. Just like I wouldn’t want to just take a programmer’s word on what makes for a great piece of consumer software, I wouldn’t take a marketer’s word - or my own word! - on what makes a compelling online shopping experience. We’re not the typical user, and we think about it differently than the typical user would. It’s precisely to get that perspective from the other side of the interaction that you do user research and focus groups in the first place. 

I went home that night annoyed in the way that only a job badly done can annoy you, and wrote an angry bunch of notes for this post. Then I unsubscribed from the site’s mailing list. I spent the honorarium on framing art that I bought somewhere else.

* The only downsides I have ever found to being in an observation room in a testing facility are (a) overdosing on M&Ms and (b) since Mad Men started, people who are new to user research come in and say “I feel just like Don Draper!”  

Looking like itself: Product design beyond the touchscreen

The last product I worked on at frog has been out for a couple of months now - the Virtuo jukebox for Touchtunes. Touchtunes was a fantastic client, with an insanely dedicated team, and I was sorry to leave mid-project. So I’m really happy to see how great the final version looks and to see it get good press.

There are two key things I learned on this project about jukeboxes, which you can see for yourself in any bar you walk into with an actively-used jukebox. One is that they are still social centers. People gather around them in groups, they choose songs together, and they spend time thinking about how their choices will improve, or ruin, other people’s experience in the bar itself. The other is that modern jukeboxes, freed of the need for racks of CDs, freed in fact from the need for any physical controls other than the cash and credit card acceptance slots, have become unremarkable-looking. I’ve walked past jukeboxes in bars, thinking they were ATMs. And if the customer doesn’t know that the jukebox is there, they aren’t going to play it.

There are physical cues from old jukeboxes that newer jukes can use to tell you what they are: the columns of bright lighting up the sides, the domed top that feels like something out of Happy Days. Those cues get the point across, but they say “I’m like this other device you used to use” as much as “I’m a jukebox.” As a result, I think they have the side-effect of making the jukeboxes feel weirdly dated from the first day they’re on the market.

What I really love about the Virtuo is the way the physical form says “I am a jukebox” in a new way. And central to that is the big physical “Play” button. Even if the screen is running ads, or off entirely, you have no doubt what this device is for, because that play symbol is so universal. The cues are no longer the cues of the old jukebox, because the way we experience music has changed so radically. In incorporating a symbol that means “play music” today, the Virtuo marks itself as both a jukebox and as entirely up-to-date.

When the iPhone was new, I had a conversation with Luke Williams in which we wondered aloud about what would happen to industrial design in the brave new world of the touchscreen. What this project taught me is that sometimes, one button is all you need, as long as it’s the right one. 

(For more on the importance of buttons, and their changing role in interaction design, go check out the work of my friend Bill DeRouchey, and in particular his great presentation on the history of the button from SxSW.)

Yale School of Management case study on Project Masiluleke

The Yale School of Management has done an interactive case study on Project Masiluleke, the project I was working on in South Africa in 2008. It walks through all the missteps and changes of plans as well as the successes of the project, and it’s a more compelling read as a result.

I’m featured talking about design research - I am delighted to be associated with the headline “Design Starts With The User” on YouTube! And as I sort of get at in the interview, one of the great values of the trip ended up being the fairly massive case of culture shock I got from it.

The questions you ask about a place when you haven’t been there, and you impose your own rhythms and patterns and expectations on it, and the questions you ask when you have at least scratched the surface, those are different things. And I don’t pretend we were able to do anything more than scratch the surface in the ethnographic research I did in Kwa-Zulu Natal, but the depth of the local knowledge, and quantitative scientific expertise, of our partners at iTeach meant that all we needed was someone who knew enough to have a sense of where they were coming from.