« June 2010 | Main | August 2010 »

July 19, 2010

Words with Friends doesn't like "Gaydar".

Words with Friends

And if you're wondering, yes, I lost the game.

July 8, 2010

Adventures in remote usability, Part 2: GoToMeeting

Basic technique

Qualitative discount usability studies. I'd phone the participant at the designated time and walk them through the process of getting GoToMeeting hooked up. I used to try to get them to call in on the official GoToMeeting line so we could record audio, but honestly it was usually too much of a pain in the ass. I didn't even bother trying to get them to use their computer's audio: Yeah, how many people have a microphone and headphone set up with their machine? Not many, and I didn't want that to become a screening criteria.

Participant recruiting

Hagen / Sinclair screened, recruited, and scheduled 6 - 8 participants per study.

Cost

  • $135 recruitment fee per user
  • $50 - $100 Amazon.com gift certificate incentive per user
  • $50/month GoToMeeting subscription

Obviously there are the costs of my time, my equipment, etc. Not counting those, the direct costs are about $1,500 per study.

Consent forms and particpant agreement

I used to have a really nice PHP clickthrough introduction to the study, where the user would type their name and email in to signify acceptance of the form, and it would email them a copy. This became a pain in the ass to maintain, so now I just email them a copy and have them email it back as signification of their consent. I also verbally go over the agreements at the beginning of the study and emphasize the crucial points: Their reward, the duration of the study, and they can take a break or stop at any time.

Getting the study started

I've gone through many iterations of this to try to come up wtih the process that is the lowest burden for the study participant. Often this is at the expense of the recording.

  1. 10 minutes before the study, send an email with GoToMeeting link, a link to the study website, my phone number, and a note saying that I will call them at the designated time.
  2. 10 minutes before the study, I also send them the participant consent form or (previously) a link to the consent form mini-site.
  3. at the designated time I phone the participant. Sometimes they ask me to call back on another line. I introduce myself and remind them that I'm calling about the usability study. I always ask if it's still a good time for them. I've only had about one person in 50 cancel, and I think it's polite and creates goodwill; it lowers the transactional nature of the test.
  4. I explain what a usability test is, both from a technical and philosophical point of view: We're going to sign in to a screen-sharing tool (GoToMeeting) and I'm going to ask them to use a website and some new features that we're thinking of building.
  5. I start the process of getting connected to GoToMeeting. I have them go to their email and click on the link, and as that's launching...
  6. I explain that we're trying to find all the things that are hard to use or confusing about the website so we can fix them before we launch, and it's not a test of their abilities -- we just need them to tell us about anything that doesn't make sense. I then reiterate that I'll be recording the screen and our voice, but it's all internal and just for my notes.
  7. I review thinkaloud procedure -- "As you go through the site I'll ask you to think aloud. So for example, 'Hmm, I see this button. I'm not sure what the label means, but it seems like the only thing I can click on, so I'll go ahead and click.'" Then I explain why it's useful -- "This gives me a better understanding of what you're seeing and thinking, so I can tell the engineers what to fix."
  8. By this point the GTM is sharing my screen. They've given verbal consent and had a chance to ask any questions about the process.
  9. I like to use the GTM call-in feature because that gives us a pretty good audio recording of the session. On the other hand, it's one more step, and if the user seems on the confused side, I often skip this part just to simplify the process. I wish I could use GTM to dial their number -- so all they have to do is pick up the phone once, rather than me calling them on their line and then having them call into a conference system.
  10. I give control of the GTM to them (so they can browse natively on their computer), ask them to pull up the URL to the study, and then I start recording.

GTM fucks up the recording about half the time. Sadly, their Mac software doesn't record sessions, so I have to boot into Windows a/o use a separate windows machine to deal with the recordings.

On the upside, GTM means that other people in the office can dial in to the meeting from their desktops. This is pretty cool, but they have to be absolutely certain to mute themselves.

Downsides of GTM / remote studies

The only downside is making time for technical issues in the beginning of the study. All told, it takes about 15 minutes of study time -- so in an hourlong study you only get 45 minutes of study time.

Upsides

  • Largely unintended. I find that it's much less emotionally draining to conduct a study that's not face-to-face, largely because I dont have to control my body language and facial expressions. When you conduct a study you have to be perfectly neutral, encouraging, polite: the participant needs to feel at ease, and you need to be able to elicit their real opinions. You have to really engage with the other person while presenting as much more detached so you don't invade their space.
  • It's less time for the participant. They spend about an hour -- they have no travel time. Thus, it's a lower barrier to entry for the participant when they agree to the study.
  • Thus, you get higher quality of participants. When you need people to come to your office during business hours, you tend to get people who can take two hours out of their day, or aren't employed, or work in jobs that have odd hours. There's nothing implicitly wrong with such testers, of course, but if your product is targeted at people with run-of-the-mill "go into an office for 8 to 12 hours per day" jobs, you're not likely to find someone who wants to interrupt their day with a 2-hour errand. It's a lot easier for someone to block an hour of their time while they sit in front of their computer in their office.

Next: Adventures in remote usability, Part 3: Basic ClickTale