And if you're wondering, yes, I lost the game.

Qualitative discount usability studies. I'd phone the participant at the designated time and walk them through the process of getting GoToMeeting hooked up. I used to try to get them to call in on the official GoToMeeting line so we could record audio, but honestly it was usually too much of a pain in the ass. I didn't even bother trying to get them to use their computer's audio: Yeah, how many people have a microphone and headphone set up with their machine? Not many, and I didn't want that to become a screening criteria.

Participant recruiting

Hagen / Sinclair screened, recruited, and scheduled 6 - 8 participants per study.

Cost

• $135 recruitment fee per user
• $50 - $100 Amazon.com gift certificate incentive per user
• $50/month GoToMeeting subscription

Obviously there are the costs of my time, my equipment, etc. Not counting those, the direct costs are about $1,500 per study.

Consent forms and particpant agreement

I used to have a really nice PHP clickthrough introduction to the study, where the user would type their name and email in to signify acceptance of the form, and it would email them a copy. This became a pain in the ass to maintain, so now I just email them a copy and have them email it back as signification of their consent. I also verbally go over the agreements at the beginning of the study and emphasize the crucial points: Their reward, the duration of the study, and they can take a break or stop at any time.

Getting the study started

I've gone through many iterations of this to try to come up wtih the process that is the lowest burden for the study participant. Often this is at the expense of the recording.

1. 10 minutes before the study, send an email with GoToMeeting link, a link to the study website, my phone number, and a note saying that I will call them at the designated time.
2. 10 minutes before the study, I also send them the participant consent form or (previously) a link to the consent form mini-site.
3. at the designated time I phone the participant. Sometimes they ask me to call back on another line. I introduce myself and remind them that I'm calling about the usability study. I always ask if it's still a good time for them. I've only had about one person in 50 cancel, and I think it's polite and creates goodwill; it lowers the transactional nature of the test.
4. I explain what a usability test is, both from a technical and philosophical point of view: We're going to sign in to a screen-sharing tool (GoToMeeting) and I'm going to ask them to use a website and some new features that we're thinking of building.
5. I start the process of getting connected to GoToMeeting. I have them go to their email and click on the link, and as that's launching...
6. I explain that we're trying to find all the things that are hard to use or confusing about the website so we can fix them before we launch, and it's not a test of their abilities -- we just need them to tell us about anything that doesn't make sense. I then reiterate that I'll be recording the screen and our voice, but it's all internal and just for my notes.
7. I review thinkaloud procedure -- "As you go through the site I'll ask you to think aloud. So for example, 'Hmm, I see this button. I'm not sure what the label means, but it seems like the only thing I can click on, so I'll go ahead and click.'" Then I explain why it's useful -- "This gives me a better understanding of what you're seeing and thinking, so I can tell the engineers what to fix."
8. By this point the GTM is sharing my screen. They've given verbal consent and had a chance to ask any questions about the process.
9. I like to use the GTM call-in feature because that gives us a pretty good audio recording of the session. On the other hand, it's one more step, and if the user seems on the confused side, I often skip this part just to simplify the process. I wish I could use GTM to dial their number -- so all they have to do is pick up the phone once, rather than me calling them on their line and then having them call into a conference system.
10. I give control of the GTM to them (so they can browse natively on their computer), ask them to pull up the URL to the study, and then I start recording.

GTM fucks up the recording about half the time. Sadly, their Mac software doesn't record sessions, so I have to boot into Windows a/o use a separate windows machine to deal with the recordings.

On the upside, GTM means that other people in the office can dial in to the meeting from their desktops. This is pretty cool, but they have to be absolutely certain

Downsides of GTM / remote studies

The only downside is making time for technical issues in the beginning of the study. All told, it takes about 15 minutes of study time -- so in an hourlong study you only get 45 minutes of study time.

Upsides

• Largely unintended. I find that it's much less emotionally draining to conduct a study that's not face-to-face, largely because I dont have to control my body language and facial expressions. When you conduct a study you have to be perfectly neutral, encouraging, polite: the participant needs to feel at ease, and you need to be able to elicit their real opinions. You have to really engage with the other person while presenting as much more detached so you don't invade their space.
• It's less time for the participant. They spend about an hour -- they have no travel time. Thus, it's a lower barrier to entry for the participant when they agree to the study.
• Thus, you get higher quality of participants. When you need people to come to your office during business hours, you tend to get people who can take two hours out of their day, or aren't employed, or work in jobs that have odd hours. There's nothing implicitly wrong with such testers, of course, but if your product is targeted at people with run-of-the-mill "go into an office for 8 to 12 hours per day" jobs, you're not likely to find someone who wants to interrupt their day with a 2-hour errand. It's a lot easier for someone to block an hour of their time while they sit in front of their computer in their office.

Next (Coming soon!): Adventures in remote usability, Part 3: Basic ClickTale

I've dealt with our user research and user info needs for the past year and a half at ChoiceVendor . Having to manage a usability program is nothing like plunking down on the couch in one of Google's usability lab observation rooms and kicking back while one of the researchers proctors the session from the other side of a one-way mirror.

For the sake of clarity, I define remote usability as "any method of gathering usage and behavioral observations about site users who are not physically coming to your office." In other words, it doesn't just mean Nielsen-style discount usability studies that gather qualitative feedback from 5 or 6 people.

Remote studies get you a little closer to users' real experiences.

We all feel more comfortable when we're in our own environment rather than in someone else's office. I've had users who put me on hold to take another call, who have to move into another room because their kid is yelling, who ask me to hold on just a moment to answer a question from an employee. They're also on their own computer, which they know how to use and operate, and it's got their email and passwords set up already.

We have special users and can't test just anyone.

We have a moderately specialized user base: people who own businesses, or are managers, or are some other sort of purchasing decisionmaker at their company. Because of this, we can't plunk ourselves down in a coffee shop and wave $10 gift certificates at people in exchange for 15 minutes of study time. (That's what we did for Mopho.)

It's true that we probably could have found a decent number of issues just by doing coffeeshop usability. But the problem with doing hallway usability is that most of the people in our hallways are either in the software industry or they live and work in the Bay Area: the epicenter of the technology world. It's not that I don't want these people to use my site -- it's that these people have about tenth of the problems as non-Bay Area professionals. I know that they can figure out how to use the site, and they're flexible enough to not get too frustrated when something goes wrong.

So I used a market research firm (Hagen / Sinclair) to recruit qualified users from anywhere. Even these participants were relatively sophisticated: they had to have high-speed internet, a relatively modern computer capable of running GoToMeeting, and they had to be sufficiently articulate to pass the screening questions.

This costs about $135 per user, plus $50 - $100 in Amazon.com gift certificates depending. Local participants were welcome to come into the office; everyone else we used GoToMeeting to screen share.

I specifically wanted people who had never used or heard of our site before, which is why we didn't recruit from our existing user base on most of these. We wanted qualitative feedback not just on the site UI, but on our brand, messaging, and the like. Plus, when we talk to actual members of our target audience, we also get to ask them lots of interesting contextual-inquiry and focus-group types of questions.

Note: One of our participants couldn't figure out how to add the gift card codes to her Amazon account. Yet she owns her own business: she's smart at what she knows, and just isn't an expert at the Internet.

We were able to get a higher quality of user by going remote.

Again with the theme: our user base is specialized, so I wanted to remove all other barriers to participating. If you require people to come into your office, you restrict your pool to people in your geographical area who are also able to take 2 hours out of their day to come visit. (Half an hour on either side of the 1-hour study for transportation, parking, etc.)

Few people with typical business-type jobs (much less business owners or VPs) are going to ever do that, especially for $100 in Amazon gift certificates.

Personally, I'm usually not eligible as a study participant because I work in UX, but occasionally I am part of the target audience for a given study. But the last time, they wanted me to phone someone on a weekend to set up a time to drive to Cupertino by 6pm later that week. Who doesn't have better things to do with their time than that?

Facial expressions are bullshit.

I'm with Nate Bolt on this one. I don't think I've seen the participant's face during a study in, let's see, the past 8 years? You don't need to. Everything you need to know about the participant's emotional state you can tell from their voice.

Plus, the thought of adding one more technical barrier to a study -- requiring the user to have a webcam or video, and getting it properly turned on and functioning -- in the course of an hour long study that also requires voice, an explanation of the study itself, screen sharing, and recording? Not worth the effort. I'd rather spend the time asking them more questions about their business and how they currently accomplish the tasks that we're trying to support.

Thus, I rejected any software or testing system that made a big hype out of its video recording capabilities.

Remote can be faster and cheaper.

It's generally a bit more work to set up a remote study -- technically you have to have more pieces in place. But for unproctored remote usability, you can run a dozen people through a 5 minute test, make some changes, and run another dozen people. You don't have to bring someone in and come up with an hour-long study to get full use of their time.

Conclusion

For the past year and a half I've tried a number of different approaches & methods. This series of articles outlines pros, cons, funny stories, and what I've learned from different techniques.

Next: Adventures in remote usability, Part 2: GoToMeeting

so that vaguely works, but i've been unable to find a decent list of all the possible formatting characters that work in whatever frankenstein jinja-django-python monster we've got running on the site. Not all of the django ones work (http://docs.djangoproject.com/en/dev/ref/templates/builtins/).

• %a -- 3-char week abbr
• %A -- Full day of week
• %b -- 3-char month abbr
• %B -- full month name
• %c -- Thu Feb 18 08:56:31 2010
• %C -- this outputs "20" for every line on the page, no idea
• %d -- day of month

Pattern matching seems to indicate that you can format jinja datetime / timestamps using python date formatting ...

Crowdsourcing fails and usually sucks, unless you have a community of people who actually know what they're talking about. So while everyone in the industry has been hyping social for several years now, it's not because authoritative, editorialized, expert content is actually going to die or go away. It's because we're just now getting the technology that ALSO affords other people the ability to create and partcipate.

In other words, I'm not arguing that the mode of collaboration will replace information-seeking behaviors -- i.e. seeking authoritative information from editorially controlled, expert-to-novice systems. Even if it's entertainment we'll still have experts who'll create widely disseminated works that other people consume without offering any collaboration. And now, joy, we can also watch people getting kicked in the nuts on YouTube.

In HCI we used to hear (like in the 90s and early naughties) that a major failing of PCs was lack of collaborative features. Lots of companies tried -- anyone remember the awful, clunky whiteboard-sharing professional applications from back then? It was a holy grail of sorts -- there was such a perceived efficiency increase in offering true remote-collaboration tools that lots of software companies spent a lot of effort writing a lot of software to solve the problem. (When your company creates an efficiency in the marketplace, you probably get a lot of money.) It hasn't been until recently, i think, that technology has gotten good enough to support truly useful collaborative tools:

• Wireless. Between smartphones and scads of 802.x networks, it's pretty easy to always be talking to the network.
• Device size. Extremely powerful devices are extremely small.
• Bandwidth: there's enough of it that you can screen share, desktop share, video chat, whatever -- all while recording your screen and audio AND browse the web.
• Browsers & web technology: XMLHttp and really powerful javascript have (1) made web front ends a suitably dignified and powerful system for hardcore engineers to spend years of their careers on. (2) they are able to invent and build robust, fast applications that do really cool stuff.

So it's not that the former needs to research and consume information are obviated. It's just that finally, since maybe 2005 or 2006, we're really able to start satisfying the collaborative needs that have been extant and unsolved since the beginning of personal computing. (Since the beginning of time if you want to get even more philosophical.)

In certain types of systems the whole is more than the sum of its parts. That's why you have collaboration; and if you can create a collaborative dynamic

Generally this is what Web 2.0 has been saying all along: the 2.0 web has highly personalized applications and a focus on user-generated content. This is different from crowdsourcing, however. Yahoo Answers! sucks because it's crowdsourced and most people are, shall we say, not exactly rocket scientists, so having them answer your question probably leaves you far worse off than having access to information from authoritative sources. (Even for something simple like "how long after each airing does iTunes release each episode of Lost?") But it's cheaper for Yahoo! to have its users make up answers to questions than for them to build a better search engine, much less hiring actual experts.

Facebook, however, isn't crowdsourced. Or at least it's not crowdsourced in the same way. FB doesn't replace authoritative content with the chatter of whichever eager-beaver member feels like commenting. If you put some work into FB -- in the form of setting up your contacts and maybe listing your preferences -- and occasionally post a photo or a status update, you are rewarded with a fairly low-key current of updates about people you know. If that is not a reward for you, I don't want to hear about it -- that's the purpose of the site, and if you don't actually find FB worth it, we both know you wouldn't be using it; thus, those who do use it obviously find it sufficiently rewarding to justify the effort.

For some communities, crowdsourcing works, primarily in a community of experts. Slashdot is good at this. They have enough really expert people and enough of a huge user base anyway that really robust, high-quality comments tend to move to the top of the stack in a reliable manner. In other communities, like, say, StackOverflow, there were very high-quality, difficult questions at first, getting answered with some truly robust and detailed technical discussions; in my view the quality of the conversation has gotten diluted a bit now that use of the site has spread a bit beyond the borders of hardcore experts. (It indicates that there's more of a benefit to asking questions than answering them; answering takes a fair bit of effort and the only reward is the artificial points & reputation system.)

The benefit that you get back depends a lot more on the expertise of the other people involved, rather than on your own effort.

Lampe and Resnick published a fantastic paper on Slashdot's community filtering at CHI 2004, and IIRC they said that it takes a community size of about 50,000 users to get sufficiently robust crowdsourced filtering.

---

Which is why at work I'm taking on the (currently not-unmanageable) job of responding to customer support requests.

However, I'm finding that gmail (we use google apps) is not really a good interface for doing this quickly. I want to be able to do things like bulk respond to a set of emails with the same message... Gmail has the canned responses plugin, but that doesn't cut it -- it's too slow to use.

So I just hooked Thunderbird into our support queue, and now I'm trying to figure out how to configure or optimize it for processing customer support emails. I'd also love it if it automatically kept stats or some how output to a CSV file on the number of emails processed through canned responses.

Or maybe I just need some lightweight CRM software. Hmm... I wonder what the best CRM solutions are for a small business? and how much can I expect to pay? If only there were a website with some reviews and a listing of top providers that would give me a sense of what to expect... (Okay, this is a shameless plug, since that's what ChoiceVendor does.)

• Office Interactive: "All you need to run your business using one system". That scares me; I don't want to run my whole business on this. NEXT.
• BatchBook: "Your Social CRM". I don't really want my CRM to be social. I want it to be efficient. This seems to be about within-company contact management, not CRM (remember, that would be CUSTOMER relationship management).
• Relenta: "Can your email do this?" Again, seems more salesforce / lead tracking-focused.

Let's try investigating "Customer Support Software for small business".

• Business.com -- ooh, that looks promising, they have a result in Google for "Small Business CRM Software: Customer Service Solutions | Business.com" (and I'm not going to link to it because I don't want to make them even more credible for their stupid site). Oh right. Business.com is a crappy AdWords linkfarm.
• WorkEtc: Okay, again they want to manage my entire company. First, dont' want that, second, not authorized to make that decision, third, we're generally pretty happy with Google Apps. (Most of us worked at Google after all, and we're pretty used to their stuff.) However, maybe their Help Desk Software is what I want.

Why are there SO MANY crappy sites for small businesses, by the way?

Okay, now I'm back to Salesforce Service Cloud 2. I can't really see if it does what I need since I have to register to walk through their product demo. Annoying. I'm pretty sure I even have an account with them already, but I'm not going to bother to look it up just to see if they even remotely have what I need.

Open source help ticket system:

Kill me now. I don't want this attached to our bug tracking and I care about the interface and how fast it'll let me get through the emails in our queue. I don't want to have to install anything anywhere, either. Back to paid / hosted services....

Searching through the Thunderbird extensions for business, CRM, helpdesk, and related keywords yielded no results.

Okay, I'm hosed; I've now spent about 2 hours researching this and haven't come up with a satisfactory discussion or solution, so it's time for me to ditch the effort and just process the emails manually.

Le sigh.

Here's the question on Quora, but haven't yet gotten much of a response:
http://www.quora.com/q/What_are_the_best_fresh_up_to_date_academic_type_resources_for_mobile_UI_UX_design

Today we launched our Private Beta release. We've still got a long ways to go -- On to usability testing!

www.choicevendor.com

Anyway, I wanted to leave a review on the product page in the Apple store. I can't figure out how to post my review:

"Some words in your review cannot be published. Please revise your review." Huh?

Their guidelines state "avoid comments about... service and support", but how the hell are you supposed to review a product that doesn't work if you can't describe anything about the customer service response to the issue? I've tried about 8 versions of text for this review, and I have no idea why they won't let me publish this.

So on my own blog I'll say it:

Fuckers.

Love,
Ellen

The Church - A Box of Birds; Snow Patrol - Eyes Open

And now, not only does Digg do it too, but the sites are so dumb that they're double stacking these stupid toolbars.

This is crap primarily because it breaks basic web behavior: the URL on your web browser no longer represents the page you're viewing, or trying to view. Big juju nono. We've known this since 1996, people!

Also, I don't want to send someone to http://www.facebook.com/ext/share.php?sid=89405539347&h=1e79i&u=2fnBr&ref=nf. I want to send them the link that's www.youtube.com so they know where they're going and what they're going to get.

The one case where Google does something like this -- a frame with another site inside its toolbar -- is when a user clicks on an image search result. That frame bothers a lot of site owners that Google links to, but it was a necessary tradeoff because image search is kind of sketchy and occasionally unreliable. Users click on an image result and get taken to pages that no longer have the image they were looking for, and the toolbar exists to help them identify the image they wanted -- NOT as a marketing-sticky thing for Google. And even though it's often beneficial, it still confuses a lot of users.

Note that mismatched content and URLs is a usability problem for Google Maps. The URL that many users copy from their browser address bar usually doesn't correctly represent the state that the user's in due to the way the site's AJAX works. AJAX or FRAME, it's the same problem.

Over the years, I’ve learned the following heuristics about my typical user base. With 14 dinner guests, only 11 will require chicken-of-doom. In Northern California, one person ends up vegan, and two vegetarian. Two are Jewish (no shellfish or pork), and one, despite mild alcoholism, is nominally Muslim and doesn’t eat pork either, so that rules out the bacon wrapping on the steaks. Once in a while, I have a guest who is severely allergic to eggs, gluten, and/or nuts, just to keep life interesting. Killing your guests is the opposite of entertaining, unless you happen to be Hannibal Lecter, so when in doubt about allergens, stick to tofu. Oh wait! people can be allergic to soy, too...

The Dinner Program: Throwing a dinner party with one knife, 11 militant chickens, and your design degree [PDF]. Ambidextrous, Summer 2007.

I listen to a fair amount of symphonic music.

I think of symphonic music first in terms of the composer (e.g. Bartok, Beethoven), then the work (5th Piano Concerto in E Flat Major), and then, in the cases where I have multiple performances of the same work, the conductor & performing orchestra (Julius Katchen & London Symphony Orchestra; Hélène Grimaud, Staatskapelle Dresden & Wladimir Jurowski).

However, music metadata format lists the conductor and symphony in the "artist" field on CDDB / iTunes. This is moderately annoying to deal with in the iTunes interface (I have to display and then sort by the correct columns of metadata) but absolutely intolerable on my iPod. Trying to listen to Smetana's Ma Vlast means that I have to remember the name of the composer and orchestra? Oh come on.

So I wrote an Applescript to fix the metadata on my symphonic catalog. It searches my library for all tracks whose genre is "symphonic" and swaps around some of the field values. I still retain the performance info, but as "album artist" instead of "artist".

(It similarly annoys me when other albums list both performers of a one-off song (e.g. Madonna and Justin Timberlake), which breaks that one song apart from the rest of the album when I sort by artist. In these cases I manually reset the "artist" field.)

* What people generally call "classical," but I refer to it as "symphonic" because of a snobby old habit of mine: "classical" refers to a specific period / style of symphonic music in the Western canon, e.g. Haydn and Mozart, so I use the more general "symphonic". I guess "orchestral" might be more appropriate... anyway, the organizing genre in my iTunes library is "symphonic".