How Aggregate Displays Change User Behavior

A fascinating study demonstrates how simply displaying aggregate data like Top 10 lists heavily influences the way people make decisions on social web sites.

Aggregate displays are everywhere, from the book ratings at Amazon.com to the most-emailed articles at the New York Times to the number of diggs at Digg.com. They’re a primary element of social design. They not only let people know how their actions relate to others, but they also alter the behavior of those who view them.

Columbia sociology professor Duncan Watts has written the fascinating piece Is Justin Timberlake the product of Cumulative Advantage?, describing a sociology experiment that has huge implications for the display of aggregate data on social web sites. (thankfully, the article isn’t about Justin Timberlake at all. The author doesn’t even mention his name…probably titled by an editor at the Times)

Description of the Experiment

Watt’s study demonstrates how socially influenced we are as we use software and make decisions online. He describes an experiment in which they built two web sites. In one web site they showed a list of songs to users and had the users rate the songs as they listened to them, allowing the users to download the songs if they wanted to. In the other, they also allowed the users to rate and download the songs but they also showed the download numbers to the users. In the second group the users could see how often the songs had been downloaded as they used the site. The difference was in the display: users either saw the aggregate display of downloads (called the social influence group) or they did not (called the independent group).

In addition, they split up the social influence group into 8 parts. They replicated the same test 8 times to see if there were any differences over time.

Watts describes how the download numbers could be used to test:

“This setup let us test the possibility of prediction in two very direct ways. First, if people know what they like regardless of what they think other people like, the most successful songs should draw about the same amount of the total market share in both the independent and social-influence conditions — that is, hits shouldn’t be any bigger just because the people downloading them know what other people downloaded. And second, the very same songs — the “best” ones — should become hits in all social-influence worlds.”

In other words, the mere fact that users can see the downloads is being tested. A simple display difference. A design decision.

Does Seeing Aggregate Data Change Behavior?

Does displaying aggregate download data change the behavior of users? The answer is Yes. Their findings:

1. The most popular songs in the social influence group were way more popular than those in the independent group. In other words, the rich got richer when people could see the aggregate data.
2. The popular songs in the 8 social influence groups were not the same! That is, the download numbers affected the popularity of the songs. Early download leaders continued to lead not just because they were good songs, but because they were already leading.
3. The independent group was considered the test for quality. The songs that were most downloaded were considered the highest quality because everybody voted independently…there was no social influence since download numbers were not displayed. These songs did correlate slightly with the songs in the social influence group that did well, but did not have much of an effect overall.
4. The social influence group was influenced much more by the number of downloads than by the quality of the songs. The social influence had a stronger effect on the song downloads than independent, unbiased decision making.

This result could be seen as a confirmation of the bandwagon effect, a known bias resulting from our tendency to follow the crowd. This bias is probably the result of ignorance…if we don’t know something we tend to rely on the opinion of others. In this case users probably paid attention to the download numbers because they didn’t have any prior experience with the music.

Outcome is Unpredictable

One outcome is that predicting what will happen in social influenced situations is impossible. Watts explains:

“In our artificial market, therefore, social influence played as large a role in determining the market share of successful songs as differences in quality. It’s a simple result to state, but it has a surprisingly deep consequence. Because the long-run success of a song depends so sensitively on the decisions of a few early-arriving individuals, whose choices are subsequently amplified and eventually locked in by the cumulative-advantage process, and because the particular individuals who play this important role are chosen randomly and may make different decisions from one moment to the next, the resulting unpredictably is inherent to the nature of the market.”

In other words, early leaders tend to stay in the lead simply because people see they are leading and are influenced by it. If a song gets an early lead then people assume it is because of quality, but the case may simply be that it happened to be one of the first listened to.

Huge Implications for Social Site Design

This result has huge implications for all social web sites, especially those that show aggregate data. Digg, for example, shows aggregate data everywhere on the site. This experiment, in addition to several other issues that I wrote about in Digg’s Design Dilemma, suggest that the results there are socially influenced to such an extent that it would be hard indeed to know where the quality lies…

It also leads to interesting questions for those building social sites. What data do we aggregate? What should we display? Where? What influence will it have on the future behavior of those who see it? Does it influence the quality of content? How so?….etc…etc…

Another level of complexity can be added on top of this. What if the users knew more about the download numbers? For example, what if a user knew their best friend hated one of these songs? Of course that’s going to affect our decision as well, and maybe moreso, because the user trusts their friends more than a simple download number.

A Note of Caution

Finally, a note of caution. Over at Publishing 2.0 Scott Karp extrapolates this finding further, suggesting (somewhat tongue-in-cheek) that it explains other phenomena like Web 2.0 and the blogging A-List. He wonders if the A-List is just riding a wave of initial popularity.

We should be careful to push this that far, however, because some things that Karp mentions happen over a lot longer a period of time and have many other factors involved. The A-Listers, for example, drop like a rock if they don’t continue to post and get linked to. They quickly lose their advantage in the face of low post output and other changes over time. In the study Watts tested the ratings for song preferences. Songs, of course, don’t change over time. My guess is that the result will apply relatively well to things that don’t change…songs, movies, texts…and less to things that don’t.

If Watt’s experiment is solid, however, it should help inform anybody building a social web site. If people are unduly influenced by aggregate data, then we cannot make any assumptions about the quality of it. In other words, much of the success in social systems is based on that most horrible of reasons…luck.

CUE: A Usability Testing Bake-Off

By Jared M. Spool

Originally published: Aug 19, 2005

Practicing usability testing has great similarities with baking an apple pie.

If you've never baked a pie and you're like most people, you'll probably learn from a friend or family member, such as Grandma, or you'll learn from a cookbook recipe or TV cooking show. The first few pies you bake—while still tasting pretty good—probably won't come out as good as you'd like, but, with practice, you'll get better and better.

As you continue to practice, you'll experiment—sometimes deliberately and sometimes through creative accidents, like leaving out a key ingredient—and from each experiment you'll learn more about what makes a great pie. As your pie baking skills become more developed, people will beg you to bake more pies and will show great excitement when another one comes out of the oven.

At least, that's been my experience with baking apple pies. Also, coincidentally, it's been my experience with usability testing.

The Parallels

Many folks learn usability testing from a colleague or they read about it in a book or attend a course. The first few tests they conducted—while producing interesting results—probably didn't come out as good as they would have liked. However, with practice, they got better and better.

As they continued to practice, they started to experiment and from each experiment, they learned more about what produces a success usability test. As their testing skills became more developed, their coworkers begged them to conduct more and more tests and showed great excitement with every new test.

Solitary Activity

Both baking apple pies and conducting usability tests are solitary activities, for the most part. The baker and the test organizer typically produce the results by themselves.

Rarely do we get to see how someone else bakes their pies. Rarely do we get to work alongside other usability test organizers to see how they do their work.

Therefore, we develop our craft by ourselves, only really learning from our own experiences. Because of our own trial and error, our testing techniques and skills evolve in a unique direction

The Value of the Bake-Off

In the world of baking apple pies, communities of pie-bakers gather at events known as "bake-offs." These festivals give the bakers a chance to compare their results, often under the guise of choosing the best pie.

While it's great to win the bake-off, the real benefit comes from learning how others created their entries. Seeing the different techniques and ingredients they used can give great insight into ways to improve your pies in the future. The bake-off is an opportunity to compare and contrast your approach against the approaches of your peers.

CUE: A Usability Testing Bake-off

In 1998, Rolf Molich held what we could call the first usability testing bake-off. Instead, he called it a Comparative Usability Evaluation or CUE. However, the goals are the same.

So far, Rolf has held four CUE sessions, each one comparing top-notch practitioners demonstrating their skills. In each session, each practitioner evaluated the same interface, using his or her own personal techniques. The results have been fascinating.

In an apple pie bake-off, the chefs often compare their recipes to their own. In Rolf's sessions, participants compared their techniques, looking for interesting differences.

For example, practitioners each wrote a report that Rolf turned over to the client whose design they were evaluating. The participants could compare their reports to the other teams and see what they did differently.

In fact, all the teams produced very different reports. They also planned the projects differently, recruited their participants differently, and created different tasks, even though they were all testing the same interface.

Comparing Leads to Improvement

By comparing our work against others doing the same thing, we can easily see opportunities to improve our own work. For example, reading the various practitioners' reports, we can get ideas on new sections to add to our own reports and clever ways to describe certain problems.

Even if we didn't participate in the CUE session ourselves, there are still interesting things to learn. We can look at the instructions Rolf gave each participating practitioner and then ask ourselves, "How would we have executed this project?"

We can think about how we'd recruit users. How many users would we recruit? Would we exclude users already experienced with the interface, focusing only on newbies? Alternatively, would we try to balance the test for both experienced and inexperienced users?

How would we design the tasks? Would we focus purely on structured tasks, trying to collect timing information and success rates? Alternatively, would we look at problem discovery, trying to uncover every problem available?

By comparing what we would do against what the CUE practitioners actually did, we can see how our techniques differ and get ideas on how we could do things differently going forward.

Rating the Problems

Every apple pie tastes a little different. It's neat to taste a sample from multiple pies, one after the other. You can really see how different approaches produce different results. Some our more tart and some are sweeter. Some have a chunky filling, while others have a crisp topping.

The same is true when you compare the problems that the CUE practitioners found. You can look at a problem, such as a particular function seeming unintuitive, and decide if you would have rated that as a severe problem or as something not worthy of much attention. Reading the CUE problem reports will make you think about how you report problems and rank their severity.

It is fascinating to see how some practitioners rated certain problems as critically important, while other practitioners rated the same problems as low priority issues. Many colleagues, after looking at the CUE results, have revisited their severity ranking process, to make sure they are accurately rating the problems they discover.

Striving for Constant Improvement

Because we're often working by ourselves, having an external reference point to compare our work to is important. For apple pie bakers, the bake-off is where we can learn from others. In the world of usability testing, the CUE studies are quickly becoming the focal point for these comparisons.

If you're thinking about conducting or already conduct your own usability tests, you'll definitely want to attend this year's UI12 Conference, where Rolf Molich will present his highly-acclaimed session, Advanced Methods for Usability Testing.

Usability -- Not User-Friendliness -- Is The Key To ERP Success

A software solution that scores high in usability will shorten implementation time frames in turn enable a faster return on investment. By Sharon Ward, Worldwide Manufacturing Industry Director, Microsoft Business Solutions, Microsoft Corp.

April 18, 2007 -- Buyers of manufacturing software have been looking for the "holy grail" of user-friendliness for years. User-friendliness is a frequent discussion topic when companies evaluate enterprise resource planning (ERP) but it is one of those things nobody can really define, but everybody knows it when they see it. To begin to define usability, it is best to start with the anticipated benefits such software solutions can deliver. A software solution that scores high in usability will shorten implementation time frames and reduce the amount of training required to go live, in turn enabling a faster return on investment and delivering benefits more quickly. Such a system will result in a lower total cost of ownership, is likely to change and grow with your company, allows for easy upgrades and interoperability and makes it much less likely that it will need to be replaced to enable future business processes. Software vendors are adept at making their ERP look user-friendly. Carefully scripted and well-prepared demos can mask the more cumbersome aspects of day-to-day business processes. Selection committees are not usually made up of people who process transactions all day, so extra screens, background processes or required keystrokes may go unnoticed. And matching up the software business flows with your in-house business processes doesn't really tell you anything about the system's ability to support your future needs. Even a hands-on session won't help you make the right choice in this crucial decision. So how can you judge whether software is truly usable? Flexible IT Architecture Surprisingly, the vendor's IT architecture is just as important as the functionality of the system to your future success. First and foremost, the architecture should use industry standards. In addition, a good IT architecture should be lean, just like a manufacturing facility. It should have everything that you need -- including strong security features -- but shouldn't include or require you to purchase a lot of stuff you don't need. Next, look for application flexibility. Your business will change; how easily can the software change with you? A role-based development process and real-life usability testing are two key things to look for when interviewing a vendor. Business processes should be easily changeable through switches or table entries. The underlying design philosophy should make it easy to customize the application if necessary to support a differentiating business process, and the application should be easy and inexpensive to upgrade as new releases come out, even after customization. In addition, there should be built-in workflows, analytics and a world-class portal with role-based metrics available. If your prospective vendor can't show you evidence of these, then it is not likely to make the grade in terms of usability. Works The Way You Do We all know that business processes don't remain static for very long. Who reading this article does business in exactly the same way they did even five years ago? Nobody -- or you wouldn't still be in business. So much has changed so rapidly that companies are continuously reinventing themselves -- and you need systems that can keep pace with the changes. Even recently it was inconceivable that customers would enter their own sales orders and configure products themselves over the Web. Today they demand this ability. Suppliers release their own purchase orders against contracts, or participate in vendor-managed inventory programs. Who knows what changes will occur in the future? Software needs to be flexible and adaptable as new ways of doing business emerge. Today more and more companies compete based on the speed of their supply chain. Flexibility, communication, collaboration and visibility are the most important enablers of supply chain velocity and transparency. How can you ensure that you provide supply chain transparency? In most cases, it means providing portal access and other collaboration tools to customers and suppliers. A portal is the best and fastest way to share documents and to collaborate around the world at all hours of the day or night, every single day. A portal must be easy for users to access and set up without requiring a lot of IT help, and yet must include strong security. Also look for pervasive workflows that notify users, customers or suppliers of events or delays that can affect business results. Workflows should be able to communicate across company boundaries, have easily set thresholds and be easy to create or modify without requiring IT intervention. Familiar Interfaces It is important that systems are easy to navigate for people who may use them only occasionally as well as for power users. Information workers today spend a large portion of their workday using their e-mail applications, probably the most widely deployed applications in the world. Messages from co-workers, customers and suppliers constantly flow through e-mail inboxes. Rather than jump from an ERP system to e-mail, it's simpler if business applications not only look and feel like e-mail but are actually integrated with it. A familiar interface enables workers to feel comfortable with an application immediately. And since industry analysts agree that one of the biggest costs involved in implementing new ERP systems is training, which often equals or exceeds the cost of the actual software purchase, eliminating the need for some training can speed up the implementation time frame dramatically, leading to faster ROI and time to benefit from the ERP investment. Confident Decision Making One of the lesser-known disadvantages of earlier generations of ERP is that software rarely communicated information to users about upcoming problems, unless the user was savvy enough to ask the right question at the right time. If the user was lucky or smart enough to stumble on a problem, deciding how to solve it was more often a matter of intuition than data analysis. The tools available were simply too slow and cumbersome, especially as the pace of business got faster and faster. Data warehouses and other analytical tools usually rely on periodic updates, forcing users to work with less-than-current data. Disparate systems often yield conflicting information. No wonder most manufacturers ran by the seat of their pants. They simply had no tools. Today it's possible for analytics engines to be embedded in applications or in the IT stack. Engines can access information from multiple disparate systems instantly and present the user with up-to-date information in a graphical format. Some engines and search tools can combine both structured data, like that found in ERP databases, and unstructured data, like that found in documents like contracts or RFPs (requests for proposal), to present the user with an immediate and complete picture. Conclusion It's rare today that a company has no IT systems in place, so it's important to select applications that run on industry-standard platforms so that applications can easily interoperate. Investigating the technology that surrounds required business functionality can be even more important than the functionality itself. After all, it's relatively easy to add new features to a system, but it's often impossible -- or at least difficult and expensive -- to change your IT architecture. Sharon Ward is worldwide manufacturing industry director for the Microsoft Business Solutions group at Microsoft Corp. In this role, she is responsible for setting the global industry strategy and ensuring communication throughout Microsoft's global industry community. http://www.microsoft.com/dynamics

Yes, you should be using personas

Personas seem to go in and out of fashion. Not long ago, people were advocating hyper-researched personas done in painstaking detail, these days designers seem more inclined to leave them out of the process.

So, are personas actually useful or should we stop wasting time and ditch them?

I first came into contact with personas in an academic context. They seemed like a nice idea but I tended to use them to justify my design rather than to guide design, which seemed kind of back to front.

Since then, I’ve worked in places where personas are more or less embraced as part of the process, and then longer I use them, the more I’ve come to the opinion that personas are incredibly valuable, but not for the reasons that many people think they are.

If I’m working on a UCD project (and thankfully, these days that is pretty much every project I do), then I would much prefer to include persona development in the process than not.

But, having said that - I find personas virtually useless when it comes to design, and I very rarely reference them in making design decisions. For me, personas aren’t about design, but that doesn’t mean they’re not incredibly powerful in other ways.

Personas communicate the user centred process like no other method

Having your clients view user research and testing is incredibly powerful in helping them realise that there is a problem in the way they’ve been approaching things to date (if you’re not encouraging stakeholders to actively participate in observing research and testing you’re missing out on a lot). But to get them to actually understand what user centred design is about - you need personas.

Personas should always be developed collaboratively with key stakeholders - as many as possible. They can often be derived from existing marketing personas or profiles (but, don’t be mistaken that personas that the marketing department gives you are personas you can use without any work). You should try to validate your personas with some kind of user research if at all possible - this can be in the form of some contextual interviews, lab based studies, or by talking to people in the company who interact with user directly on a regular basis (I’ve found people who work in call centres can often provide invaluable insight to what users are really like).

Personas should define the boundaries for which you will design. It’s a common misconception that personas are about creating a set of ‘typical’ or ’stereotypical’ users. Much more useful is to use personas who are edge cases. Creating ‘edge case’ personas and then prioritising personas and their goals is much more useful in helping decide what functionality goes in and what doesn’t. And which of the tasks or outcomes are most common, or more important. Decisions which are critical foundations to allowing good design work.

Personas are powerful in guiding requirements definition.

Even before you start designing you need to know what the product is and what it does. What are it’s core feature and what are it’s advanced functions. Personas really help guide discussion and decision making on requirements.

It’s generally agreed that one of the biggest factors for bad user experience is featuritis. Products that do too many things, or that prioritise advanced functionality over core functionality.

Putting each function to the test using personas before letting it in the door is very helpful. Then giving features relative priority using the number and importance of personas who are using them helps guides the overall application or site’s design priorities.

Using this evaluation process makes putting the user at the centre of the design process habitual and natural, and is also very helpful in reaching concensus within a project team.

Personas kill the ‘elastic user’

If I’m managing to inspire you to use personas more or differently, and you haven’t read it recently, let me encourage you to take the time to read Allan Cooper’s ‘The Inmates Are Running The Asylum‘. It’s a classic text that is renown for making programmers angry because Allan isn’t particularly flattering to them, but it gives real insight into the original ideas behind personas and I think you’ll find yourself even more inclined to use personas regularly.

I re-read it earlier this year and I particularly took away the idea of the Elastic User. This is where stakeholders make statements about what ‘users’ want, what ‘users’ do, what ‘users’ prefer, and because the ‘user’ in that context is so undefined and broad, they are able to say almost anything they like and there is no real way to contradict that opinion.

The creation of personas means that user groups are much more defined, so broad sweeping statements about what users want can actually be tested against something. Rather than having a free pass to do anything to the requirements or design by just using the word ‘user’, these assertions can now be tested and validated against more closely defined user characteristics and goals.

Use what you’ve got to build personas

Sometimes you’ll have a whole lot of research, sometimes you’ll have practically nothing. Either way, personas can be useful. Obviously the more you know the better, but even just a quick verbal sketch can be extremely powerful.

Not long ago I used the following persona in a design workshop:

‘Betty is a pensioner who lives in suburban Newcastle and is in charge of the local Neighbourhood Watch chapter’

That’s it. But it made a huge difference because it created an edge case persona for a site that was originally targeted at statisticians. ‘Creating’ Betty allowed the team to see that designs would work perfectly well for people who understood the language of statistics would be completely useless for Betty. And we found that pages that were really *for* Betty hadn’t actually be designed appropriately for her at all.

Sometimes, a pencil sketch persona can provide massive bang for buck.

You should care what car your persona drives

One of the things that make personas easy to criticise is that sometimes they feel like an exercise in creative writing.

‘Clare is 28 and lives in Primrose Hill. She drives a Prius but only on the weekends and she has a puppy that she takes walking twice a day. Clare is single but dates frequently and likes to travel.’

For as long as you think personas are about design you’re right - what does this have to do with design? When you use personas predominantly as a communication tool, then these small details become all important. After all - what you are trying to do is humanise the ‘target audience’. As humans, it is exactly these little details we care about, it’s what helps us to relate to other people and what makes them real. Does it really *matter* what car Clare drives? Well… kind of. People choose things like cars to drive and places to live as a means of communicating who they are. It adds depth to Clare’s personality. But, I’d never advocate actually *researching* those details. They’re communication devices, if people don’t know what Prius *means* then there’s no point using it. Make sense?

Don’t design for personas

As I mentioned before - personas should be edge cases. From your personas you can work out what functionality is included and what is the core functionality. Once you’ve got that information - and you’ve immersed yourself in understanding the users through the creation of the personas - put them away and use your skill as a designer.

If you use the personas to closely guide your design you will end up supporting a series of edge cases. This will invariably mean that your CORE functionality is compromised. That’s bad design.

Once you’ve designed the product, you can then use the personas to evaluate the design, and to help communicate to your stakeholders how and why your design supports the goals and tasks of your personas.

Don’t be tempted to use the goals and task lists you generate for your personas as a jigsaw puzzle to layout on a page. That way failure lies.

So. Personas. I’m a fan, but you have to know where their power lies and use them appropriately, and don’t get too caught up in the process of their creation - do what you can with what you’ve got, bring your stakeholders on board and teach them how to be user centred, and get on with the important business of designing great user experience.

(Need more background on personas. Austin Govella has collected a bunch of useful links)