Too Much Data?

A senior designer at Google recently left the company for an interesting reason. That reason? Design is dictated to too large a degree by data.

This is not a problem I have ever heard from libraries and their staff, but perhaps there are still some lessons we can pick up here. For the most part I am a data fiend. I rely on my usage statistics to judge the effectiveness of my library, I arrange my hours based on them (despite the fact that I would prefer to open up at 10:00 AM or later, my statistics just won't allow it), and I cancel journals and other publications because of statistics. This is not necessarily a bad thing. However, even I can see that demanding statistics to decide "whether a border should be 3, 4 or 5 pixels wide" is a bit ridiculous.

So where do you draw that line? When does data stop becoming important and start becoming nit-picky, or worse yet, a waste of everyone's time? I've just started thinking about the issue, but I think the answer has to do with when data collection takes more time and effort than the best possible solution is worth. Admittedly, that is an extremely hard problem to quantify and this is probably why Google can continue to justify their data demands. When a search engine is used by millions of people a day and generates billions of dollars in revenue, very small differences in usage by a very small percentage of users can still mean that a million people are negatively effected and you've lost hundreds of thousands, or even millions, of dollars. This is not the problems that most libraries face and for most libraries this kind of decision cannot be based on cost because they are non-profit organizations. So what do you base these decisions on? Mostly, I would say research. Look at your benchmarks or even unrelated organizations with similar audiences and see what similar changes have done for them. Off the top of my head, I would consider a 10% positive change to be worth a couple of weeks of work hours. Be sure to adjust that number based on how large your audience is and how many staff members you have. Also, be aware that if what you thought would be a 10% increase turns out to be a 10% decrease, then it may be time to revert to what you had before.

On a similar note, how big a number of complaints trumps how big of an increase in usage? Two sites that I use on a regular basis have recently gone through redesigns, Woot and Facebook. Neither site had a very positive response from current users regarding the changes. Facebook, in particular, had a huge public backlash against their new layouts with 100s of "I hate the new Facebook" groups popping up on the site itself, some of them with tens of thousands of members. Did many people actually stop using Facebook since the change? That's harder to say and I haven't yet found the numbers to tell me for certain, but my guess is that there wasn't much actual change in usage. Still, Facebook is responding to complaints since they've been getting a lot of press and they are trying to placate their disgruntled users. Should Facebook have bothered responding if the changes actually brought in more people? What if the number of users stayed the same? Complaining users tend to be those who will stick around and complain until they decide you're not listening and then they jump ship. If the number of complainers is significantly less than the increase in users, then I'd argue that you're safe to ignore them or at least not do anything drastic to appease them. You'll probably still try to address some of the concerns since happy customers are, after all, always better than angry ones. Most of this stuff I'm talking about is data driven, primarily by the usage data. On the other hand, people are fickle and all those students who complained when Facebook stopped being a site for students only and started being open to the public don't seem to have run away too far. In general, I do think you're better off looking at the usage trends rather than responding to the squeaky wheels.

Alright, I've pretty much gone off the rails now and stopped talking about when you can make decisions without data. Before I stray any further afield, I'll go ahead and wrap this up. When it comes right down to it you need to decide if collecting data is worth the cost of the collection. You may have to figure out your own formula to decide how this decision is made, but don't forget that sometimes data isn't worth the cost.