I was at a party this weekend and listened to our self-described non-techical hostess introduce me as, “this is Alistair. He does computer stuff.”

Which, while accurate, doesn’t really lead to a rousing chat over drinks.

I began with the usual explanation: “The company I work for makes web performance monitoring equipment.” There were a couple of nods, but mostly the same blank-faced, I-don’t-care look I get when I try to explain a recipe to someone who hates to cook.

I went to my back-up explanation: “You know how you go to a website, and it’s really slow? Or you get one of those 404 errors?” (nods from everyone, with a couple of MySpace and YouTube complaints thrown in.) “Well, we fix that.” A bit more understanding, but not a lot of excitement.

So I tried a new one. It’s not really a new explanation, but as more CIOs and directors get their hands on TrueSight and our Real User Monitoring data, it’s a conversation I’m having more and more. I took a drink, and continued:

“You know how geeks like me are always talking about networks, and computers, and browsers … and you don’t care?”

Vigorous nods from everyone (eager for me to shut up so they can go get another glass of wine or look at our hostess’ spectacular apartment.)

“And you know how business, rather than caring about the geek stuff, worries about people — customers, employees, and so on?”

More nods.

“Well, we let the geeks talk about users, so the business finally understands them. In other words, watching real users makes IT matter to the business.”

Much better. All of a sudden they cared.

It’s true. We used to sell to the web operators who had to handle incidents and answer phones, who lived a life of blame and conference calls. They’re our most active users and eager proponents.

But now, along with those guys, the executives in the company are getting involved. They’re frustrated by the huge disconnect between what the IT team does — servers, networks, and applications — and what the business is about. And they’re looking for ways to get IT and the business working together.
The first step is to get them talking. For years now, IT and business haven’t been speaking the same language. And it’s been a source of huge problems; IT has useful information that business doesn’t care about; and the business has goals for its IT infrastructure that IT can’t turn into meaningful, measurable goals.

As more and more people in an organization get access to Real User data (what we call, Pervaisive Real User Intelligence,) it’s amazing just what an impact it has on the way the organization operates. Not only does it get easier to fix problems and resolve disputes, but businesspeople start being engaged in the IT systems themselves.

I think the British TV show The IT Crowd said it best. I haven’t seen the episode in a while, so some of it may be NSFW. But the first five minutes of this episode pretty much sum up a traditional business view of IT. If you’re in IT, and you’re not talking about real users, the business probably doesn’t understand what you’re saying.

Most of our customers have three different tools for determining the health of their web applications. Component monitoring tools look at the elements themselves, such as RAM or queue depth. There’s always a value there (even if it’s “0″). Testing tools generate activity, and look at the results. These are used primarily to load sites and measure capacity, or see if a known function is working. The number of measurements is directly related to the number of tests. And Traffic monitoring — whether it’s packets, weblogs, or, in our case, user sessions — looks at activity generated on a site. This can include real users as well as tests.

What’s interesting is how complementary these three are. Component tools are a lousy way to assess the user experience, but they’re absolutely essential in localizing to root cause. And the other two — tests and traffic measurements — serve very different purposes.

At first blush, synthetic testing and real user monitoring seem competitive. We’ve certainly had customers say, “with you, I don’t need to spend money on testing.” But let’s look a little more closely.

At this point I’m going to get a bit philosophical. I took some fascinating philosophy of language courses in university (from a professor who was freakishly like Stephen Colbert at the time), and much of it revolved around the precision of knowledge. There’s a huge difference between what we know to be true, and what we can’t say we know not to be true.

Try to answer two questions. The first: Is your website working? To answer it, you might try the site. If it works, you answer, “yes!” For you, at that time, in that place, with that page, it worked.

Now answer, “is your website broken?” It would seem at first blush that the answer is a resounding, “no!” After all, you just tried it, and it worked. But do you know that it isn’t broken for someone else, on some other page, from some other place?

Philosopher Bertram Russell used the example of a teapot, in orbit around Mars, to discuss the latter point. Do you know there isn’t a teapot orbiting Mars? British upbringing to the contrary, I doubt very seriously that there is a teapot flying around the red planet. But I don’t know there isn’t. In fact, I’m certain that I don’t know. In the same way, most web operators don’t know that a user isn’t stuck or a part of their site isn’t broken. And this is one of the crucial differences between testing and watching traffic.

Right now, I’m pretty sure my Blackberry (where I’m typing this) is working. I can see and touch it; and I know how to check if it’s working: press keys and see text appear. I would say I’m certain that the typing function on my Blackberry is working right now. (And I know what broken’s like; I busted my last keyboard a couple of months ago. Might come from blogging on it too often.)

Many websites have a “test page” built by the development team that exercises the application. When requested, it hits a number of back-end functions, writes to and reads from a database, and generally returns a comforting message that all is well. This is analagous to knowing my Blackberry can type. It’s within my control, and I can assess it at regular intervals.

Real users, on the other hand, are like teapots. They definitely exist; I’ve met some in person. They might be doing something unexpected (like orbiting a planet.) We won’t know they aren’t unless we watch every possible user in every possible place. This is the function Real User monitoring provides. At the same time, if there are no real users, there are no results to analyze; which is why we need repeatable tests of known functions to detect problems quickly (there are other good reasons, like baselining and measuring reachability, CDNs, and DNS resolution.)

Put another way, testing does the same, limited, specific thing over and over to see if it varies or is different somehow (test.asp is unreachable from Kansas.) And real user monitoring watches many different things — each of them unique — and tries to find patterns that indicate a trend or problem (everyone is suddenly getting 10 seconds of delay; all requests for page.jsp get a 500 error.)

Unfortunately for web operators, the chances that a user is doing something weird, getting stuck, or having a bad experience are much higher than the likelihood of interplanetary dishware. Which is why (according to a recent Forrester survey) more than 70 percent of problens are still reported by end users.

Who knew web operations could be so existential?