Web analytics practitioners, myself included, often express site behavior in terms of the number of unique visitors to a site.  Statements such as “we had 14,000 unique visitors to our site between the 1^st and the 15^th of February 2010” are quite common and, on the face of it, impressive.  After all, if unique visitor volume, in the lingua franca of web analytics, is considered a proxy for site popularity – why not use it?

However, the concept of a “unique” visitor gets confusing.  The more I think about it, the more I wonder if this concept is accurate. For example, if I go to the espn.com site from my desktop in the morning and then read a story in the afternoon from my mobile device, and return to buy merchandise late in the evening from my laptop – can I really be considered as three different individuals? Or should I be considered as one person making three visits in the course of a day?

In the absence of any reliable way to connect my identity across three independent visits to the espn.com site, a unique visitor count of three is wrong.  Second, even when repeat visits are from the same device, the counting of unique visitors becomes problematic given that cookies may get blocked or deleted.  Web analytics experts estimate that after about four weeks, about a third of cookies are missing (see http://www.advanced-web-metrics.com/accuracy-whitepaper).  This implies that when a visitor returns to a site from the same device after four weeks she is accounted for as another unique visitor. That’s bizarre, but true.

So, what is an analyst to do? The simple solution is to forget the unique visitor metric and simply use the visit count. In most cases, for sites that do not invite individual visitors to self identify simply doing away with the unique visitor count and using the visit metric allows the site owners to take the same business decisions.  For sites that do invite self identification a solution would be to take the ratio of logged in visits to the total visit volume among that group and apply the same split to the non-identified visitors to estimate an overall visitor volume. The overall point, however, is still the same.  Most decisions in today’s business environment that leverage web analytics data are not likely to be negatively impacted simply because we choose to substitute unique visitor data with visit information.

Coradiant started out several years ago with a single clear message – that understanding the end user experience (whether actual human beings or back-office clients) was fundamental to managing any application.  Since that time, the industry has evolved significantly, as it always does, leaving in its wake the broken remains of once-rising companies and rejected approaches.  Often, as experience and depth-of-understanding increases, initial ideas are overthrown in favor of more accurate theories that reflect lessons learned in the field.  Not so with end user experience.  Not only has this idea won through unscathed, it has seen wide adoption at every level of the performance monitoring space.  Analysts write about its importance and vendors of all sizes have integrated it into their solutions.  There is not a single first-tier or second-tier vendor of APM solutions that has not built or acquired end-user monitoring technology and added it to their solution – e.g. CA, HP, Compuware and Quest all have an end-user component in their solutions.  Companies in North America no longer doubt its value and expect to have such intelligence out-of-the-box.  Europe is trailing behind this trend but I expect that they will rapidly catch up over the next two years.

The fact that end user experience has survived indicates that it is a fundamental truth of application monitoring – if you don’t understand how your user is being impacted by your application, you cannot effectively assess whether you have an actual problem.  For example, if a primary database that has a backup goes down over a holiday weekend, should the IT manager agree to pay double overtime to have it fixed immediately?  If, in an attempt to prevent a repeat of the failure, the IT manager presents the CTO with a purchase order for a clustered set of industrial servers with fibre-channel connections, should he agree to that very expensive purchase?  These questions cannot be answered unless the impact of the problem is well understood and this impact must be expressed in terms of how it affected the end users of the business.  No other metric is nearly as important.  If few or no end users were affected, the expense of an emergency repair or a massive upgrade may not be truly justified.  The days of watching CPU, RAM and disk usage as a means of determining real impact are over.  Those metrics, although good to know and important in their own right, cannot effectively reflect the impact to the business the same way that user experience does.

The fact is that some truths never change.

Companies spend an inordinate amount of money setting up complex IT systems to manage and optimize service offerings to customers.  Often these management systems only look at a single metric, like conversions or performance. What if both could be correlated?

Imagine you are a high-traffic on-line ticket vendor for major events nationwide.  As the head of IT operations, you want to maximize conversions by ensuring that you can handle the traffic volumes generated by a major concert event or marketing push and that the response times for customer queries remain optimal. In spite of your best efforts, however, the latest concert promotion brought your site to its knees and some of your systems went down for a short period of time.  The impact of that outage on your users might be unacceptably slow response times or actual system errors in their browser.  In either case, if they risk losing out on good seats for the concert while waiting for your systems to recover, they will abandon your site in favor of a competitor that is more responsive and you will have lost that revenue. On the assumption that you could have sold 1000 tickets at $100 apiece during that outage, you could lose up to $100k of revenue, depending on the number of users affected.  That’s a lot of crumpets, as they say.

Poor response times and unreachable sites are bad for revenue. But there is also a third reason why your site traffic could be impacted that is not caused by the performance issues described above.  It may be that your site content is unappealing to the average eye or that your offerings are priced out of reach for the average consumer.  This is the typical explanation that marketing gives to conversion dips, but it is clearly not the only root cause.

This is where Coradiant Analytics In A Box can help.  Our passive capture technology combined with the included Google Urchin software collects performance and availability metrics about your site while it is tracking user behavior. This allows you to look at conversions (e.g. ticket purchases) in the context of performance and availability, thus allowing you to categorize conversion dips into those that are infrastructure related and those that are marketing and content related.  For example, your analytics data tells you that the 22% of your users bought tickets between 12-2 PM on Monday afternoon. You also see that this is about 35% less than the norm for a corresponding period.  Coradiant allows you to correlate this event with an increase in page load time or availability problems.

End-user monitoring coupled with analytics provides a powerful solution that helps to maximize customer visits and conversions.  It places those responsible for business results and for performance in the driver’s seat by providing them with the data they need to understand the real business impact of system outages, thus enabling them to provide the best possible service to their Line-of-Business managers.

I feel real sympathy for Marketing departments.  They are saddled with the enormously difficult task of profiling users in order to achieve more effective marketing efforts.  In order to do that, they must understand who the user is, what they did on the site and how they reacted to various marketing efforts.  Up to now, they have been relying on traditional Web Analytics solutions to get that data.  However, the price for that data is the insertion of page tags and tracking cookies.  Cloud-based services add on an additional price – the exporting of analytic data out to the cloud.  All of that might seem like a small price, but it is actually a true Faustian deal.

To start with, Marketing must subject itself to the busy timelines of Development, QA and IT whenever it wants to change its tagging or cookies.  Since those departments are often busy rolling out new features, Marketing often has to wait weeks or even months to get their new data.  In many cases, Marketing discovers a problem with the page tagging during the course of a campaign and are unable to roll out a fix quickly enough. In companies where this problem is recognized, the problem is reversed and Marketing is allowed to hijack the roll-out process with an emergency patch to its analytics tagging, often negatively impacting the delivery of important new features.  Both Marketing and IT would benefit if this link could be severed.

The other problem that Marketing is facing comes from Europe, where a wave of privacy regulation is forcing existing Web Analytics solutions to run for cover, leaving Marketing departments with little to help them.  Germany has passed very strict laws prohibiting the use of page tagging and tracking cookies without the user’s consent.  Moreover, shipping analytics data to a hosted service for processing is specifically forbidden.  Although privacy is often talked about in every part of the world, Europe is the first to have passed these kinds of laws about it – a trend that could easily spread outside EMEA.

The irony is that most of the data that Marketing often requires is already contained within the traffic stream before any tagging or tracking takes place – where the user is from, what browser and OS they use, which ISP they used, where they came from, where they went, what they looked at, what they bought, etc.  It’s either embedded in the HTTP protocol or as part of a web server’s natural ability to maintain a stateful application or it is directly within the request or response content.

Consequently, as announced last week, Coradiant has teamed up with Google to create our Analytics In A Box (AIB) solution.  This revolutionary new product uses Coradiant’s existing technology to passively process all user traffic to and from the web site, producing full-featured web analytics data that remains in-house.  Instead of inserting page tags or using special cookies, Marketing can define AIB rules that will extract the information directly from the traffic stream. This approach will allow Marketing to define new metrics whenever they want to – even in the middle of a campaign during peak hours.  Moreover, the solution refreshes its data every hour, providing Marketing with immediate results from any changes.  No other department has to be involved.  Moreover, the data is 100% secure and kept in-house, which means no privacy violations.

As Bogey said, “this could be the start of a beautiful friendship”.

TrueSight Load Test Adapter has been selected as a Network World Product of the Week.  This add-on VM for TrueSight allows Operations and QA departments to quickly convert production traffic into real-world usage patterns, rated by popularity and path length.  Examples of those usage patterns can be imported into HP’s LoadRunner test suite as a basis for building test scenarios.  This cuts short the debate about what to test or how to generate the important background noise required in any reasonable QA environment.  More importantly, it gives Operations and QA a means of talking about acceptable test scenarios using the same language.

Coradiant is holding an informative afternoon executive event in the Penthouse at Gary’s Loft in Manhattan focused on end-to-end Web application visibility.

You can register for the event here (https://www.123signup.com/event?id=jbxhq).
Presenters include:

• Dave Anderson, Principal Architect/Co-founder of Peopleclick,
• Dennis Callaghan of The 451 Group,
• Fred Dumoulin of Coradiant
• Andreas Grabner, dynaTrace software,

Gary’s Loft in Manhattan features a stunning 360 degree view of the Manhattan skyline.  Light food, beverages and hospitality will be provided by Tip Of The Tongue NYC.

Date: Thursday, November 12, 2009
Time: 4:30 PM to 6:30 PM ET

Gary’s Loft
28 West 36th Street
Penthouse
New York, New York 10018

I am astonished by the number of customers I have spoken with that are having trouble creating accurate load testing scripts.  Even worse is the heavy lifting they must go through to create realistic scenarios.  It’s time-consuming.  It’s expensive.  And it doesn’t always produce the right results either.  It can be quite discouraging.  Although there are other products that produce LoadRunner scripts from real traffic – e.g. TeaLeaf or HP RUM – they only do so based on a single session.  Although this is fine for reproducing a single one-off scenario (i.e. functional testing), this does nothing to help with load testing since it still leaves the user in doubt as to whether this single scenario represents a popular or common click path.
Our flagship product, TrueSight, is able to see all of the production traffic and recreates click paths, which we display in the Session Browser.  Leveraging this ability, we decided to help eliminate guesswork and put back the accuracy into load testing.  I am pleased to announce Coradiant’s first product that will help in that effort.
Today, we officially introduced the TrueSight Load Test Adapter, packaged as a VM that runs in VMWare Server or ESXi.  It takes session data from any TrueSight appliance and converts it into click path statistics, from which our customers can generate endless varieties of load testing scripts that can be directly imported to LoadRunner.  Users can sort the click paths by popularity or by number of pages.  They can filter to find click paths with a minimum length or that pass through a URL that contains specific text.  They can even create week-long data sets of click paths that focus on specific applications (using Watchpoints) and then generate their scripts by focusing on specific time periods within that data set.

This product has a robust road map and will see additional releases over the coming year.
If you are having trouble generating realistic scripts or simply need a script that reproduces production behavior as a background to your current stress testing, you owe it to yourself to check this out.

Another traffic jam this morning. As I inched forward, I brooded over the similarity between highway design engineering and the design of Web applications.  It may look good on paper, but when the real world intervenes, all bets are off.

In the world of the Web it’s the same way. Sure the testing and pre-deployment steps are critical, but as Web applications become more complex, it’s not enough. It’s important to thoroughly test your Web applications.  But it’s even more important to “test” your Web applications after deployment to make sure that they continue to run correctly.

In the pre-deployment world there is a lot to manage; make sure that all the application logic and algorithms are sound, have users run through the UI, hunt and kill a load of bugs, load-test the application, perform integration testing to ferret out conflicts with other applications, make sure all the system components required to run the application are there and that you’re running the right version in the targeted infrastructure, and so on and so forth. It’s a daunting task. But at least then you have full control.

But after it goes live, you’ve lost some of that control. That’s the unpredictable nature of the Web.  You don’t have much say in how networks perform, or which devices may be running your application on the user side.  And Web users are fickle, with hard to predict usage patterns. Turn on your perfect application and soon you’ll get a call from your IT operations people passing on a customer complaint relating to some totally unexpected thing.  Real users always do the unexpected. You can’t always predict every scenario. Similarly, no amount of testing using traditional IT tools can possibly find all the hidden problems that can pop out in live deployment situation.  Throw in the complexity of the Web application infrastructure, and you’ll be spending a lot of time looking for hidden problems that might have nothing to do with your code.

Knowing the end-user experience is key. Organizations that run web applications have discovered that the end-user experience is the key metric for success. By understanding what is being delivered in real-time, you can prevent dissatisfaction and application abandonment.  If you serve your users well, the application is perceived as successful. If you fail in delivering the service expected by users the application is deemed a failure.

OK – so we can’t really test our highways and make dynamic changes after it’s built  – yet, but you can certainly do that with Web applications.

The key is to provide the right tools. Web Application Performance Management solves the dilemma of gaining real-time visibility into end-user performance, and providing the actionable information you need to make decisions and changes to keep everything running at the speed limit.

Contrary to popular belief, the brain is not a Personal Video Recorder, recording everything submitted by your various senses.  That would be too much data for any brain to handle.  Instead, it sifts through sensory input looking for relevant data points that it can trust and throws everything else away.  The important words in that last sentence are “relevant” and “trust”.

If a data point is not relevant, then it is considered to be a distraction.  There are well-known studies on Inattentional Blindness and Change Blindness which demonstrate that even large-scale events can be filtered out by the brain if they are considered irrelevant to the task at hand.  Similarly, if the data point cannot be trusted, the brain tosses it out as well (whether your senses can be trusted has been a heated debate in philosophy for centuries, but I digress).  Trust and relevance are crucial to the brain’s ability to eliminate useless noise and derive good results.

These same principles apply to monitoring your web applications.  Instead of monitoring the universe, you should be reducing your data flood to those points that are relevant.  Moreover, you should only be using the most trusted tools and methodologies to draw conclusions.

For web applications, the most relevant data is the data that directly describes or explains your user’s experience and places it in context.  In order to identify that data, you must be able to draw a direct line from your user’s experience to those data points.  If you cannot do that, you are probably chasing your tail and wasting a lot of valuable resources. It is important to realize that a lot of tools cannot draw a direct line from user experience to monitoring data without leaving a few gaps and logical leaps of faith.

As an example, operations teams love to know whether a database is down.  Although this is valuable data, is it relevant?  If users experienced worse performance around the same time, does that mean that fixing the database will solve the performance problem?  In fact, in a well-architected environment, the loss of a web server, app server or database should have little, if any, effect on the end user’s experience due to clustering and load-balancing. A lot of solutions love to use time correlation as a magnificent leap of faith, but it simply makes unreliable conclusions look enticing.

To draw that line between user experience and environmental monitoring, you need a tool that can see the actual users’ experience and is able to relate it directly to problems in your network, application design, deployment, code quality, etc.  Moreover, it must prove itself to be a trusted source of information, returning results quickly and reliably without drowning you in irrelevant data.  In other words, it must be trusted to extract and analyze relevant information and return high-quality results.