I feel real sympathy for Marketing departments.  They are saddled with the enormously difficult task of profiling users in order to achieve more effective marketing efforts.  In order to do that, they must understand who the user is, what they did on the site and how they reacted to various marketing efforts.  Up to now, they have been relying on traditional Web Analytics solutions to get that data.  However, the price for that data is the insertion of page tags and tracking cookies.  Cloud-based services add on an additional price – the exporting of analytic data out to the cloud.  All of that might seem like a small price, but it is actually a true Faustian deal.

To start with, Marketing must subject itself to the busy timelines of Development, QA and IT whenever it wants to change its tagging or cookies.  Since those departments are often busy rolling out new features, Marketing often has to wait weeks or even months to get their new data.  In many cases, Marketing discovers a problem with the page tagging during the course of a campaign and are unable to roll out a fix quickly enough. In companies where this problem is recognized, the problem is reversed and Marketing is allowed to hijack the roll-out process with an emergency patch to its analytics tagging, often negatively impacting the delivery of important new features.  Both Marketing and IT would benefit if this link could be severed.

The other problem that Marketing is facing comes from Europe, where a wave of privacy regulation is forcing existing Web Analytics solutions to run for cover, leaving Marketing departments with little to help them.  Germany has passed very strict laws prohibiting the use of page tagging and tracking cookies without the user’s consent.  Moreover, shipping analytics data to a hosted service for processing is specifically forbidden.  Although privacy is often talked about in every part of the world, Europe is the first to have passed these kinds of laws about it – a trend that could easily spread outside EMEA.

The irony is that most of the data that Marketing often requires is already contained within the traffic stream before any tagging or tracking takes place – where the user is from, what browser and OS they use, which ISP they used, where they came from, where they went, what they looked at, what they bought, etc.  It’s either embedded in the HTTP protocol or as part of a web server’s natural ability to maintain a stateful application or it is directly within the request or response content.

Consequently, as announced last week, Coradiant has teamed up with Google to create our Analytics In A Box (AIB) solution.  This revolutionary new product uses Coradiant’s existing technology to passively process all user traffic to and from the web site, producing full-featured web analytics data that remains in-house.  Instead of inserting page tags or using special cookies, Marketing can define AIB rules that will extract the information directly from the traffic stream. This approach will allow Marketing to define new metrics whenever they want to – even in the middle of a campaign during peak hours.  Moreover, the solution refreshes its data every hour, providing Marketing with immediate results from any changes.  No other department has to be involved.  Moreover, the data is 100% secure and kept in-house, which means no privacy violations.

As Bogey said, “this could be the start of a beautiful friendship”.

I spend a portion of my time working with Coradiant customers on session tracking strategies that ensure a complete view of the end-user experience. Session tracking gives us the ability to see clearly every action, every page and every object associated with a user. Proper session tracking achieves what we refer to as user awareness. This is different from identity awareness, which allows us to put a face to the session. I’ve written in more detail about the differences between identity awareness and user awareness, but I can recap briefly for the purpose of this article.

Identity awareness means that you know the unique ID of a specific user on the site, allowing you to put a name (or even a face) to that user’s activity – e.g., user XYZ can now be referred to as “Jonathan” or “jginter”.  Oddly enough, this is fairly simple since it only requires that users identify themselves (by providing an ID) at least once during their session – e.g., as part of a login procedure, etc.  If even one hit during the session carries the user’s personal ID, we can be configured to pull it out and place it on the session we are tracking.

I just said a mouthful and didn’t underscore it, though.  I said that we were tracking a session already, which means we would have already achieved user awareness.  To do this is no small feat, since it requires that we be able to identify the subset of hits that represent the activity of a single unique user.  This is the foundation of any good monitoring solution.  Moreover, we must find these related needles out of the haystack that is the river of traffic we are monitoring.  Ironically, this could be very simple, but most sites make it very hard.  However, we can resolve this through a couple of simple suggestions.

As I have said before, most sites host traffic that is unintentionally anonymous – i.e., there is nothing on the hit that would identify who it belonged to.  Even sites that are diligently trying to place identifiers in their traffic are likely to have anonymous traffic that they are not aware of.  Why is that?  The simple fact is that most sites suffer from tunnel vision.  They focus on their most important traffic – JSP, HTML, ASP, etc.  These hits represent their application, as they perceive it.  When they track a user’s progression through their site, these are the URLs that they care most about.  What are often overlooked are the support files – stylesheets, javascript, images, etc.  Although secondary in importance, they can be critical when it comes to understanding the performance seen by the user.  Organizations must solve this problem in order to monitor their traffic properly.

There are two basic problems that need to be addressed:

1. Failing to use cookies to identify sessions
2. Failing to control the domain of those cookies

Whatever identifier is being used on the primary URLs, it should also be present on all secondary URLs as well.  For those applications that use something other than a cookie to carry session IDs, this can be a challenge if not downright impossible.  Switching to cookies allows the browser – the most authoritative source for identifying a unique user – to properly label every hit using a natural aspect of the protocol.  Even when browsers do not support cookies, web servers have been designed to rewrite the URLs in the content being sent to that the cookie values are embedded in the URL paths themselves.  The beauty of this is that browsers and web servers are designed to do this with little effort on the part of the application developer.  All that is required is that the web server be configured to open a new session for any traffic it sees – something that is not usually done if the developers are trying to maintain a stateless application.

Recommendation #1: all web servers should be configured to track sessions even if the application is stateless.  Opening a session does not mean you are required to maintain any state.

So now your web servers are diligently placing a session cookie on all traffic that they are serving up, which will solve the problem … most of the time.  You can still be easily defeated by deployments in which some or all of the secondary files are hosted on an alternate server.  This is a problem because it places those files in a different domain and cookies are sensitive to domains (and paths).  It is considered a bad practice to ask browsers to send cookies to servers that don’t care about them (it bloats the network traffic and forces web servers to do extra work for nothing).  Therefore, web servers often configure their cookies so that browsers only send them back to the servers (i.e., domains) that issued them.  Thus, browsers usually do not send cookies set by the primary server to the secondary servers as well.  This is easy to fix, however, by changing the configuration of the web servers.  Most of the time, the secondary servers have the same root domain – e.g., www.coradiant.com vs. images.coradiant.ca – so a simple rule that includes both domains (*.coradiant.*) can be associated to the session ID cookie, allowing the browser to send it on all traffic to the monitored site.  That will ensure that all secondary traffic is also clearly tagged with a unique session ID.  Although this goes against best practice, the session ID cookie is typically very small and the best practice was meant to avoid needlessly sending multiple cookies and / or fat cookies with heavy payloads.  Moreover, what I am suggesting is hardly “needless”, since it solves a significant problem.

Recommendation #2: define domain patterns that cover your whole site and associate those patterns to the cookie(s) that will be carrying the session IDs.  Each web server vendor may do this differently, so consult the administration manuals for your specific web servers for more information on how this can be done.

Reconstructing sessions reliably does not have to be complicated, but people often get in their own way.  These simple recommendations should help solve a world of problems, no matter what monitoring solution you are using.

Julie Craig from Enterprise Management Associates published a very interesting article entitled “The End-to-End Challenge“. In this article, she reveals some disturbing statistics, among which were the following (I am paraphrasing here):

Gartner released their June 2008 version of their venerable “Magic Quadrant” ranking for E-Recruitment Software.

Six of the eight leaders are already Coradiant customers.

This is a clear indication that leaders in this burgeoning field are taking End-User Experience Management seriously.

Coradiant is consistently chosen by leaders in online HR and in a number of other industries whose business relies on web applications.

With Coradiant TrueSight, businesses know how they are treating every single one of their web visitors, whether they are a small business interfacing with a few high valued users or a large enterprise interacting with hundreds of individuals online every second.

Leaders in E-recruitment software and HR Software-as-a-Service businesses are all focused on providing a high-quality end user experience. And Coradiant is the overwhelming choice among the leaders for End User Experience Management. Gartner ranked leading vendors on the completeness of their model and on their ability to execute.

Copies of Gartner’s E-Recruitment Software Magic Quadrant report are available from Gartner, Inc. (www.gartner.com).   

The holy grail of web monitoring – whether for real user experience or web analytics or any other purpose – is to be able to reliably recognize users.  You can only accomplish this goal by inspecting the traffic itself.   However, as I pointed out in a previous posting, you will always be as blind as your own applications.

Naturally, applications will only inject such identifiers when they are interested in identifying the user in some fashion.  Not all of them are.  Moreover, due to the undisciplined nature of web development, some of them are horrendously inconsistent in their intentions.There are really three levels of user awareness:

• Identity: an application is identity-aware if they require the user to authenticate themselves in some fashion.  This is typical of on-line banking, insurance sites, etc.
• User: an application is user-aware if they track the user’s session but cannot actually identify the user.  For example, most on-line stores allow anonymous users to buy items from a catalog, requiring a session state so that the site can remember the contents of the shopping cart.
• Anonymous: these applications do not track the user specifically and they do not place anything in the unique in the traffic.

Most sites that are interested in tracking users believe that they are either identity-aware or user-aware.  In fact this is frequently not the case.  Some applications appear to be user-aware because they use cookies to remember preferences or state.  However, none of those cookies are unique to a given user, so those applications are actually entirely anonymous.  Most other sites are at least partially anonymous, failing to place anything unique in the traffic unless absolutely required.  Thus, users are allowed to remain anonymous through large sections of those sites, only becoming traceable when they enter a transaction or log into the application.

Anonymous users cannot be tracked.  The HTTP protocol does not do it natively and the evolution of the internet is only making that more apparent.  If you are allowing traffic to be anonymous, then you must either accept that you cannot track that traffic reliably (you can make reasonable attempts using things like the client IP, but it will be seriously flawed) or you must alter your applications to build in the level of awareness that you need.

The Web Analytics industry is in the midst of a debate about how to identify and count Unique Users.  Some people are starting to suggest that we should abandon the idea of Unique Users in favor of counting something easier.  At the heart of that debate is the question of whether we will ever be able to uniquely identify users on the web.
 

Surely I can trust the client IP?

The problems with the client IP have been public knowledge for a long time.  This is an excerpt from a tutorial about Web Analytics, published by Summary.net (a log analysis tool) back in 2002:
“The majority of Internet users connect through dial-up services of some kind. In order to preserve IP numbers (there are a limited number available right now), the dial-up providers will assign each user a number when he connects and then reuse the number when he is done with it. So a dial-up service may have 100 IP numbers that they select from and use to serve 2000 users. This gets even more complicated with caches and proxies that many providers now use to improve performance …”
With the introduction of mega-proxies (like AOL), this problem gets even worse.  Mega-proxies will spray their traffic across multiple gateways.  Since the internet was designed to treat each hit as a stand-alone transaction, this means that every request making up a single page can be routed through a different client IP and port.  So, instead of a one-to-many relationship between the IP and the users, we have a many-to-many relationship.
Entities like corporate firewalls are rendering the client IP extremely weak and unreliable as a user identifier.  Mega-proxies completely destroy its reliability.
 

What about the user agent?

A lot of people choose to set aside the concern about mega-proxies and talk about combining the user agent with the client IP as a differentiator.  The problem with this is that there are a finite number of user agents in the world.  Admittedly, they number in the thousands.  However, these user agents are shared by millions of web users, which means that tons of users are being represented by the same user agents.  In fact, this understates the problem since most people are running the same browsers and plugins on the same basic operating systems, reducing the pool of popular user agents.  Combining IP and user agent will still result in users that are sharing the same combination.
If you are expecting to use this as a means for identity tracking – as in “this is Bob” – then you are going to be disappointed.  Since the user agent contains information about the browser and the OS, it can easily mutate over time as users upgrade their browser, download plugins, install service packs, etc.  Moreover, users are not limited to one browser – I use Firefox but am occasionally forced to use IE on specific sites – or one system.  I surf from my laptop, my wife’s computer and my iPod, so I’m using three different platforms as well as three different browsers.
 

Enter the plugin

At this point, you may be thinking that the user agent will at least improve your odds.  This would be true if it weren’t for plugins.  Plugins within a browser are allowed to request their own resources from the server.  When they do so, they send a user agent and it does not have to be the same one used by the browser.  The Java plugin is a classic example.
 

Grab your bootstraps and pull

This problem – as with all others – begins at home.  If you want to track users, do not expect the HTTP protocol to help you.  It was originally designed for anonymous traffic.  Deploy your own tracking IDs that are tailored to your needs.  Most web servers have mastered the art of injecting user awareness into the traffic (via cookies or URL-rewriting).  If you need identity awareness, then you need to take the next step and have your developers build that into your application.
There is no magic bullet.  You need to solve this problem for yourself.

When the web first appeared on the IT scene, it seemed like a wonderful solution to the main problem facing IT at the time – how to push software upgrades onto hundreds (or thousands) of desktops.  It reduced the supported infrastructure down to a single set of web servers, etc.  Unfortunately, as applications were migrated to the web, IT departments lost their relationship with the users and help desks were put in place to take over that role.  Some IT departments may even have considered this to be a benefit.  At first.  They completely failed to see the down side before the storm hit.

Who are you and what are you doing to my web site?

It is important to note that up until this point, the IT department enjoyed an intimate relationship with their users.  They knew everyone that was using the application and could call them up directly, if necessary.  More importantly, the users knew the IT staff and could speak to them directly, if they wished.  The web took away that vital relationship.

Web users are a faceless mob – completely anonymous whenever they choose to be.  Even if they are registered users, you may be forced to consider them as “anonymous” due to privacy laws (even if they work for your own company).  Even worse, you can’t speak to them directly.  Gone are the days when the IT technician could call up a user and say, “I’m having trouble reproducing your problem.  Can you give me some more details?”  Moreover, web users do not have your phone number either.  They do not even know that you exist.  You might never be able to answer the most important IT questions:

- Who are you?
- What is your environment and how is it different?
- What were you doing when this problem occurred?

If you can’t get the answers to those questions, you have severely limited your ability to solve problems when they arise.  Your job has just moved several notches up the “difficulty” meter.

The only people that web users might interact with are Customer Support agents.  Help desks have taken over the IT department’s traditional relationship with the end user.  That means that all of the details that the IT department needs in order to do their job must be gathered by the help desk.  In fact, all communication with the end user is filtered through them.  A survey done by HDI in 2007 indicated that 70% of all known issues are reported via the help desk.  In other words, of the known issues, a small minority – less than one third – is being found by the IT staff and the majority is being reported by users after they’ve had a chance to get upset.  Moreover, since many CS departments are staffed by non-technical people in a high-turnover environment, the likelihood of your vital debugging information being the victim of a broken telephone effect is very high.  And don’t forget to factor in the impact of having an out-sourced help desk that might not even be clear on what your application is all about.

The loss of the relationship with the user is one whose negative impact cannot be adequately measured.

If you only knew what you don’t know …

And it gets worse.  A report released by Transversal last December indicates that the average corporate web site in the UK can take anywhere from 30 hours to 116 hours to respond to support requests via email.  This is driving customers to switch to phone support.  This, in turn, discourages users from reporting problems, since the phone is a more time-intensive method.  Since most IT departments – whether they realize it or not – are largely relying on users to find the problems, this is a serious issue.  How many problems are going unreported?

Several years ago, I was working as a consultant for a large telecommunications company with a base of about 11 million users.  Needless to say, they were concerned with the welfare of such a large customer base.  They relied on an internally produced survey indicating that users only reported 10% of the problems they experienced.  Of the ones that reported problems, a minority consisted of extremely diligent users who felt it was their duty to report issues whereas the majority tended to be extremely disgruntled users who were so angry that they insisted on taking the time to be heard.  This implies that everyone else felt that it wasn’t worth the time or effort to report a web glitch and either tried their transaction later or gave up entirely.

If we take that survey seriously, then the real pool of issues is an order of magnitude larger than you believe it is – e.g., if you know of 100 issues, then there are another 900 that you might never find out about.  This also means that the IT department’s internal visibility into the health of their applications is an order of magnitude smaller than they believe – i.e., only 3% of actual issues are being found by the IT department, with another 7% being reported by the help desk.  This gigantic blind spot should be an unacceptable liability to any IT department.

If, in the absence of real contact with your users, you are relying on your QA lab and your help desk to find and address your web problems, I wish you much luck.  Your work is cut out for you.  Instead, you need to start looking at ways of monitoring what is happening to your users in real time.  I would strongly recommend that you start thinking about adding Real User Performance Monitoring to your tool box.  Don’t wait for them to tell you that something’s wrong.  Get ahead of that tsunami.  There are excellent tools out there to help you with this problem. 

Patrick Gardella, of Discovery Communications, recently spoke to Network World and said about Coradiant TrueSight, “Basically, it allows us to identify very rapidly what is happening with actual users on the site, and then it helps us debug those things.”

“With our huge online shopping site — and other Web systems that require major user interaction – users have problems. When that happens, we get e-mails saying, ‘Your site is broken’ and not much more. The reason I like Coradiant is that it offers a very simple, easy-to-use appliance that can find out what’s happening with those individual users, as well as how many other people are having those same problems.”

For the full article see: Network World