Coradiant Blog

End user impact is still the most important metric

Jonathan Ginter — Wed, 03 Feb 2010 18:33:44 +0000

Coradiant started out several years ago with a single clear message – that understanding the end user experience (whether actual human beings or back-office clients) was fundamental to managing any application. Since that time, the industry has evolved significantly, as it always does, leaving in its wake the broken remains of once-rising companies and rejected approaches. Often, as experience and depth-of-understanding increases, initial ideas are overthrown in favor of more accurate theories that reflect lessons learned in the field. Not so with end user experience. Not only has this idea won through unscathed, it has seen wide adoption at every level of the performance monitoring space. Analysts write about its importance and vendors of all sizes have integrated it into their solutions. There is not a single first-tier or second-tier vendor of APM solutions that has not built or acquired end-user monitoring technology and added it to their solution – e.g. CA, HP, Compuware and Quest all have an end-user component in their solutions. Companies in North America no longer doubt its value and expect to have such intelligence out-of-the-box. Europe is trailing behind this trend but I expect that they will rapidly catch up over the next two years.

The fact that end user experience has survived indicates that it is a fundamental truth of application monitoring – if you don’t understand how your user is being impacted by your application, you cannot effectively assess whether you have an actual problem. For example, if a primary database that has a backup goes down over a holiday weekend, should the IT manager agree to pay double overtime to have it fixed immediately? If, in an attempt to prevent a repeat of the failure, the IT manager presents the CTO with a purchase order for a clustered set of industrial servers with fibre-channel connections, should he agree to that very expensive purchase? These questions cannot be answered unless the impact of the problem is well understood and this impact must be expressed in terms of how it affected the end users of the business. No other metric is nearly as important. If few or no end users were affected, the expense of an emergency repair or a massive upgrade may not be truly justified. The days of watching CPU, RAM and disk usage as a means of determining real impact are over. Those metrics, although good to know and important in their own right, cannot effectively reflect the impact to the business the same way that user experience does.

The fact is that some truths never change.

Correlating End-User Performance and Conversions

Sri Raghavan — Tue, 12 Jan 2010 03:39:40 +0000

Companies spend an inordinate amount of money setting up complex IT systems to manage and optimize service offerings to customers. Often these management systems only look at a single metric, like conversions or performance. What if both could be correlated?

Imagine you are a high-traffic on-line ticket vendor for major events nationwide. As the head of IT operations, you want to maximize conversions by ensuring that you can handle the traffic volumes generated by a major concert event or marketing push and that the response times for customer queries remain optimal. In spite of your best efforts, however, the latest concert promotion brought your site to its knees and some of your systems went down for a short period of time. The impact of that outage on your users might be unacceptably slow response times or actual system errors in their browser. In either case, if they risk losing out on good seats for the concert while waiting for your systems to recover, they will abandon your site in favor of a competitor that is more responsive and you will have lost that revenue. On the assumption that you could have sold 1000 tickets at $100 apiece during that outage, you could lose up to $100k of revenue, depending on the number of users affected. That’s a lot of crumpets, as they say.

Poor response times and unreachable sites are bad for revenue. But there is also a third reason why your site traffic could be impacted that is not caused by the performance issues described above. It may be that your site content is unappealing to the average eye or that your offerings are priced out of reach for the average consumer. This is the typical explanation that marketing gives to conversion dips, but it is clearly not the only root cause.

This is where Coradiant Analytics In A Box can help. Our passive capture technology combined with the included Google Urchin software collects performance and availability metrics about your site while it is tracking user behavior. This allows you to look at conversions (e.g. ticket purchases) in the context of performance and availability, thus allowing you to categorize conversion dips into those that are infrastructure related and those that are marketing and content related. For example, your analytics data tells you that the 22% of your users bought tickets between 12-2 PM on Monday afternoon. You also see that this is about 35% less than the norm for a corresponding period. Coradiant allows you to correlate this event with an increase in page load time or availability problems.

End-user monitoring coupled with analytics provides a powerful solution that helps to maximize customer visits and conversions. It places those responsible for business results and for performance in the driver’s seat by providing them with the data they need to understand the real business impact of system outages, thus enabling them to provide the best possible service to their Line-of-Business managers.

How can Marketing free itself from IT bondage and privacy regulation?

Jonathan Ginter — Mon, 14 Dec 2009 00:52:12 +0000

I feel real sympathy for Marketing departments. They are saddled with the enormously difficult task of profiling users in order to achieve more effective marketing efforts. In order to do that, they must understand who the user is, what they did on the site and how they reacted to various marketing efforts. Up to now, they have been relying on traditional Web Analytics solutions to get that data. However, the price for that data is the insertion of page tags and tracking cookies. Cloud-based services add on an additional price – the exporting of analytic data out to the cloud. All of that might seem like a small price, but it is actually a true Faustian deal.

To start with, Marketing must subject itself to the busy timelines of Development, QA and IT whenever it wants to change its tagging or cookies. Since those departments are often busy rolling out new features, Marketing often has to wait weeks or even months to get their new data. In many cases, Marketing discovers a problem with the page tagging during the course of a campaign and are unable to roll out a fix quickly enough. In companies where this problem is recognized, the problem is reversed and Marketing is allowed to hijack the roll-out process with an emergency patch to its analytics tagging, often negatively impacting the delivery of important new features. Both Marketing and IT would benefit if this link could be severed.

The other problem that Marketing is facing comes from Europe, where a wave of privacy regulation is forcing existing Web Analytics solutions to run for cover, leaving Marketing departments with little to help them. Germany has passed very strict laws prohibiting the use of page tagging and tracking cookies without the user’s consent. Moreover, shipping analytics data to a hosted service for processing is specifically forbidden. Although privacy is often talked about in every part of the world, Europe is the first to have passed these kinds of laws about it – a trend that could easily spread outside EMEA.

The irony is that most of the data that Marketing often requires is already contained within the traffic stream before any tagging or tracking takes place – where the user is from, what browser and OS they use, which ISP they used, where they came from, where they went, what they looked at, what they bought, etc. It’s either embedded in the HTTP protocol or as part of a web server’s natural ability to maintain a stateful application or it is directly within the request or response content.

Consequently, as announced last week, Coradiant has teamed up with Google to create our Analytics In A Box (AIB) solution. This revolutionary new product uses Coradiant’s existing technology to passively process all user traffic to and from the web site, producing full-featured web analytics data that remains in-house. Instead of inserting page tags or using special cookies, Marketing can define AIB rules that will extract the information directly from the traffic stream. This approach will allow Marketing to define new metrics whenever they want to – even in the middle of a campaign during peak hours. Moreover, the solution refreshes its data every hour, providing Marketing with immediate results from any changes. No other department has to be involved. Moreover, the data is 100% secure and kept in-house, which means no privacy violations.

As Bogey said, “this could be the start of a beautiful friendship”.

TrueSight LTA is NetworkWorld’s Product of the Week

Jonathan Ginter — Mon, 23 Nov 2009 19:22:58 +0000

TrueSight Load Test Adapter has been selected as a Network World Product of the Week. This add-on VM for TrueSight allows Operations and QA departments to quickly convert production traffic into real-world usage patterns, rated by popularity and path length. Examples of those usage patterns can be imported into HP’s LoadRunner test suite as a basis for building test scenarios. This cuts short the debate about what to test or how to generate the important background noise required in any reasonable QA environment. More importantly, it gives Operations and QA a means of talking about acceptable test scenarios using the same language.

Coradiant presents Web Application Visibility solution in Manhattan

Jonathan Ginter — Sat, 31 Oct 2009 00:09:08 +0000

Coradiant is holding an informative afternoon executive event in the Penthouse at Gary’s Loft in Manhattan focused on end-to-end Web application visibility.

You can register for the event here (https://www.123signup.com/event?id=jbxhq).
Presenters include:

Dave Anderson, Principal Architect/Co-founder of Peopleclick,
Dennis Callaghan of The 451 Group,
Fred Dumoulin of Coradiant
Andreas Grabner, dynaTrace software,

Gary’s Loft in Manhattan features a stunning 360 degree view of the Manhattan skyline. Light food, beverages and hospitality will be provided by Tip Of The Tongue NYC.

Date: Thursday, November 12, 2009
Time: 4:30 PM to 6:30 PM ET

Gary’s Loft
28 West 36th Street
Penthouse
New York, New York 10018

Load Testing using Real Traffic Patterns

Jonathan Ginter — Wed, 28 Oct 2009 22:21:48 +0000

I am astonished by the number of customers I have spoken with that are having trouble creating accurate load testing scripts. Even worse is the heavy lifting they must go through to create realistic scenarios. It’s time-consuming. It’s expensive. And it doesn’t always produce the right results either. It can be quite discouraging. Although there are other products that produce LoadRunner scripts from real traffic – e.g. TeaLeaf or HP RUM – they only do so based on a single session. Although this is fine for reproducing a single one-off scenario (i.e. functional testing), this does nothing to help with load testing since it still leaves the user in doubt as to whether this single scenario represents a popular or common click path.
Our flagship product, TrueSight, is able to see all of the production traffic and recreates click paths, which we display in the Session Browser. Leveraging this ability, we decided to help eliminate guesswork and put back the accuracy into load testing. I am pleased to announce Coradiant’s first product that will help in that effort.
Today, we officially introduced the TrueSight Load Test Adapter, packaged as a VM that runs in VMWare Server or ESXi. It takes session data from any TrueSight appliance and converts it into click path statistics, from which our customers can generate endless varieties of load testing scripts that can be directly imported to LoadRunner. Users can sort the click paths by popularity or by number of pages. They can filter to find click paths with a minimum length or that pass through a URL that contains specific text. They can even create week-long data sets of click paths that focus on specific applications (using Watchpoints) and then generate their scripts by focusing on specific time periods within that data set.

This product has a robust road map and will see additional releases over the coming year.
If you are having trouble generating realistic scripts or simply need a script that reproduces production behavior as a background to your current stress testing, you owe it to yourself to check this out.

Customers Lose Patience with Poorly Performing Retail Sites

Tony Tissot — Thu, 17 Sep 2009 23:12:05 +0000

Online retail customers have less and less patience for poorly performing Web applications.

Forrester’s recent report on “eCommerce Web Site Performance Today “clearly shows that customer frustration leads to lost sales. The report was commissioned by Akamai Technologies and is available (after registering) at www.akamai.com/2seconds.

- 69 % of dissatisfied online shoppers indicated that they are less likely to buy from a poorly performing site again.
- 64 % would simply purchase from another online store.
- 47 % of consumers expect a Web page to load in 2 seconds or less
- 40 % of consumers will wait no more than 3 seconds for a Web page to render before abandoning a site.

These are sobering statistics indeed, particularly for etailers that don’t proactively manage and monitor site performance and actual, delivered service levels. Synthetic testing alone does not provide the needed level of visibility into actual, delivered performance because it misses far too much of the action.

The clear trend is that consumers are even more demanding now than in the past. The study reveals that forty-seven percent of consumers expect a Web page to load in 2 seconds or less. That consumer expectation is a sea change from a similar 2006 Forrester study which showed that the majority of customers expected page loads of 4 seconds or less.

Most revealing is the finding that forty percent of consumers will wait no more than 3 seconds for a Web page to render before abandoning the site.

The report concludes: “It is clear that there are serious consequences for an online retailer with an underperforming site. However, by taking steps to improve site features and performance, online retailers can look to increase overall consumer satisfaction and ultimately increase sales. Forrester recommends that online retailers test their Web site performance, fix easy site features and performance issues before attempting to address larger problems, as well as improve the multichannel experience by addressing content and functionality issues on the retail site.”

Every business that relies on the Web needs to understand exactly the service quality they are providing to online customers. How can you accomplish that?

Coradiant provides TrueSight Automated Incident Management and TrueSight Edge which are specifically designed to help solve the Web application performance problem. With Coradiant, reports on performance, availability, and traffic volumes are only a click away. When problems occur, they can quickly be detected, localized, and resolved. IT operations can now manage user performance and optimize and troubleshoot important functions, including Akamai traffic, and then drill down to specific parts of the infrastructure to see how they handle transactions.

Coradiant TrueSight is the most cost-effective means to achieve a single, comprehensive “voice of the online customer.”

Another Traffic Jam – End-User Experience is Key

Hon Wong — Fri, 21 Aug 2009 03:25:37 +0000

Another traffic jam this morning. As I inched forward, I brooded over the similarity between highway design engineering and the design of Web applications. It may look good on paper, but when the real world intervenes, all bets are off.

In the world of the Web it’s the same way. Sure the testing and pre-deployment steps are critical, but as Web applications become more complex, it’s not enough. It’s important to thoroughly test your Web applications. But it’s even more important to “test” your Web applications after deployment to make sure that they continue to run correctly.

In the pre-deployment world there is a lot to manage; make sure that all the application logic and algorithms are sound, have users run through the UI, hunt and kill a load of bugs, load-test the application, perform integration testing to ferret out conflicts with other applications, make sure all the system components required to run the application are there and that you’re running the right version in the targeted infrastructure, and so on and so forth. It’s a daunting task. But at least then you have full control.

But after it goes live, you’ve lost some of that control. That’s the unpredictable nature of the Web. You don’t have much say in how networks perform, or which devices may be running your application on the user side. And Web users are fickle, with hard to predict usage patterns. Turn on your perfect application and soon you’ll get a call from your IT operations people passing on a customer complaint relating to some totally unexpected thing. Real users always do the unexpected. You can’t always predict every scenario. Similarly, no amount of testing using traditional IT tools can possibly find all the hidden problems that can pop out in live deployment situation. Throw in the complexity of the Web application infrastructure, and you’ll be spending a lot of time looking for hidden problems that might have nothing to do with your code.

Knowing the end-user experience is key. Organizations that run web applications have discovered that the end-user experience is the key metric for success. By understanding what is being delivered in real-time, you can prevent dissatisfaction and application abandonment. If you serve your users well, the application is perceived as successful. If you fail in delivering the service expected by users the application is deemed a failure.

OK – so we can’t really test our highways and make dynamic changes after it’s built – yet, but you can certainly do that with Web applications.

The key is to provide the right tools. Web Application Performance Management solves the dilemma of gaining real-time visibility into end-user performance, and providing the actionable information you need to make decisions and changes to keep everything running at the speed limit.

Blind Spots in Web Application Performance Monitoring

Jonathan Ginter — Thu, 06 Aug 2009 09:21:57 +0000

Contrary to popular belief, the brain is not a Personal Video Recorder, recording everything submitted by your various senses. That would be too much data for any brain to handle. Instead, it sifts through sensory input looking for relevant data points that it can trust and throws everything else away. The important words in that last sentence are “relevant” and “trust”.

If a data point is not relevant, then it is considered to be a distraction. There are well-known studies on Inattentional Blindness and Change Blindness which demonstrate that even large-scale events can be filtered out by the brain if they are considered irrelevant to the task at hand. Similarly, if the data point cannot be trusted, the brain tosses it out as well (whether your senses can be trusted has been a heated debate in philosophy for centuries, but I digress). Trust and relevance are crucial to the brain’s ability to eliminate useless noise and derive good results.

These same principles apply to monitoring your web applications. Instead of monitoring the universe, you should be reducing your data flood to those points that are relevant. Moreover, you should only be using the most trusted tools and methodologies to draw conclusions.

For web applications, the most relevant data is the data that directly describes or explains your user’s experience and places it in context. In order to identify that data, you must be able to draw a direct line from your user’s experience to those data points. If you cannot do that, you are probably chasing your tail and wasting a lot of valuable resources. It is important to realize that a lot of tools cannot draw a direct line from user experience to monitoring data without leaving a few gaps and logical leaps of faith.

As an example, operations teams love to know whether a database is down. Although this is valuable data, is it relevant? If users experienced worse performance around the same time, does that mean that fixing the database will solve the performance problem? In fact, in a well-architected environment, the loss of a web server, app server or database should have little, if any, effect on the end user’s experience due to clustering and load-balancing. A lot of solutions love to use time correlation as a magnificent leap of faith, but it simply makes unreliable conclusions look enticing.

To draw that line between user experience and environmental monitoring, you need a tool that can see the actual users’ experience and is able to relate it directly to problems in your network, application design, deployment, code quality, etc. Moreover, it must prove itself to be a trusted source of information, returning results quickly and reliably without drowning you in irrelevant data. In other words, it must be trusted to extract and analyze relevant information and return high-quality results.

Handling the Truth

Jonathan Ginter — Wed, 24 Dec 2008 16:04:03 +0000

Coradiant’s TrueSight End-User Experience Management product evokes a number of interesting reactions when we first start monitoring a customer’s Web traffic. One of the most common reactions is amazement at just how many bugs exist – even in the best Web applications. One customer speaking at a luncheon described the experience as being similar to turning on the light in your apartment and seeing big ugly cockroaches everywhere – you are appalled, you are embarrassed … and you feel a strong urge to simply turn off the light.

This might seem like a damning statement to make about one’s own environment. And yet we repeatedly hear how such confrontations with the ugly truth have provided insights that resulted in the correction of long-standing problems, some of which had never even made it onto the radar of Web Operations. I think you would have to struggle to find a Coradiant customer that did not have a similar story to tell. One customer discovered that 30% of their traffic consisted of cache hits (where the server reports that nothing has changed) or redirects. By simply tweaking the caching parameters returned by their web servers, they reduced the load on their servers significantly. Another customer discovered that some pages were taking up to 1.5 minutes to be handled by the server before a response was being sent back to the browser. Yikes.

How many users are hitting your site? How many errors are being returned? How slow are the pages? How reliable is the network? Some customers are clearly floundering without any real ability to answer these fundamental questions. Other customers believe they already have a solid handle on such issues. We have found that almost all of them have a real shock in store. Some of our most loyal customers are those that firmly believed they knew the truth already.

Often, though, the insight doesn’t have to be that deep to be a revelation. It never ceases to amaze me how often web sites are thrown over the fence to be supported by a team that hasn’t the first clue about what they have taken on. We offer a fairly simple feature that reports lists of traffic attributes sorted by popularity – e.g., URLs, hosts, client IP address blocks, geographic regions, cookie keys, etc. Our customers can define their own fields as well, pulling whatever they would like out of the traffic to do so (e.g., database error codes, product IDs, etc). We use this feature to help populate configuration fields. However, the contents of those lists proved to be such a revelation to our customers that we re-categorized the feature under “Reports”. Using this simple feature, one of our customers noticed that we were seeing internal traffic that we should not have been able to see. This led him to realize that his routers were improperly configured.

The customer that we had invited to speak at our luncheon finished off his presentation by advising others – somewhat jokingly – to consider carefully whether they were truly ready to handle the truth about their traffic. Ugly as it may be, facing it can reveal real solutions to real problems. I highly recommend it.