Web application specialist New Relic is claiming an industry first by introducing an "Application Speed Index" to its SaaS application performance management (APM) tool. This new benchmarking tool is intended to compare application performance across a set of "industry peers" in real-time based on real user data from over 25,000 active accounts.
NOTE: There appears to be a somewhat scant definition of who these "industry peers" might in fact be, although the company says that it has listed a public "Application Speed Index Leaderboard" as a living infographic that tracks the fastest applications and industries using New Relic. New Relic does specify that since the company was founded in 2008 over 140,000 developer, operations, and DevOps personnel have opened accounts on the company's APM platform.
50 bn x 500,000 Apps x 200,000 Domains
New Relic says that its SaaS-based solution collects over 50 billion metrics for its users each day from over 500,000 application instances that power 200,000 unique web domains on the Internet. This figure (says the company) makes New Relic the largest big data repository for application performance metrics ever built.
Company CEO Lew Cirne has stated that in his view, most companies measure speed and performance against their own subjective benchmarks, but that this is "operating in a vacuum" because they don't have insight into how other companies in their industry compare.
"With New Relic's massive big data collection and our new app speed index capability, we're giving companies a huge competitive edge by providing a real benchmark that they can measure against day after day," he said.
Gartner's Jonah Kowell agrees that industry benchmarking is critical for helping companies understand how they compare with their peers. "However, benchmarking reports are typically driven by synthetic transactions, which can assume connectivity that is often uncertain in today's mobile workforce. Newer approaches to application performance monitoring delivered as a service (APMaaS) can inject real user data into benchmarks adding valuable data for decision makers," he said.
NOTE: Application performance metrics measured by this tool will include end user response time, application availability, and error rate.
New Relic users who opt-in to the Application Speed Index are reportedly given a percentile rank for each metric as well as daily histograms of their performance compared to that of their industry peers. The index itself is divided into 14 industry categories including ecommerce, mobile, B2B and consumer internet, social networking, and others.
