
In the complex world of software and infrastructure monitoring, having robust observability tools is paramount. For Grafana Labs, the company behind the ubiquitous Grafana dashboards and a suite of powerful open-source observability projects like Loki (logs), Mimir (metrics), and Tempo (traces), this isn't just a business—it's a daily operational reality. Grafana Labs "eats its own dogfood" at an impressive scale, running its massive, multi-tenant Grafana Cloud platform on the very software it develops and offers to the world. This profound internal usage is a cornerstone of their product development philosophy, driving innovation, resilience, and a deep empathy for their users' challenges.

### Grafana Cloud: Powered by the "LGTM Stack" Itself

One of the most compelling testaments to Grafana Labs' commitment to its own technology is how it operates Grafana Cloud. Tom Wilkie, Grafana Labs CTO, has openly stated this: "**We use our own Grafana Cloud products — [powered by] Mimir, Loki, Tempo, Grafana — to monitor our Grafana Cloud SaaS, and that's what drives the reliability.**" This isn't a siloed test environment; it's the real-time, production monitoring of a critical commercial service using the tools they build.

This practice offers several distinct advantages:

* **Real-World Stress Testing:** Operating Grafana Cloud means dealing with immense scale, diverse workloads, and the constant need for high availability. By using Mimir for scalable metrics, Loki for log aggregation, and Tempo for distributed tracing to monitor this very service, Grafana Labs subjects its own tools to the most demanding conditions imaginable.
* **Identifying "Growing Pains" Firsthand:** When performance issues or scalability challenges arise within their own infrastructure, Grafana Labs' engineers are the first to experience them. A [Grafana Labs blog post on Tempo 2.7](https://grafana.com/blog/2025/01/16/grafana-tempo-2.7-release-new-traceql-metrics-functions-operational-improvements-and-more/) candidly mentions, "As Grafana Cloud Traces — the fully managed distributed tracing system powered by Grafana Tempo — has grown, we have felt some internal growing pains related to operating enormous Tempo clusters, and are working to reduce resource consumption." This direct feedback loop from internal SREs and developers to the product teams is invaluable for driving meaningful improvements.
* **Deep Operational Empathy:** By being operators of their own software at scale, Grafana Labs gains profound empathy for the challenges their users face. This understanding directly influences product roadmaps, feature prioritization, and the development of more intuitive and effective tools.

### Meta-Monitoring: A Sign of Mature Internal Use

The concept of "meta-monitoring"—monitoring your monitoring systems—is a practice well understood at Grafana Labs. The [Loki documentation](https://grafana.com/docs/loki/latest/operations/meta-monitoring/), for example, advises users to monitor their Loki cluster using a *separate* Grafana, Loki, and Prometheus setup (ideally via Grafana Cloud). This recommendation itself stems from the operational wisdom gained by running these systems internally. It acknowledges that for true resilience, the system observing your primary observability stack needs its own independent line of sight. This type of nuanced, practical advice is born from real-world operational experience.

### The "Big Tent" Philosophy, Tested Internally

Grafana Labs champions a "big tent" philosophy, aiming for its tools to be open, composable, and capable of integrating with a vast ecosystem of data sources and other observability components. This approach is also, by necessity, part of their internal reality. As they operate Grafana Cloud and integrate various open-source components and their own managed services, their internal teams are constantly testing and validating these integrations.

This ensures that when Grafana Labs promotes the interoperability of its stack, it's based on practical, internal validation. Their open-source roots and commitment to transparency mean that many of the challenges and solutions they discover internally often benefit the wider community through improved documentation, bug fixes, and new features in the open-source projects.

### From Internal Feedback to Product Evolution

The feedback loop from internal users (SREs, developers, support engineers, and potentially other teams using Grafana for business intelligence) directly impacts product development:

* **Usability Enhancements:** Daily use of Grafana dashboards, LogQL (for Loki), PromQL (for Mimir/Prometheus), and TraceQL (for Tempo) by their own teams helps identify usability friction points and drives improvements in query builders, visualization options, and overall workflow.
* **Scalability and Performance:** The need to manage Grafana Cloud efficiently pushes the boundaries of Mimir's scalability for metrics, Loki's efficiency for log ingestion and querying, and Tempo's ability to handle vast numbers of traces. The performance improvements in Tempo 2.7, for instance, were explicitly linked to addressing internal resource consumption challenges.
* **New Feature Development:** It's highly probable that many new features and even entire products (like Grafana OnCall or Grafana Incident for incident management) are born from identifying needs within their own operational teams first. Building tools to solve their own problems, then generalizing them for the broader community and customers, is a common and effective pattern in open-source-centric companies. The development of "observability as code" features in Grafana 12 and the new Grafana Assistant AI (as mentioned at [GrafanaCON 2025 by Business Wire](https://www.businesswire.com/news/home/20250507135781/en/Grafana-Labs-Demonstrates-Open-Source-Leadership-at-GrafanaCON-2025)) likely stems from a desire to streamline their own complex observability configurations and make data interrogation more accessible.

### Benefits for the Entire Ecosystem

Grafana Labs' commitment to "dogfooding" its entire observability stack offers significant benefits that extend beyond its own operational resilience:

* **More Robust Open Source Software:** Features and fixes tested under the demanding conditions of Grafana Cloud often make their way back into the open-source versions of Grafana, Loki, Mimir, and Tempo, benefiting the entire community.
* **Better-Informed Product Strategy:** Having a deep, firsthand understanding of user challenges allows Grafana Labs to build a more relevant and impactful product roadmap.
* **Increased Trust and Credibility:** When a company successfully runs its own critical services on the products it offers, it sends a powerful message of confidence and reliability to its customers and users.

### Conclusion: Charting the Course by Navigating It Themselves

Grafana Labs stands as a compelling example of a company that truly lives and breathes its own technology. By operating Grafana Cloud on the very LGTM (Loki, Grafana, Tempo, Mimir) stack it champions, Grafana Labs doesn't just build observability tools; it becomes its own most critical and informed user. This deep, internal reliance fuels a continuous cycle of refinement, innovation, and operational excellence that benefits their open-source community, their enterprise customers, and the broader observability landscape. They are not just charting the future of observability; they are navigating it firsthand, every single day.

Observing Themselves: How Grafana Labs Builds a Better Observability Platform by Being Its Own Most Demanding User


In the world of data analytics, security, and observability, Splunk stands as a formidable platform, enabling organizations to turn vast seas of data into actionable insights. But Splunk's commitment to its technology goes beyond its customer base; it's deeply embedded in its own operational DNA. Splunk actively employs a "Customer Zero" philosophy, where its own internal teams, particularly its Global Security Operations (GSO), are power users of the Splunk platform. This practice of "eating their own cooking" is fundamental to how Splunk innovates, refines its products, and ultimately delivers greater value to its customers.

### The Splunk Platform: A Comprehensive Toolkit

Before diving into how Splunk uses its own solutions, it's important to understand the breadth of their offerings. The Splunk platform is designed to handle the entire data lifecycle for various use cases:

* **Splunk Enterprise / Splunk Cloud Platform:** The core for searching, analyzing, monitoring, and visualizing machine-generated data from virtually any source.
* **Splunk Enterprise Security (ES):** A market-leading Security Information and Event Management (SIEM) solution.
* **Splunk SOAR (Security Orchestration, Automation, and Response):** Empowers security teams to automate tasks and orchestrate complex workflows.
* **Splunk Observability Cloud:** Provides full-stack visibility into applications and infrastructure, including Application Performance Monitoring (APM) and Real User Monitoring (RUM).
* **Splunk AI:** A suite of artificial intelligence and machine learning capabilities woven throughout the portfolio, including AI Assistants for security and observability, designed to enhance human decision-making and automate processes.
* **Specialized solutions** for areas like IT Service Intelligence (ITSI), Asset and Risk Intelligence, and Threat Intelligence Management.

This powerful array of tools isn't just outward-facing; it's the same technology Splunk's internal teams rely on daily.

### Customer Zero: The Ultimate Proving Ground

Splunk is vocal about its "Customer Zero" initiative. As highlighted in a [Splunk blog post titled "Splunk Security Ops: Building the Blueprint for Success"](https://www.splunk.com/en_us/blog/security/splunk-security-operations-blueprint-for-success.html), their own Global Security Operations (GSO) team doesn't just use Splunk products—they "put our technology through the wringer — just as any customer would."

This means Splunk's internal environment serves as a demanding, real-world testbed. The benefits of this approach are manifold:

1.  **Real-Time, Real-World Feedback:** Splunk's GSO team, acting as an advanced customer, provides continuous, invaluable feedback directly to the product development teams. This isn't theoretical testing; it's feedback born from tackling actual operational and security challenges faced by a large, global technology company. This helps "shape features, improve usability, and ultimately reflect the needs of the broader security community."
2.  **Accelerated Product Improvement:** Issues encountered internally, whether they are bugs, performance bottlenecks, or usability friction points, are identified and addressed with high priority. This direct feedback loop significantly shortens the iteration cycle and leads to more robust and refined products.
3.  **Validation of Efficacy:** When Splunk's own SOC can successfully leverage Splunk Enterprise Security and Splunk SOAR to, for example, reduce the time to triage a phishing email to under seven minutes (as mentioned in their blog), it's a powerful testament to the platform's capabilities.
4.  **Driving Innovation in AI:** Splunk's internal teams are prime users of their AI capabilities. For instance, the Splunk AI Assistant for Observability is used by their engineers to analyze instance health, trace issues, and even perform predictive analysis for system capacity. ([Splunk's AI Assistant: Top 7 Use Cases for AI-Driven Observability](https://www.splunk.com/en_us/blog/artificial-intelligence/top-splunk-ai-use-cases-for-observability.html)). This internal application allows Splunk to rapidly iterate on its AI-driven features, making them more practical and powerful for customers.
5.  **Developing Best Practices and Playbooks:** The methodologies and successes of Splunk's internal teams often become blueprints for customers. The "Splunk Security Ops" blog itself aims to share these playbooks, helping other organizations replicate their approach to data-driven security.

### Splunk's Internal Use Cases: A Glimpse Inside

While Splunk is a vast platform with countless applications, we can see how their internal teams would leverage it:

* **Security Operations (SecOps):**
    * **Threat Detection and Response:** Using Splunk ES for advanced threat detection, leveraging risk-based alerting (RBA) to prioritize threats, and using Storylines to understand complex attack chains.
    * **Automation:** Employing Splunk SOAR to automate repetitive tasks, orchestrate incident response workflows, and integrate with a wide array of security tools.
    * **Threat Hunting:** Proactively searching for indicators of compromise (IoCs) and novel attack patterns within their own environment using Splunk's powerful search processing language (SPL).
    * **Asset and Identity Monitoring:** Keeping track of their digital assets and user behaviors to detect anomalies and potential compromises.
* **IT Operations (ITOps) and Observability:**
    * **Infrastructure Monitoring:** Ensuring the health, performance, and availability of Splunk's own extensive IT infrastructure, including servers, networks, and cloud services.
    * **Application Performance Monitoring (APM):** Using Splunk Observability Cloud to monitor the performance of their internal and customer-facing applications, troubleshoot issues, and optimize user experience.
    * **Log Management and Analysis:** Ingesting and analyzing logs from across their entire technology stack to gain operational insights, identify root causes of problems, and ensure service reliability.
    * **Cloud Monitoring:** Optimizing the performance and cost of their cloud deployments and ensuring the security of their cloud assets.
* **Business Analytics and Product Insights:**
    * Analyzing product usage data (from their own internal deployments and anonymized telemetry where appropriate) to understand how features are being used, identify areas for improvement, and inform the product roadmap.

### The "No Secret Handshakes" Policy

A crucial aspect of Splunk's "Customer Zero" approach, as stated by their GSO team, is that they build their internal service "the same way any customer would—no secret handshakes, no back-channel agreements—just a team using the best product in the world to solve real problems." This commitment ensures that the challenges they face and the solutions they develop are directly translatable to their customers' experiences.

### Challenges and The Path to Refinement

Operating as "Customer Zero" isn't without its inherent challenges. Exposing internal teams to early or beta versions of software can mean encountering bugs or unfinished features. However, for a company like Splunk, these are not seen as mere inconveniences but as critical opportunities. The internal team's expertise allows them to provide detailed, actionable feedback, turning potential frustrations into drivers for improvement. The alternative—releasing less-tested software to the public—is far riskier.

The focus is always on turning operational problems—be it alert fatigue, data sprawl, or the need for faster insights—into measurable outcomes that demonstrate the value and drive the evolution of the Splunk platform.

### Conclusion: Building Resilience from the Inside Out

Splunk's dedication to being "Customer Zero" is a powerful strategy that significantly benefits its users. By rigorously using its own comprehensive platform for its demanding security, IT operations, and observability needs, Splunk ensures its products are not just feature-rich but also battle-tested, scalable, and refined through real-world application. This internal crucible of innovation means that when customers invest in Splunk, they are getting solutions shaped by practitioners who face the same complex data challenges, ensuring that what works for Splunk, works even better for the world.

Splunk on Splunk: How the Data Giant Powers Its Own Resilience and Innovation

For decades, security has often been treated as a final, fortified gate in the software development lifecycle—a checkpoint managed by a specialized team before code ships to production. But in an era of high-velocity CI/CD pipelines and complex cloud-native architectures, that model is fundamentally broken. The modern approach, DevSecOps, calls for security to be a shared responsibility, woven into every stage of development. For a company like New Relic, whose very business is providing deep insight into complex systems, this isn't just a philosophy they sell; it's a principle they must live by.

To build a credible security product in today's landscape, a company must first look inward. It needs to confront the same challenges its customers face: tool sprawl, alert fatigue, and the constant tension between speed and safety. New Relic's journey into security observability is a prime example of this, where the development of its own tools is intrinsically linked to the needs of its own engineering teams.

### **The Internal Problem: A Flood of Signals, A Puddle of Context**

Imagine being an engineer at New Relic. You operate a planet-scale platform that ingests over 120 petabytes of data every month, running on thousands of nodes across dozens of Kubernetes clusters. The performance and reliability of this system are paramount. But so is its security.

In such an environment, security signals come from everywhere: static code analysis tools, dependency scanners, cloud configuration monitors, and runtime threat detectors. The result is often a deluge of alerts, each siloed within its own tool. This creates a cascade of problems that New Relic's own teams would inevitably face:

* **Alert Fatigue:** When developers are bombarded with low-context security warnings, they quickly become noise. Legitimate threats can get lost in the flood.  
* **Lack of Prioritization:** Is a 'critical' vulnerability in a non-production, internal-only service more important than a 'medium' vulnerability in a core production library? Without context from an observability platform, it's nearly impossible to know.  
* **Developer Friction:** Tossing a security report "over the wall" to a developer without actionable context forces them to stop coding, switch tools, and begin a time-consuming investigation.

Solving this for their customers meant first solving it for themselves. This necessity appears to be the driving force behind **New Relic Vulnerability Management**.

### **Building the Solution: From APM to IAST**

New Relic's history is rooted in Application Performance Monitoring (APM). Their agents already live deep within an application's runtime, observing performance metrics and tracing requests. This existing footprint provided a unique advantage. Why couldn't that same agent also watch for security vulnerabilities?

This logical extension led to the development of their own security offerings. A key move was the acquisition of K2 Cyber Security, which brought powerful **Interactive Application Security Testing (IAST)** technology into the fold. As detailed in a [DevOps.com article](https://devops.com/new-relics-bolsters-observability-platform/), IAST allows New Relic to use its agents to detect and validate vulnerabilities *as the application runs*, without requiring separate, disruptive scans.

In their blog post, ["Streamlined security: Integrating IAST into your DevSecOps strategy,"](https://newrelic.com/blog/best-practices/iast-devsecops) New Relic champions a vision where security is not a separate step but an integrated part of the workflow. The post states, "IAST operates live, immediately spotlighting vulnerabilities," and emphasizes "fewer false alarms." This isn't just marketing copy; it's a direct solution to the internal pain points of alert fatigue and developer friction. By building these capabilities into the platform their own engineers use daily, New Relic could refine a system that provides:

* **Unified Visibility:** Bringing native vulnerability detection alongside third-party security signals into the same platform where performance data lives.  
* **Context-Rich Prioritization:** Correlating a vulnerability with the specific application entity, its environment (production, staging), and its performance impact.  
* **A "Single Pane of Glass":** Allowing developers, SREs, and security teams to look at the same data, breaking down the silos that create so many issues.

### **The Risk of a Unified Vision**

Did this strategy lead them astray? There's no evidence of a major misstep, but it does introduce a new, more nuanced set of challenges. By championing a platform that unifies observability and security, New Relic is also championing a significant cultural shift. The tool is designed to empower developers to take on more security responsibility.

The potential pitfall here is not in the technology itself, but in the implementation of the culture it requires. For an organization—whether it's New Relic internally or one of their customers—that isn't ready for this "shift-left" mindset, a unified tool can feel complex. It asks developers and operations engineers to engage with a domain that has traditionally been someone else's problem. If the organizational structure and training aren't in place to support this shift, the tool's power can become a source of complexity rather than a solution.

The case study of their customer, Dustin, as detailed on the [New Relic blog](https://newrelic.com/blog/how-to-relic/monitoring-solution-merger-dustin), highlights the goal: to use a unified tool to "make sure that when we did get an alert, it was accurate and valuable." The unstated challenge is the organizational work required to get to that point.

### **Conclusion: Charting a Secure Path Forward**

New Relic's foray into security is a natural evolution of its core mission. They are applying the principles of observability to one of the most complex and critical domains in modern software. By building the security tools they need internally, they've created a powerful feedback loop. The daily challenges faced by their own engineers in securing a massive, dynamic platform directly inform the features and philosophy of their Vulnerability Management product.

This approach of "securing thyself" is more than just a development strategy. It’s a testament to the belief that in the world of DevSecOps, you can't effectively solve a problem for your customers until you've first solved it for yourself. The result is a platform that doesn't just find vulnerabilities, but provides the context needed to fix them, aiming to make security an integral, observable, and manageable part of everyone's workflow.

Articles tagged with "Observability"

Observing Themselves: How Grafana Labs Builds a Better Observability Platform by Being Its Own Most Demanding User

Splunk on Splunk: How the Data Giant Powers Its Own Resilience and Innovation

Secure Thyself: Inside New Relic's DevSecOps Feedback Loop