Introducing Splunk Data Links: Connect Streaming Metrics to Logs, Reduce MTTR

We’re incredibly excited to announce that Data Links are now generally available in Splunk Infrastructure Monitoring. Data Links enable DevOps teams to get to better insights and resolve issues faster by tapping into the right data from the right system at the right time. They accomplish this by enabling seamless and context-aware transitions among Splunk dashboards, and to third-party solutions in your toolchain, such as logging and APM.  

Having worked with hundreds of DevOps teams, we’ve seen that the modern operations toolchain is comprised of many best-of-breed products from a broad ecosystem. Each tool plays a critical role in the execution of complex operational workflows that in many cases start from your monitoring system.

We built Data Links to enable context-aware workflows across tools, while remaining customizable and easily manageable at scale. You can click on property values to follow links to Splunk dashboards, deep link into other systems, and create dynamic URLs for any external destination. Data Links can be efficiently added using account-level or dashboard-level controls that ensure links appear wherever they’re relevant.

How Data Links Work

Imagine getting an alert from Splunk Infrastructure Monitoring that the 99th percentile latency of your service has gone up. You follow a link to the service dashboard right from the alert modal. The service dashboard shows all the components of the service, and the charts show that something’s going wrong with the data store. Follow the link to the dashboard about the data store, and sure enough one of the instances started spiking latency about 15 minutes ago. You now know exactly where and when the problem started. Armed with the ‘what’, ‘where’ and ‘when’, now you can follow a link into the logs to discover the ‘why’, perhaps by looking at full stack traces written to logs.

With Splunk, it took one alert, two dashboards and three clicks to narrow down the source of the problem.

How did we create the links? Anywhere that a data link could appear on a property, there’s a prompt to configure data links. You can configure links locally, on just one dashboard, or globally throughout Splunk.

Use Case: Bringing Time Context from Metrics to Logs

Logging systems are a fine example of an area where deep linking is incredibly helpful. Log systems are based on searches, but it gets harder as they scale to know what you should be searching for. The point of a log search is to identify a message or messages emitted by a failing component, in order to identify the primary cause of the failure. Your likelihood of success in finding that error message depends on how much you already know about the problem you have.

• First, do you know there’s a problem at all? It’s possible the error message didn’t make it to the logfile. 
• If you know there’s a problem, do you know when it started? Splunk’s docs tell us “Time is the single most important search parameter that you specify.” 
• Do you know what systems are affected? What services they’re running? Which hosts are involved and what they have in common?
   

These are the answers that you need in order to approach the root cause, and recognize it when you find it. While all of the data to answer these preparatory questions is likely present in the logs themselves, it’s not in a consumable form. In a system at scale, it is wildly inefficient to read through individual log messages to identify the source and impact of a problem. Log vendors understand this problem, which is why they support tools in their own applications that turn log entries into metrics for visualization and alerting.

Splunk Infrastructure Monitoring answers exactly these questions. We designed our alerts so that it’s straightforward to detect genuine alert conditions, and so you can find out in seconds that there’s a problem you need to pay attention to. Time-based visualizations, event overlays, and responsive time controls mean it’s obvious when a problem started. Dynamic dimension-based filtering, grouping, and aggregation means you can check for trends quickly. Using Splunk features like the clipboard, you can collect all the evidence you need to pinpoint the source of the problem based on the order of events and the scope of the systems affected.

And now, using Data Links, you can follow the gun smoke directly to the smoking gun.

What’s Special About Splunk Data Links?

Links are Based on Data, Not UI Objects

If all you can do is add a link to one chart or alert at a time, that means every time you make a new chart, you have to copy and paste the same configuration. Those links will inevitably need to redirect somewhere else, which forces you to edit them one by one.

To avoid this problem, data links in Splunk Infrastructure Monitoring are triggered by your data, not the UI objects that display it: 

This is a serious boost to efficiency: the same data appears in many different charts and detectors, and relevant links should appear in all of them. If you add a data link to a dashboard once, it’s available on every chart where the trigger appears, even when you add new ones.

Links Can be Customized to Specific Workflows

Splunk Data Links link one trigger, which might appear in many places, to one URL that’s dynamically configured using relevant time and metadata values based on the context that the trigger appeared in. This promotes a better dynamic experience, because the path forward might not be the same for all the data that appears in a given chart or alert. The right link to follow depends more on the data you’re looking at than the particular chart you saw it in.

Imagine a chart that lists all the apps running on a host. You need the ability to link to the right destination for each app. With data links, you can not only go to the best destination for each value, but also include more general-purpose destinations as always-on options. All the link destinations for a value are shown in the Actions menu: 

Links Can be Mapped to Properties for Consistency Across Environments

If you monitor your environment using data from different sources, inevitably there will come a time when one source says “host” and the other says “server”, or one service says “LoadBalancer” and the other says “LoadBalancerName”. Sometimes you can go to the source and enforce the same naming scheme everywhere, but other times it’s out of your hands. To smooth out translation errors between systems, we included Property Mapping:

Name a key or key:value pair at the source, and specify how to map it at the destination. This dynamic mapping takes effect wherever the link appears.

Links Make Useful Content Discoverable Across the Organization

If you create a global data link, it’s available on all relevant dashboards and alerts, and it’ll appear automatically even on dashboards, charts and detectors that are created later.

As an individual user in a large organization, this is a great way to discover connections and content you didn’t even know existed. If you work on a centralized observability or platform team, Data Links make it easier to drive continued improvements to the monitoring content you author for everyone else.

Experience Seamless Workflows Without Painful Configuration

Our particular implementation of cross-linking is unique – other vendors’ efforts in this area either link a well-known source (like a hostname) to a well-known destination known in advance, or let you link one custom starting point to one custom destination. If they have customizable links, they’re difficult to maintain. If they have maintainable links, they aren’t customizable.

• Datadog lets you navigate between its offerings via "View related," but doesn’t let you customize where they should take you
• New Relic supports links between dashboards, but only within a small segment of its offering — Data apps. And these links are added to individual charts, which is hard to maintain.
• New Relic and Datadog both appear to “linkify” values of ‘host’, which is nice, but again lack the ability to add links to other interesting fields or customize the destination. They both use known dimensions like ‘host’ as a value to pivot into infrastructure metrics, but that isn’t sufficient in a world of custom dimensions and metadata specific to particular integrations.
   

Splunk Data Links link one trigger, which might appear in many places, to one destination that’s colored by the place it appeared. Once you’ve set up a link, no additional effort is required to make the link appear in a newly relevant place.

Easily Combine the Right Tools for the Job

Another important consideration is that linked dashboards in Splunk Infrastructure Monitoring aren’t limited to Splunk-specific destinations. Our approach to Data Links means that you won’t be forced to switch from your preferred logging or APM solutions in order to take advantage of this feature. Instead, we enable you to build interconnected workflows between any tools of your choosing. This allows you and your team to remain as vendor-agnostic as possible, and keep using the solutions that fit your specific requirements.

Get Started with Data Links in Splunk Infrastructure Monitoring

We recognize that monitoring (in our case, streaming metrics) is just one part of a larger toolchain when it comes to building and operating modern applications, and in most cases people will choose what they think are best-of-breed tools covering specific use cases.

Our goal at Splunk is to enable the best user experience for monitoring, and that means empowering our customers to use the right tools for the job. If you have a solution that works for you, keep it, and use Splunk Data Links to support your connected workflow and make it even better. If you’re not already using Splunk Infrastructure Monitoring, get started with a 14-day trial.

This post features contributions from Rebecca Tortell and Aaron Sun.