With the New Relic One observability platform, modern software teams can combine log data, agent-based APM and Infrastructure data, and third-party telemetry data to create an entity-centric system of record that illuminates their entire stack. Earlier this year, we kicked off our open instrumentation initiative, and now DevOps and Ops teams can use exporters and adapters to send telemetry data from open source protocols, formats, and APIs to New Relic, including from sources like the popular monitoring tool, Prometheus.

From our Kubernetes integration to the cluster explorer, we’ve built tools that give Kubernetes operators and developers clear visibility into their environments. And now, as part of our open platform, we’re excited to announce our new Prometheus OpenMetrics integration.

This integration collects telemetry data from the many services (such as Traefik, Envoy, and etcd) that expose metrics in a format compatible with Prometheus. In fact, with the integration you’ll be able to monitor key aspects of your Kubernetes environments, such as etcd performance and health metrics, Kubernetes horizontal pod autoscaler (HPA) capacity, and node readiness.

The Prometheus OpenMetrics integration gathers all your data in one place, and New Relic stores metrics from Prometheus, removing the overhead of managing storage and availability of the Prometheus server.

When troubleshooting issues in your Kubernetes clusters, the metrics collected by this integration are accessible alongside those gathered from New Relic APM and New Relic Infrastructure.

About Prometheus

Prometheus, part of the Cloud Native Computing Foundation (CNCF), is an open-source toolkit that provides monitoring and alerting for services and applications running in containers. It’s widely used to collect metrics data from Kubernetes environments.

Prometheus has done a significant amount of work with the open source community to standardize how Prometheus formats and exposes metrics, especially within Kubernetes clusters. In fact, Prometheus’ scheme for exposing metrics has become the de facto standard for Kubernetes.

How Prometheus works

Prometheus uses a pull-based system to pull multidimensional timeseries metrics from services over HTTP endpoints, instead of relying on services to push metrics out to Prometheus. Because of this pull-based system, third parties like New Relic, can build integrations that work with Prometheus’ metric exporters to gather valuable data for storage and visualization.

While deploying the Prometheus server is easy, managing Prometheus at scale can be a real challenge for organizations as they grow. New Relic takes care of scaling and managing the storage of metrics over time, and provides the tools to visualize and alert on these metrics.

Prometheus uses key-value tagging to organize metric data, which allows users to build strong queries to interrogate their data. For example, this is what a metric looks like when queried over HTTP:

myservice_requests_total{service="catalogue", env="production"} 100

Core components of Prometheus

The Prometheus architecture is created from a handful of components, the following of which are essential:

  • Prometheus server: scrapes and stores time series metric data
  • Pushgateway: caches metrics from short-lived jobs before sending to Prometheus server
  • Exporters: pull metrics from third-party services, such as Redis, etcd, and Grafana
  • Client libraries: used to instrument services and applications to be monitored with Prometheus

Prometheus and Kubernetes

Prometheus has seen a significant increase of usage within Kubernetes deployments in part due to its status within the CNCF. Prometheus also supports a large number of open source exporters, and projects like the Prometheus operator make it very easy to deploy within Kubernetes environments. It’s a proven and effective way to get metrics from Kubernetes hosts and processes.

Getting started with the New Relic Prometheus OpenMetrics integration in Kubernetes

New Relic’s Prometheus OpenMetrics integration supports both Docker and Kubernetes, using Prometheus version 2.

Installing the Prometheus OpenMetrics integration within a Kubernetes cluster, for instance, is as easy as changing two variables in a manifest and deploying it in the cluster:

  1. Download the integration manifest YAML file:
    curl -O https://download.newrelic.com/infrastructure_agent/integrations/kubernetes/nri-prometheus-latest.yaml
  2. Edit the nri-prometheus-latest.yaml manifest file, and add a cluster name to identify your Kubernetes cluster (required) and your New Relic license key (required).
    env: - name: LICENSE_KEY
           value: "<YOUR_LICENSE_KEY>"
    [...]
    config.yaml: |
      cluster_name: "<YOUR_CLUSTER_NAME>"
  3. Deploy the integration in your Kubernetes cluster:
    kubectl apply -f nri-prometheus-latest.yaml

The nri-prometheus-latest.yaml manifest file includes the nri-prometheus-cfg config map, which shows an example configuration. You can use the example file to configure, for example, how endpoints are scraped and metrics are filtered.

Once you’ve installed the integration for Docker or Kubernetes, you can begin building queries to track and visualize your Prometheus data in New Relic.

See the New Relic docs for more on compatibility and requirements, installation options, data limits, configuration, metric queries, troubleshooting, metric transformation, and more.

How the Prometheus OpenMetrics integration is used in New Relic

When you have the integration in place, you’ll first gather metrics from Prometheus, and then query and view them.

Gathering metrics

The integration automatically discovers which targets to scrape. By default, metrics are queried in the /metrics path on port 8080. However, you can use the prometheus.io/port and prometheus.io/path annotations or labels in your Kubernetes pods and services to specify the port and endpoint path the integration should use when constructing the target (but note that annotations take precedence over labels).

For example, if you have a deployment in your cluster, and the pods expose Prometheus metrics on port 8080 in the path my-metrics, set the labels prometheus.io/port to "8080" and prometheus.io/path to "my-metrics" in the PodSpec metadata of the deployment manifest, as shown:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "my-metrics"

The integration will retrieve metrics from your pods by sending a request to http://<pod-ip>:8080/my-metrics.

Viewing and querying your metrics

Once collected, you can query your Prometheus metrics using the New Relic Query Language (NRQL) in New Relic Insights or the New Relic One chart builder.

For example, to get metric names for a specific cluster, namespace, or pod, run one of the following commands:

  • Cluster:
    FROM Metric SELECT uniques(metricName) WHERE clusterName='<cn>'
  • Namespace:
    FROM Metric SELECT uniques(metricName) WHERE namespaceName='<ns>'
  • Pod:
    FROM Metric SELECT uniques(metricName) WHERE podName='<pod>'

From there, you can get attributes and attribute values for a metric

  • FROM Metric SELECT keyset() WHERE metricName='<mn>'
  • FROM Metric SELECT uniques(<attribute>) WHERE metricName='<mn>' AND podName='<pod>'

Using the metric name and attributes you retrieved, you can now query your Prometheus data.

For example:

  • To get raw metric values:
    FROM Metric SELECT <metricname> WHERE <attribute>='<value>'
  • To get a graph of the metric (possible aggregators are average, min, max, and sum):
    FROM Metric SELECT <aggregator>(<metricname>) WHERE <attribute>='<value>' TIMESERIES
  • To view average memory usage for all pods in a deployment:
     FROM Metric SELECT average(container_memory_usage_bytes) WHERE deploymentName='my-app-deployment' AND namespaceName='default'

Read more about viewing and querying metric data in the documentation.

Examples of using Prometheus data in New Relic

There are any number of ways to use Prometheus data in New Relic, but consider the following use cases:

Monitoring etcd

Etcd is key-value data store that’s essential for running Kubernetes clusters. Prometheus pulls metrics from etcd, so to ensure your clusters are healthy, you can use the Prometheus OpenMetrics integration to monitor etcd server, disk, and network metrics such as:

  • etcd_server_has_leader
  • etcd_server_proposals_failed_total
  • etcd_network_peer_sent_bytes_total
  • etcd_disk_wal_fsync_duration_seconds

Kubernetes Horizontal Pod Autoscaler (HPA)

HPA automatically scales up a Kubernetes deployment based on user-configured limits. After installing the Prometheus OpenMetrics integration, you can use the following query in the New Relic One chartbuilder (or New Relic Insights) to build a dashboard widget that monitors remaining HPA capacity.

FROM Metric select latest(kube_hpa_status_current_replicas),latest(kube_hpa_spec_max_replicas) where clusterName = '<YOUR CLUSTER NAME>'  facet hpa

Using chart builder to create a dashboard widget to monitor HPA capacity

Node readiness

In Kubernetes, a node is marked ready when it can accept workloads (pods). If a node is having issues, Kubernetes will label it as “not ready.” To create an alert condition for this, using the integration, use the following query:

FROM Metric select latest(kube_node_status_condition) where condition='Ready' and status = 'true' and clusterName = '<YOUR CLUSTER NAME>' facet nodeName

Setting an alert condition for node readiness

Let New Relic manage and scale your Prometheus data

Whether you’re getting started with Prometheus, or you have been using it in conjunction with a dashboard tool like Grafana to monitor your Kubernetes environment, New Relic can help you get started and scale your data without the hassles of managing Prometheus and your dashboard tool. The New Relic Prometheus OpenMetrics integration allows you to store and visualize those crucial metrics on one platform. With New Relic One, you can more easily combine metrics data with events, traces, and log data (or, as we call them, M.E.L.T.) from all of your entities across your entire software stack—from your Kubernetes backend to your frontend browser UI—for fully connected view of the relationships between your data and applications.

Install the integration and get started today!

Contributing to the Prometheus OpenMetrics integration

New Relic has contributed the Prometheus integration to the open source community under an Apache 2.0 license.

We welcome contributions to this integration or any of our open source exporters and adapters. If you’d like to contribute, please review our Contributors Guide. Keep in mind that when you submit your pull request, you’ll need to sign our CLA. If you’d like to execute our corporate CLA, or if you have any questions, please drop us an email at opensource@newrelic.com.

As a product manager for Kubernetes at New Relic, JF helps customers make sense of, troubleshoot, and optimize their Kubernetes environment. Previously, he architected, deployed, managed, and automated global and large scale Infrastructure as a Service offerings for a telecommunications company and has worked as a product manager in a startup developing open-source network virtualization and analytics software. View posts by .

Interested in writing for New Relic Blog? Send us a pitch!