Sample Grafana dashboard graphs

In the Run Grafana steps, you used a JSON file to set up a sample Grafana dashboard. You can customize this dashboard by clicking the title of any graph and clicking Edit.

Graph name Description
Active requests This graph serves as a "health check" for the Puppet Server. It shows a flat line that represents the number of CPUs you have in your system, a metric that indicates the total number of HTTP requests actively being processed by the server at any moment in time, and a rolling average of the number of active requests.

If the number of requests being processed exceeds the number of CPUs for any significant length of time, your server might be receiving more requests than it can efficiently process.

Request durations This graph breaks down the average response times for different types of requests made by Puppet agents. This indicates how expensive catalog and report requests are compared to the other types of requests. It also provides a way to see changes in catalog compilation times when you modify your Puppet code.

A sharp upward curve for all request types indicates an overloaded server. Expect these to trend downward after the server load is reduced.

Request ratios This graph shows how many requests of each type that Puppet Server has handled. Under normal circumstances, you'll see about the same number of catalog, node, or report requests, because these all happen once per agent run. The number of file and file metadata requests correlate to how many remote file resources are in the agents' catalogs.
External HTTP Communications This graph tracks the amount of time it takes Puppet Server to send data and requests for common operations to, and receive responses from, external HTTP services, such as PuppetDB.
File Sync This graph tracks how long Puppet Server spends on File Sync operations, for both its storage and client services.
JRubies This graph tracks how many JRubies are in use, how many are free, the mean number of free JRubies, and the mean number of requested JRubies.

If the number of free JRubies is often less than one, or the mean number of free JRubies is less than one, Puppet Server is requesting and consuming more JRubies than are available. This overload reduces Puppet Server's performance. While this might simply be a symptom of an under-resourced server, it can also be caused by poorly optimized Puppet code or bottlenecks in the server's communications with PuppetDB if it is in use.

If catalog compilation times have increased but PuppetDB performance remains the same, examine your Puppet code for potentially unoptimized code. If PuppetDB communication times have increased, tune PuppetDB for better performance or allocate more resources to it.

If neither catalog compilation nor PuppetDB communication times are degraded, the Puppet Server process might be under-resourced on your server. If you have available CPU time and memory, increase the JRuby max active instances to allow it to allocate more JRubies. Otherwise, consider adding additional compilers to distribute the catalog compilation load.

JRuby Timers This graph tracks these JRuby pool metrics:
  • Borrow time: The mean amount of time that Puppet Server uses (or "borrows") each JRuby from the pool.
  • Wait time: The total amount of time that Puppet Server waits for a free JRuby instance.
  • Lock held time: The amount of time that Puppet Server holds a lock on the pool, during which JRubies cannot be borrowed. This occurs while Puppet Server synchronizes code for File Sync.
  • Lock wait time: The amount of time that Puppet Server waits to acquire a lock on the pool.
These metrics help identify sources of potential JRuby allocation bottlenecks.
Memory Usage This graph tracks how much heap and non-heap memory that Puppet Server uses.
Compilation This graph breaks catalog compilation down into various phases to show how expensive each phase is on the primary server.

Example Grafana dashboard excerpt

The following example shows only the targets parameter of a dashboard. It demonstrates:

  • The full names of Puppet's exported Graphite metrics
  • A way to add targets directly to an exported Grafana dashboard's JSON content

This example assumes the Puppet Server instance has a domain of primary.example.com.

"panels": [
    {
        "span": 4,
        "editable": true,
        "type": "graphite",

...

        "targets": [
            {
                "target": "alias(puppetlabs.primary.example.com.num-cpus,'num cpus')"
            },
            {
                "target": "alias(puppetlabs.primary.example.com.http.active-requests.count,'active requests')"
            },
            {
                "target": "alias(puppetlabs.primary.example.com.http.active-histo.mean,'average')"
            }
        ],
        "aliasColors": {},
        "aliasYAxis": {},
        "title": "Active Requests"
    }
]

Refer to the complete Grafana dashboard JSON sample file for a complete, detailed example of how a Grafana dashboard accesses these exported Graphite metrics.