How To Use Metric Collectors

Diffusion reports operational metrics about network performance, topics, client sessions, and server log messages to provide visibility of server and application performance. This information is useful to system administrators responsible for monitoring and managing Diffusion systems, and to developers who need to diagnose and troubleshoot application issues.

Diffusion 6.3 added metric collectors, which enable you to record metrics for custom subsets of client session or topics.

This tutorial will show you how to set up metric collectors on a Diffusion 6.3 server.

Accessing the metrics

Metrics can be accessed in various ways: through the developer web console, Java Management Extensions (JMX), and (with a Scale and Availability license), the Prometheus gateway. The web console only reports the metrics from the server which hosts it and does not record metrics, so does not support historical queries and trending. Generally, the web console is used during development or for operational monitoring and problem diagnosis focused on a single server. For monitoring multiple Diffusion servers, we recommend exporting metrics to an external system via JMX or Prometheus. There is a wide variety of commercial and software products that can be connected using JMX and/or Prometheus, including AppDynamics, DataDog, Grafana, Nagios, New Relic, Splunk, and Sysdig.

A fresh Diffusion install initially has no metric collectors. Each server produces a set of standard metrics that are scoped to the server. For example, the network metric inbound_bytes records the cumulative number of bytes received by the server over the network.

The following is a screenshot of the console from an out-of-the-box Diffusion installation with the Sports Book demo deployed, showing the standard server-scope topic metrics. If you download and install Diffusion, you can follow along by starting the server and opening the console at http://localhost:8080/console.

Metric collectors are user-defined scopes that allow a more specific subset of the metrics to be recorded. There are two types of metric collector: session metric collectors and topic metric collectors. Session metric collectors can be configured to record metrics for subsets of the sessions. Each session metric collector specifies the target sessions using a session filter, and the metrics can be further partitioned by an arbitrary list of session properties. Topic metric collectors can be configured to record metrics for subsets of the topics. Each topic metric collector specifies the target topics using a topic selector, and the metrics can optionally be further partitioned by topic type.

Like server-scoped metrics, metrics recorded by metric collectors are published to the console, JMX, and (optionally) the Prometheus gateway. Metric collectors can be defined using the console or JMX. The server will persist the configuration, so the metric collectors will be recreated if the server is restarted. If the server belongs to a cluster, the configuration will be replicated across all members of the cluster.

Let’s use the console to add a topic metric collector. Our server is running the example Sports Book application, so the partially expanded topic tree looks like:

Suppose we are interested in the topics for the branch of the topic tree Demos/Sportsbook/Football/England/League One. Selecting the Metric Collectors sub-tab, we can define a metric collector as follows:

The branch is identified by topic selector ?Demos/Sportsbook/Football/England/League One//. The trailing // characters indicate that we want to select the topics at the path Demos/Sportsbook/Football/England/League One and all of its descendant paths. Topic selectors are commonly used in the Diffusion API; you can read about the full syntax in the manual.

We’ve checked the option boxes to specify that we want to break down the results into groups by the topic type and that the metrics should be exported via the Prometheus gateway.

If we add the metric collector and return to the Metrics sub-tab, the results produced by the metric collector are visible:

Note the new lines for the two types of topic found in the topic branch. Although each of the new lines is for a subset of the topics, the set of recorded metrics is the same. In this example, we can see that the branch has five JSON topics storing a total of 255 bytes of data in topic values. There is a single BINARY topic which has no value. In the time between adding the metric collector and taking the screenshot, there were 49 updates. These updates all provide a complete value, indicating that the Sports Book application is not using the update stream API. If there were subscribers, they would benefit from a 60% saving in data transfer due to delta streaming.

Here’s the same view a little while later, having defined a second metric collector for the Demos/Sportsbook/Football/England/League Two topic tree branch. We’ve also created two Sports Book sessions in other browser tabs by accessing the Sports Book demo page (http://localhost:8080/demos/sportsbook/ if you’re following at home).

Now we can see that for each branch, there are two sessions that subscribe to one or more topics in the branch, with ten subscriptions in total. That makes sense – each of the two Sports Book sessions is subscribing to the five JSON topics in each branch.

Let’s move on to look at session metric collectors. These are defined in much the same way as topic metric collectors, but use a session filter to narrow the sessions of interest, and allow the results to be grouped by one or more session properties. Combined with custom session properties, session metric collectors provide insights into the behavior of your application users.

We’ll define a session metric collector that collects information on all sessions, grouped by two of the standard session properties: $Principal which records the security principal the session used to authenticate, and $ClientType which records which client library the session uses. Here’s the session metric collector being added:

The Session / Metrics sub-tab displays the results:

The breakdown by security principal and client library makes it straightforward to identify the purpose of the four connected sessions. The browser session using the admin principal is the one we are using to access the web console. There are two browser sessions using the client principal; these are the two Sports Book sessions we opened earlier. Finally, there is a Java control client – the Sports Book demo publisher that generates the game data.

So now you know how to create topic and session metric collectors. Let’s finish by showing the metric collectors in JConsole, retrieved using JMX:

We’re expanded the MBean tree to show how SessionMetrics and TopicMetrics MBeans are named. There is a SessionMetrics MBean for the standard, server-scope session metrics; as well as further instances for each set of session metrics collected by our metric collector. Similarly, there is a TopicMetrics MBean for the standard, server-scope topic metrics; and further instances created by the two topic metric collectors.

You can also see the MetricCollectors MBean. This allows the current metric collectors to be retrieved and provides operations to create and remove metric collectors.