Tutorial¶

Varying the Summary Function¶

By default the summary statistic function used in each time-series is the average, you can change this by setting the function option to the series call to one of min, max, count, sum:

series("cpu.load.*", "server*", {function:"sum"})

Varying the Resolution¶

You can also alter the granularity of the summarized data. By default a query for the last 14 days will return time-series with hourly datapoints i.e. period=”3600”. With the period option you can specify an arbitrary resolution in seconds to which the datapoints should be rolled up. For example, to specify daily resolution using sum as the summary function:

series("cpu.load.*", "server*", {function:"sum",period:"86400"})

Shorthand Notation¶

As every composite metric definition contains at least one invocation of series() and more sophisticated defintions may contain several, our grammar supports s() as an equivalent shorthand notation e.g.:

s("cpu.load.*", "server*", {function:"sum",period:"86400"})

Sets of sets¶

Finally we can create sets of sets by square-bracketing series() calls. We can use this to specify multiple series as a single argument to functions like divide(), as we’ll see shortly, or to make more specific typeglobs, for example to specify the 5 and 15 minute load averages, but not the ten minute average we could use:

[s("collectd.load.load.short*","*"),
  s("collectd.load.load.long*","*")]

Dynamic sources¶

When you embed a metric in an instrument, our user interface supports the concept of dynamic sources, which is a fancy way of saying that you’ll specify the source later on when you actually want to view the data. You can use dynamic sources in your composite metric definitions by using % in lieu of the source argument to series(). We could make our set of load averages dynamic like so:

[ s("collectd.load.load.short*","%"),
  s("collectd.load.load.long*","%")]

sum()¶

The sum() function aggregates the input set down to a single series by adding together the measurements at each time interval. You get a single time series consisting of all the input series added together.

If, for example, you had several metrics that tracked the occurrences of HTTP response codes in your logs, you could get a total count of HTTP 400-series errors using the sum function like so:

sum(s("prod.log.http.4*","*"))

You could use a set of sets to capture 400 and 500’s like so:

sum([s("prod.log.http.4*","*"),s("prod.log.http.5*","*")])

subtract()¶

The subtract() function takes a set of exactly two time series and returns the result of subtracting the second from the first.

Continuing from the last example, if you tracked the occurrences of HTTP response codes in your logs, you could get a count of all non-200 responses by combining sum() and subtract() in a set of sets like so:

subtract([sum(s("prod.log.http.*","*")),sum(s("prod.log.http.2*","*"))])

max()¶

The max() function aggregates the input set down to a single series by discarding all but the largest measurement at each time interval. You get a single time series consisting of the largest single measurement from each of the input series.

If you wanted to monitor load average site-wide with a single line, you could use the max() function to display the site-wide maximum load average.

max(s("collectd.load.load.*","*"))

min()¶

The min() function aggregates the input set down to a single series by discarding all but the smallest measurement in each time interval. You get a single time series consisting of the smallest single measurement from each of the input series.

You could add a lower bound to the site-wide load average graph with a call to min() like so…

min(s("collectd.load.load.*","*"))

mean()¶

The mean() function aggregates the input set down to a single series by computing the mean of all measurement in each time interval. You get a single time series consisting of the mean of the input series.

Adding the mean to our site-wide load average graph with mean() like so…

mean(s("collectd.load.load.*","*"))

… yields a graph that represents the CPU load of every host in our entire production infrastructure:

derive()¶

The derive() function is useful when you want to view the rate of change of a given metric - how much it increases or decreases from one measurement to the next. It transforms the input set by computing the derivative of each series in the set. Unlike the functions we’ve discussed thus far, derive() doesn’t aggregate, or combine the input set, you get as many series out as you put in. In order to avoid certain edge cases involving late-arriving data, we recommend that you push calls to derive() to the lowest, or innermost, possible level, e.g:

sum(derive(series()))

is preferable to:

derive(sum(series()))

You can track the rate of customer signups by using derive() to graph the rate of change in a total user count metric.

derive(s("prod.myapp.users.count", "*"))

divide()¶

It’s often useful to calculate the ratio of two metrics. The divide() function takes a set of exactly 2 time series and returns the result of dividing the first by the second (The first set is used as the dividend and the second, the divisor). Continuing the last example, if we were tracking sign-ups and churn on a per-server basis, you could track the ratio of new customers to cancellations by first aggregating all of the per-server stats with sum(), and then dividing the result like so:

divide([sum(s("prod.myapp.signups", "*")), sum(s("prod.myapp.cancels", "*"))])