Fluentd is an open source data collector to unify log management.  It uses a JSON log format to enable data collection from multiple, disparate input sources which can then be sent to any number of output destinations, making it a great fit for Librato metrics.


Here’s an example of how you might use Fluentd to send Apache 2xx and 4xx access log counts to Librato

Example using Apache logs

## Tail apache logs with tag=apache.access
  type tail
  format apache
  path /var/log/apache2/access.log
  pos_file /var/log/apache2/tmp/access.log.pos
  tag apache.access

# Filter logs for 2xx and 4xx
 <match apache.access>
  type grepcounter
  count_interval 3
  threshold 1
  input_key code
  # This matches 2xx or 4xx
  regexp ^(2|4)\d\d$
  add_tag_suffix counted

<match apache.access.counted>
  type record_reformer
  tag librato.apache
  # Metric name starts with first character of the first status code
  # "message":["200","200","200"...]
  key apache.request.${message[0][0]+"xx"}
  # Measurement value
  value ${count}
  source dockerton

# Output to Librato
<match librato.**>
  type librato
  email example@librato.com
  apikey ABCD1234

This example uses the following Ruby gems:

Use with Docker

You can use the following plugin to collect Docker metrics: https://github.com/kiyoto/fluent-plugin-docker-metrics (thanks to Kiyoto Tamura at Treasure Data for providing this!).

Also refer to the case study on our blog: http://blog.librato.com/posts/docker-container-data-with-fluentd.

@bdehamer has created a simple Docker container which actually includes the Docker input plugin + the Librato output plugin: