Fluentd¶
Fluentd is an open source data collector to unify log management. It uses a JSON log format to enable data collection from multiple, disparate input sources which can then be sent to any number of output destinations, making it a great fit for Librato metrics.
Here’s an example of how you might use Fluentd to send Apache 2xx and 4xx access log counts to Librato
Example using Apache logs¶
## Tail apache logs with tag=apache.access
<source>
type tail
format apache
path /var/log/apache2/access.log
pos_file /var/log/apache2/tmp/access.log.pos
tag apache.access
</source>
# Filter logs for 2xx and 4xx
<match apache.access>
type grepcounter
count_interval 3
threshold 1
input_key code
# This matches 2xx or 4xx
regexp ^(2|4)\d\d$
add_tag_suffix counted
</match>
<match apache.access.counted>
type record_reformer
tag librato.apache
# Metric name starts with first character of the first status code
# "message":["200","200","200"...]
key apache.request.${message[0][0]+"xx"}
# Measurement value
value ${count}
source dockerton
</match>
# Output to Librato
<match librato.**>
type librato
email example@librato.com
apikey ABCD1234
</match>
This example uses the following Ruby gems:
Use with Docker¶
You can use the following plugin to collect Docker metrics: https://github.com/kiyoto/fluent-plugin-docker-metrics (thanks to Kiyoto Tamura at Treasure Data for providing this!).
Also refer to the case study on our blog: http://blog.librato.com/posts/docker-container-data-with-fluentd.
@bdehamer has created a simple Docker container which actually includes the Docker input plugin + the Librato output plugin: