Docker Infrastructure Snippets

Logs with docker and logstash

It would be nice to have all container logs from a docker cluster sent to … let’s say, an ELK stack. Right?


So we did:

  • on each host in the cluster, we use the GELF log driver to send all logs to a logstash instance
  • the logstash instance clones each request using type “ELK”
  • to the “ELK” clone, it adds the token for the external ELK service
  • the “ELK” clone goes out to the external ELK cluster
  • the original event goes to S3.

Here’s how.


Logstash, clone filter & add_field mysteries

That’s a really great piece of documentation. This does not work:

# let's clone each event, one goes to somewhere, one goes to somewhere else.
# note this was copy-pasted from the docs!
# see here:
filter {
  clone {
    add_field => { "token" => "ABCDEF" }
output {
  if [token] {
    # go somewhere
    tcp { ... }
  } else {
    # go to somewhere else
    s3 { ... }

Why? Because the clone filter will not clone anything. And the documentation is super unclear on this. If you know it, you can read it – if you don’t know this, you’ll … google.

For it to actually clone anything you have to specify the ‘clones => [“one”, …]’ parameter. Then it will clone, and add the token field as expected. Like this:

filter {
  clone {
    clones => ["logz"]     # NOW it will clone.
    add_field => { "token" => "ABCDEF" }

Interestingly the “clones =>” parameter is optional, which just confuses the shit out of me.

The reasoning that I don’t just add the field altogether is that this is the access token for our externally hosted ELK service. This should only be there for the external path, and not be put in S3 in parallel.


Testing logstash configs with Docker

Now this is really not rocket science, but since I might do this more often, I don’t want to google every time.

Prepare your directories

./tmp                   # THIS IS YOUR WORKING DIRECTORY
  |- patterns/          # optional
  |   |- patternfile1   # optional
  |   |- patternfile2   # optional
  |- logs.log
  |- logstash.conf

Prepare your logstash config

# logstash.conf
input {
  file {
    path => '/stash/logs.log'

filter {
  # whatever config you want to test
  grok {
    match        => [ "message", "%{WHATEVER}" ]
    patterns_dir => '/stash/patterns'              # optional :)

output {
  stdout { codec => rubydebug }

Run logstash

docker run --rm -ti -v $(pwd):/stash logstash logstash -f /stash/logstash.conf


Done. 🙂