Tagged: logstash Toggle Comment Threads | Keyboard Shortcuts

  • penguin 13:31 on 2017-01-12 Permalink | Reply
    Tags: , logging, logstash,   

    Logs with docker and logstash 

    It would be nice to have all container logs from a docker cluster sent to … let’s say, an ELK stack. Right?


    So we did:

    • on each host in the cluster, we use the GELF log driver to send all logs to a logstash instance
    • the logstash instance clones each request using type “ELK”
    • to the “ELK” clone, it adds the token for the external ELK service
    • the “ELK” clone goes out to the external ELK cluster
    • the original event goes to S3.

    Here’s how.

    (More …)

    • David Sanftenberg 09:30 on 2017-07-04 Permalink | Reply

      Multiline gelf filters are no longer supported in 5.x of Logstash it seems. I’m considering downgrading to 4.x for this, as we use a lot of microservices and many JSONs are logged simultaneously, really messing up our logs. Thanks for the writeup.

  • penguin 10:49 on 2017-01-09 Permalink | Reply
    Tags: , logstash   

    Logstash, clone filter & add_field mysteries 

    That’s a really great piece of documentation. This does not work:

    # let's clone each event, one goes to somewhere, one goes to somewhere else.
    # note this was copy-pasted from the docs!
    # see here: https://is.gd/QSHNps
    # again. THIS DOES NOT WORK.
    filter {
      clone {
        add_field => { "token" => "ABCDEF" }
    output {
      if [token] {
        # go somewhere
        tcp { ... }
      } else {
        # go to somewhere else
        s3 { ... }

    Why? Because the clone filter will not clone anything. And the documentation is super unclear on this. If you know it, you can read it – if you don’t know this, you’ll … google.

    For it to actually clone anything you have to specify the ‘clones => [“one”, …]’ parameter. Then it will clone, and add the token field as expected. Like this:

    filter {
      clone {
        clones => ["logz"]     # NOW it will clone.
        add_field => { "token" => "ABCDEF" }

    Interestingly the “clones =>” parameter is optional, which just confuses the shit out of me.

    The reasoning that I don’t just add the field altogether is that this is the access token for our externally hosted ELK service. This should only be there for the external path, and not be put in S3 in parallel.

  • penguin 16:22 on 2016-06-28 Permalink | Reply
    Tags: , logstash, ,   

    Testing logstash configs with Docker 

    Now this is really not rocket science, but since I might do this more often, I don’t want to google every time.

    Prepare your directories

    ./tmp                   # THIS IS YOUR WORKING DIRECTORY
      |- patterns/          # optional
      |   |- patternfile1   # optional
      |   |- patternfile2   # optional
      |- logs.log
      |- logstash.conf

    Prepare your logstash config

    # logstash.conf
    input {
      file {
        path => '/stash/logs.log'
    filter {
      # whatever config you want to test
      grok {
        match        => [ "message", "%{WHATEVER}" ]
        patterns_dir => '/stash/patterns'              # optional :)
    output {
      stdout { codec => rubydebug }

    Run logstash

    docker run --rm -ti -v $(pwd):/stash logstash logstash -f /stash/logstash.conf


    Done. 🙂

compose new post
next post/next comment
previous post/previous comment
show/hide comments
go to top
go to login
show/hide help
shift + esc