Ingest Logstash Messages to Scalyr in a Docker Environment

Scalyr recently introduced Logstash plugin, so now Scalyr users can import Logstash messages by configuring the Logstash configuration file (i.e logstash.conf). Here is our official documentation link.

In this article, I am gonna cover the steps to set up the same thing in a containerized environment. 

Install the Logstash Plugin into a Docker Container

The easiest way to add Logstash Scalyr plugin in a Docker environment is to build a custom Logstash image. 

• Modifying Dockerfile

# https://github.com/elastic/logstash-docker
FROM docker.elastic.co/logstash/logstash:6.8.10

# Add your logstash plugins setup here
RUN bin/logstash-plugin install logstash-output-scalyr

• Build the custom Logstash image with Scalyr output plugin.

docker build -t scalyr-logstash .

Modify Logstash Configuration

Now, you have a Logstash Docker image with the tag scalyr-logstash. You can add Scalyr API key and other attributes to the configuration file. Here is an example config that reads file input (i.e scalyr-plugin.log) through Logstash and ingests lines to Scalyr.  

input {
    file {
        path => "/usr/share/logstash/scalyr-genlog/scalyr-plugin.log"
        start_position => beginning
        sincedb_path => "/dev/null"
        type => "plugin"
    }
    stdin { } 
}

filter {
    mutate {
        add_field => { "parser" => "logstash_parser" }
        add_field => { "serverHost" => "my hostname" }
        rename => { "path" => "logfile" }
        rename => { "data" => "message" }
    }
}

output {
    if [type] == "plugin" {
         scalyr {
            api_write_token => "<LOG_WRITE_API_TOKEN>"
         }
    }
    stdout { codec => "rubydebug"}

}

Run the Logstash Container and Send Messages to Scalyr

I use docker-compose to manage the Logstash container.

  logstash:
    image: scalyr-logstash:latest
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
      - ./logstash/scalyr-genlog:/usr/share/logstash/scalyr-genlog
    command: sh -c "logstash -f /usr/share/logstash/pipeline/logstash.conf"
    ports:
      - "5000:5000"
      - "9600:9600"
    networks:
      - front
    logging:
      driver: "json-file"
      options:
        max-size: "2g"

After the Logstash Scalyr container successfully starts, I add a few lines to the file and confirm that the messages were successfully ingested to Scalyr.