Saturday, 22 August 2015

Using Docker Compose to setup Elasticsearch, Kibana and Packetbeat

This post is based on work done by Alex :

I started looking at packetbeat earlier today and wanted to use docker-compose to simplify the setup of Elasticsearch, Kibana and Packetbeat.

The beauty of Docker is that I know nothing about Elasticsearch, Kibana or packetbeat but within a few minutes it is up and running and it's play time :)

The code for this blog can be found here :

Firstly ensure you have docker and docker-compose installed.

Next up, grab the code and run docker-compose.
  1. git clone
  2. cd packetbeat
  3. docker-compose up -d
Output :

Creating packetbeat_test_1...
Creating elasticsearch...
Creating kibana...
Creating packetbeat...

That's it.

So what just happened?  We now have 4 containers running.  

To verify this run :

docker-compose ps

Output :

      Name                     Command               State                        Ports                      
elasticsearch       / elas ...   Up>9200/tcp,>9300/tcp 
kibana              / kibana     Up>5601/tcp                         
packetbeat          /bin/bash -c                     Up                                                      
                        echo Wai ...                                                                         
packetbeat_test_1   bash -c apt-get update           Exit 0   

The Elasticsearch container is started first.  The Kibana container is started second and linked to elasticsearch.  The packetbeat container then waits for Kibana to become available before running the packetbeat script.  This is a crude netcat command but it works for this example.

The packetbeat_test container just puts some test data into elasticsearch to ensure everything is working.

Open a browser and go to : http://localhost:5601

Set the index pattern to : packetbeat-*

If no items are returned after entering this pattern it means the packetbeat_test container ran too soon.  To rerun the test container just run :

docker-compose up -d test

Go back to the browser, update the pattern to packetbeat-* and the data packets will be available.

Click on the "Discovery" tab and you will see details of the data packets from the packetbeat_test container.

There are lots of cool filters, visualizations and search capabilities available.  That's as far as I've got. 

One last thing.  You can use the docker-compose scale option to populate elasticsearch with alot more data.  Worth noting that if you are in Starbucks using your mobile internet and you scale to 100 test containers that run apt-get install then you very quickly hit your data limit. 

This command will start 20 test containers, each running apt-get update.

docker-compose scale test=20

No comments:

Post a Comment