ELK Stack Notes

Russell Bateman
August 2018
last update:

Table of Contents

Forward and Summary
Preliminaries
Download, shallow configuration and installation (launch)
Elasticsearch
Nginx
Logstash
Kibana
Filebeat
Reach ELK stack from browser
Draining data from ELK
Appendix: Collected ELK links
Appendix: Securing ELK

Foreword

This tutorial displays better if you have the Candara font available (so, from a browser on Windows or after installing this font on Linux.) The Candara font and my use of the Ergonomic Keyboard 4000 are practically my only concessions to Microsoft's product line.

These notes are intended for an Ubuntu 18.04.1 Bionic headless server installation (and I have done this), but the day I wrote this, I did it from scratch using a spare desktop.

System specifications

Operating System: Linux Mint 18.2 Cinnamon 64-bit (desktop)
Linux Kernel:     4.13.0-45-generic
Processor:        Intel© Core™ i5 CPU 750 @ 2.67GHz × 4
Memory:           8Gb
Hard Drive:       1Tb

I began an ELK installation on host tuonela (at home). Note that commands that require root access are simply prefixed with # instead of $. As I'm acquiring the relevant packages, there is no possibility of automatic update.

Summary

  1. Elasticsearch
  2. Nginx
  3. Logstash
  4. Kibana
  5. Filebeat (to be installed on my Plex Media server, not on ELK stack host itself)

Preliminaries

You need to install Java 8 or later and set up JAVA_HOME. If you install something newer than Java 9, some Elastic 6.4.2 agents will complain of an unrecognized VM option (UseParNewGC). This option disappeared in Java 9 and you may safely ignore this error. You may also safely ignore error, "chmod: cannot access '/etc/default/logstash': No such file or directory' if you see it.

These notes assume you're running Ubuntu Server 18.04.1 Bionic (headless). If something else, your mileage may vary. For example, earlier versions of Ubuntu, that is, prior to Ubuntu 15 Vivid, did not offer systemd, so the systemctl commands below will have to be done otherwise (at very least you can pop open a shell and run them from the command line).

Here is where the various Elastic component binaries are installed by their Debian packages. This is interesting to know as you follow steps in the next section:

  1. /usr/share/elasticsearch/bin/elasticsearch
  2. /usr/sbin/nginx
  3. /usr/share/logstash/bin/logstash
  4. /opt/kibana/bin/kibana
  5. /usr/share/filebeat/bin/filebeat

Download, shallow configuration and installation (launch)

  1. Preparation
    $ mkdir -p elk/done
    $ cd elk
    $ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.2.deb
    $ wget http://archive.ubuntu.com/ubuntu/pool/universe/n/nginx/nginx-light_1.14.0-0ubuntu1_amd64.deb
    $ wget http://archive.ubuntu.com/ubuntu/pool/main/n/nginx/nginx-common_1.14.0-0ubuntu1_all.deb
    $ wget http://archive.ubuntu.com/ubuntu/pool/universe/n/nginx/libnginx-mod-http-echo_1.14.0-0ubuntu1_i386.deb
    $ wget https://artifacts.elastic.co/downloads/logstash/logstash-6.4.2.deb
    $ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-amd.deb
    $ wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.4.2-amd64.deb
    
  2. Elasticsearch
    # dpkg --install elasticsearch-6.4.2.deb
    $ mv elasticsearch-6.4.2.deb done/
    # vim /etc/elasticsearch/elasticsearch.yml (configure network.host)
    
    network.host: 0.0.0.0
    # vim /etc/elasticsearch/jvm.option (configure Java heap)
    -Xms4g -Xmx4g
    # systemctl start elasticsearch

    Elasticsearch is a service that runs as user elasticsearch (used to be user elastic.) Once installed, you need to change the password (using root) to changeme for this tutorial.

  3. Nginx light. Not strictly a member of the ELK stack, this is to host Kibana. It's difficult to find package downloads, but if you can, these are the ones that will give you the lightest footprint. Or, you can just install the canonical (lowercase—no pun intended) package (see apt-get command just below).
    # apt-get install nginx
    # dpkg --install nginx-common.1.14.0-0ubuntu1_all.deb
    # dpkg --install libnginx-mod-http-echo_1.14.0-0ubuntu1_amd64.deb
    # dpkg --install nginx-light_1.14.0-0ubuntu1_amd64.deb
    # mv nginx-common.1.14.0-0ubuntu1_all.deb done/
    # mv libnginx-mod-http-echo_1.14.0-0ubuntu1_amd64.deb done/
    # mv nginx-light_1.14.0-0ubuntu1_amd64.deb done/
    # ufw allow from any to any port 80 proto tcp (no need for this if you installed with apt-get)
    # vim /etc/nginx/conf.d/kibana.conf           (put these lines in the new file)
    
    server { listen 80; server_name whatever you want here; location / { proxy_pass http://localhost:5601; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } }
    # systemctl start nginx.service
  4. Logstash
    # dpkg --install logstash-6.4.2.deb
    $ mv logstash-6.4.2.deb done/
    # vim /etc/logstash/conf.d             (sample configuration for Apache logs)
    
    input { # tells Logstash to pull logs from Apache file { path => "/var/log/apache/access.log" start_position => "beginning" sincedb_path => "/dev/null" } # tells Logstash to ingest anything you type at the keyboard (Logstash's stdin) # --if you run Logstash from the command line (so, good for brute debugging) stdin {} } filter { # parses the log string and populates the event with information from... grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } # date (defines timestamp field) date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } # and the client's geographical information geoip { source => "clientip" } } output { # tells Logstash to send extracted and transformed data to Elasticsearch elasticsearch { hosts => ["localhost:9200"] } }
    # vim /lib/systemd/system/logstash.service (how to set up Logstash for systemd)
    [Unit] Description=Logstash Service [Service] ExecStart=/usr/share/logstash/bin/logstash -f /etc/logstash/logstash.yml StandardOutput=null [Install] Alias=logstash
    # systemctl daemon-reload (warns systemd to reload for changes) # systemctl enable logstash.service Created symlink /etc/systemd/system/logstash → /lib/systemd/system/logstash.service. # systemctl start logstash.service # systemctl status logstash.service
  5. Kibana
    # dpkg --install kibana-6.4.2.deb
    $ mv kibana-6.4.2.deb done/
    # vim /etc/kibana/kibana.yml (configure server.host)
    
    server.host: host's IP address
    # systemctl start kibana.service
  6. Filebeat —install on tol-eressea, my Plex Media server, redirect what this finds to host tuonela, running Elasticsearch*, equivalent to Splunk's forwarder
    # dpkg --install filebeat-6.4.2-amd64.deb
    $ mv filebeat-6.4.2-amd64.deb done/
    # vim /etc/filebeat/filebeat.yml
    
    output.elasticsearch: hosts: ["tuonela:9200"]* protocol: "https" username: "elasticsearch" password: "changeme"**

    Information in /etc/filebeat/filebeat.yml would lead you to believe that Elasticsearch's user/password is elastic/changeme, but the user is now elasticsearch and you should have changed the password to changeme for this tutorial.

* Yes, Elasticsearch is up and running on tuonela:

russ@tuonela ~ $ systemctl status elasticsearch
 elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: active (running) since Sat 2018-08-25 09:27:56 MDT; 1h 35min ago
     Docs: http://www.elastic.co
 Main PID: 8403 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─8403 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSI
           └─8467 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Aug 25 09:27:56 tuonela systemd[1]: Stopped Elasticsearch.
Aug 25 09:27:56 tuonela systemd[1]: Started Elasticsearch.

** Yes, I changed Elasticsearch's password to changeme for now.

Tutorial stopped! tol-eressea, my Plex Media Server, is running this version:

Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-48-generic x86_64)

This version does not support systemd services. I must run Filebeat from the command line.

Reach ELK stack from browser

The next thing to do is to see if Elasticsearch is up, using curl:

$ curl tuonela:9200
{
  "name" : "u-tiTM2",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "Ic1V1rBCR32KHh06-xjzZA",
  "version" : {
    "number" : "6.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "595516e",
    "build_date" : "2018-08-17T23:18:47.308994Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

We could also use a browser and we must use a browser to reach Kibana. A caveat first, however. When I attempt to reach Elasticsearch or Kibana via a browser from a host at work running either of my VPN clients, it sometimes doesn't work unless and until I open a Chrome incognito window, but sometimes it does work. (I'm clueless, but it probably has nothing to do with the VPN.)

In a browser, for Elasticsearch, I see the same thing as from curl. Note that if I make my browser address line http://tuonela, I just get Nginx' welcome message, which is exactly what I'd expect.

For Kibana, here's my browser address line: https://tuonela:5601

...and, here's what I see:



Draining data from ELK

In early experimentation and testing, it might be useful to know how to drain ELK without having to obliterate the installation and start over from scratch. This amounts to draining data out of Elasticsearch. There are a couple of ways, the right one is to use the curl command below, but I'm giving other details too.

  1. Use an Elasticsearch API:
    # systemctl stop filebeat.service
    # curl -XDELETE localhost:9200/_all
    
  2. Remove all the indices in Elasticsearch's data: # systemctl stop filebeat.service # rm -rf /var/lib/elasticsearch/nodes/indices/*

Appendix: Collected ELK links

Elastic links:

Other links:

Appendix: Securing ELK

A common security precaution is to allow access to Elasticsearch from only localhost and to restrict access to Kibana by using Nginx as a reverse proxy to Kibana running on localhost. Use either HTTP authentication to restrict access to Kibana or use LDAP authentication from Nginx.

Here are some useful links:

Appendix: Random notes

Kibana since 6.2 has something called application performance monitoring (APM); it's a module that's loaded. See APM with Elasticsearch, Elastic Stack (ELK Stack).