Deploying Graylog

Graylog is an Open Source log ingestion platform, similar to Splunk.
Its a great way to centralize logs and analyze, alert and act upon events for networks ranging from a few home devices, to full corporate deployments.
Almost every device & app has some sort of logging facility built-in, that can be routed easily (or with some configuration) to a platform like Graylog.

Why is this not being deployed to Unraid like the other projects?

One of my drivers for this was to get Unraid logs prior to a system failure - Unraid logs are stored in RAM, and are lost on reboot, power outage, etc. - not great for diagnosing the cause!
Having Unraid set to send logs to a remote system, in my case a Raspberry Pi 4, it means that logs keep getting 'logged' up until the last moment of the system going down.
TL:DR - you can't rely on a system you need the logs on, to log itself.

Pre-Req's

A docker host
Docker compose

Considerations

Whilst for small usage scenarios apps like Graylog are not resource intensive on most modern systems, with all 3x containers using less than 2.5Gb RAM and a few % of CPU time on a Raspberry Pi 4 for example, the bigger concern is the disk IO and its effect on your storage. Logs are small for most file system & disk block sizes, and can cause excessive wear on solid storage devices, like MicroSDs & SSDs.
As such, you should consider where you are writing your elastic data too -

Do you care about its longevity? If so, write it instead to a HDD.
Do you not care about longevity, but care about keeping the data? If so, work out a backup method for said data?
Do you not care about either? Write to a SSD or MicroSD.
Do you care about having the OS operational, but don't care what happens to your data? Separate out the OS and data storage devices.
etc.

Docker Compose

Copy/paste the below code block into your text editor of choice, edit the variables listed underneath, then run docker-compose to deploy -

note

Why Graylog v4 and not v5 ?
Graylog v5 needs Mongo v5+, Mongo v5 & 6 (at the time of writing) does not support the ARMv8 instruction set that Raspberry Pi's use.
If you are using x86 hardware however, you can modify the below to the latest releases if you so choose.

docker-compose -up

version: '3'
services:
    # MongoDB:
    mongo:
      image: mongo:4.2
      networks:
        - graylog
      restart: unless-stopped
      volumes:
        - mongo_db:/data/db
        - mongo_config:/data/configdb       

    # Elasticsearch:
    elasticsearch:
      image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
      environment:
        - http.host=0.0.0.0
        - transport.host=localhost
        - network.host=0.0.0.0
        - "ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true -Xms512m -Xmx512m"
      limits:
        memlock:
          soft: -1
          hard: -1
      deploy:
        resources:
          limits:
            memory: 1g
      volumes:
        - graylog-elasticsearch_data:/usr/share/elasticsearch/data
      networks:
        - graylog
      restart: unless-stopped
    # Graylog:
    graylog:
      image: graylog/graylog:4.3
      environment:
        # Timezone (important!)
        - GRAYLOG_ROOT_TIMEZONE=America/New_York
        - TZ=America/New_York
        # Root usernames and password (below applies to user 'admin')
        # Password Secret
        - GRAYLOG_PASSWORD_SECRET=password_secret
        # Password SHA2
        - GRAYLOG_ROOT_PASSWORD_SHA2=root_password_sha2
        # Email Settings
        - GRAYLOG_TRANSPORT_EMAIL_ENABLED=true
        - GRAYLOG_TRANSPORT_EMAIL_HOSTNAME=email_server_address
        - GRAYLOG_TRANSPORT_EMAIL_PORT=email_server_port
        - GRAYLOG_TRANSPORT_EMAIL_USE_AUTH=true
        # Enable only one of TLS or SSL.
#        - GRAYLOG_TRANSPORT_EMAIL_USE_TLS=true
#        - GRAYLOG_TRANSPORT_EMAIL_USE_SSL=false
        - GRAYLOG_TRANSPORT_EMAIL_AUTH_USERNAME=email_username
        - GRAYLOG_TRANSPORT_EMAIL_AUTH_PASSWORD=email_password
        - GRAYLOG_TRANSPORT_EMAIL_SUBJECT_PREFIX=[graylog]
        - [email protected]
        # Other Settings
        - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
        - GRAYLOG_TRUSTED_PROXIES=127.0.0.1/32
        - GRAYLOG_ALLOW_LEADING_WILDCARD_SEARCHES=true
      entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
      networks:
        - graylog
      restart: unless-stopped
      depends_on:
        - mongo
        - elasticsearch
      ports:
        # Graylog web interface and REST API
        - 9900:9000
        # Syslog TCP
        - 1514-1530:1514-1530
        # Syslog UDP
        - 1514-1530:1514-1530/udp
        # GELF TCP
        - 12201-12220:12201-12220
        # GELF UDP
        - 12201-12220:12201-12220/udp
        # Filebeats and API
        - 5044:5044
      volumes:
        - app_data:/usr/share/graylog/data
        - app_plugins:/usr/share/graylog/plugin
networks:
    graylog:
      driver: bridge
volumes:
  mongo_config:
  mongo_db:
  app_data:
  app_plugins:
  graylog-elasticsearch_data:
      

Edit the following variables -

Variable	Edit to	Notes
GRAYLOG_ROOT_TIMEZONE	Enter your timezone in PHP format, for example, Americas/New_York	Lookup here
TZ	Enter your timezone in PHP format, for example, Americas/New_York	Lookup here
GRAYLOG_PASSWORD_SECRET	You MUST set a secret that is used for password encryption and salting. The server refuses to start if this value is not set
GRAYLOG_ROOT_PASSWORD_SHA2	A SHA2 hash of the password you will use for your initial login
GRAYLOG_TRANSPORT_EMAIL_HOSTNAME	Address of your email server
GRAYLOG_TRANSPORT_EMAIL_USE_AUTH	Use auth for your email server. True/False
GRAYLOG_TRANSPORT_EMAIL_PORT	Port to use for email
GRAYLOG_TRANSPORT_EMAIL_USE_SSL	Use SSL
GRAYLOG_TRANSPORT_EMAIL_USE_TLS	Use TLS
GRAYLOG_TRANSPORT_EMAIL_AUTH_USERNAME	Email Username
GRAYLOG_TRANSPORT_EMAIL_AUTH_PASSWORD	Email Password
GRAYLOG_TRANSPORT_EMAIL_FROM_EMAIL	"From Email" Address
GRAYLOG_TRUSTED_PROXIES	Comma separated list of proxies/subnets to accept for incoming proxy requests	127.0.0.1/32, 1.2.3.4/32 - 127 should be specified regardless of additional entries
GRAYLOG_ALLOW_LEADING_WILDCARD_SEARCHES	Allow wildcard searches, true/false	Can be detrimental to performance on large/busy installations. Adjust as you see fit.

danger

Timezones
Its important that timezones are set accurately to where you are, as both the container & graylog variables for timezone will affect the timestamped messages and ultimately affect your ability to filter data correctly.

Password SHA2 Generation

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

Password Secret Generation

pwgen -N 1 -s 96

Port Ranges

At a basic level, one port is should be allocated for every input on Graylog.
With this in mind, and to give a level of expansion in future, the docker-compose file has a range of 20-30 ports for the two main 'types' of incoming connections, GELF & Syslog.
You can and should reduce this down to just required ports in your final deployment.

You can technically just use the one port for this, and instead separate out messages based on streams and rules, however you lose some of the granularity with this method.

After a few minutes, everything will start and you can browse to the URL of your docker host, with port 9900.
For example: http://docker_host_ip:9900
You will be presented with the Graylog login screen, where you can login with -

Username: admin
Password: the password you entered as part of the variable for 'root password sha2'.

Creating your first input

Graylog configuration

Once logged in, on the menu at the top select System, then select inputs.
This page is where we can manage inputs, can do basic monitoring, etc.

At the top, select Select Input, choose Syslog TCP from the list, then click Launch new input on the right.

The screen that appears allows us to setup the settings for a new input.
Under title give it a name that describes what the input is, for example "UniFi"
Edit the port to be one of the unique ports listed as a syslog port in the docker-compose, for example, "1530".
Everything else can be left as default. Click Save in the bottom right.

You will now see your newly created input, showing a summary of settings on the left, and the throughput of metrics on the right.
Right now, there is nothing coming in to see.

Device configuration

This is highly dependant on the device being input, however the steps should be mostly the same.
In your device/app of choice, configure the syslog settings to be the IP of your graylog instance (docker host) port that you chose above, then apply it.
At this point, your device/app should start logging its syslog events to Graylog.

Unraid Example

Generic Linux Example

/etc/rsyslog.conf
(bottom of file)

# Log to Graylog
*.* @@logserver:1522

We can check this by going back to graylog, and viewing the input again, which should now show some IO and totals on the right -

We can now view what is being inputted, click Show received messages on the input.
On the screen that shows, ensuring that the time variable in the top left is set to 'All time', the bottom All Messages section should populate with any messages it has received. You can then click on one to show additional parsed info -

Troubleshooting

I don't get any input on the message list?

First, check that your device/app is configured correctly, then check that it actually has a log entry to send.

The timestamp is wrong

Check you have inputted the correct timezone and TZ variable during deployment.
If not, correct & redeploy.
Note: existing data will not re-update, only newly received messages
Next, check the time on your device/app is also correct, with the correct timezone.

Creating Windows Inputs

Coming Later!

Tips & Notes

Indexes & Streams

Incoming messages can (and arguably should) be separated into its their indices and streams.
This will separate the data out into its own indexes and allow you to run separate pipelines against each data type being received, allowing you to be more granular with what you do.

Creating new indexes

On the menu at the top select System, then select Indices.

On the top right, click Create Index Set.

On this screen we can configure the index and rotation settings for the index.
Enter a title & Index prefix, the edit the rotation and retention configuration at the bottom to your liking, using the examples it gives as a reference.
For 'prefix', I would recommend a format of "title_".
Click Save at the bottom.

We now need to configure data from one (or many) inputs to go into the newly created index.
On the menu at the top select Streams.

On the top right, click Create Stream.

On the screen that shows, enter a Title, Description & use the drop down to select the Index Set you just created.
Tick Remove matches from 'All Messages' stream, then click Save.

On your newly created stream, on the right, click Manage Rules.

On this screen, is where we configure the stream to gather messages based on rules, to direct into your index.
Under Recent Message, click the Select an Input drop down, and select your input.
Then click Load Message.

On the right, click Add stream rule.
This is where we can set the rule to intercept the messages and redirect.
Under Field type in Source.
Set Type to match exactly.
Set Value to the name of your input.
Click Save.

info

You need to have recent messages in your input to be able to test rules against, otherwise this screen will not work.

You will now be returned to the previous screen, but with the rule/s at the bottom (that you created) highlighting in green or red to indicate if they have applied.
Check that they are applied as you wish, as this is what will ensure that the messages go to your new index.

Click I'm done! - at this point you literally are.
Messages matching the rule in the stream, from the input you selected, will redirect to the index you created, with the rules on retention & rotation you set.

Pipelines & Rules

Pipelines and rules can be used to transform/drop/filter messages as they hit your instance, allowing you to filter out whitenoise automatically, correct issues with messages, etc.
As a basic/common rule, lets deploy a pipeline to drop messages of a certain type.

Create the pipeline

On the menu at the top select System, then select Pipelines.

Next, on the right, click New Pipeline.

Give the pipeline a title (minimum), then click Save.

Next, click Edit connections on the right.

In the box that shows, select the stream you want the pipeline to apply to in the dropbox box, then press Save.

Create a basic rule

On the top right, click Manage rules.

On the top right, click Create Rule.

This is where we can input the logic for rules, which is done in code.
The description box at the top is entirely optional, as the 'title' is taken from the rule code itself.

Use the following code block as a starting point, changing the string value XXXX to what you want to filter.

rule "Test - Drop - Drop messages with XXXX in"
when
  contains(to_string($message.message), "XXXX", false)
then
  drop_message();
end

Next, click Save & Close

On the top right, click Manage pipelines.

On the right of your pipeline, click Edit.

We now need to define how many stages we need for our pipelines, however in this basic example, a one stage pipeline is fine.
Click Edit on the right of Stage 0.

On this screen we need to add the rule you just created from the dropdown, followed by Save in the bottom right.

tip

You can add multiple rules to each stage if needed.

You should now have a fully functioning pipeline that filters the messages out!

Troubleshooting

Messages not being filtered

Check the Message Processors Configuration, under System->Configurations, ensuring that Message Filter Chain is above Pipeline Processor. You can edit the order with the Update button.

Messages are being received, but not processed

You can add another rule to your pipeline stage that will output what its doing to the system log (or docker log) -

rule "debug log all"
when
    true
then
    debug("Pipeline got: " + to_string($message.message));
end

Indexing Failures

Indexing failures are exactly as they sound - a failure to index.
Unfortunately, there is no function in Graylog to clear out the failures in its webGUI, so we will need to drop into MongoDB's CLI to resolve.

First, bash into your container -

docker exec -it [mongo container name] /bin/bash

Next, run the following commands -

mongo
use graylog
db.index_failures.drop()
exit

tip

You can use show collections both before the collection drop, and after to verify both name, and that its been dropped.

Further reading

"Deflector exists as an index and is not an alias"

See also: "Elasticsearch cluster is yellow/unassigned shards" This can be caused by different things, such as an indexing error, missing indices, etc.
Which themselves can each have different causes.
The solution to this is centered around Elasticsearch itself.

Shutdown the Graylog container.
Connect a shell session to the Elasticsearch container.
docker exec -it elasticsearch /bin/bash
List problematic indices.
curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
Delete problematic indices.
curl -X DELETE "localhost:9200/NAMEofINDEX?pretty"

Replace NAMEofINDEX with the name of the index from the previous command.
5. Exit the shell, and restart Graylog again.
6. Done!

Deploying Graylog

Why is this not being deployed to Unraid like the other projects?​

Pre-Req's​

Considerations​

Docker Compose​

Port Ranges​

First Login​

Creating your first input​

Graylog configuration​

Device configuration​

Unraid Example​

Generic Linux Example​

Troubleshooting​

I don't get any input on the message list?​

The timestamp is wrong​

Creating Windows Inputs​

Tips & Notes​

Indexes & Streams​

Creating new indexes​

Pipelines & Rules​

Create the pipeline​

Create a basic rule​

Troubleshooting​

Messages not being filtered​

Messages are being received, but not processed​

Indexing Failures​

"Deflector exists as an index and is not an alias"​

Why is this not being deployed to Unraid like the other projects?

Pre-Req's

Considerations

Docker Compose

Port Ranges

First Login

Creating your first input

Graylog configuration

Device configuration

Unraid Example

Generic Linux Example

Troubleshooting

I don't get any input on the message list?

The timestamp is wrong

Creating Windows Inputs

Tips & Notes

Indexes & Streams

Creating new indexes

Pipelines & Rules

Create the pipeline

Create a basic rule

Troubleshooting

Messages not being filtered

Messages are being received, but not processed

Indexing Failures

"Deflector exists as an index and is not an alias"