So a while ago i was in a situation were i had to centrally collect logs for auditing purposes, after much testing, poking and prodding i decided to use a piece of open-source software called Graylog2, initially not the easiest software to setup due to it only being available on Linux (not my speciality), but with some persistence and and a few “lessons learnt” i managed to get there in the end.
This software does precisely what it says on the tin, with the help of nxlog installed on the client machines it collects all logs and lists them by device, along with giving you some pretty helpful dashboards. In my case there was a lot of data, collecting well over 2000 logs per minute, so the facility to search and visualise the logs within a web interface is very useful.
Back to my initial point however, this server had been ticking away for well over a year with no major issues except the occasional reboot. However following a recent penetration test it was revealed it was behind on patches, as well as Graylog being a number of versions out.
So the task was to update Ubuntu upto version 17, and upgrade Graylog to version 2.3.1, which after the previous “lessons learnt” could have been a tricky process. However having a read though the upgrade guide it appears things have changed a bit in the newer versions, a major part of Graylog is both the MongoDB database which holds all the data, along with Elasticsearch which is used to do a lot of the heavy searching and indexing, both are prerequisites for Graylog and must be installed and setup prior to the main software.
For installation i used the following site for reference, i will detail the steps below but credit for this is with Digital Ocean for a very good step by step guide to follow, i found this very helpful with not being overly familiar with Ubuntu.
The software as you can imagine a log of disk space, although compression is very good make sure you put as much space in your server as possible.
We need to modify the Elasticsearch configuration file so that the cluster name matches the one set in the Graylog configuration file. To keep things simple, we’ll set the Elasticsearch cluster name to the default Graylog name of
graylog. You may set it to whatever you wish, but make sure you update the Graylog configuration file to reflect that change.
Open the Elasticsearch configuration file in your editor:
Find the following line:
cluster.name: <CURRENT CLUSTER NAME>
cluster.name value to
Save the file and exit your editor.
Since we modified the configuration file, we have to restart the service for the changes to take effect.
Now that you have configured Elasticsearch, let’s move on to installing Graylog.
In this step, we we’ll install the Graylog server.
First, download the package file containing the Graylog repository configuration. Visit the Graylog download page to find the current version number. We’ll use version
2.2 for this tutorial.
Next, install the repository configuration from the
.deb package file, again replacing
2.2 with the version you downloaded.
Now that the repository configuration has been updated, we have to fetch the new list of packages. Execute this command:
Next, install the
Lastly, start Graylog automatically on system boot with this command:
Graylog is now successfully installed, but it’s not started yet. We have to configure it before it will start.
Now that we have Elasticsearch configured and Graylog installed, we need to change a few settings in the default Graylog configuration file before we can use it. Graylog’s configuration file is located at
/etc/graylog/server/server.conf by default.
First, we need to set the
password_secret value. Graylog uses this value to secure the stored user passwords. We will use a randomly-generated 128-character value.
We will use
pwgen to generate the password, so install it if it isn’t already installed:
Generate the password and place it in the Graylog configuration file. We’ll use the
sed program to inject the
password_secret value into the Graylog configuration file. This way we don’t have to copy and paste any values. Execute this command to create the secret and store it in the file:
For more information on using
sed, see this DigitalOcean sed tutorial.
Next, we need to set the
root_password_sha2 value. This is an SHA-256 hash of your desired password. Once again, we’ll use the
sed command to modify the Graylog configuration file so we don’t have to manually generate the SHA-256 hash using
shasum and paste it into the configuration file.
Execute this command, but replace
password below with your desired default administrator password:
Note: There is a leading space in the command, which prevents your password from being stored as plain text in your Bash history.
Now, we need to make a couple more changes to the configuration file. Open the Graylog configuration file with your editor:
Find and change the following lines, uncommenting them and replacing
graylog_public_ip with the public IP of your server. This can be an IP address or a fully-qualified domain name.
... rest_listen_uri = http://your_server_ip_or_domain:9000/api/ ... web_listen_uri = http://your_server_ip_or_domain:9000/ ...
Save the file and exit your editor.
Since we changed the configuration file, we have to restart (or start) the
graylog-server service. The restart command will start the server even if it is currently stopped.
Next, check the status of the server.
The output should look something like this:
● graylog-server.service - Graylog server Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2017-03-03 20:10:34 PST; 1 months 7 days ago Docs: http://docs.graylog.org/ Main PID: 1300 (graylog-server) Tasks: 191 (limit: 9830) Memory: 1.2G CPU: 14h 57min 21.475s CGroup: /system.slice/graylog-server.service ├─1300 /bin/sh /usr/share/graylog-server/bin/graylog-server └─1388 /usr/bin/java -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSCon
You should see
active for the status.
If the output reports that the system isn’t running, check
/var/log/syslog for any errors. Make sure you installed Java when you installed Elasticsearch, and that you changed all of the values in Step 3. Then attept to restart the Graylog service again.
If you have configured a firewall with
ufw, add a firewall exception for TCP port
9000 so you can access the web interface:
Once Graylog is running, you should be able to access
http://your_server_ip:9000 with your web browser. You may have to wait up to five minutes after restarting
graylog-server before the web interface starts. Additionally, ensure that MongoDB is running.
Now that Graylog is running properly, we can move on to processing logs.
Let’s add a new input to Graylog to receive logs. Inputs tell Graylog which port to listen on and which protocol to use when receiving logs. We ‘ll add a Syslog UDP input, which is a commonly used logging protocol.
When you visit
http://your_server_ip:9000 in your browser, you’ll see a login page. Use
admin for your username, and use the password you entered in Step 3 for your password.
Once logged in, you’ll see a page titled “Getting Started” that looks like the following image:
To view the inputs page, click the System dropdown in the navigation bar and select Inputs.
You’ll then see a dropdown box that contains the text Select Input. Select Syslog UDP from this dropdown, and then click on the Launch new input button.
A modal with a form should appear. Fill in the following details to create your input:
Linux Server Logs.
8514. Note that we are using port
8514for this tutorial because ports
1024can be only used by the root user. You can use any port number above
1024should be fine as long as it doesn’t conflict with any other services.
Click Save. The local input listing will update and show your new input, as shown in the following figure:
Now that an input has been created, we can send some logs to Graylog.
We have an input configured and listening on port
8514, but we are not sending any data to the input yet, so we won’t see any results.
rsyslog is a software utility used to forward logs and is pre-installed on Ubuntu, so we’ll configure that to send logs to Graylog. In this tutorial, we’ll configure the Ubuntu server running Graylog to send its system logs to the input we just created, but you can follow these steps on any other servers you may have.
If you want to send data to Graylog from other servers, you need to add a firewall exception for UDP port
Create and open a new
rsyslog configuration file in your editor.
Add the following line to the file, replacing
your_server_private_ip with your Graylog server’s private IP.
Save and exit your editor.
rsyslog service so the changes take effect.
Repeat these steps for each server you want to send logs from.
You should now be able to view your logs in the web interface. Click the Sources tab in the navigation bar to view a graph of the sources. It should look something like this:
You can also click the Search tab in the navigation bar to view a overview of the most recent logs.
You can learn more about searches in the Graylog searching documentation.
You now have a working Graylog server with an input source that can collect logs from other servers.
Next, you might want to look into setting up dashboards, alerts, and streams. Dashboards provide a quick overview of your logs. Streams categorize messages, which you can monitor with alerts. To learn more about configuring the more advanced features of Graylog, you can find instructions in the Graylog documentation.
Content take from Digital Ocean at the following website
I will produce further articles as i progress with Graylog, but for now i would advise setting up a server and doing some testing, Graylog has a very strong support section on there website, along with a marketplace giving you hundreds of plugins and content packs to play with