Graylog Collector Sidecar

Required Graylog version: 2.0 and later + installed graylog-plugin-collector (this plugin available in install package - folder plugins)
The Graylog Collector Sidecar is a supervisor process for 3rd party log collectors like NXLog. The Sidecar program is able to fetch configurations from a Graylog server and render them as a valid configuration file for various log collectors. You can think of it like a centralized configuration management system for your log collectors.

Download Graylog Collector Sidecar at
https://github.com/Graylog2/collector-sidecar/releases

Installation

Download a package and install it on the target system.

Ubuntu

Install the NXLog package from the offical download page

  $ sudo /etc/init.d/nxlog stop
  $ sudo update-rc.d -f nxlog remove
  $ sudo gpasswd -a nxlog adm

  $ sudo dpkg -i collector-sidecar_0.0.8-1_amd64.deb
Edit /etc/graylog/collector-sidecar/collector_sidecar.yml, you should set at least the correct URL to your Graylog server and proper tags. The tags are used to define which configurations the host should receive.

  $ sudo graylog-collector-sidecar -service install
  $ sudo start collector-sidecar

CentOS

  $ sudo service nxlog stop
  $ sudo chkconfig --del nxlog
  $ sudo gpasswd -a nxlog root

  $ sudo rpm -i collector-sidecar-0.0.8-1.x86_64.rpm
Activate the Sidecar as a system service

  $ sudo graylog-collector-sidecar -service install
  $ sudo systemctl start collector-sidecar

Windows

Install the NXLog package from the offical download page

  $ C:\Program Files (x86)\nxlog\nxlog -u

  $ graylog_collector_sidecar_installer.exe
It's also possible to run the installer in silent mode with

  $ graylog_collector_sidecar_installer.exe /S
Edit C:\Program Files (x86)\graylog\collector-sidecar\collector_sidecar.yml, you should set at least the correct URL to your Graylog server and proper tags.

  $ C:\Program Files (x86)\graylog\collector-sidecar\graylog-collector-sidecar.exe -service install
  $ C:\Program Files (x86)\graylog\collector-sidecar\graylog-collector-sidecar.exe -service start
You can also run the Sidecar in foreground for debugging purposes. Simply call it like this and watch for error messages:

  $ graylog-collector-sidecar -c /etc/graylog/collector-sidecar/collector_sidecar.yml

Configuration

There are a couple of configuration settings for the Sidecar:

Parameter	Description
server_url	URL to the Graylog API, e.g. http://127.0.0.1:12900
tls_skip_verify	Ignore errors when the REST API was started with a self-signed certificate
node_id	Name of the Sidecar instance, will also show up in the web interface
collector_id	Unique ID (UUID) of the instance. This can be an ID string or a path to an ID file
tags	List of configuration tags. All configurations on the server side that match the tag list will be fetched and merged by this instance
log_path	A path to a directory where the Sidecar can store the output of each running collector backend
log_rotation_time	Rotate the stdout and stderr logs of each collector after X seconds
log_max_age	Delete rotated log files older than Y seconds
update_interval	The interval in seconds the sidecar will fetch new configurations from the Graylog server
backends	A list of collector backends the user wants to run on the target host

Each backend can be enabled/disabled and should point to a binary of the actual collector and a path to a configuration file the Sidecar can write to:

Parameter	Description
name	The type name of the collector
enabled	Weather this backend should be started by the Sidecar or not
binary_path	Path to the actual collector binary
configuration_path	A path for this collector configuration file Sidecar can write to

Sample configuration

server_url: http://10.0.10.10:12900

node_id: client-hostname

collector_id: file:/etc/graylog/collector-sidecar/collector-id

tags: linux

update_interval: 10

log_path: /var/log/graylog/collector-sidecar

backends:

    - name: nxlog

      enabled: true

      binary_path: /usr/bin/nxlog

      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf

Use the Graylog web interface to configure remote collectors

Navigate to System → Collectors → Manage configurations, this is the entry point for all Sidecar configurations. Multiple configurations can be created. Because not all connected Sidecars should fetch all configurations, it’s essential to provide tags for each configuration. Every Sidecar is only fetching the configuration with the tag it was started with. See also the tags parameter in the section before. Each configuration can hold parts for multiple collector backends.

So you can create one configuration with the tag linux and this include e.g. an input section for a NXlog collector and one for a Filebeat collector. The Sidecar will then pick the right parts based on the backends that are enabled for the host system.

Outputs, Inputs and Snippets

In the example above, Sidecar is instructing NXlog to create a GELF output that writes log messages back to Graylog. The two inputs are for reading in /var/log/syslog as a file input and listening on the UDP port 514 for incoming syslog messages. Both inputs route their messages to the GELF output.

There are three sections in a configuration: Outputs, Inputs and Snippets.

Step 1: Create outputs - Once data is collected by NXLog, the data is transmitted to this IP or address and port. You need to configure a GELF “Input” (System->Inputs) to capture data on the port.

Step 2: Create inputs - Data collected by NXLog. Think of this as a source of log data. For example, it could be a file or a syslog.

In this step we will assign the Collector Output in step 1 we have created.

Step 3: Snippets - Snippets can be used to represent more complicated collector configurations. Simply paste the whole content of your NXlog configuration into a snippet or use it as an extension to the inputs and outputs defined before. All snippets will be copied directly to the generated collector configuration, no matter if there inputs or outputs defined.

Step 4: Fill the tag of Collector, this step will apply the configure we doing to the Collector Sidecar. So we can manage which client should use which configuration.

tags: linux -> this tag is linux.

Step 5: Click on Update tags to push the configure to client.

Configure Collector get log from Syslog services

In Collector Configuration > Configure Collector Inputs > Create new Input

Name: Syslog

Forward to: Output in Step 1

Type: [NXLog] UDP Syslog listener

Host: 127.0.0.1

Port: 514

Next I will configure Rsyslog on Client machine, which running Graylog collector sidecar.

vi /etc/rsyslog.conf

Update remote host then restart rsyslog services

*.* @127.0.0.1:514

Issues

Log folder (centos 7): /var/log/graylog/collector-sidecar

Log file: nxlog_stderr.log nxlog_stdout.log nxlog.log

2016-07-04 05:57:06 ERROR Failed to load module from /usr/lib/nxlog/modules/extension/xm_gelf.so, /usr/lib/nxlog/modules/extension/xm_gelf.so: cannot open shared object file

Resolve:

- Find missing file

# rpm -ql nxlog-ce | grep xm_gelf

/usr/libexec/nxlog/modules/extension/xm_gelf.so

Create symbolink

# ln -s /usr/libexec/nxlog/modules/* /usr/lib/nxlog/modules/.