Install and configure rsyslog Centralized logging server in CentOS 6.5

In this tutorial we will learn, how to install and configure rsyslog 7.6 version on RHEL 6.5/CentOS 6.5 .The scenario is,install and setup rsyslog Centralized Logging Server in RHEL/CentOS 6.5. All the logs from client servers will be sent to Centralized logging server i.e rsyslog server.

Check Pre-installed rsyslog package

Step 1: First of all check the rsyslog package is installed in your system.Generally by-default we get rsyslog version 5.x , after minimal installation of CentOS 6.x / RHEL 6.x 
We will install the latest rsyslog package. At the time of writing this post, rsyslog stable version 7.6 was available. You can find the latest package information from rsyslog official website
Note: By default, RHEL 6.x and CentOS 6.x has rsyslog version 5.x. So here we will update the rsyslog with new version.
You can get the rsyslog version information,by using below given two commands
rpm -qa|grep rsyslog
And
rsyslogd -v
See the below given screenshot

Installing / Updating the rsyslog version 7.6 in RHEL 6.x / CentOS 6.x

For installing rsyslog version 7.6. Create a new yum client repo file and paste the below given contents. (With the same method,you can install other version of rsyslog [information link])
Creating new file /etc/yum.repos.d/rsyslog.repo (You can use your favorite editor )

vi /etc/yum.repos.d/rsyslog.repo

Paste the below given contents in file /etc/yum.repos.d/rsyslog.repo (In vi editor, for inserting content press i key,then paste content in file , for saving press key esc :wq )

[rsyslog-v7-devel]
name=Adiscon Rsyslog v7-devel for CentOS-$releasever-$basearch
baseurl=http://rpms.adiscon.com/v7-devel/epel-$releasever/$basearch
enabled=0
gpgcheck=0
protect=1

[rsyslog-v7-stable]
name=Adiscon Rsyslog v7-stable for CentOS-$releasever-$basearch
baseurl=http://rpms.adiscon.com/v7-stable/epel-$releasever/$basearch
enabled=1
gpgcheck=0
protect=1

For installing the rsyslog,in case the package is not available (rpm -qa|grep rsyslog). Run below given command

yum install rsyslog

For updating to new rsyslog version, run below given command

yum update rsyslog

Configure rsyslog in RHEL 6.x / CentOS 6.x

Step 1 : Enable the module.We will uncomment the below given line by removing #
Take the backup of original file

cp -pv /etc/rsyslog.conf /etc/rsyslog.conf.orig

Edit the file /etc/rsyslog.conf

vi /etc/rsyslog.conf

Uncomment by removing # in front of these module names

module(load="imuxsock") # provides support for local system logging (e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously done by rklogd)

Now, in same file , search for line *.emerg *. Modify the action (i.e *) with :omusrmsg:* . See below given reference

*.emerg                        :omusrmsg:*

Enable UDP port no. 514 for rsyslog .Uncomment below given lines by removing # sign

module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

Now, at the end of file /etc/rsyslog.conf, paste the below given code(These are rsyslog templates)

$template TmplAuth, "/var/log/rsyslog_custom/%HOSTNAME%/%PROGRAMNAME%.log" 
$template TmplMsg, "/var/log/rsyslog_custom/%HOSTNAME%/%PROGRAMNAME%.log" 
authpriv.*   ?TmplAuth
*.info,mail.none,authpriv.none,cron.none   ?TmplMsg

Now , save and exit from file vi /etc/rsyslog.conf
Below given is the reference from our Server,after editing /etc/rsyslog.conf, it looks as given below (verify your file with below given reference)
Here, egrep -v '^#|^$' command will show only uncommented lines from the file.

[root@localhost /]# egrep -v '^#|^$' /etc/rsyslog.conf -v 
module(load="imuxsock") # provides support for local system logging (e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously done by rklogd)
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  /var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
$template TmplAuth, "/var/log/rsyslog_custom/%HOSTNAME%/%PROGRAMNAME%.log"
$template TmplMsg, "/var/log/rsyslog_custom/%HOSTNAME%/%PROGRAMNAME%.log"
authpriv.*   ?TmplAuth
*.info,mail.none,authpriv.none,cron.none   ?TmplMsg
[root@localhost /]# 

Step 4: Now edit the file /etc/sysconfig/rsyslog .And set the SYSLOGD_OPTIONS with no parameter

SYSLOGD_OPTIONS=""

See the below given reference from our Server

[root@localhost ~]# cat /etc/sysconfig/rsyslog
# Options for rsyslogd
# Syslogd options are deprecated since rsyslog v3.
# If you want to use them, switch to compatibility mode 2 by "-c 2"
# See rsyslogd(8) for more details
SYSLOGD_OPTIONS=""
[root@localhost ~]# 

Step 5: Start/Restart the rsyslog service
For starting rsyslog

 /etc/init.d/rsyslog start 

For restarting rsyslog

 /etc/init.d/rsyslog restart 

[for stoping rsyslog, /etc/init.d/rsyslog stop]

Create new directory in /var/log

Create new directory in /var/log called rsyslog_custom. so that we will keep all Servers log in this directory.

mkdir -p /var/log/rsyslog_custom

Set SELINUX rules for rsyslog

Some system admin, disable the selinux .
If you want to keep SELINUX enable. Use the below given command
(Read this post,in case semanage command not found)

semanage fcontext -a -t syslogd_exec_t /sbin/rsyslogd
restorecon /sbin/rsyslogd

/usr/sbin/semanage fcontext -a -t var_log_t "/var/log/rsyslog_custom(/.*)?"
/sbin/restorecon -R -v /var/log/rsyslog_custom

Set IPTABLES for rsyslog

rsyslog service uses the UDP port number 514 .Hence we will set the iptable only for this port
Edit /etc/sysconfig/iptables

vi /etc/sysconfig/iptables

Put the given below RULE always above of any REJECT INPUT rule

-A INPUT -m state --state NEW -m udp -p udp --dport 514 -j ACCEPT

Now, save and exit. Restart the iptables service

/etc/init.d/iptables restart

Use iptables -nL command to check the iptables rule

Restart rsyslog and verify listening status of 514 port

/etc/init.rsyslog restart

Check listening status of port 514

 netstat -uanp|grep rsyslog

See below given reference from my server

[root@localhost ~]# netstat -uanp|grep rsyslog
udp        0      0 0.0.0.0:514                 0.0.0.0:*                               3266/rsyslogd       
udp        0      0 :::514                      :::*                                    3266/rsyslogd       
[root@localhost ~]# 

Configure rsyslog in client servers

To fetch the logs from remote client servers.We will edit the rsyslog.conf file in client machine. The method is applicable to Red Hat based and Debian based Operation system (eg. RHEL,CentOS,Debian,Ubuntu)
Syntax:
*.* @ip-address-of-rsyslog-server:514

Example:
Edit file

vi /etc/rsyslog.conf

Paste the below line (replace 192.168.56.102 with your rsyslog server ip address, here 514 is UDP port no.)

*.* @192.168.56.102:514

Save and exit from file /etc/rsyslog.conf and restart the rsyslog service

/etc/init.d/rsyslog restart

Now relogin in Client system so that we will capture the log.And same log information we will see in rsyslog server

Verify the log in rsyslog server

Change to directory /var/log/rsyslog_custom . You must see,a directory with hostname of your client machine. And inside that directory you will see some logs.

cd /var/log/rsyslog_custom

Reference from my system (Output will be different in your case)

[root@localhost ~]# ls -l /var/log/rsyslog_custom/
total 8
drwx------. 2 root root 4096 Mar  1 07:52 localhost
drwx------. 2 root root 4096 Mar  1 07:47 tuxworld
[root@localhost ~]# 
[root@localhost ~]# ls -l /var/log/rsyslog_custom/tuxworld/
total 32
-rw-------. 1 root root 193 Mar  1 07:46 CRON.log
-rw-------. 1 root root 619 Mar  1 07:47 dbus.log
-rw-------. 1 root root 255 Mar  1 07:47 dhclient.log
-rw-------. 1 root root   0 Mar  1 07:46 kernel.log
-rw-------. 1 root root 659 Mar  1 07:47 NetworkManager.log
-rw-------. 1 root root 120 Mar  1 07:46 rsyslogd-2039.log
-rw-------. 1 root root 149 Mar  1 07:46 rsyslogd.log
-rw-------. 1 root root 296 Mar  1 07:44 sudo.log
-rw-------. 1 root root 316 Mar  1 07:44 su.log
[root@localhost ~]# 

IMPORTANT Note: After setting all things,it is good practice to check /var/log/messages .In case of any rsyslog related error/warning is found
Note: This rsyslog server setup is recommended to use in secure internal network. The rsyslog server should not be publicly exposed.
In case you want to secure the rsyslog server.Use the iptable which only accept logs from your desired particular ip address/ network .
For EXAMPLE Only allow particular ip address to connect with rsyslog server for logging. The client machine has ip address 192.168.56.1 (replace 192.168.56.1 with your client machine ip address [ifconfig])

vi /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m udp -p udp -s 192.168.56.1 --dport 514 -j ACCEPT

Restart the iptable after this /etc/init.d/iptables restart
For EXAMPLE. Only allow particular NETWORK to connect with rsyslog server for logging. The Network subnet is 10.0.0.0/255.255.255.0 (replace 10.0.0.0/255.255.255.0 from your network information )

vi /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m udp -p udp -s 10.0.0.0/24 --dport 514 -j ACCEPT

Restart the iptable /etc/init.d/iptables restart