User Tools

Site Tools


aix:aix_rsyslog

RSYSLOG for AIX

Rsyslog is able to send and receive messages using UDP and TCP, on customized ports.

Basic setup on AIX

Rsyslog bff package is available by IBM:

https://www-01.ibm.com/marketing/iwm/iwm/web/dispatcher.do?source=aixbp

After having installed rsyslogd, we have a new command, which is helpful to convert from the normal syslogd version :

syslog_ssw

First convert /etc/syslog.conf to /etc/rsyslog.conf

[root@prnim01]/etc> syslog_ssw -c /etc/syslog.conf /etc/rsyslog.conf
[root@prnim01]/etc> cat /etc/rsyslog.conf
################################################################
# Rsyslog is free software

# Load the unix socket for local communication
$ModLoad imuxsock

# Load the UDP module for remote communication and Run the UDP server on the default port 514
$ModLoad imudp
$UDPServerRun 514

# Load the TCP module for remote communication and Run the TCP server on the default port 514
$ModLoad imtcp
$InputTCPServerRun 514
################################################################

aso.notice /var/log/aso/aso.log
aso.info /var/log/aso/aso_process.log
aso.debug /var/log/aso/aso_debug.log
*.emerg /var/log/syslog/emerg.log
*.alert /var/log/syslog/alert.log
*.crit /var/log/syslog/crit.log
*.err /var/log/syslog/error.log
*.warning /var/log/syslog/warning.log
*.notice /var/log/syslog/notice.log
*.info /var/log/syslog/info.log
*.debug /var/log/syslog/debug.log
caa.debug /var/adm/ras/syslog.caa
auth.debug /var/log/syslog/auth.log

Then, to replace syslog with rsyslog, you can use the command syslog_ssw with the -r option.

[root@prnim01]/var/log/syslog# syslog_ssw -r 
0513-077 Subsystem has been changed.
Start daemon: syslogd
0513-059 The syslogd Subsystem has been started. Subsystem PID is 37027944.
[root@prnim01]/var/log/syslog# ps -ef | grep syslog
root 26869770 4128770 0 04:19:37 - 0:00 /usr/sbin/rsyslogd

Which syslogd is in use:

[root@prnim01]/var/log/syslog# odmget -q "subsysname = 'syslogd'" SRCsubsys
SRCsubsys:
subsysname = "syslogd"
synonym = ""
cmdargs = ""
path = "/usr/sbin/rsyslogd" <<<< rsyslogd is enabled

Now test, you can see the date format has changed:

[root@prnim01]/var/log/syslog# logger -p notice "this is a RSYSLOG test"

[root@prnim01]/var/log/syslog# tail -20 notice.log
Sep 29 11:13:46 prnim01 daemon:notice bootpd[8061212]: hardware address not found: 5CF3FC9F7580
Sep 29 11:13:54 prnim01 syslog:err|error syslogd: Good Bye

2017-09-29T11:14:06.067495+02:00 prnim01 bootpd[8061212]: hardware address not found: 5CF3FC9F7580
2017-09-29T11:14:22.070448+02:00 prnim01 bootpd[8061212]: hardware address not found: 5CF3FC9F7580
2017-09-29T11:14:29.506642+02:00 prnim01 root: this is a RSYSLOG test 
2017-09-29T11:14:38.073427+02:00 prnim01 bootpd[8061212]: hardware address not found: 5CF3FC9F7580

Now rsyslog can be start and stop using standard AIX service commands:

startsrc -s rsyslogd
stopsrc -s rsyslogd

Example:

[root@prnim01]/var/log/syslog# startsrc -s syslogd
 0513-059 The syslogd Subsystem has been started. Subsystem PID is 18809342.
[root@prnim01]/var/log/syslog# ps -ef | grep sysl
 root 18809342  3801372   0 11:22:00      -  0:00 /usr/sbin/rsyslogd

To switch back to standard AIX syslog:

syslog_ssw -r

Notice
RSYSLOG has no rotation mechanism, you have to find your own solution, like logrotate.

Rotate logs

Do not forget to rotate logs

In the default rsyslog logrotate /etc/logrotate.d/rsyslog config I see the following:

/var/log/syslog/warning.log
{
        rotate 7
        daily
        missingok
        notifempty
        delaycompress
        compress
        postrotate
                reload rsyslog >/dev/null 2>&1 || true
        endscript
}

Copytruncate is also an option instead reloading rsyslog, but you could lose some entries

Example in Linux

manu-opensuse:/etc/logrotate.d # cat syslog 
/var/log/warn /var/log/messages /var/log/allmessages /var/log/localmessages
/var/log/firewall /var/log/acpid /var/log/NetworkManager
/var/log/mail /var/log/mail.info /var/log/mail.warn /var/log/mail.err
/var/log/news/news.crit /var/log/news/news.err /var/log/news/news.notice
{
    compress
    dateext
    maxage 365
    rotate 99
    missingok
    notifempty
    size +4096k
    create 640 root root
    sharedscripts
    postrotate
        /usr/bin/systemctl reload syslog.service > /dev/null
    endscript
}

Advanced setup

Debug output sent to central server:

[root@aix001]/etc#  tcpdump  -i en0  host 10.10.10.10 -v

        Msg: May 18 16:15:56 aix001 "AIX_AUDIT" 18 May 2022 16:15:46.652129  FS_Chdir        rsyslogd                        root     root     OK          34931838 4260108
16:15:56.664128 IP (tos 0x0, ttl 30, id 54514, offset 0, flags [none], proto UDP (17), length 117)
    aix001.38075 > syslogsrv01.syslog: SYSLOG, length: 89
        Facility local7 (23), Severity debug (7)
...

To revert back to old date format, add this option in /etc/rsyslog.conf

Sep 29 15:04:55 prnim01 root: this is a RSYSLOG test 
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

Custom Time Format:

2013-11-30 13:39:59 localhost sshd[17331]: Accepted publickey for kaz from 192.1...
$template CustomFormat,"%timestamp:::date-year%-%timestamp:::date-month%-%timestamp:::date-day% %timestamp:::date-hour%:%timestamp:::date-minute%:%timestamp:::date-second% %HOSTNAME% %syslogtag%%msg%0\n"
$ActionFileDefaultTemplate CustomFormat

Send info messages to central server using TCP port 222:

info.* @@syslogserver:222

Send info messages to central server using UDP port 222:

info.* @syslogserver:222

This will be the server that will receive all the syslog messages. you can add some parameters, according to rsyslog wiki. But a good starting point is the following options (you have to create /var/rsyslog before launching rsyslog) :

$WorkDirectory /var/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1G # 1Gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
*.* @@ip_central_server:port

Advanced config using Rulesets

################
# Modules
################
$ModLoad imtcp
$ModLoad imudp
$ModLoad imuxsock

################
# Templates
################
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# log every host in its own directory
$template RemoteHost,"/var/syslog/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/syslog.log"

################
### Rulesets
################

################
# Local Logging
$RuleSet local
aso.notice /var/log/aso/aso.log
aso.info /var/log/aso/aso_process.log
aso.debug /var/log/aso/aso_debug.log
*.emerg /var/log/syslog/emerg.log
*.alert /var/log/syslog/alert.log
*.crit /var/log/syslog/crit.log
*.err /var/log/syslog/error.log
*.warning /var/log/syslog/warning.log
*.notice /var/log/syslog/notice.log
*.info /var/log/syslog/info.log
*.debug /var/log/syslog/debug.log
caa.debug /var/adm/ras/syslog.caa
auth.debug /var/log/syslog/auth.log
# use the local RuleSet as default if not specified otherwise
$DefaultRuleset local


################
# Remote Logging
$RuleSet remote
*.* ?RemoteHost
# Send messages we receive to another syslog server using TCP port 514
*.* @@W.X.Y.Z:514
# Send messages we receive to another syslog server using UDP port 514
*.notice @10.10.10.10
auth,authpriv.info @10.20.20.20:1234


################
### Listeners
################

# bind ruleset to tcp listener, must be at the end
$InputTCPServerBindRuleset remote
$InputTCPServerRun 514

$InputUDPServerBindRuleset remote
$UDPServerRun 514

debugging :

To debug rsyslog is quite simple :

stop daemon via stopsrc command. export the two RSYSLOG parameters :

export RSYSLOG_DEBUG="DebugOnDemand NoStdOut"
export RSYSLOG_DEBUGLOG=/tmp/rsyslog_debug.out

then launch rsyslog on the command line as follow :

# rsyslogd  -n -c5

then, send the usr signal via the kill command on the running process

# kill -USR1 syslogpid

you will then, have all the debug info into the indicated file :

root/datestaix7 / #cat /tmp/rsyslog_debug.out
7600.312486000:1:
7600.312524000:1:
7600.312537000:1: ********************************************************************************
7600.312550000:1: Switching debugging_on to true at 14:46:40
7600.312561000:1: ********************************************************************************
7635.540547000:203: Message from UNIX socket: #4
7635.540627000:203: MsgSetTAG in: len 3, pszBuf: ege
7635.540661000:203: MsgSetTAG exit: pMsg->iLenTAG 3, pMsg->TAG.szBuf: ege
7635.540709000:203: main Q: entry added, size now log 2, phys 3 entries
7635.540759000:203: main Q: EnqueueMsg advised worker start
7635.540821000:203: --------imuxsock calling select, active file descriptors (max 4): 4
7647.435687000:102: file netstrms.c released module 'lmnsd_ptcp', reference count now 1
7647.435722000:102: Action 20059608 transitioned to state: rtry

http://gileb-aix.blogspot.fr/2013/03/rsyslogd-on-aix.html

Redirect output of specific file

#/etc/rsyslog.conf
$ModLoad imfile
$InputFileName /data/mysql/error.log
$InputFileTag mysql-error
$InputFileStateFile stat-mysql-error
$InputFileSeverity error
$InputFileFacility local3
$InputRunFileMonitor
local3.* @@hostname:<portnumber>

# Set the default permissions for all log files.
#
#$FileOwner syslog
#$FileGroup adm
#$FileCreateMode 0640
#$DirCreateMode 0755
#$Umask 0022
#$PrivDropToUser syslog
#$PrivDropToGroup syslog

centralize login

If you switch from auth_type = STD_AUTH to auth_type = PAM_AUTH in /etc/security/login.cfg

You can enable pam debug by creating an empty file: /etc/pam_debug

Now you are able to collect to syslog all login informations:

  • .debug /var/log/auth.log

Custom message format

Template names beginning with “RSYSLOG_” are reserved for rsyslog use. Do NOT use them if, otherwise you may receive a conflict in the future (and quite unpredictable behaviour). There is a small set of pre-defined templates that you can use without the need to define it:

  • RSYSLOG_TraditionalFileFormat - the “old style” default log file format with low-precision timestamps
  • RSYSLOG_FileFormat - a modern-style logfile format similar to TraditionalFileFormat, buth with high-precision timestamps and timezone information
  • RSYSLOG_TraditionalForwardFormat - the traditional forwarding format with low-precision timestamps. Most useful if you send messages to other syslogd's or rsyslogd below version 3.12.5.
  • RSYSLOG_ForwardFormat - a new high-precision forwarding format very similar to the traditional one, but with high-precision timestamps and timezone information. Recommended to be used when sending messages to rsyslog 3.12.5 or above.
  • RSYSLOG_SyslogProtocol23Format - the format specified in IETF's internet-draft ietf-syslog-protocol-23, which is assumed to be come the new syslog standard RFC. This format includes several improvements. The rsyslog message parser understands this format, so you can use it together with all relatively recent versions of rsyslog. Other syslogd's may get hopelessly confused if receiving that format, so check before you use it. Note that the format is unlikely to change when the final RFC comes out, but this may happen.
  • RSYSLOG_DebugFormat - a special format used for troubleshooting property problems. This format is meant to be written to a log file. Do not use for production or remote forwarding.

String-based Template Samples This section provides some sample of what the default formats would look as a text-based template. Hopefully, their description is self-explanatory. Note that each $Template statement is on a single line, but probably broken accross several lines for display purposes by your browsers. Lines are separated by empty lines.

A template that resembles traditional syslogd file output:
    $template TraditionalFormat,”%timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n”
    
A template that tells you a little more about the message:
    $template precise,”%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%, %syslogtag%,%msg%\n”

A template for RFC 3164 format:
    $template RFC3164fmt,”<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%”

A template for the format traditionally used for user messages:
    $template usermsg,” XXXX%syslogtag%%msg%\n\r”

And a template with the traditional wall-message format:
    $template wallmsg,”\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated%

A template that can be used for the database write (please note the SQL template option)
    $template MySQLInsert,”insert iut, message, received at values (‘%iut%’, ‘%msg:::UPPERCASE%’, ‘%timegenerated:::date-mysql%’) into systemevents\r\n”, SQL

The following template emulates WinSyslog format:
    $template WinSyslogFmt,"%HOSTNAME%,%timegenerated:1:10:date-rfc3339%,
%timegenerated:12:19:date-rfc3339%,%timegenerated:1:10:date-rfc3339%,
%timegenerated:12:19:date-rfc3339%,%syslogfacility%,%syslogpriority%,
%syslogtag%%msg%\\n"
   
$template FileFormat,"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"

$template ForwardFormat,"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"

$template TraditionalForwardFormat,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%"

$template StdSQLFormat,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL

To assign a msg to a format:

kern.crit @server.example.net;RFC3164fmt

Example

[root@aixclt01]/var/log/syslog> cat /etc/rsyslog.conf
################ Order of the lines is important ################# 
# Load the unix socket for local communication                   #
  $ModLoad imuxsock                                              #
# Load the UDP module for remote communication                   #
  $ModLoad imudp                                                 #
# Run the UDP server on the default port 514                     #
  $UDPServerRun 514                                              #
# Load module to read file
  $ModLoad imfile                                                                #
#################################################################

$WorkDirectory /var/log/syslog/

################
# Local Logging1
aso.notice /var/log/aso/aso.log
aso.info /var/log/aso/aso_process.log
aso.debug /var/log/aso/aso_debug.log
caa.debug /var/adm/ras/syslog.caa
*.emerg /var/log/syslog/emerg.log
*.alert /var/log/syslog/alert.log
*.crit /var/log/syslog/crit.log
*.err /var/log/syslog/error.log
*.warning /var/log/syslog/warning.log
*.notice /var/log/syslog/notice.log
*.info /var/log/syslog/info.log
auth,authpriv.debug /var/log/syslog/auth.log
daemon.* /dev/null
lpr.* /dev/null

################
# Remote Logging

# Read a specific file
$InputFileName /audit/stream.out
$InputFileTag "AUDIT_AIX"
$InputFileStateFile audit-log
$InputFileSeverity debug
$InputFileFacility local7
$InputRunFileMonitor

local7.debug,*.warning,daemon.info,auth,authpriv.debug,*.emerg @10.10.10.10:514

#PollingInterval 10

################
# Local Logging2
# Exclude tags from local saving 
:syslogtag, contains, "AUDIT_AIX" ~
*.debug /var/log/syslog/debug.log

Misc examples

The fourth line tells rsyslogd to save all kernel messages that come with priorities from info up to warning in the file /var/adm/kernel-info. Everything from err and higher is excluded.

# The tcp wrapper loggs with mail.info, we display all the connections on tty12
mail.=info /dev/tty12

This directs all messages that uses mail.info (in source LOG_MAIL | LOG_INFO) to /dev/tty12, the 12th console. For example the tcpwrapper tcpd(8) uses this as it’s default.

# Store all mail concerning stuff in a file
mail.\*;mail.!=info /var/adm/mail

This will store all messages with the priority crit in the file /var/adm/critical, except for any kernel message.

# A template for RFC 3164 format:
$template RFC3164fmt,”<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%”
# Kernel messages are first, stored in the kernel
# file, critical messages and higher ones also go
# to another host and to the console. Messages to
# the host server.example.net are forwarded in RFC 3164
# format (using the template defined above).
kern.* /var/adm/kernel
kern.crit @server.example.net;RFC3164fmt
kern.crit /dev/console
kern.info;kern.!err /var/adm/kernel-info
aix/aix_rsyslog.txt · Last modified: 2022/11/21 17:31 by manu