====== Howto debug AIX commands ======

===== Debug a process =====

==== Use the trace command ====

Give an output very low level, hard to interpret. To debug into a WPAR, you have to start the trace from Global, with option -@:
<cli prompt='#'>
# trace -a -o /tmp/my_trace_log -@sefasblludev ; clogin sefasblludev enable PHO302C-PS  ; trcstop
# trcrpt /tmp/my_trace_log > /tmp/my_trace_log.txt
</cli>

==== Use the truss command ====

More friendly: truss <options> <command to trace>

Debug with detailed info
<cli prompt='#'>
# truss -adeflo /tmp/dcons_truss.out dconsole -t -n systemr
</cli>

<cli prompt='#'>
# truss -f -t !close -o /tmp/truss.out /usr/sbin/sshd -p 4444 -d 
</cli>
    * -t !close : suppress all close system call (many unsusable line into the log file).
    * -f : trace also children processes from main process.
    * -o /tmp/truss.out :output file, directly readable.

Basically trace a process ID:
<cli prompt='#'>
# truss -p 348468
</cli>

===== Trace applications =====

==== Debug PAM ====

To enable PAM debug output, complete the following steps:

Create an empty file named pam_debug in the **/etc/pam_debug** directory by using the touch command, if the file does not exist. The PAM library checks for the /etc/pam_debug file and enables syslog output if it is found.

Edit the /etc/syslog.conf file to identify a file where it will log the auth syslog messages at the priority level you want. For example, to send PAM debug-level messages to the /var/log/auth.log file, add the following text as a new line in the syslog.conf file:
<code>
*.debug /var/log/auth.log
</code>

Create the output file /var/log/auth.log, by using the touch command, if it does not exist.

To restart the syslogd daemon so that configuration changes are recognized, complete the following steps:
<cli prompt='#'>
# stopsrc -s syslogd
# startsrc -s syslogd
</cli>

==== Debug SSH connection ====

To start a ssh server in debug mode (will stop after first connection)
<cli prompt='#'>
# stoprs -s sshd
# /usr/sbin/sshd -p 22 -d
</cli>

Start the client in debug mode
<cli prompt='#'>
# ssh -vvv user@server1
</cli>

==== Debug LDAP client ====

Add rotation for the debug file
<cli prompt='#'>
# cat /etc/logrotate.d/system.conf
/var/log/syslog/ldap.debug {
    copytruncate
    rotate 3
    size=100M
}
</cli>

Create the debug file
<cli prompt='#'>
# touch /var/log/syslog/ldap.debug
</cli>

Restart the LDAP daemon in debug mode
<cli prompt='#'>
[root@aix01]/root# export LDAP_DEBUG=65535
[root@aix01]/root# export LDAP_DEBUG_FILE=/var/log/syslog/ldap.debug
[root@aix01]/root# stop-secldapclntd
The secldapclntd daemon is successfully terminated.
[root@aix01]/root# start-secldapclntd
Starting the secldapclntd daemon.
The secldapclntd daemon started successfully.
</cli>

==== Debug DNS ====

For info, only the 3 first DNS are answered, even if more are availble int /etc/resolv.conf
<cli prompt='#'>
[root@labosrv]/root# RES_OPTIONS=debug host myhost01
;; res_setoptions("debug", "env")..
;;      debug
;;      calling process id = 20840598
;; res_nquerydomain(myhost01, test.lu, 1, 1)
;; res_query(myhost01.test.lu, 1, 1)
;; res_nmkquery(QUERY, myhost01.test.lu, IN, A)
;; res_send()
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7136
;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;;      myhost01.test.lu, type = A, class = IN
;; Querying server (# 1) address = 10.10.10.100
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7136
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1
;;      myhost01.test.lu, type = A, class = IN
myhost01.test.lu.    1M IN A         172.19.28.37
test.lu.                  1M IN NS        dns.test.lu.
dns.test.lu.             1M IN A         10.10.10.100
myhost01.test.lu is 10.10.10.37
</cli>

===== Debug LPAR startup =====

First connect to the HMC in SSH using putty, and enable the --> session --> logging

Then open a console on the LPAR that doesn't boot (mkvterm, or vtmenu), and start the LPAR from web interface in advanced mode
  Back in the SSH console session window, wait for the Open Firmware prompt "0>"
  At the 0> prompt, enter "boot -s verbose"

  For cdrom boot debug enter:
    0> boot cdrom:\ppc\chrp\bootfile.exe -s verbose

At this point, the LPAR will continue to boot and debug information will be sent to the console. While the LPAR is booted in this debug state, all commands that are run will output debug information, such as exec() system calls.