Trace command call chain on Linux

We have serveral D&T(Developing and Testing) server, which runs several internal system that generate files and then copy/sync to other server. Because those system were developed by many developers and without good documentation. No one knows exactly how each system works.

One day we want to change the user account that was used to copy files via ssh, we know that the basic commands are scp and rsync. However, no one knows exactly which background job, or web interface, or user scripts may call these commands. How can we find out all those things? A full disk grep may help, but it’s too time-consuming. How about command call history? If we know which command called scp, we can search through the call chain and find the related program/script that need to be changed. But wait, how can we log command call history? There are several known approach:

1. Use auditd
2. Patch kernel with grsecurity
3. Use Bash history

Our running kernel doesn’t support audit, and it is unrealistic to compile the kernel in our situation. The bash command history only includes interactive shell commands, so it’s useless here. At last, we decide to take a “cheap” approach: replace the original scp command with a script, which logs the command call chain and then exec the original scp command with original arguments, this change is transparent to the caller, so it is safe.

Here is how we accomplish our target with the help of trace_call.pl:

start_trace.sh

#!/bin/bash
ORIG_DIR=/usr/local/orig_bin
RSYNC=/usr/bin/rsync
SCP=/usr/bin/scp
TRACE_SCRIPT=$ORIG_DIR/trace_call.pl
TRACE_LOG=/tmp/trace_call.log
#create  directory to hold original program and the trace script
mkdir -p $ORIG_DIR
cp trace_call.pl $ORIG_DIR
#setup log file permission so that everyone can write to it
touch $TRACE_LOG && chmod 0666 $TRACE_LOG
chmod +x $TRACE_SCRIPT
#intercept the original command
mv $SCP $ORIG_DIR && ln -s $TRACE_SCRIPT $SCP
mv $RSYNC $ORIG_DIR && ln -s $TRACE_SCRIPT $RSYNC

Then we can check /tmp/trace_call.log and see all the command call history.

To stop trace, just move the program from $ORIG_DIR to their original place. eg:

mv /usr/local/orig_bin/scp /usr/bin/scp
mv /usr/local/orig_bin/rsync /usr/bin/rsync

Click to get trace_call.pl from my google code project.

This entry was posted in Perl, System Administration and tagged , . Bookmark the permalink.

Leave a Reply