We have serveral D&T(Developing and Testing) server, which runs several internal system that generate files and then copy/sync to other server. Because those system were developed by many developers and without good documentation. No one knows exactly how each system works.
One day we want to change the user account that was used to copy files via ssh, we know that the basic commands are scp and rsync. However, no one knows exactly which background job, or web interface, or user scripts may call these commands. How can we find out all those things? A full disk grep may help, but it’s too time-consuming. How about command call history? If we know which command called scp, we can search through the call chain and find the related program/script that need to be changed. But wait, how can we log command call history? There are several known approach:
1. Use auditd
2. Patch kernel with grsecurity
3. Use Bash history
Our running kernel doesn’t support audit, and it is unrealistic to compile the kernel in our situation. The bash command history only includes interactive shell commands, so it’s useless here. At last, we decide to take a “cheap” approach: replace the original scp command with a script, which logs the command call chain and then exec the original scp command with original arguments, this change is transparent to the caller, so it is safe.
Here is how we accomplish our target with the help of trace_call.pl:
#!/bin/bash ORIG_DIR=/usr/local/orig_bin RSYNC=/usr/bin/rsync SCP=/usr/bin/scp TRACE_SCRIPT=$ORIG_DIR/trace_call.pl TRACE_LOG=/tmp/trace_call.log #create directory to hold original program and the trace script mkdir -p $ORIG_DIR cp trace_call.pl $ORIG_DIR #setup log file permission so that everyone can write to it touch $TRACE_LOG && chmod 0666 $TRACE_LOG chmod +x $TRACE_SCRIPT #intercept the original command mv $SCP $ORIG_DIR && ln -s $TRACE_SCRIPT $SCP mv $RSYNC $ORIG_DIR && ln -s $TRACE_SCRIPT $RSYNC
Then we can check /tmp/trace_call.log and see all the command call history.
To stop trace, just move the program from $ORIG_DIR to their original place. eg:
mv /usr/local/orig_bin/scp /usr/bin/scp mv /usr/local/orig_bin/rsync /usr/bin/rsync
Click to get trace_call.pl from my google code project.