Bash quote trap: why rsync exclude pattern not work

A friend wanted to sync all his files to a remote host, exluding the source files(those with suffix .c, .cpp, .h, .hpp). First I wrote this script

#!/bin/bash
####sync.sh (version 1): sync files to remote host###
SYNC_DST='xxx@example.com::sample_project'
SYNC_OPT=' --exclude="*.[ch]" --exclude="*.[ch]pp" '
if [[ $# -eq 0 ]]; then
        echo "Usage $0 /src/path"
        exit 1
fi
src=$1
#for debug ...
echo rsync -avz $SYNC_OPT "$src" "$SYNC_DST"
rsync -avz $SYNC_OPT "$src" "$SYNC_DST"

Now we try to sync the ‘src’ directory:

./sync.sh src
####output########################################
rsync -avz --exclude="*.[ch]" --exclude="*.[ch]pp" src xxx@example.com::sample_project
sending incremental file list
src/
src/main
src/main.c
src/main.o
src/lib/
src/lib/lib.c
src/lib/lib.h
src/lib/lib.o
####output########################################

We can see that the source files are not excluded,we can also see the full sync command. After removing all the files in the remote host, I copy and run the sync command manually:

rsync -avz --exclude="*.[ch]" --exclude="*.[ch]pp" src xxx@example.com::sample_project
sending incremental file list
src/
src/main
src/main.o
src/lib/
src/lib/lib.o

It works great! But wait, why, why, the same command was used in sync.sh, why didn’t it work as expected?
After some painful test and failure, I changed the line from

SYNC_OPT=' --exclude="*.[ch]" --exclude="*.[ch]pp" '

to

SYNC_OPT=' --exclude=*.[ch] --exclude=*.[ch]pp '

just remove the double quote in $SYNC_OPT variable and everything works as expected. Then I realize that it’s the quote in variable that caused rsync pattern to fail. Because in Bash, quotes have the quoting function only when they appears literally, otherwise, they are just quote character, nothing special.
Let’s make it clear with some example.

#!/bin/bash
###show_arg.sh: used to show arguments provided to this script##
for arg in "$@"; do
   echo "==$arg=="
done

Let’s see how the show_arg.sh works

./show_arg.sh arg1 "arg 2" arg3
==arg1==
==arg 2==
==arg3==

We can see the that the show_arg.sh will print out each argument provided to it, one argument per line, surrounded by two “==”.
replace rsync with show_arg.sh in sync.sh

#!/bin/bash
####sync.sh (version 2): show the real arguments###
SYNC_DST='xxx@example.com::sample_project'
SYNC_OPT=' --exclude="*.[ch]" --exclude="*.[ch]pp" '
if [[ $# -eq 0 ]]; then
        echo "Usage $0 /src/path"
        exit 1
fi
src=$1
./show_arg.sh -avz $SYNC_OPT "$src" "$SYNC_DST"

Test it again:

./sync.sh src

Output:

rsync -avz --exclude="*.[ch]" --exclude="*.[ch]pp" src xxx@example.com::sample_project
==-avz==
==--exclude="*.[ch]"==
==--exclude="*.[ch]pp"==
==src==
==xxx@example.com::sample_project==

It’s now obvious that the quotes are passed to rsync command literally, which definitely cause the pattern to fail. You may be happy and think that remove all the useless quotes will solve all problem. However, what if we really need the quotes?

For example, if we want to exclude files named “test a”(test space a, without quotes), we can’t write it like this:

SYNC_OPT=' --exclude=test a'

this will only exclude files named test, and takes a as file to sync. We must write it like this:

SYNC_OPT=' --exclude="test a"'

We know for now that with sync.sh version 1, it will definitely fail because of the quotes problem. Fortunately, Bash has a valuable eval command, it will process the quotes in variable as you would normally expect.

#!/bin/bash
####sync.sh (version 3): sync files, exclude file name with space###
SYNC_DST='dst'
SYNC_OPT=' --exclude="*.[ch]" --exclude="*.[ch]pp" --exclude="test a"'
if [[ $# -eq 0 ]]; then
        echo "Usage $0 /src/path"
        exit 1
fi
src=$1
#rsync -avz $SYNC_OPT "$src" "$SYNC_DST"
eval rsync -avz $SYNC_OPT "$src" "$SYNC_DST"

Test result
rsync exlude pattern works when use the eval command

Conclusion: In Bash shell, quotes have the quoting capability only when they appears literally, if they appear in variable, they are just the the quote character, nothing special. If you want the quotes in variable to have quoting function, eval them.

This entry was posted in Bash, Programming, System Administration and tagged , , , . Bookmark the permalink.

2 Responses to Bash quote trap: why rsync exclude pattern not work

  1. alnav says:

    Thanks, man!

  2. Alex says:

    Thanks a lot.

Leave a Reply