I promised to write this blog post long time ago at one of conferences in Russia. Don't know why I delayed this, but finally I did.
We, members of MySQL bugs verification group, have to verify bugs in all currently supported versions. We use not only version reported, but test in development source tree for each of supported major versions and identify recent regressions.
You can imagine that even if I would do so for simple bug report about wrong results with perfect test case, which requires me simply run few queries I would have to start 4 or more MySQL servers: one for each of currently supported versions 5.0, 5.1, 5.5 plus one for current development. And unknown number of servers if I could not repeat or if I want to check if this is regression.
Even if I have all these basic 4 servers running I still should type all these queries at least 4 times. How much time it would take to verify single bug report if I did so?
I know some members of my group preferred this way, because typing queries manually is same action which our customers do. Again, some bugs are repeatable only if you type queries manually.
But I prefer to test manually erroneous exceptions only and don't make it my routine job.
So how do I test bug reports?
Every version of MySQL server comes with regression test suite: a program, called mtr (mysql-test-run.pl), its libraries, mysqltest program (you should not call it directly, though) and set of tests. Good thing with MySQL test suite is that you can create your own test cases. So do I.
I write my tests in MTR format, then run MTR with record option and examine result. Actually this is kind of hack, because users expected to create result file first, then compare output of running test with that result file. But my purpose is to repeat bug report, not to create proper test case for it, so I can be lazy.
But simply running MTR manually still takes time. And I found a way to automate this process as well.
I created a BASH script, called do_test.sh, which run through all my MySQL trees and runs tests for me automatically, then prints result.
Let me explain it a little bit.
$ cat ~/scripts/do_test.sh
#!/bin/bash
# runs MySQL tests in all source directories
# prints usage information
usage ()
{echo "$VERSION"
echo "
do_test copies MySQL test files from any place
to each of source directory, then runs them
Usage: `basename $0` [option]... [testfile ...]
or `basename $0` [option]... -d dirname [test ...]
or `basename $0` [option]... [-b build [build option]... ]...
Options:
-d --testdir directory, contains test files
I have a directory, there I store test files. It has subdirectory t where tests to be run are stored, subdirectory r, where results, sorted by MySQL server version number, are stored, and directory named archive, there test are stored for archiving purpose.
-s --srcdir directory, contains sources directories
This is path to the directory where MySQL package is located. I called it srcdir, but this is actually not so strict: program will work with binary packages as well.
-b --build mysql source directory
Name of MySQL source directory. You can specify any package name. For example, to run tests in 5.6.9 package in my current dir I call the program as do_test -s . -b mysql-5.6.9-rc-linux-glibc2.5-x86_64
-c --clean remove tests from src directory after execution
-t --suite suite where to put test
MTR can have test suites with their own rules of how to run test case. If you want to run your tests in specific suite, specify this option. You can also have directory for your own suite, but in this case you need to create directories your_suite, your_suite/t and your_suite/r in mysql-test/suite directory of your MySQL installation prior doing this.
As I told I am lazy, so I run tests in main test suite mostly. This can be not good idea if you use MySQL installation not only for tests of its own bugs, but for some other tests.
Rest of the code speaks for itself, so I would not explain it. What you need to do to run this program is simply call it: do_test.sh and pass paths to your test, src dir and MySQL installation.
-v --version print version number, then exit
-h --help print this help, then exit
You can also pass any option to mysqltest program.
"
}
# error exit
error()
{printf "$@">&2
exit $E_CDERROR
}
# creates defaults values
initialize()
{This probably not very obvious. These are my default paths and, most importantly, default set of servers I test
TESTDIR=/home/sveta/src/tests
SRCDIR=/home/sveta/src
BUILDS="mysql-5.0 mysql-5.1 mysql-5.5 mysql-trunk"
CLEAN=0 #false
MYSQLTEST_OPTIONS="--record --force"
TESTS_TO_PASS=""
TESTS=""
SUITE=""
SUITEDIR=""
OLD_PWD=`pwd`
VERSION="do_test v0.2 (May 28 2010)"
}
# parses arguments/sets values to defaults
parse()
{TEMP_BUILDS=""
while getopts "cvhd:s:b:t:" Option
do
case $Option in
c) CLEAN=1;;
v) echo "$VERSION";;
h) usage; exit 0;;
d) TESTDIR="$OPTARG";;
s) SRCDIR="$OPTARG";;
b) TEMP_BUILDS="$TEMP_BUILDS $OPTARG";;
t) SUITE="$OPTARG"; SUITEDIR="/suite/$SUITE"; MYSQLTEST_OPTIONS="$MYSQLTEST_OPTIONS --suite=$SUITE";;
*) usage; exit 0; ;;
esac
done
if [[ $TEMP_BUILDS ]]
then
BUILDS="$TEMP_BUILDS"
fi
}
# copies test to source directories
copy()
{cd "$TESTDIR/t"
TESTS_TO_PASS=`ls *.test 2>/dev/null | sed s/.test$//`
cd $OLD_PWD
for build in $BUILDS
do
#cp -i for reject silent overload
cp "$TESTDIR"/t/*.{test,opt,init,sql,cnf} "$SRCDIR/$build/mysql-test$SUITEDIR/t" 2>/dev/nulldone
}
# runs tests
run()
{for build in $BUILDS
do
cd "$SRCDIR/$build/mysql-test"
./mysql-test-run.pl $MYSQLTEST_OPTIONS $TESTS_TO_PASS
done
cd $OLD_PWD
}
# copies result and log files to the main directory
get_result()
{for build in $BUILDS
do
ls "$TESTDIR/r/$build" 2>/dev/null
if [[ 0 -ne $? ]]
then
mkdir "$TESTDIR/r/$build"
fi
for test in $TESTS_TO_PASS
do
cp "$SRCDIR/$build/mysql-test$SUITEDIR/r/$test".{log,result} "$TESTDIR/r/$build" 2>/dev/nulldone
done
}
# removes tests and results from MySQL sources directories
cleanup()
{if [[ 1 -eq $CLEAN ]]
then
for build in $BUILDS
do
for test in $TESTS_TO_PASS
do
rm "$SRCDIR/$build/mysql-test$SUITEDIR/r/$test".{log,result} 2>/dev/nullrm "$SRCDIR/$build/mysql-test$SUITEDIR/t/$test.test"
done
done
fi
}
# shows results
show()
{for build in $BUILDS
do
echo "=====$build====="
for test in $TESTS_TO_PASS
do
echo "=====$test====="
cat "$TESTDIR/r/$build/$test".{log,result} 2>/dev/nullecho
done
echo
done
}
E_CDERROR=65
#usage
initialize
parse $@
copy
run
get_result
cleanup
show
exit 0
After I finished with test I copy it to archive directory, again, with a script, named ar_test.sh:
$ cat ~/scripts/ar_test.sh
#!/bin/bash
# moves MySQL tests from t to archive directory and clean ups r directories
# prints usage information
usage ()
{echo "$VERSION"
echo "
ar_test copies MySQL test files from t to archive folder
Usage: `basename $0` [-v] [-d dirname] [test ...]
Options:
-d directory, contains test files
-v print version
-h print this help
"
}
# error exit
error()
{printf "$@">&2
exit $E_CDERROR
}
# creates defaults values
initialize()
{TESTDIR=/home/sveta/src/tests
TESTS_TO_MOVE=""
OLD_PWD=`pwd`
VERSION="ar_test v0.2 (Dec 01 2011)"
}
# parses arguments/sets values to defaults
parse()
{ while getopts "vhd:" Option
do
case $Option in
v) echo "$VERSION"; shift;;
h) usage; exit 0;;
d) TESTDIR="$OPTARG"; shift;;
*) usage; exit 0;;
esac
done
TESTS_TO_MOVE="$@"
}
# copies test to source directories
copy()
{if [[ "xx" = x"$TESTS_TO_MOVE"x ]]
then
cp "$TESTDIR"/t/* "$TESTDIR"/archive 2>/dev/null
else
for test in $TESTS_TO_MOVE
do
cp "$TESTDIR/t/$test".{test,opt,init,sql} "$TESTDIR"/archive 2>/dev/nulldone
fi
}
# removes tests and results from r directories
cleanup()
{if [[ "xx" = x"$TESTS_TO_MOVE"x ]]
then
rm "$TESTDIR"/t/* 2>/dev/null
rm "$TESTDIR"/r/*/* 2>/dev/null
else
for test in $TESTS_TO_MOVE
do
rm "$TESTDIR/t/$test".{test,opt,init,sql} 2>/dev/null rm "$TESTDIR/r/"*"/$test".{test,opt,init,sql} 2>/dev/nulldone
fi
}
E_CDERROR=65
initialize
parse $@
copy
cleanup
exit 0
But most important part: what to do if I want to test on some specific machine which is not available at home? Fortunately, we have shared machines to run tests on, so I can simply move them to my network homedir, then choose appropriate machine and run. Since this is BASH script and test cases in MTR format this would work on any operating system.
$ cat ~/scripts/scp_test.sh
#!/bin/bash
# copies MySQL tests to remote box
# prints usage information
usage ()
{echo "$VERSION"
echo "
scp_test copies MySQL test files from t directory on local box to MySQL's XXX
Usage: `basename $0` [-v] [-d dirname] [-r user@host:path] [test ...]
Options:
-d directory, contains test files
-r path to test directory on remote server, default: USERNAME@MACHINE_ADDRESS:~/PATH/src/tests/t
-v print version
-h print this help
"
}
# error exit
error()
{printf "$@">&2
exit $E_CDERROR
}
# creates defaults values
initialize()
{TESTDIR=/home/sveta/src/tests
MOVETO='USERNAME@MACHINE_ADDRESS:~/PATH/src/tests/t'
TESTS_TO_MOVE=""
OLD_PWD=`pwd`
VERSION="scp_test v0.2 (Dec 1 2011)"
}
# parses arguments/sets values to defaults
parse()
{ while getopts "vhd:" Option
do
case $Option in
v) echo "$VERSION"; shift;;
h) usage; exit 0;;
d) TESTDIR="$OPTARG"; shift;;
r) MOVETO="$OPTARG"; shift;;
*) usage; exit 0;;
esac
done
TESTS_TO_MOVE="$@"
}
# copies test to source directories
copy()
{if [[ "xx" = x"$TESTS_TO_MOVE"x ]]
then
scp "$TESTDIR"/t/* "$MOVETO"
else
for test in $TESTS_TO_MOVE
do
scp "$TESTDIR/t/$test".{test,opt,init,sql} "$MOVETO"done
fi
}
E_CDERROR=65
initialize
parse $@
copy
exit 0
Wanted to put them to Launchpad, but stack with name for this package. Does anybody have ideas?