Skip to content
Snippets Groups Projects
Commit 74d03f97 authored by OMID ROSTAMABADI's avatar OMID ROSTAMABADI
Browse files

Add tests

parent 1ed87c57
Branches main
No related tags found
No related merge requests found
*.swp
## Running Tests
You can run all the test cases by simply invoking the `run-tests.sh` script
Some of the test cases to help you get started are given in `tests/tests/`
directory. It holds the expected return code, standard output, and standard
error in files called `n.rc`, `n.out`, and `n.err` (respectively) for each
test `n`.
The files needed to describe a test number `n` are:
- `n.rc`: The return code the program should return (usually 0 or 1)
- `n.out`: The standard output expected from the test
- `n.err`: The standard error expected from the test
- `n.run`: How to run the test (which arguments it needs, etc.)
- `n.desc`: A short text description of the test
- `n.pre` (optional): Code to run before the test, to set something up
- `n.post` (optional): Code to run after the test, to clean something up
There is also a single file called `pre` which gets run once at the
beginning of testing; this is often used to do a more complex build
of a code base, for example. To prevent repeated time-wasting pre-test
activity, suppress this with the `-s` flag (as described below).
The options for `run-tests.sh` include:
* `-h` (the help message)
* `-v` (verbose: print what each test is doing)
* `-t n` (run only test `n`)
* `-c` (continue even after a test fails)
* `-d` (run tests not from `tests/` directory but from this directory instead)
* `-s` (suppress running the one-time set of commands in `pre` file)
## Adding New Tests
The testing framework just starts at `1` and keeps incrementing
tests until it can't find any more or encounters a failure. Thus, adding new
tests is easy; All files mentioned below should be added in the `tests/tests/`
directory.
1. Add `test_n.c` (this will have the main function which would like to invoke
to test your code)
2. Add `n.rc`, `n.out`, and `n.err` based on intended behaviour
3. Modify `pre` by adding your test in the first command and adding a command to
copy your file (follow what is done for `test_1`)
Now running `run-tests.sh` should call your newly added test too!
You can verify that by passing flag -t with value `n` (n is the number of the
test case you just added).
## Note
There are two additional scripts, `edit-makefile.sh` and `run-xv6-command.exp`.
The second one is an [`expect`](https://en.wikipedia.org/wiki/Expect) script
which launches the qemu emulator and runs the relevant testing command in the
xv6 environment before automatically terminating the test.
You would generally not need to modify these to add your own test cases.
#! /bin/bash
infile=$1
testnames=$2
gawk -vtestnames=$testnames '
($1 == "_mkdir\\") {
n = split(testnames, x, ",");
for (i = 1; i <= n; i++) {
printf("\t_%s\\\n", x[i]);
}
}
{
print $0;
}' $infile
#! /usr/bin/env bash
GREEN='\033[0;32m'
RED='\033[0;31m'
NONE='\033[0m'
# run_test testdir testnumber
run_test () {
local testdir=$1
local testnum=$2
local verbose=$3
# pre: execute this after before the test is done, to set up
local prefile=$testdir/$testnum.pre
if [[ -f $prefile ]]; then
eval $(cat $prefile)
if (( $verbose == 1 )); then
echo -n "pre-test: "
cat $prefile
fi
fi
local testfile=$testdir/$testnum.run
if (( $verbose == 1 )); then
echo -n "test: "
cat $testfile
fi
eval $(cat $testfile) > tests-out/$testnum.out 2> tests-out/$testnum.err
echo $? > tests-out/$testnum.rc
# post: execute this after the test is done, to clean up
local postfile=$testdir/$testnum.post
if [[ -f $postfile ]]; then
eval $(cat $postfile)
if (( $verbose == 1 )); then
echo -n "post-test: "
cat $postfile
fi
fi
return
}
print_error_message () {
local testnum=$1
local contrunning=$2
local filetype=$3
builtin echo -e "test $testnum: ${RED}$testnum.$filetype incorrect${NONE}"
echo " what results should be found in file: $testdir/$testnum.$filetype"
echo " what results produced by your program: tests-out/$testnum.$filetype"
echo " compare the two using diff, cmp, or related tools to debug, e.g.:"
echo " prompt> diff $testdir/$testnum.$filetype tests-out/$testnum.$filetype"
echo " See tests/$testnum.run for what is being run"
if (( $contrunning == 0 )); then
exit 1
fi
}
# check_test testdir testnumber contrunning out/err
check_test () {
local testdir=$1
local testnum=$2
local contrunning=$3
local filetype=$4
# option to use cmp instead?
returnval=$(diff $testdir/$testnum.$filetype tests-out/$testnum.$filetype)
if (( $? == 0 )); then
echo 0
else
echo 1
fi
}
# run_and_check testdir testnumber contrunning verbose printerror
# testnumber: the test to run and check
# printerrer: if 1, print an error if test does not exist
run_and_check () {
local testdir=$1
local testnum=$2
local contrunning=$3
local verbose=$4
local failmode=$5
if [[ ! -f $testdir/$testnum.run ]]; then
if (( $failmode == 1 )); then
echo "test $testnum does not exist" >&2; exit 1
fi
exit 0
fi
if (( $verbose == 1 )); then
echo -n -e "running test $testnum: "
cat $testdir/$testnum.desc
fi
run_test $testdir $testnum $verbose
rccheck=$(check_test $testdir $testnum $contrunning rc)
outcheck=$(check_test $testdir $testnum $contrunning out)
errcheck=$(check_test $testdir $testnum $contrunning err)
othercheck=0
if [[ -f $testdir/$testnum.other ]]; then
othercheck=$(check_test $testdir $testnum $contrunning other)
fi
# echo "results: outcheck:$outcheck errcheck:$errcheck"
if (( $rccheck == 0 )) && (( $outcheck == 0 )) && (( $errcheck == 0 )) && (( $othercheck == 0 )); then
echo -e "test $testnum: ${GREEN}passed${NONE}"
if (( $verbose == 1 )); then
echo ""
fi
else
if (( $rccheck == 1 )); then
print_error_message $testnum $contrunning rc
fi
if (( $outcheck == 1 )); then
print_error_message $testnum $contrunning out
fi
if (( $errcheck == 1 )); then
print_error_message $testnum $contrunning err
fi
if (( $othercheck == 1 )); then
print_error_message $testnum $contrunning other
fi
fi
}
# usage: call when args not parsed, or when help needed
usage () {
echo "usage: run-tests.sh [-h] [-v] [-t test] [-c] [-s] [-d testdir]"
echo " -h help message"
echo " -v verbose"
echo " -t n run only test n"
echo " -c continue even after failure"
echo " -s skip pre-test initialization"
echo " -d testdir run tests from testdir"
return 0
}
#
# main program
#
verbose=0
testdir="tests"
contrunning=0
skippre=0
specific=""
args=`getopt hvsct:d: $*`
if [[ $? != 0 ]]; then
usage; exit 1
fi
set -- $args
for i; do
case "$i" in
-h)
usage
exit 0
shift;;
-v)
verbose=1
shift;;
-c)
contrunning=1
shift;;
-s)
skippre=1
shift;;
-t)
specific=$2
shift
number='^[0-9]+$'
if ! [[ $specific =~ $number ]]; then
usage
echo "-t must be followed by a number" >&2; exit 1
fi
shift;;
-d)
testdir=$2
shift
shift;;
--)
shift; break;;
esac
done
# need a test directory; must be named "tests-out"
if [[ ! -d tests-out ]]; then
mkdir tests-out
fi
# do a one-time setup step
if (( $skippre == 0 )); then
if [[ -f tests/pre ]]; then
echo -e "doing one-time pre-test (use -s to suppress)"
source tests/pre
if (( $? != 0 )); then
echo "pre-test: failed"
exit 1
fi
echo ""
fi
fi
# run just one test
if [[ $specific != "" ]]; then
run_and_check $testdir $specific $contrunning $verbose 1
exit 0
fi
# run all tests
(( testnum = 1 ))
while true; do
run_and_check $testdir $testnum $contrunning $verbose 0
(( testnum = $testnum + 1 ))
done
exit 0
\ No newline at end of file
#! /usr/bin/env expect
proc shutdown {} {
# send command to halt qemu (ctrl-a x)
# https://stackoverflow.com/questions/27050473/how-to-send-ctrl-a-then-d-in-expect
send "\x01"; send "x"
# make sure to wait for it all to stop
# (without this, script was terminating before qemu quit -> bad)
expect eof
}
# turn off timeout (perhaps make this flexible later)
set timeout -1
# clean the solution?
#spawn make -f [lindex $argv 2] clean
# build and launch xv6 on qemu
spawn make [lindex $argv 0] [lindex $argv 1] -f [lindex $argv 2] qemu-nox
trap {
shutdown
exit 0
} SIGINT
# wait for initial prompt
expect "init: starting sh\r"
expect "$ "
# send command as per command line
send "[lindex $argv 3]\r"
# wait for it to be done
expect "$ "
# shutdown qemu properly (avoid runaways)
shutdown
Basic usage of getpinfo - check in-use and tickets
P4_TESTER: TEST PASSED
0
cd ../solution; ../tests/run-xv6-command.exp CPUS=1 SCHEDULER=STRIDE Makefile.test test_1 | grep -E 'P4_TESTER'; cd ../tests
Check if pass increases in proportion to stride
P4_TESTER: TEST PASSED
0
cd ../solution; ../tests/run-xv6-command.exp CPUS=1 SCHEDULER=STRIDE Makefile.test test_2 | grep -E 'P4_TESTER'; cd ../tests
Check if rtime is proportional to # tickets with two processes
P4_TESTER: TEST PASSED
0
cd ../solution; ../tests/run-xv6-command.exp SCHEDULER=STRIDE CPUS=1 Makefile.test test_3 | grep -E 'P4_TESTER'; cd ../tests
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment