After master created some tasks on a shared filesystem, slave must take and execute them. * slave/bin/task-taker is used for that task: $ [ -n "$BASS_ROOT" ] || BASS_ROOT=/path/to/bass BASS_RC=/path/to/rc $ $BASS_ROOT/slave/bin/task-taker It is also expected to be run under some kind of supervisor. It saves lastnum file in current directory with the latest task's counter value it processed. You may run multiple task-takers to run jobs in parallel. * task-taker runs slave/bin/job-starter on a taken task. * Initially job-starter takes the task and checks does it have appropriate architecture and probably optional hostname set. It exits successfully if task is not for us. Another slave will take it instead. * Then it creates the job $JOBS/cur/$task state. Various metainformation is filled in it, like path to temporary directory, hostname and so on. build/bin/mk-skelenv creates the [Build/skelenv] in that temporary directory and installs slave-base package, that depends on various utilities needed for running the testing steps. * tmux executable file is created in that directory, which you can use to attach to the job's tmux instance. * code.tar and steps.tar are unpacked to that directory under code/ and steps/ paths. * Background heartbeat process is started, that touches $job/alive file every second. * Then the tmux is started in steps/ directory and runs slave/bin/steps-runner. * If steps-runner succeeds, then temporary directory is removed. Otherwise tmux is left running and waiting for someone to attach it and press Enter. If no input happens for $FAILED_JOB_WAITTIME seconds (one hour be default), it exits and removes the temporary directory. What exactly does steps-runner? * For each step, sorted lexicographically, it creates corresponding output directory in job's directory with started, stdout.txt, stderr.txt files. * Step is run with its stdout/stderr redirected through tai64n utility, prepending the timestamps for each output line. Its exitcode it saved in exitcode.txt. It is always run in the code/ directory. * Background process is also started to look for step's output progress. Every second it checks if any stdout/stderr output happened for for the last $LINE_TIMEOUT (ten minutes by default) seconds. If step is stuck (no output for that time), then it is killed and "timeout" is written to exitcode.txt.