Master node(s) are intended to create tasks. As a rule they are created
as an event on someone pushing the commit. There are no specialised
committed daemons running on them, because each project's task making
process can vastly differ in details. Only atomic counter utilities are
commonly used by "task makers" as a rule.
Let's see how example example/goredo CI pipeline is created.
We want to run the tests when someone pushes the commit.
* Prepare necessary directories at the very beginning:
mkdir -p /nfs/revs/goredo
mkdir -p /nfs/tasks/ctr/0
mkdir -p /nfs/tasks/{cur,old,tmp}
mkdir /nfs/jobs
* First thing to do is to create Git's post-receive hook, that will
touch files with the revision needed to be tested.
$ cat >goredo.git/hooks/post-receive <<EOF
#!/bin/sh -e
REVS=/nfs/revs/goredo
ZERO="0000000000000000000000000000000000000000"
read prev curr ref
[ "$curr" != $ZERO ] || exit 0
[ "$prev" != $ZERO ] || prev=$curr^
git rev-list $prev..$curr | while read rev ; do
mkdir -p $REVS/$ref
echo BASSing $ref/$rev... >&2
touch $REVS/$ref/$rev
done
EOF
After pushing a bunch of commits, corresponding empty files in
revisions directory will be created. Each filename is a commit's
hash. Those are basically a notification events about the need to
create corresponding tasks.
* Someone has to process those events. Each project has its own
task-maker, because there are so many variations how code and build
steps can be retrieved and created. Let's create one:
#!/bin/sh -e
[ -n "$BASS_ROOT" ]
sname="$0" . $BASS_ROOT/lib/rc
[ -n "$REVS" ] || {
echo '"REVS"' is not set >&2
exit 1
}
[ -n "$PROJ" ] || {
echo '"PROJ"' is not set >&2
exit 1
}
[ -n "$STEPS" ] || {
echo '"STEPS"' is not set >&2
exit 1
}
[ -n "$ARCHS" ] || {
echo '"ARCHS"' is not set >&2
exit 1
}
cd $REVS
rev=$(find . -type f | sed -n 1p)
[ -n "$rev" ]
rev_path=$(realpath $rev)
rev=$(basename $rev)
task_proj=goredo
task_version=$(cd $PROJ ; $BASS_ROOT/master/bin/version-for-git $rev)
[ -n "$task_version" ]
task=":$task_proj:$task_version:"
mkdir $TASKS/tmp/$task
trap "rm -fr $TASKS/tmp/${task}*" HUP PIPE INT QUIT TERM EXIT
cd $STEPS
$BASS_ROOT/master/bin/version-for-git >$TASKS/tmp/$task/steps-version.txt
git rev-parse @ >$TASKS/tmp/$task/steps-revision.txt
# $TAR cf - --posix * | $COMPRESSOR >$TASKS/tmp/$task/steps.tar
git archive @ | $COMPRESSOR >$TASKS/tmp/$task/steps.tar
cd $PROJ
echo $task_version >$TASKS/tmp/$task/code-version.txt
git show --no-patch --pretty=fuller $rev >>$TASKS/tmp/$task/code-version.txt
echo $rev >$TASKS/tmp/$task/code-revision.txt
git archive $rev | $COMPRESSOR >$TASKS/tmp/$task/code.tar
tasks=$($BASS_ROOT/master/bin/clone-with-ctr $task
$(for arch in $ARCH ; do echo ${task}${arch} ; done))
[ -n "$tasks" ]
for t in $tasks ; do
echo $t
mv $t ../cur
done
rm $rev_path
* Source $BASS_ROOT/lib/rc to get all possibly useful environmental
variables. Expect $REVS (set by $BASS_RC sourced file) point to the
directory filled by post-receive hook. Expect $PROJ point to the Git
repository where we can read the code. Expect $STEPS point to the
Git repository with build steps for that project. Expect $ARCHS to
hold whitespace separated list of architectures to create tasks for.
* Take one file from $REVS directory. Then go to project's root
and use version-for-git to get human readable name of the commit.
* "task" variable holds partly created name of the future task.
* Create temporary directory in $TASKS/tmp.
* Go to $STEPS, save its Git's commit version in
$task/steps-revision.txt and save all its code in $task/steps.tar.
* Go to $PROJ and similarly save its code version and code itself.
* Go to temporary directory for tasks and call clone-with-ctr. It
copies your specified temporary directory to directories with the
architecture in their name. One directory per architecture specified
in $ARCHS.
Why not ordinary "cp -a"? It fsyncs your source directory and
hardlinks all files, taking virtually no additional space for each
of your task.
* At last move you fsynced tasks outside the tmp/. That way
they will appear atomically for processed looking at cur/.
* That task-maker is expected to be run under some kind of supervisor,
like [CI/Daemontools].
* Well, task is created, event is removed. Master finished its job.
Now it is time for slave to acquire one of appeared tasks.
Note that you can easily create tasks on a cron events, just by touching
files at specified time. Whatever workflow you wish!