Master node(s) are intended to create tasks. As a rule they are created as an event on someone pushing the commit. There are no specialised committed daemons running on them, because each project's task making process can vastly differ in details. Only atomic counter utilities are commonly used by "task makers" as a rule. Let's see how example example/goredo CI pipeline is created. We want to run the tests when someone pushes the commit. * Prepare necessary directories at the very beginning: mkdir -p /nfs/revs/goredo mkdir -p /nfs/tasks/ctr/0 mkdir -p /nfs/tasks/{cur,old,tmp} mkdir /nfs/jobs * First thing to do is to create Git's post-receive hook, that will touch files with the revision needed to be tested. $ cat >goredo.git/hooks/post-receive <<EOF #!/bin/sh -e REVS=/nfs/revs/goredo ZERO="0000000000000000000000000000000000000000" read prev curr ref [ "$curr" != $ZERO ] || exit 0 [ "$prev" != $ZERO ] || prev=$curr^ git rev-list $prev..$curr | while read rev ; do mkdir -p $REVS/$ref echo BASSing $ref/$rev... >&2 touch $REVS/$ref/$rev done EOF After pushing a bunch of commits, corresponding empty files in revisions directory will be created. Each filename is a commit's hash. Those are basically a notification events about the need to create corresponding tasks. * Someone has to process those events. Each project has its own task-maker, because there are so many variations how code and build steps can be retrieved and created. Let's create one: #!/bin/sh -e [ -n "$BASS_ROOT" ] sname="$0" . $BASS_ROOT/lib/rc [ -n "$REVS" ] || { echo '"REVS"' is not set >&2 exit 1 } [ -n "$PROJ" ] || { echo '"PROJ"' is not set >&2 exit 1 } [ -n "$STEPS" ] || { echo '"STEPS"' is not set >&2 exit 1 } [ -n "$ARCHS" ] || { echo '"ARCHS"' is not set >&2 exit 1 } cd $REVS rev=$(find . -type f | sed -n 1p) [ -n "$rev" ] rev_path=$(realpath $rev) rev=$(basename $rev) task_proj=goredo task_version=$(cd $PROJ ; $BASS_ROOT/master/bin/version-for-git $rev) [ -n "$task_version" ] task=":$task_proj:$task_version:" mkdir $TASKS/tmp/$task trap "rm -fr $TASKS/tmp/${task}*" HUP PIPE INT QUIT TERM EXIT cd $STEPS $BASS_ROOT/master/bin/version-for-git >$TASKS/tmp/$task/steps-version.txt git rev-parse @ >$TASKS/tmp/$task/steps-revision.txt # $TAR cf - --posix * | $COMPRESSOR >$TASKS/tmp/$task/steps.tar git archive @ | $COMPRESSOR >$TASKS/tmp/$task/steps.tar cd $PROJ echo $task_version >$TASKS/tmp/$task/code-version.txt git show --no-patch --pretty=fuller $rev >>$TASKS/tmp/$task/code-version.txt echo $rev >$TASKS/tmp/$task/code-revision.txt git archive $rev | $COMPRESSOR >$TASKS/tmp/$task/code.tar tasks=$($BASS_ROOT/master/bin/clone-with-ctr $task $(for arch in $ARCH ; do echo ${task}${arch} ; done)) [ -n "$tasks" ] for t in $tasks ; do echo $t mv $t ../cur done rm $rev_path * Source $BASS_ROOT/lib/rc to get all possibly useful environmental variables. Expect $REVS (set by $BASS_RC sourced file) point to the directory filled by post-receive hook. Expect $PROJ point to the Git repository where we can read the code. Expect $STEPS point to the Git repository with build steps for that project. Expect $ARCHS to hold whitespace separated list of architectures to create tasks for. * Take one file from $REVS directory. Then go to project's root and use version-for-git to get human readable name of the commit. * "task" variable holds partly created name of the future task. * Create temporary directory in $TASKS/tmp. * Go to $STEPS, save its Git's commit version in $task/steps-revision.txt and save all its code in $task/steps.tar. * Go to $PROJ and similarly save its code version and code itself. * Go to temporary directory for tasks and call clone-with-ctr. It copies your specified temporary directory to directories with the architecture in their name. One directory per architecture specified in $ARCHS. Why not ordinary "cp -a"? It fsyncs your source directory and hardlinks all files, taking virtually no additional space for each of your task. * At last move you fsynced tasks outside the tmp/. That way they will appear atomically for processed looking at cur/. * That task-maker is expected to be run under some kind of supervisor, like [CI/Daemontools]. * Well, task is created, event is removed. Master finished its job. Now it is time for slave to acquire one of appeared tasks. Note that you can easily create tasks on a cron events, just by touching files at specified time. Whatever workflow you wish!