---

Master ¶

Master node(s) are intended to create tasks. As a rule they are created as an event on someone pushing the commit. There are no specialised committed daemons running on them, because each project’s task making process can vastly differ in details. Only atomic counter utilities are commonly used by "task makers" as a rule.

Let’s see how example example/goredo CI pipeline is created. We want to run the tests when someone pushes the commit.

• Prepare necessary directories at the very beginning:

  mkdir -p /nfs/revs/goredo
  mkdir -p /nfs/tasks/ctr/0
  mkdir -p /nfs/tasks/{cur,old,tmp}
  mkdir /nfs/jobs
  

• First thing to do is to create Git’s post-receive hook, that will touch files with the revision needed to be tested.

  $ cat >goredo.git/hooks/post-receive <<EOF
  #!/bin/sh -e
  
  REVS=/nfs/revs/goredo
  ZERO="0000000000000000000000000000000000000000"
  
  read prev curr ref
  [ "$curr" != $ZERO ] || exit 0
  [ "$prev" != $ZERO ] || prev=$curr^
  git rev-list $prev..$curr | while read rev ; do
      mkdir -p $REVS/$ref
      echo BASSing $ref/$rev... >&2
      touch $REVS/$ref/$rev
  done
  EOF
  

  After pushing a bunch of commits, corresponding empty files in revisions directory will be created. Each filename is a commit’s hash. Those are basically a notification events about the need to create corresponding tasks.

• Someone has to process those events. Each project has its own task-maker, because there are so many variations how code and build steps can be retrieved and created. Let’s create one:

  #!/bin/sh -e
  
  [ -n "$BASS_ROOT" ]
  sname="$0" . $BASS_ROOT/lib/rc
  [ -n "$REVS" ] || {
      echo '"REVS"' is not set >&2
      exit 1
  }
  [ -n "$PROJ" ] || {
      echo '"PROJ"' is not set >&2
      exit 1
  }
  [ -n "$STEPS" ] || {
      echo '"STEPS"' is not set >&2
      exit 1
  }
  [ -n "$ARCHS" ] || {
      echo '"ARCHS"' is not set >&2
      exit 1
  }
  
  cd $REVS
  rev=$(find . -type f | sed -n 1p)
  [ -n "$rev" ]
  rev_path=$(realpath $rev)
  rev=$(basename $rev)
  
  task_proj=goredo
  task_version=$(cd $PROJ ; $BASS_ROOT/master/bin/version-for-git $rev)
  [ -n "$task_version" ]
  task=":$task_proj:$task_version:"
  mkdir $TASKS/tmp/$task
  trap "rm -fr $TASKS/tmp/${task}*" HUP PIPE INT QUIT TERM EXIT
  
  cd $STEPS
  $BASS_ROOT/master/bin/version-for-git >$TASKS/tmp/$task/steps-version.txt
  git rev-parse  >$TASKS/tmp/$task/steps-revision.txt
  # $TAR cf - --posix * | $COMPRESSOR >$TASKS/tmp/$task/steps.tar
  git archive  | $COMPRESSOR >$TASKS/tmp/$task/steps.tar
  
  cd $PROJ
  echo $task_version >$TASKS/tmp/$task/code-version.txt
  git show --no-patch --pretty=fuller $rev >>$TASKS/tmp/$task/code-version.txt
  echo $rev >$TASKS/tmp/$task/code-revision.txt
  git archive $rev | $COMPRESSOR >$TASKS/tmp/$task/code.tar
  
  tasks=$($BASS_ROOT/master/bin/clone-with-ctr $task
      $(for arch in $ARCH ; do echo ${task}${arch} ; done))
  [ -n "$tasks" ]
  for t in $tasks ; do
      echo $t
      mv $t ../cur
  done
  
  rm $rev_path
  

  • Source $BASS_ROOT/lib/rc to get all possibly useful environmental variables. Expect $REVS (set by $BASS_RC sourced file) point to the directory filled by post-receive hook. Expect $PROJ point to the Git repository where we can read the code. Expect $STEPS point to the Git repository with build steps for that project. Expect $ARCHS to hold whitespace separated list of architectures to create tasks for.
  • Take one file from $REVS directory. Then go to project’s root and use version-for-git to get human readable name of the commit.
  • "task" variable holds partly created name of the future task.
  • Create temporary directory in $TASKS/tmp.
  • Go to $STEPS, save its Git’s commit version in $task/steps-revision.txt and save all its code in $task/steps.tar.
  • Go to $PROJ and similarly save its code version and code itself.
  • Go to temporary directory for tasks and call clone-with-ctr. It copies your specified temporary directory to directories with the architecture in their name. One directory per architecture specified in $ARCHS.

    Why not ordinary cp -a? It fsyncs your source directory and hardlinks all files, taking virtually no additional space for each of your task.

  • At last move you fsynced tasks outside the tmp/. That way they will appear atomically for processed looking at cur/.
• That task-maker is expected to be run under some kind of supervisor, like daemontools.
• Well, task is created, event is removed. Master finished its job. Now it is time for slave to acquire one of appeared tasks.

Note that you can easily create tasks on a cron events, just by touching files at specified time. Whatever workflow you wish!

Next: Slave, Previous: Daemontools, Up: CI   [Index]