I was helping an Open Source Project, the Puppet Editor Services project, to move the automated CI pipelines from Travis and AppVeyor over to GitHub Actions. The project uses different combinations of Operating Systems, Ruby versions and Puppet versions to test on. This was achieved using Travis Build Matrix and AppVeyor Build Matrix. But when I tried to the same in GitHub Actions Matrix it just didn't work the same 😢.

In short - You can't add matrix entries that share the same keys. Instead you have to explicitly list out all combinations

In my case, I had the following test cases in the matrix

  os: ['windows-latest', 'ubuntu-latest']
  ruby: ['2.4', '2.5', '2.7']
  command: ['... run all tests ...']

This would run the tests for all the combinations of Operating System (Windows and Ubuntu) and Ruby Versions (2.4, 2.5 and 2.7). This worked great, however I also needed to add a specific Puppet Version and Ruby Version to test, as there is a regression in Puppet that needed to be tested. Also, we only needed to run a subset of the tests, not the whole suite. So the matrix now looked like:

  os: ['windows-latest', 'ubuntu-latest']
  ruby: ['2.4', '2.5', '2.7']
  command: ['... run all tests ...']
  include:
    - os: 'ubuntu-latest'
      ruby : '2.5'
      puppet_version: '5.1.0'
      command : ['... only run unit tests ...']

HoweverGitHub Actions would then not run the "All Tests" command for Ruby 2.5 on ubuntu-latest. This was because the matrix was not adding, but overwriting. A quick search on this and I discovered this a known "thing" with the matrix feature in GitHub Actions. I tried to re-architect the matrix and rethink how I would do the testing but nothing I came up with really worked or was just too horrible to maintain.

Partial solution

The solution offered by GitHub Actions was to not use a matrix at all, but define all the combinations explicitly. This meant turning my example 3 line matrix into a 24 line hardcoded list. This didn't take into account the other tests that I hadn't mentioned yet! I tried YAML anchors to reduce the copying of information but GitHub Actions doesn't support YAML anchors.

Sure this could work but it seemed like hard-coding is a bad way to go. So more searching and I came across a post about sharing matrix information between jobs, in particular this YAML example:

job1:
  runs-on: ubuntu-latest
  outputs:
    matrix: ${{ steps.set-matrix.outputs.matrix }}
  steps:
  - id: set-matrix
    run: |
      echo "::set-output name=matrix::[{\"go\":\"1.13\",\"commit\":\"v1.0.0\"},{\"go\":\"1.14\",\"commit\":\"v1.2.0\"}]"

builder:
  needs: job1
  runs-on: ubuntu-latest
  strategy:
    matrix:
      cfg: ${{fromJson(needs.job1.outputs.matrix)}}
  steps:
  - run: |
      echo bin-${{ matrix.cfg.go }}-${{ matrix.cfg.commit }}

So what's going on here? Well this line looks very interesting:

  matrix:
    cfg: ${{fromJson(needs.job1.outputs.matrix)}}

Instead of the matrix configuration being defined in YAML, it was consuming the JSON output from another job?!?! So the execution of the job looked like:

• job1 starts
  • Runs the step set-matrix
  • The set-matrix step outputs a step variable called matrix. This is a compact JSON string. See the set-output documentation for more information about how to set output parameters
• The builder job starts after job1 completes
  • The matrix configuration is deserialised from the JSON string value needs.job1.outputs.matrix
  • The steps are then run for each entry on the matrix

Note - This example uses the matrix.cfg setting, but GitHub Actions now supports using matrix.include directly.

This means we can use an arbitrary JSON file to configure the matrix and I had partial solution to my probelm! But so what? Having a hardcoded JSON string that's read by a job is no different than hardcoding the YAML in the first place!

... But

... What if the JSON string wasn't hardcoded?

... What if the JSON string was created on the fly? by a PowerShell script? 🤔

And in Part 2 we'll explore how to create a script with the required JSON String for GitHub Actions.

Resources

• Part 2 of this blog

• GitHub Actions Matrix

• Sharing matrix information between jobs

• Setting an output paramter

You can find the original version of this post on Glenn's personal blog.