GitLab: don’t execute a job if previous run was successful

With built-in Auto DevOps enabled, GitLab tends to run a full pipeline whenever anything of the following happens:

• pushing
• branching
• merging (even if “Fast-forward merge” is configured)
• tagging

And that tireless building on every possible occasion consumes all kinds of resources: CPU, disk space, developers’ time.

I guess there are many ways to solve this problem, but I’ve come up with two so far.

First way: change the flow

One can make sure that certain events are meaningful in the context of one’s workflow. Here is an example:

1. Start build and test jobs only when new commits are pushed to feature/* branches.
2. Start deployment jobs only when new commits are pushed to develop branch, or when tag is created.
3. Do not allow direct pushing to develop, but enforce merge requests.

This flow might or might not be good for any particular project. It is good in my case, and is rather easy to implement via .gitlab-ci.yml:

build:
  script: ...
  rules:
    - if: '$CI_COMMIT_BRANCH =~ /^feature\/.*$/'
test:
  script: ...
  rules:
    - if: '$CI_COMMIT_BRANCH =~ /^feature\/.*$/'
deploy-dev:
  script: ...
  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'
deploy-test:
  script: ...
  rules:
    - if: '$CI_COMMIT_TAG == "test"'
deploy-prod:
  script: ...
  rules:
    - if: '$CI_COMMIT_TAG == "prod"'

So:

• building and testing happens only on feature branches;
• merges to develop are meant to happen after successful feature branch commit (build succeeded, tests passed);
• every commit do develop branch is automatically deployed to dev environment;
• every test and prod tags are deployed to corresponding environments — these tags meant to be created by developer manually, and tagged commits are meant to be successful merges (in other words, one could go to develop branch history and tag any commit to get immediate deployment).

Second way: check if job is already done

So, it’s simple: given the repository holds all info required to build a project, why would two builds of the same git commit be different? Same goes for unit tests and other similar procedures. And this means that after first successful pipeline we can mark next runs as successful immediately. Or, to make it more granular and precise, do that on a job level.

To do this, we can use GitLab API /projects/<project id>/jobs to fetch job status which yields something similar to this (showing only relevant fields here, more complete example can be found in GitLab API reference):

[
  {
    "commit": {
      "id": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd"
    },
    "id": 7,
    "name": "teaspoon",
    "status": "success"
    }
  }
]

Of course, this kind of info is protected, and CI_JOB_TOKEN cannot be used for that. So, a new token must be created in Access Tokens section of Settings.—read_api should be enough for this particular task. The token can be added as a variable in corresponding section of project’s CI/CI settings and then used in .gitlab-ci.yml.

Anyhow, here is the first step — a curl command that fetches last 10 jobs as JSON like in the example above:

curl \
  -s \
  -H "PRIVATE-TOKEN: ****" \
  "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs?page=1&per_page=10"

We need to check if there is a job record that refers to the same commit hash, same job name, and is successful, and maybe job ID — just for reference. Here is jq call to do that, i.e. it returns ID of first job which matches given criteria, or null:

jq \
  --arg sha "${CI_COMMIT_SHA}" \
  --arg job "${CI_JOB_NAME}" \
  '[ .[] | select(.commit.id == $sha and .status == "success" and .name == $job) ][0] | .id'

And here is how it can be added to .gitlab-ci.yaml:

# This is a global one, all jobs will inherit this code:
before_script:
  - |
  last_successful_job_id=$(curl \
    -s \
    -H "PRIVATE-TOKEN: ****" \
    "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/jobs?page=1&per_page=10" \
  | jq \
    --arg sha "${CI_COMMIT_SHA}" \
    --arg job "${CI_JOB_NAME}" \
    '[ .[] | select(.commit.id == $sha and .status == "success" and .name == $job) ][0] | .id')
  [ "${last_successful_job_id}" == "null" ] || ( echo "Found previous successful job #$last_successful_job_id" && exit )