GitLab Profile Multi-project Pipeline

Source

I my previous post I relayed setting up a GitLab scheduled profile to add blog posts from this site's RSS feed into my personalised GitLab profile. I discussed setting up a new profile project, creating the README, fetching and parsing the RSS and inserting it into the README, and creating a pipeline to add the changed README back to the profile project. Finally I scheduled the pipeline to run once daily.

Since I don't actually write a post every single day, this mainly runs to no purpose, consuming CI minutes and filling the job logs with pointless runs. It would be better if the pipeline had a job to update the README only after I've actually made a blog post.

Multi-project pipelines

Since the blog itself is built from a pipeline in it's own project, then I would need a way to start my GitLab profile project's pipeline from there, after the new page is deployed.

Well, turns out that there is a way! It's called multi-project pipelines.

To work with multi-project pipelines, set up a job in one project's pipeline to trigger another project's pipeline.

In my blog's pipeline, I added a new stage, to occur after the Pages deploy stage, which triggers my profile project:

pipeline stages visualisation

Here's the commit on the blog project:

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8c12e1ca77bbdd0f32c124003b835b1e2a8d16cf..17858e024a2c1f7ca7908c1f00ae9fc97d78dd48 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,6 +1,7 @@
 ---
 stages:
   - deploy
+  - update

 image: registry.gitlab.com/milohax/milosophical-me

@@ -25,4 +26,10 @@ pages:
     paths: !reference [.download_artifacts_and_build, paths]
   except:
     changes: !reference [.exceptions, changes]
-...
\ No newline at end of file
+
+update-profile:
+  stage: update
+  except:
+    changes: !reference [.exceptions, changes]
+  trigger: milohax/milohax
+...

In my profile project's pipeline, I added a new job which doesn't have the rules:if for $CI_PIPELINE_SOURCE == "schedule". At first I tried $CI_PIPELINE_SOURCE == "pipeline" so that the job only runs when the pipeline is triggered, but this runs too soon: there is a deploy job on Pages CI which takes roughly 5-6 minutes to complete, until then the RSS doesn't have the new post. So instead I have delayed the start of this job by 8 minutes, which is long enough for the blog deployment to complete:

update-profile:
  when: delayed
  start_in: 8 minutes
  script:
    - ruby blog-read.rb
    - !reference [.git-clone-add-push, script]

(Also note the !reference tag, where I pulled out the git cloning code above to keep the code DRY, Rule 2).

With these changes in place, my blog post deployment pipeline now looks like this:

Multi-project pipeline visualisation

And the new post is added to the profile after being published. I can remove the scheduled job.

A custom Docker container

A final little touch: if you have to make changes to stock containers (which I did in my last post, by adding git to the ruby-alpine container), it's always a Good Thing to use your own container image, and store it in GitLab's Container Registry. Each project in GitLab has it's own container registry which can store the containers you use. This provides three benefits:

  1. It's faster, because you don't repeatedly make the change on every job run
  2. You won't be downloading from Docker Hub each time, so don't have to think about download limits or dependency caching
  3. You remove at least two dependencies (the external container image, and the thing you added to it, which in my case comes from the Alpine CDN)

So to do that, I made a very simple Dockerfile:

FROM ruby:2.7-alpine
RUN apk add git

Then built and pushed to GitLab. It's important when you tag the image, to use a name that matches an existing project. I actually decided to store this image in my hax project rather than the profile project, as I suspect I'll use it more widely in future:

docker build -t registry.gitlab.com/milohax/hax/ruby:2.7-alpine .
docker push registry.gitlab.com/milohax/hax/ruby:2.7-alpine

Then, in the CI/CD pipeline code, specify this new container image:

image: registry.gitlab.com/milohax/hax/ruby:2.7-alpine

And there we have it, an efficient update to my GitLab profile whenever I publish a new blog post, like this one! The final version of the CI is this:

---
image: registry.gitlab.com/milohax/hax/ruby:2.7-alpine

stages:
  - schedule
  - deploy

.git-clone-add-push:
  script:
    - |
      git clone https://oauth2:${SCHEDULE_TOKEN}@gitlab.com/${CI_PROJECT_NAMESPACE}/${CI_PROJECT_NAME}
      cp README.md ${CI_PROJECT_NAME}
      cd ${CI_PROJECT_NAME}
      git config user.email "${GITLAB_USER_EMAIL}"
      git config user.name "${GITLAB_USER_NAME}"
      git add README.md
      git commit -m "Update blogs list"
      git push

job:add-blogs:
  stage: schedule
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
  script:
    - ruby blog-read.rb
    - !reference [.git-clone-add-push, script]

update-profile:
  stage: deploy
  when: delayed
  start_in: 8 minutes
  script:
    - ruby blog-read.rb
    - !reference [.git-clone-add-push, script]
...

I hope that you enjoy following this and use it to build your own personalised GitLab profile, with dynamic elements.

Happy Hacking!

Logs in this blog post

Again, there are a lot of links in this post! Here they all are in one place:

Also I've added a snippet for how to commit repository changes within a pipeline.