A secure and scalable Renovate service on GitLab¶

This is a repost of my original article in Siemens' blog with some formatting enhancements.

Self-hosting a per-project Renovate service on GitLab with a project access token and CI approval gate enables secure and scalable dependency updates even with non-public dependencies.

Abstraction plays a crucial role in software engineering, enabling developers to manage complexity, hide implementation details, and create modular, reusable components. As Grady Booch said:

The entire history of software engineering is one of rising levels of abstraction.

— Grady Booch

These abstractions extend to integrating external functionality from software libraries or modules, especially from open-source and inner-source software ecosystems. Managing such dependencies manually incurs significant effort and requires discipline, contributing to technical debt when neglected by causing outdated dependencies and thereby lack of new features and optimizations, compatibility issues, security vulnerabilities, and reduced maintainability. (Semi-)automated dependency updating solutions facilitate the update process by discovering new versions of dependencies and generating merge requests (MRs) for updates.

Renovate is a popular, highly configurable, open-source software for dependency update automation, supporting a comprehensive set of package managers, data sources, and DevOps platforms. At Siemens, we use Renovate on our company-wide self-hosted GitLab instance at code.siemens.com. Running Renovate on GitLab is delicate though because of GitLab's CI security model as disclosed by fellow Siemens colleagues in 2020, such that hosting a central, instance-wide Renovate service is susceptible to privilege escalation. Instead, a dedicated Renovate service is typically run per GitLab group, which reduces the attack surface when all maintainers of projects within the group are trusted and diligent MR reviewers. Yet, this approach is only a shift of the balance between security and convenience towards security.

But running Renovate per GitLab project with a project access token (PrAT) is secure, as the token is scoped to the project, enables updating public dependencies out of the box, and even supports semi-automated updating of non-public dependencies on the same GitLab instance due to a creative implementation of a CI approval gate.

Primer¶

Let's assemble the necessary technical background about GitLab and Renovate first.

GitLab CI job token¶

GitLab provides each CI job with a unique token via the predefined CI variable $CI_JOB_TOKEN that grants it seamless access to a curated set of features on the GitLab server such as the package registry. This token is scoped to the GitLab project by default and has limited permissions based on the privileges of the user who triggers the CI job.

Unfortunately, the lack of granular control over CI job token permissions (gitlab&3559) combined with some poorly calibrated role permissions and missing protection features may ensue severe security risks including the following: The "Developer" role – also via the CI job token – permits write access to the package registry, which makes it not only susceptible to (accidental) package updates or deletions (gitlab&5574) but even introduces opportunities for contamination with unauthorized packages containing malicious code. Likewise, the container registry is vulnerable to these threats (gitlab&9825). And the "Developer" role – also via the CI job token – permits read access to all Terraform states of a project without configurable protection (gitlab#227108), which risks secrets leakage even for production infrastructure. That said, fellow Siemens colleagues have begun contributing package protection to GitLab with experimental support for NPM packages available via feature flag at the time of writing.

Despite increased security through the CI job token allowlist for inbound access, privilege escalation vulnerabilities exist with particular relevance to running Renovate on GitLab as a central service.

Renovate¶

Renovate is a CLI application which automates the monitoring and updating of dependencies in software projects. It supports a wide range of package managers and data sources for discovering updates, ensuring comprehensive coverage across various ecosystems, and integrates with major DevOps platforms including GitLab to create MRs with updated dependency versions. These are the high-level steps that Renovate performs:

1. Dependency extraction: Renovate scans the files in a Git repository to extract dependency information from configuration files specific to each supported programming language or package manager.
2. Dependency update discovery: Renovate checks the corresponding data source for newer versions of an identified dependency, comparing the current version used in the Git repository with the latest available version according to a configurable versioning scheme.
3. Dependency update request: Renovate updates the relevant files in the Git repository to reflect the new version of a dependency and submits a MR with the changes for review and merging into the target branch.

For this, Renovate needs read access to the data sources of all relevant dependencies and comprehensive read/write permissions on the project for, e.g., reading project settings, creating/updating/deleting branches, creating/updating/closing MRs, and more.

On github.com, Renovate's GitHub app is a popular way of enabling Renovate on a project; for public projects, its sister app offers enhanced security by submitting MRs from repository forks, which require only read permissions. However, no equivalent integration for GitLab exists at the time of writing due to security issues on GitLab.

Renovate on GitLab¶

On GitLab, Renovate must be self-hosted. The typical setup involves running it in a dedicated project on a scheduled CI pipeline with the permissions of a functional user (dedicated regular account, group access token or service account), that is added as a member in a target project with the "Developer" role or higher for onboarding and henceforth performing dependency updates. However, this setup is insecure for non-public² projects because of GitLab's CI job token privilege escalation exploit.

The general attack pattern involves an attacker performing illicit actions in a CI job run beyond their originally assigned privileges when the CI job is triggered by a higher-privileged user. The exploit workflow involving Renovate is as follows:

An attacker creates an initially innocuous project Malicious which depends on resources from a non-public project Compromised. Naturally, the attacker also is a project member of Compromised with the "Reporter" role granting read permissions. Additionally, Compromised includes Malicious in its CI job token allowlist for inbound access to grant CI jobs of Malicious access to Compromised. This means, a CI job of Malicious triggered by the attacker can read resources from Compromised. To this point, the setup is secure; however, the exploit gets introduced when a central Renovate service is onboarded to both Malicious and Compromised by inviting Renovate's functional user to both projects with privileged access, e.g. the "Maintainer" role. When Renovate submits a dependency update to Compromised after the attacker has committed malicious code, the triggered CI pipeline runs with Renovate's elevated permissions in Compromised, authorizing illicit actions via the CI job token in Compromised despite the attacker's lack of privileges.

Solution¶

Instead of running a central Renovate service that performs updates on multiple GitLab projects, giving rise to cross-project privilege escalation attacks as discussed above, a dedicated Renovate service is set up in each project using a PrAT with scoped permissions to perform actions on the project. We present the setup and workflow for projects with only public dependencies as well as for projects with non-public dependencies on the same GitLab instance. In addition, we lay out a solution to out-of-band rebase requests of branches/MRs created by Renovate, which unlike with a central Renovate service is feasible with a per-project Renovate setup.

Public dependencies¶

In a per-project Renovate setup, Renovate's CI job and the project's main CI pipeline co-exist but are independent; in fact, they are mutually exclusive, i.e., running the main CI pipeline does not comprise a Renovate run and vice versa. Unfortunately, GitLab CI does not idiomatically support targeting a specific CI job or pipeline configuration but only branches and tags. That said, a solution is created by including Renovate's CI job specification from the default branch in a dedicated orphan branch named renovate-bot and creating a CI pipeline schedule with this target branch.

To begin, add the following per-project Renovate CI job specification to the new file .gitlab/ci/renovate.yml on the default branch:

renovate:
  stage: deploy
  image: renovate/renovate:<version>
  variables:
    GIT_STRATEGY: none # (1)!
    RENOVATE_PLATFORM: gitlab # (2)!
    RENOVATE_ENDPOINT: $CI_API_V4_URL # (3)!
    RENOVATE_REPOSITORIES: '["$CI_PROJECT_PATH"]' # (4)!
    RENOVATE_HOST_RULES: | # (5)!
      [
        {
          "matchHost": "$CI_SERVER_HOST",
          "hostType": "<datasource>",
          "username": "gitlab-ci-token",
          "password": "$CI_JOB_TOKEN"
        },
        {
          "matchHost": "$CI_REGISTRY",
          "username": "gitlab-ci-token",
          "password": "$CI_JOB_TOKEN"
        }
      ]
    RENOVATE_REGISTRY_ALIASES: | # (6)!
      {
        "$$CI_REGISTRY": "$CI_REGISTRY",
        "$${CI_REGISTRY}": "$CI_REGISTRY",
      }
    RENOVATE_IGNORE_PR_AUTHOR: "true" # (7)!
    RENOVATE_CACHE_DIR: $CI_PROJECT_DIR/.cache/ # (8)!
  script: # (9)!
    - renovate
  cache: # (10)!
    key: renovate
    paths:
      - .cache/
  interruptible: false # (11)!
  environment: # (12)!
    name: renovate
    action: access
  resource_group: renovate # (13)!
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule" # (14)!

1. The CI job variable $GIT_STRATEGY overrides the default Git strategy with none, causing the CI runner to skip all Git operations during CI job initialization, thereby optimizing CI performance, as Renovate is self-contained when configured exclusively via environment variables.
2. The CI job variable $RENOVATE_PLATFORM enables Renovate's GitLab integration.
3. The CI job variable $RENOVATE_ENDPOINT provides GitLab's REST API base URL.
4. The CI job variable $RENOVATE_REPOSITORIES provides the path of the project for which to automate dependency updating – the same project that runs Renovate.
5. The CI job variable $RENOVATE_HOST_RULES provides credentials for authenticating requests to GitLab's package registries and container image registry using the CI job token. Substitute the host type placeholder <datasource> by a Renovate data source identifier supported by GitLab (e.g., npm, pypi, maven, etc.) and add more similar host rules as needed. Even when all dependencies are public, these host rules are important for Renovate to make authenticated requests to GitLab and thus avoid hitting rate limits.
6. The CI job variable $RENOVATE_REGISTRY_ALIASES allows Renovate to expand the variable references $CI_REGISTRY and ${CI_REGISTRY} during its dependency extraction phase. A common use case for these aliases is updating container images specified in the CI configuration such as image: ${CI_REGISTRY}/<namespace>/<project>:<tag>. Extend the registry aliases according to your needs.
7. The CI job variable $RENOVATE_IGNORE_PR_AUTHOR, when set to true, disables an optimization to fetch only MRs created by Renovate's GitLab user. This setting is useful when Renovate's GitLab user is not stable, which may happen when, e.g., the PrAT is not rotated before expiry but re-created, resulting in a different user. Unless Renovate's performance drops notably, setting it to true is preferable for ensuring correct behavior.
8. The CI job variable $RENOVATE_CACHE_DIR configures Renovate to store its cache in the .cache/ directory of the the CI job's working directory as a prerequisite for declaring a CI cache.
9. In the script step, the renovate executable is called.
10. The cache keyword declares a CI cache for Renovate's cache to speed up subsequent Renovate runs with a warm cache.
11. The interruptible keyword disables cancellation of the CI job before completion due to auto-cancellation of redundant pipelines.
12. The environment keyword declares a static environment named renovate for limiting the scope of sensitive CI variables to this CI job. The environment:action keyword declares the job to only access the environment (i.e., its environment-scoped CI variables) for semantic correctness and slightly better developer experience by omitting an entry in the environments list.
13. The resource_group keyword ensures that Renovate does not run concurrently across multiple CI pipelines to avoid conflicting actions.
14. The rules:if clause limits the CI job to run only for scheduled CI pipelines.

This CI job specification is inspired by Renovate's official GitLab CI template. However, the CI template is intended for hosting a central Renovate service in a dedicated GitLab project with some settings being non-applicable to or suboptimal for a per-project Renovate setup. We prefer a self-contained, explicit, and tailored CI job configuration over an indirection via an external resource with its own lifecycle.

By default, Renovate's GitLab CI manager will only check any files matching the regular expression \.gitlab-ci\.ya?ml$. To consider also .gitlab/ci/renovate.yml for updates – even all YAML files within the .gitlab/ci/ directory –, add the following configuration to Renovate's repository configuration file (e.g., renovate.json):

{
  "gitlabci": {
    "fileMatch": ["\\.gitlab-ci\\.ya?ml$", "(^|/)\\.gitlab/ci/.+\\.ya?ml$"]
  }
}

Commit the two files and push them to the remote Git repository on GitLab as follows:

git add .gitlab/ci/renovate.yml renovate.json
git commit
git push origin main

Then, create and switch to the empty renovate-bot orphan branch:

git switch --orphan renovate-bot

Create the new file .gitlab-ci.yml and add the following content to include Renovate's CI configuration from the tip of the default branch:

include:
  - project: $CI_PROJECT_PATH
    ref: $CI_DEFAULT_BRANCH
    file: .gitlab/ci/renovate.yml

Commit this file and push it to the remote Git repository on GitLab as follows:

git add .gitlab-ci.yml
git commit
git push origin renovate-bot

Renovate needs permissions to create branches and MRs, for which a PrAT is created in the GitLab project at Settings > Access Tokens with the following form inputs:

Then, the PrAT is exposed to Renovate's CI job by adding a CI project variable at Settings > CI/CD > Variables with the following form inputs:

For Renovate to be able to access the protected CI variable, the renovate-bot branch must be protected at Settings > Repository > Protected branches > Add protected branch with the following form inputs:

By default, the CI pipeline name is the subject line from the message of the commit for which it runs. To better identify a scheduled Renovate CI pipeline run and distinguish it from regular runs, the CI pipeline name may be customized using the workflow:name keyword:

workflow:
  name: $WORKFLOW_NAME

include:
  # ...

For Renovate to run periodically, create a CI pipeline schedule at Build > Pipeline schedules > New schedule with the following settings:

We recommend creating the CI pipeline schedule with the PrAT (which is possible only via GitLab's API at the time of writing), so the scheduled CI pipeline runs with the PrAT's scoped permissions for increased security:

curl https://gitlab.example.com/api/graphql \
  --request POST \
  --header "PRIVATE-TOKEN: glpat-xxxxxxxxxxxxxxxxxxxx" \
  --header "Content-Type: application/json" \
  --data-raw '
{
  "operationName": "createPipelineSchedule",
  "variables": {
    "input": {
      "description": "Renovate",
      "cron": "0 4 * * 1-5",
      "cronTimezone": "Etc/UTC",
      "ref": "renovate-bot",
      "variables": [
        {
          "key": "WORKFLOW_NAME",
          "value": "Renovate",
          "variableType": "ENV_VAR"
        }
      ],
      "active": true,
      "projectPath": "<project_path>"
    }
  },
  "query": "mutation createPipelineSchedule($input: PipelineScheduleCreateInput!) { pipelineScheduleCreate(input: $input) { errors } }"
}'

Note the use of GitLab's GraphQL API to create the CI pipeline schedule, as GitLab's REST API does not support including CI variables in a single request at the time of writing.

Renovate may be configured to auto-merge branches/MRs (with some limitations):

Given the standard branch protection settings for the default branch, the PrAT must have the "Maintainer" role on GitLab Free at the time of writing; on GitLab Premium/Ultimate, a PrAT with the "Developer" role may auto-merge MRs when the protection rule for the default branch includes the PrAT user under "Allowed to merge".

Due to the added support for PrAT avatar customization released in GitLab v17.0, the visual experience may be enhanced by uploading a custom PrAT avatar (which is possible only via GitLab API at the time of writing):

curl https://gitlab.example.com/api/v4/user/avatar \
  --request PUT \
  --header "PRIVATE-TOKEN: glpat-xxxxxxxxxxxxxxxxxxxx" \
  --header "Content-Type: multipart/form-data" \
  --form avatar=@/path/to/avatar.png

Internal & private dependencies¶

The PrAT's limited scope is concomitant with its lack of permissions for accessing internal¹ and private dependencies on the same GitLab instance. Thus, a CI pipeline triggered by Renovate upon branch or MR creation is not able to install internal and private dependencies because the CI job token, associated with Renovate's PrAT, lacks the necessary permissions.

A solution in this scenario is achieved by setting up an approval gate via a manual CI job that triggers the main CI pipeline in a child pipeline only upon manual action, leveraging the fact that the manual CI job and child pipeline run with the permissions of the user who triggers them.

For this, factor out any existing CI jobs' specifications from the CI pipeline configuration in the .gitlab-ci.yml file into a separate file .gitlab/ci/main.yml and conditionally include the latter in the former, excluding unapproved CI pipelines on branches created by Renovate. Further, add the approval gate CI job specification to the .gitlab-ci.yml file:

include:
  - local: .gitlab/ci/main.yml
    rules:
      - if: $CI_COMMIT_REF_NAME !~ /^renovate\//
      - if: $CI_APPROVED == "true"

renovate:approve:
  stage: deploy
  trigger:
    include: $CI_CONFIG_PATH # (1)!
    strategy: depend # (2)!
  inherit:
    variables: false # (3)!
  rules: # (4)!
    - if: $CI_COMMIT_REF_NAME !~ /^renovate\//
      when: never
    - if: $CI_APPROVED
      when: never
    - if: $GITLAB_USER_NAME == "renovate[bot]"
      when: manual
      variables:
        CI_APPROVED: "true"
    - variables:
        CI_APPROVED: "true"

1. The trigger:include keyword declares the path to the project's CI configuration file.

2. The trigger:strategy keyword makes the trigger job run the child pipeline synchronously and mirror its status.

3. The inherit:variables keyword disables inheritance of global CI variables in the CI job and its child pipeline to avoid side effects caused by leakage from the parent pipeline into the child pipeline.

4. The rules keyword specifies rules for running the CI job. There are two preconditions for the CI job to run: The branch must be created by Renovate (i.e., the branch name prefix is renovate/) and the CI variable $CI_APPROVED must not be set, indicating no preceding approval. Given those preconditions are met, the CI variable $CI_APPROVED is set to true and forwarded to the child pipeline, marking it approved. Also, the CI job is declared as "manual" if Renovate's PrAT user creates the CI pipeline, making the execution of the child pipeline contingent on an authorized user taking action.

   Note that this rule set offers a smooth developer experience when a CI pipeline is triggered by a user pushing additional commits to the branch, as the approval gate automatically starts the child pipeline without requiring manual action.

Unlike when updating only public dependencies, Renovate's CI pipeline schedule must be created (and thus, owned) by a regular project member – not by the PrAT – because the CI job token exported to Renovate's CI job must be allowed to access all relevant dependencies for Renovate to discover internal and private dependency updates. Any member with the "Maintainer" role may assume ownership of the CI pipeline schedule, as accessing all dependencies is a natural prerequisite of the role. It is worth pointing out that, at the time of writing, GitLab's CI pipeline schedule ownership model is subject to identity theft among privileged project members (typically having the "Maintainer" role). Whether this risk is acceptable must be decided case-by-case. If it is unacceptable, as a last resort the CI pipeline schedule may be disabled and triggered manually, as the triggered CI pipeline will run with the permissions of the user who triggers it, but Renovate's merit diminishes. In any case, untrusted privileged project members pose a threat, so the security implications of a regular project member owning Renovate's CI pipeline schedule must be put into perspective.

Due to the integral need for manual action in this setup, Renovate's auto-merging feature does not work.

Rebase requests¶

In addition to creating MRs for dependency updates, Renovate must rebase its branches on occasion, e.g. to resolve merge conflicts or update outdated branches. Since Renovate runs on a scheduled CI pipeline on GitLab, rebasing branches occurs by default only at the next run – requesting a rebase out-of-band is ineffective – which, depending on the schedule interval, may incur prohibitive delays. A better solution is achieved by manually triggering a Renovate run with a rebase-only configuration, which is feasible because Renovate is part of the project and project members with the "Maintainer" role or higher are permitted to trigger Renovate's CI pipeline on the renovate-bot branch.

For this, create an additional inactive CI pipeline schedule at Build > Pipeline schedules > New schedule:

To initiate the rebase process, trigger the CI pipeline schedule at Build > Pipeline schedules by pressing the button of the schedule "Renovate: Rebase".

Limitations¶

PrAT availability¶

At the time of writing, PrATs have mixed availability across GitLab deployment options and tiers:

PrAT rotation¶

At the time of writing, PrATs can be rotated via API only. In addition, PrATs cannot self-rotate; instead, a personal access token (PAT) and a PrAT ID are required, which weakens the user experience. That said, feature requests related to PrAT self-rotation have been submitted, comprising the addition of an API endpoint for token self-rotation and the addition of the rotate_self scope.

To rotate a PrAT, begin by creating a PAT with the api scope. Then, retrieve the metadata of the active PrAT whose token name is renovate[bot]:

curl https://gitlab.example.com/api/v4/projects/<project_id>/access_tokens \
  --header "PRIVATE-TOKEN: glpat-xxxxxxxxxxxxxxxxxxxx" |
  jq '.[] | select(.name == "renovate[bot]" and .active)'

Note the token ID in the printed JSON response body's id field. Finally, rotate the PrAT with an appropriate expiration date:

curl https://gitlab.example.com/api/v4/projects/<project_id>/access_tokens/<token_id>/rotate" \
  --request POST \
  --header "PRIVATE-TOKEN: glpat-xxxxxxxxxxxxxxxxxxxx" \
  --form expires_at=YYYY-MM-DD |
  jq

Find the new PrAT in the token field of the printed JSON response body.

Commit signing¶

At the time of writing, a PrAT user cannot sign commits because it has no signing key, so Renovate's unsigned commits will be rejected when the push rule "Reject unsigned commits" (available on GitLab Premium/Ultimate) is enabled. However, this limitation can be worked around with a creative combination of SSH-based commit signing and deploy keys by leveraging the fact that commits signed with a deploy key pass commit signature verification.

Generate a new SSH key pair without a passphrase for Renovate:

ssh-keygen -t ed25519 -C "renovate[bot]" -N "" -f id_renovate

Then, export the private key, which will be used for signing commits, to Renovate's CI job by adding a CI project variable at Settings > CI/CD > Variables with the following form inputs:

-----BEGIN OPENSSH PRIVATE KEY-----
...
-----END OPENSSH PRIVATE KEY-----

Next, add the public key, which will be used for verifying commit signatures, as a deploy key to the project using the PrAT to associate the deploy key with the PrAT user:

curl https://gitlab.example.com/api/v4/projects/<project_id>/deploy_keys \
  --request POST \
  --header "PRIVATE-TOKEN: glpat-xxxxxxxxxxxxxxxxxxxx" \
  --header "Content-Type: application/json" \
  --data '
{
  "title": "Renovate'\''s public signing key",
  "key": "ssh-ed25519 AAAA... renovate[bot]",
  "can_push": "false"
}'

Auto-merging with internal & private dependencies¶

When Renovate is set up to allow updating projects with internal and private dependencies, auto-merging Renovate's MRs without human intervention is not supported whenever the ability to merge is contingent on a successful CI pipeline run, because the approval gate requires manual action by a project member to trigger the CI pipeline of the MR.

Role for rebase requests¶

Rebase requests can be initiated only by project members with the "Maintainer" role or higher although checking the "If you want to rebase/retry this MR, check this box" checkbox in the description of a MR created by Renovate requires no elevated permissions. This limitation may conceivably be overcome by using a project webhook listening to MR events with a CI pipeline receiver that initiates a Renovate rebase run when the checkbox gets checked. However, using a CI pipeline receiver for frequent webhook events is very inefficient at the time of writing, as the webhook payload cannot be included in a rules:if clause to efficiently avoid obsolete CI job creation, which incurs inherent overhead.

Conclusion¶

Renovate is a widely used open-source dependency update automation tool, which integrates with popular DevOps platforms including GitLab. Unfortunately, GitLab's CI security model is susceptible to privilege escalation, rendering a central, instance-wide Renovate service insecure; in fact, any multi-project Renovate service is vulnerable to exploitation.

But a secure setup is achieved by running a dedicated per-project Renovate service with a project access token for authenticating to the GitLab server. Projects with only public dependencies work seamlessly out of the box. And a creative implementation of a CI approval gate also enables semi-automated dependency updates for projects with non-public dependencies on the same GitLab instance. As a side effect, a per-project Renovate service supports out-of-band rebase requests of branches/MRs created by Renovate. A comprehensive discussion about security, limitations, and workarounds complements the proposed solutions.

If you find this article useful, please consider sharing your feedback and experience via the comment box below. 🙏

Comments