Migration to GitLab CI Components #56
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Create catalog-info.yml with component metadata - Add templates directory structure for components - Create initial machine-pipeline component template This is the first step in migrating from include-based YAML files to GitLab CI Components for better versioning and discoverability. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Convert utilities/preliminary-ignore-draft-pr.yml to component - Convert utilities/skip-branch-not-a-pr.yml to component - Add spec inputs for GitHub configuration and branch patterns - Maintain original functionality with improved parameterization These components can now be consumed with type-safe inputs instead of relying solely on environment variables. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Convert pipelines/performances.yml to component - Add machine-specific performance templates (.perf_on_*) - Include processing and reporting templates for GitHub integration - Support Caliper-based benchmark conversion Projects can now use this component to run performance measurements with parameterized configuration instead of copy-pasting files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Create lassen-pipeline component (LSF scheduler) - Create dane-pipeline component (SLURM scheduler) - Create matrix-pipeline component (SLURM scheduler) - Create corona-pipeline component (flux scheduler) - Create tioga-pipeline component (flux scheduler) - Create tuolumne-pipeline component (flux scheduler) Each component provides machine-specific job templates that projects can extend, matching the original pipeline structure: - .job_on_<machine> templates for regular jobs - .on_<machine> rules for job control - Reproducer templates for debugging - Resource allocation/deallocation (SLURM/flux only) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Create base-pipeline component with common templates - Provides .machine-check template for availability checking - Provides .build-and-test template for child pipeline triggers - Provides .custom_job and .custom_perf templates - Includes GitHub status reporting for machine availability Projects can use this component to simplify their main .gitlab-ci.yml by extending these templates instead of copying boilerplate code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Create example .gitlab-ci.yml showing component usage - Create example custom-jobs-and-variables.yml with allocations - Create example jobs file for Lassen machine - Demonstrate migration from include-based to component-based setup These examples help projects understand how to adopt the new component-based architecture. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Add quick start guide for GitLab CI Components - List all available components with descriptions - Include code example showing component usage - Note legacy include-based approach is still supported - Reference examples directory for detailed configurations This helps users understand the new component-based architecture and how to adopt it in their projects. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Create detailed migration documentation in Sphinx format - Cover migration steps from legacy to component-based approach - Document all component inputs and exported templates - Include troubleshooting section for common issues - Provide complete component reference documentation This guide helps users understand how to migrate their projects from the include-based approach to GitLab CI Components. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
BREAKING CHANGE: base-pipeline component no longer defines stages. Users must now define required stages in their .gitlab-ci.yml: - prerequisites - build-and-test - performance-measurements This is necessary because GitLab CI stage definitions override rather than merge. The previous approach prevented users from adding custom stages to their pipelines. Changes: - Remove stages: from base-pipeline component - Add prominent documentation in component comments - Update example .gitlab-ci.yml to show stage definition - Update migration guide with new Step 1 for stages - Add important note in component reference docs - Update README quick start example Machine pipeline components (lassen, dane, etc.) still define their own stages because they run in isolated child pipelines. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
BREAKING CHANGE: The combined custom-jobs-and-variables.yml file has been split into two files for clarity and to avoid duplication: - custom-variables.yml - Variables only (parent pipeline) - custom-jobs.yml - Job templates only (child pipelines) Problem: The old design required including custom-jobs-and-variables.yml twice: 1. In parent pipeline (for variables to pass as component inputs) 2. In child pipelines (for .custom_job templates) This caused confusion because: - Variables were duplicated (as vars AND as component inputs) - Templates in parent were unused (only needed in child) - It wasn't clear what was used where Solution: - custom-variables.yml: Included ONLY in parent .gitlab-ci.yml - custom-jobs.yml: Included ONLY in child pipeline triggers Benefits: - Clear separation of concerns - No duplication - Obvious where each piece is used - Variables flow: parent vars → component inputs → child pipeline Changes: - Delete examples/example-custom-jobs-and-variables.yml - Add examples/example-custom-variables.yml - Add examples/example-custom-jobs.yml - Update examples/example-gitlab-ci.yml to use split files - Update migration guide with new Step 5 explaining the split - Add prominent note explaining the file split rationale 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Component inputs stored in variables weren't being properly interpolated, causing scripts to receive literal "$[[ inputs.xxx ]]" strings instead of actual values. This fix replaces all variable references with direct input interpolation in scripts. Changes: - utility-draft-pr-filter: Remove intermediate variables, use inputs directly - utility-branch-skip: Remove intermediate variables, use inputs directly - base-pipeline: Use inputs directly in .machine-check template script The machine pipeline templates (dane, matrix, corona, tioga, tuolumne, lassen) still use the variables approach as those variables are consumed by child pipeline jobs defined in separate files, making direct input interpolation impractical. According to GitLab documentation, $[[ inputs.xxx ]] should be interpolated at component include time. Using inputs directly in scripts ensures the interpolation happens correctly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
BUILD_ROOT was being set to ${CI_PROJECT_DIR}, but this variable doesn't
exist when component inputs are resolved (too early in the pipeline
lifecycle), causing BUILD_ROOT to be empty in child pipelines.
The BUILD_ROOT variable is project-specific and not needed by the
components themselves. Projects can define it locally if needed, but
it should not be part of component inputs.
Changes:
- Remove build_root input spec from all machine pipeline components
- Remove BUILD_ROOT variable assignment from all machine pipelines
- Replace ${BUILD_ROOT} with /usr/workspace/${USER} in default reproducer
scripts for each machine (sensible default).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
The intermediate variable usage is permitted when components are used in a child pipeline. We leverage this as it allows to override this variable value in a given CI job for customization purpose.
The component version should be specified in the component path using the @Version syntax with a variable (e.g., @$RADIUSS_SHARED_CI_REF), not as a component input.
adrienbernede
changed the title
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR introduces GitLab CI Components as the recommended way to consume RADIUSS Shared CI configuration, replacing the legacy include-based approach. The components provide better versioning, type-safe inputs, cleaner syntax, and improved discoverability through the GitLab CI/CD Catalog.
- Adds component templates for all machine pipelines (Lassen, Dane, Matrix, Corona, Tioga, Tuolumne) with scheduler-specific configurations
- Implements utility components for draft PR filtering and branch skipping
- Creates comprehensive documentation and migration guide for transitioning from legacy includes to components
Reviewed Changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| templates/*/template.yml | New component templates for all machine pipelines and utilities with standardized input specifications |
| examples/example-*.yml | Example configurations demonstrating component usage patterns |
| docs/sphinx/user_guide/components_migration.rst | Complete migration guide with troubleshooting and component reference documentation |
| docs/sphinx/user_guide/setup_ci.rst | Updated to remove obsolete BUILD_ROOT variable reference |
| catalog-info.yml | GitLab CI/CD Catalog metadata for component discovery |
| README.md | Updated with component quick start guide and available components list |
| customization/gitlab-ci.yml | Removed obsolete BUILD_ROOT variable from legacy template |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tdrwenski
reviewed
Nov 4, 2025
Collaborator
tdrwenski
left a comment
tdrwenski
left a comment
There was a problem hiding this comment.
Wow, this is really cool! :D I am going to test it out on Quandary!
Co-authored-by: Tara Drwenski <[email protected]>
Co-authored-by: Copilot <[email protected]>
adrienbernede
commented
Nov 4, 2025
Member
Author
adrienbernede
left a comment
adrienbernede
left a comment
There was a problem hiding this comment.
Taking action after reviews.
In example-custom-variables we set sensible defaults for allocation variables. In the machine specific components we don't set any default for job alloc, forcing the users to find what's best for their use case.
adrienbernede
changed the title
tdrwenski
reviewed
Nov 4, 2025
tdrwenski
reviewed
Nov 4, 2025
Instead of suggesting that users should create a custom-variable file with variables for each set of allocation parameters, we define sensible defaults for the corresponding inputs of each component. Users only need to change the input value, and might choose to gather such values definition in dedicated file using variables (RADIUSS projects do so for maintenance purpose).
We define a default for those templates so that user can get started with using Shared CI more easily. Customization is still possible by overriding the template, in a dedicated file include in the child pipeline, or in each .gitlab/jobs/<machine>.yml. We adapted the docs, the examples, and fixed some things.
Only run perf jobs for a given machine in allocation information was provided. By default the allocation information is empty, which will prevent the jobs from running on this machine. We split the config to create machine specific templates that allow to apply the same behavior on other jobs the users may create for this pipeline. This is different than the `ON_MACHINE` variable since it only applies to the performance pipeline and is not meant to be used globally: e.g. pipeline jobs for a given machine cannot be turned off from the UI.
tdrwenski
reviewed
Dec 2, 2025
tdrwenski
approved these changes
Dec 5, 2025
Collaborator
tdrwenski
left a comment
tdrwenski
left a comment
There was a problem hiding this comment.
I had a few minor documentation related suggestions but overall looks good!
| Migrating to GitLab CI Components | ||
| =================================== | ||
|
|
||
| Starting with v2025.11.0, RADIUSS Shared CI provides GitLab CI Components as the |
Collaborator
There was a problem hiding this comment.
Maybe check what release you want mentioned here and in the the components examples below (and throughout the template.yml examples)
tdrwenski
reviewed
Dec 5, 2025
Co-authored-by: Tara Drwenski <[email protected]>
Co-authored-by: Tara Drwenski <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Migrate to GitLab CI Components
Summary
This PR migrates the radiuss-shared-ci templates to GitLab CI Components (requires GitLab 17.0+). Components provide better versioning, type-safe inputs, and cleaner syntax compared to the previous template-based approach.
Motivation
The previous template-based CI system had several limitations:
GitLab CI Components address these issues by providing:
Changes Made
Rationale: Component input interpolation ($[[ inputs.xxx ]]) happens at include time. Storing these in intermediate variables and then referencing them caused the literal string $ [[ inputs.xxx ]] to be stored instead of the actual value.
Rationale: BUILD_ROOT was set to ${CI_PROJECT_DIR} but evaluated to empty string because CI_PROJECT_DIR doesn't exist at component input resolution time. Investigation revealed BUILD_ROOT was never actually used in practice and is project-specific,
not something that should be managed by shared components.
Rationale: Component version should be specified in the component path using @Version syntax (e.g., @$RADIUSS_SHARED_CI_REF), not as a component input. Projects define RADIUSS_SHARED_CI_REF as a variable and use it consistently across all component
includes.
Component Structure
The migration creates the following components:
Breaking Changes
For Projects Migrating to Components
variables:
RADIUSS_SHARED_CI_REF: "v2025.10.0"
include:
- component: $CI_SERVER_FQDN/radiuss/radiuss-shared-ci/base-pipeline@$RADIUSS_SHARED_CI_REF
3. Split configuration files: Projects should split .gitlab/custom-jobs-and-variables.yml into:
- .gitlab/custom-jobs.yml - Job templates (included in child pipelines)
- .gitlab/custom-variables.yml - Machine allocation variables (included in parent pipeline)
Migration Status
The following projects have been migrated to use these components, tests are ongoing:
Testing
All fixes have been validated through the migration of the above projects. Each project's migration includes:
Compatibility