'gh ado2gh rewire-pipeline': support test rewiring mode before actual rewiring

[georgy-gorelko]

A comprehensive test mode for the gh ado2gh rewire-pipeline command that covers the scenarios below:

Overview

The pipeline testing feature allows to validate Azure DevOps pipelines before permanently migrating them to GitHub. It provides two testing modes:

1. Single Pipeline Test (--dry-run): Test a specific pipeline
2. Batch Pipeline Test (test-pipelines command): Test multiple pipelines concurrently

Single Pipeline Testing

Usage

gh ado2gh rewire-pipeline --ado-org "MyOrg" --ado-team-project "MyProject" \
  --ado-pipeline "MyPipeline" --github-org "MyGitHubOrg" --github-repo "MyRepo" \
  --service-connection-id "12345" --dry-run --monitor-timeout-minutes 45

Parameters

• --dry-run: Enables test mode (temporarily rewires, tests, then restores)
• --monitor-timeout-minutes: How long to wait for build completion (default: 30 minutes)

Process Flow

1. Retrieve Pipeline Information: Gets current pipeline configuration and repository details
2. Temporary Rewiring: Points pipeline to GitHub repository temporarily
3. Build Execution: Queues a test build and captures the build ID
4. Restoration: Automatically restores pipeline to original ADO repository
5. Build Monitoring: Monitors build progress until completion or timeout
6. Report Generation: Provides detailed test results and recommendations

Sample Output

Starting dry-run mode: Testing pipeline rewiring to GitHub...
Step 1: Retrieving pipeline information...
Pipeline ID: 123
Original ADO Repository: MyAdoRepo
Step 2: Temporarily rewiring pipeline to GitHub...
✓ Pipeline successfully rewired to GitHub
Step 3: Queuing a test build...
Build queued with ID: 456
Build URL: https://dev.azure.com/MyOrg/MyProject/_build/results?buildId=456
Step 4: Restoring pipeline back to original ADO repository...
✓ Pipeline successfully restored to original ADO repository
Step 5: Monitoring build progress (timeout: 30 minutes)...
Build completed with result: succeeded

=== PIPELINE TEST REPORT ===
ADO Organization: MyOrg
ADO Team Project: MyProject
Pipeline Name: MyPipeline
Build Result: succeeded
✅ Pipeline test PASSED - Build completed successfully

Batch Pipeline Testing

Usage

gh ado2gh test-pipelines --ado-org "MyOrg" --ado-team-project "MyProject" \
  --github-org "MyGitHubOrg" --github-repo "MyRepo" --service-connection-id "12345" \
  --pipeline-filter "CI-*" --max-concurrent-tests 3 --report-path "test-results.json"

Parameters

• --pipeline-filter: Wildcard pattern to filter pipelines (e.g., "CI-", "-Deploy", "*")
• --max-concurrent-tests: Maximum number of concurrent tests (default: 3)
• --report-path: Path to save detailed JSON report (default: pipeline-test-report.json)
• --monitor-timeout-minutes: Timeout for each pipeline test (default: 30 minutes)

Process Flow

1. Pipeline Discovery: Finds all pipelines matching the filter criteria
2. Concurrent Testing: Runs multiple pipeline tests simultaneously (respecting concurrency limits)
3. Automatic Restoration: Ensures all pipelines are restored to ADO
4. Progress Monitoring: Tracks each pipeline's test progress independently
5. Comprehensive Reporting: Generates both console summary and detailed JSON report

Sample Output

Starting batch pipeline testing...
Step 1: Discovering pipelines...
Found 15 pipelines to test
Step 2: Testing pipelines (max concurrent: 3)...
Testing pipeline: CI-Frontend (ID: 101)
Testing pipeline: CI-Backend (ID: 102)
Testing pipeline: CI-Database (ID: 103)
...

=== PIPELINE BATCH TEST SUMMARY ===
Total Pipelines Tested: 15
Successful Builds: 12
Failed Builds: 2
Timed Out Builds: 1
Rewiring Errors: 0
Restoration Errors: 0
Success Rate: 80.0%
Total Test Time: 01:45:32

Report Structure

JSON Report Format

{
  "TotalPipelines": 15,
  "SuccessfulBuilds": 12,
  "FailedBuilds": 2,
  "TimedOutBuilds": 1,
  "ErrorsRewiring": 0,
  "ErrorsRestoring": 0,
  "TotalTestTime": "01:45:32",
  "SuccessRate": 80.0,
  "Results": [
    {
      "AdoOrg": "MyOrg",
      "AdoTeamProject": "MyProject",
      "AdoRepoName": "MyAdoRepo",
      "PipelineName": "CI-Frontend",
      "PipelineId": 101,
      "PipelineUrl": "https://dev.azure.com/MyOrg/MyProject/_build/definition?definitionId=101",
      "BuildId": 201,
      "BuildUrl": "https://dev.azure.com/MyOrg/MyProject/_build/results?buildId=201",
      "Status": "completed",
      "Result": "succeeded",
      "StartTime": "2025-01-01T10:00:00Z",
      "EndTime": "2025-01-01T10:15:00Z",
      "BuildDuration": "00:15:00",
      "RewiredSuccessfully": true,
      "RestoredSuccessfully": true,
      "IsSuccessful": true
    }
  ]
}

Error Handling and Recovery

Automatic Recovery

The testing system is designed to be resilient:

• Restoration Guarantee: Pipelines are always restored to ADO, even if tests fail
• Concurrent Safety: Multiple tests run safely without interfering with each other
• Timeout Handling: Tests that exceed timeout limits are automatically terminated
• Error Isolation: One failed test doesn't affect others in batch mode

Manual Recovery

If automatic restoration fails:

1. Check the console output for "MANUAL RESTORATION REQUIRED" messages
2. Use the provided Pipeline ID and URL to manually restore the pipeline
3. Review the detailed JSON report for specific error information

Best Practices

Before Testing

1. Verify Service Connection: Ensure the GitHub service connection works properly
2. Check Repository Access: Confirm the GitHub repository is accessible from ADO
3. Review Branch Strategy: Ensure the default branch exists in both repositories
4. Test with Small Batch: Start with a few pipelines before testing all

During Testing

1. Monitor Progress: Watch console output for any immediate issues
2. Check Build Status: Use provided build URLs to monitor detailed progress
3. Resource Management: Limit concurrent tests based on your ADO capacity

After Testing

1. Review Results: Analyze both successful and failed test results
2. Address Issues: Fix identified problems before permanent migration
3. Document Findings: Use test results to plan migration strategy
4. Validate Restoration: Ensure all pipelines were properly restored

Troubleshooting

Common Issues

Build Failures

• Check GitHub repository exists and is accessible
• Verify service connection has proper permissions
• Ensure YAML pipeline files are compatible

Restoration Failures

• Verify ADO permissions are sufficient
• Check if original repository still exists

Timeout Issues

• Increase --monitor-timeout-minutes for longer builds
• Check if builds are queued but not starting
• Verify agent pools are available

Support Information

For issues or questions about pipeline testing:

1. Review the detailed error messages in console output
2. Check the JSON report for specific failure details
3. Verify permissions and connectivity to both ADO and GitHub

[Copilot]

Pull Request Overview

This PR adds comprehensive test mode functionality to the gh ado2gh rewire-pipeline command that allows validation of Azure DevOps pipelines before permanently migrating them to GitHub.

• Adds --dry-run flag to the rewire-pipeline command for single pipeline testing
• Introduces new test-pipelines command for batch testing multiple pipelines concurrently
• Implements pipeline test monitoring with automatic restoration to ensure pipelines are always restored to their original ADO repositories

Reviewed Changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/ado2gh/Program.cs	Registers new AdoPipelineTriggerServiceFactory dependency
src/ado2gh/Factories/AdoPipelineTriggerServiceFactory.cs	Factory for creating AdoPipelineTriggerService instances
src/ado2gh/Commands/TestPipelines/*	New command implementation for batch pipeline testing
src/ado2gh/Commands/RewirePipeline/*	Updates to support dry-run mode and use new trigger service
src/Octoshift/Services/PipelineTestService.cs	Service for testing individual pipelines
src/Octoshift/Services/AdoPipelineTriggerService.cs	Service for managing pipeline trigger configurations
src/Octoshift/Services/AdoApi.cs	Adds new API methods for pipeline testing operations
src/Octoshift/Models/*	New models for pipeline test results and ADO branch policies
test files	Comprehensive test coverage for new functionality
docs/pipeline-testing.md	Documentation for the new testing features

Comments suppressed due to low confidence (3)

src/Octoshift/Services/AdoPipelineTriggerService.cs:431

• The method returns 'triggers' instead of 'result' which is the variable being built up in this method. This will return the original triggers array instead of the new YAML-controlled triggers.

    }

src/Octoshift/Services/AdoPipelineTriggerService.cs:192

• Same issue as line 162 - the URL is missing the base URL prefix. It should use $"{_adoBaseUrl}/{adoOrg.EscapeDataString()}..." to match the pattern used in other API calls.

        {

---

_{You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.}

[github-actions]

Unit Test Results

  1 files    1 suites   10m 25s ⏱️
932 tests 932 ✅ 0 💤 0 ❌
933 runs  933 ✅ 0 💤 0 ❌

Results for commit 0bfd38d.

♻️ This comment has been updated with latest results.

[georgy-gorelko]

Need to finish #1412 first

[georgy-gorelko]

@begonaguereca please review this next

[georgy-gorelko]

@boylejj can I get Integration Tests scheduled and validated so PR can be merged?

[brianaj]

What is the value add on test-pipelines command, is this still calling the rewire pipeline logic?

My only concern is it says Pipelines are always restored to ADO, even if tests fail but then says Manual restoration may be required. I am okay with dry-run behavior otherwise but that seems like a risk we should make very clear.

@begonaguereca might be good for you to take a look at this too.

[georgy-gorelko]

What is the value add on test-pipelines command, is this still calling the rewire pipeline logic?

My only concern is it says Pipelines are always restored to ADO, even if tests fail but then says Manual restoration may be required. I am okay with dry-run behavior otherwise but that seems like a risk we should make very clear.

@begonaguereca might be good for you to take a look at this too.

Yes, it does rewire from ADO to GitHub, schedule pipeline, rewire back to ADO and monitor build status. We need this exercise to ensure rewiring works as expected. For bulk migrations (100+ repos) this automation helpful to report any deviations found

[georgy-gorelko]

@begonaguereca, @brianaj can I get pipelines scheduled for validation and review?

[brianaj]

Good with this change, with the noted caveat, a user might have to manually revert back after rewiring in dry-run mode.

[brianaj]

@georgy-gorelko I will be out of office soon so @begonaguereca or whatever team member has bandwidth will have to take this over the finish line.

[georgy-gorelko]

Good with this change, with the noted caveat, a user might have to manually revert back after rewiring in dry-run mode.

No manual rewiring required. Sorry for confusion, I will update the description

[georgy-gorelko]

@begonaguereca please schedule Integration Tests and if all good, please proceed with merging

[georgy-gorelko]

@offbyone any chance you may know how to trigger Integration Test pipelines pending and required by policy?

[georgy-gorelko]

@begonaguereca can you help with validation pipelines to be scheduled?

[begonaguereca]

@georgy-gorelko its triggered here https://github.com/github/gh-gei/actions/runs/17864476137

[georgy-gorelko]

@georgy-gorelko its triggered here https://github.com/github/gh-gei/actions/runs/17864476137

Is there a way to debug/repro failures locally? Logs provided do not emit much details of the failures. Not sure if known or new failures appear.

[github-actions]

Package	Line Rate	Branch Rate	Complexity	Health
gei	80%	72%	574	✔
bbs2gh	83%	77%	653	✔
ado2gh	71%	69%	700	➖
Octoshift	83%	72%	1714	✔
Summary	80% (7628 / 9493)	73% (1799 / 2481)	3641	✔

[georgy-gorelko]

@begonaguereca any hints on Integration tests debuging/scheduling?

[begonaguereca]

@georgy-gorelko do you have access to slack at all? we have chatops commands to look more deeply at the failed migration ids eg: RM_kgDaACQ4MDY3NDNkNy1mZjM1LTQ5NTctODExNC02NTA0YTI5NWI2OGQ

[georgy-gorelko]

@georgy-gorelko do you have access to slack at all? we have chatops commands to look more deeply at the failed migration ids eg: RM_kgDaACQ4MDY3NDNkNy1mZjM1LTQ5NTctODExNC02NTA0YTI5NWI2OGQ

not in Slack

[begonaguereca]

Ok you can query failure reason this way as well https://docs.github.com/en/graphql/reference/objects#repositorymigration

@georgy-gorelko

[georgy-gorelko]

Ok you can query failure reason this way as well https://docs.github.com/en/graphql/reference/objects#repositorymigration
I need more guidance here to get the details.