feat(ci): add Bedrock integration tests with record/replay #4292

skamenan7 · 2025-12-03T19:08:02Z

What does this PR do?

Adds Bedrock integration tests to CI using a record/replay mechanism. Tests run against pre-recorded API responses, without the need for AWS credentials in CI.

The main challenge was that Bedrock's OpenAI-compatible API doesn't support everything - no tool calling, no embeddings, no dynamic model listing. So instead of running the full base suite (which would fail on ~40 tests), I created a dedicated bedrock suite with just the tests that actually work.

Changes:

New run-bedrock.yaml stack config with dummy API key for replay mode
bedrock suite in suites.py pointing to 3 test functions (6 parametrized tests total)
Config resolution fix for distro::file.yaml format in library mode
Added a docs/source/providers/inference/bedrock_recording_guide.md so contributors with AWS access can re-record tests when needed

Closes #4095

Test Plan

Run from tests/integration/inference:


  uv run pytest -v \
    test_openai_completion.py::test_openai_chat_completion_non_streaming \
    test_openai_completion.py::test_openai_chat_completion_streaming \
    test_openai_completion.py::test_inference_store \
    --setup=bedrock \
    --stack-config=ci-tests::run-bedrock.yaml \
    --inference-mode=replay \
    -k "client_with_models"

Expected: 6 passed

test_openai_completion.py::test_openai_chat_completion_non_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_com
 pletion:non_streaming_01] PASSED [ 16%]
 test_openai_completion.py::test_openai_chat_completion_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_complet
 ion:streaming_01] PASSED [ 33%]
 test_openai_completion.py::test_inference_store[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-True] PASSED                          [
  50%]
 test_openai_completion.py::test_openai_chat_completion_non_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_com
 pletion:non_streaming_02] PASSED [ 66%]
 test_openai_completion.py::test_openai_chat_completion_streaming[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-inference:chat_complet
 ion:streaming_02] PASSED [ 83%]
 test_openai_completion.py::test_inference_store[client_with_models-txt=bedrock/openai.gpt-oss-20b-1:0-False] PASSED
 [100%]

src/llama_stack/distributions/ci-tests/run-bedrock.yaml

docs/source/providers/inference/bedrock_recording_guide.md

src/llama_stack/testing/api_recorder.py

tests/integration/ci_matrix.json

mergify · 2025-12-10T14:14:18Z

This pull request has merge conflicts that must be resolved before it can be merged. @skamenan7 please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

docs/docs/contributing/testing/record-replay.mdx

src/llama_stack/providers/remote/inference/bedrock/config.py

src/llama_stack/distributions/ci-tests/config.yaml

tests/integration/conftest.py

skamenan7 · 2025-12-19T18:21:02Z

The failing tests (ollama, gpt, backward-compat) are caused by a test infrastructure issue with stream_options hashing, not by Bedrock code. This is being fixed in #4326. All Bedrock-specific tests pass. Will rebase after #4326 merges.

skamenan7 · 2025-12-22T18:31:19Z

Please review. CI is now green. cc:@derekhiggins and @cdoern

- Create dedicated bedrock suite with 3 compatible test functions - Add run-bedrock.yaml stack config for CI - Enable config resolution for distro::file.yaml format in library mode - Add test recordings for streaming, non-streaming, and inference store - Skip tool-calling tests for Bedrock (not supported by AWS) - Add recording guide documentation for contributors Tested with GPT-OSS model on us-west-2 region.

The bedrock suite uses specific test function paths like "test_file.py::test_function" in its roots. The pytest_ignore_collect hook was treating these as filesystem paths, causing 0 tests to be collected. Changes: - Strip "::test_function" suffix when checking file paths - Add pytest_collection_modifyitems to filter to specific tests Without this fix, cleanup_recordings.py marks bedrock recordings as unused and deletes them in CI.

- Remove separate run-bedrock.yaml in favor of modifying templates - Update bedrock provider config: dummy API key for replay mode, us-west-2 region - Pre-register bedrock model in ci-tests template (Bedrock /v1/models returns empty) - Update ci_matrix.json to use default stack config

Consolidated config file usage (ci-tests::run.yaml instead of run-bedrock.yaml), added Stack Configuration Choices section, fixed default region to us-west-2, and updated pytest commands to use full paths with clarification about test count.

Changed default region from us-east-2 to us-west-2 to reflect where GPT-OSS model is available. Update test expectation to match.

- Replace non-existent run-bedrock.yaml with config.yaml in recording guide - Replace non-existent run.yaml with config.yaml in record-replay.mdx - Fix test count from "4 parametrized test cases" to "6 parametrized tests" 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>

leseb

mostfly nit, thanks for your patience on this one

leseb · 2026-01-09T15:42:59Z

docs/source/providers/inference/bedrock_recording_guide.md

@@ -0,0 +1,263 @@
+# AWS Bedrock Test Recording Guide


this file looks very similar to docs/docs/contributing/testing/record-replay.mdx with more content, is there any chance to just keep one?

leseb · 2026-01-09T15:44:23Z

src/llama_stack/distributions/starter-gpu/config.yaml

    config:
-      api_key: ${env.AWS_BEARER_TOKEN_BEDROCK:=}
-      region_name: ${env.AWS_DEFAULT_REGION:=us-east-2}
+      api_key: ${env.AWS_BEARER_TOKEN_BEDROCK:=replay-mode-dummy-key}


can avoid the replay-mode-dummy-key and keep things the way they were before the change?

meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 3, 2025

skamenan7 force-pushed the feat/bedrock-ci-4095 branch 7 times, most recently from 305be8e to 0667de8 Compare December 3, 2025 22:46

derekhiggins reviewed Dec 4, 2025

View reviewed changes

skamenan7 force-pushed the feat/bedrock-ci-4095 branch 3 times, most recently from 69eb2c4 to 51121ae Compare December 4, 2025 20:13

skamenan7 marked this pull request as ready for review December 4, 2025 21:36

skamenan7 requested review from ashwinb, bbrowning, cdoern, ehhuang, franciscojavierarceo, leseb, mattf and raghotham as code owners December 4, 2025 21:36

skamenan7 requested a review from derekhiggins December 4, 2025 22:07

skamenan7 force-pushed the feat/bedrock-ci-4095 branch 3 times, most recently from bd9f2dd to d75be15 Compare December 9, 2025 14:35

mergify bot added the needs-rebase label Dec 10, 2025

skamenan7 force-pushed the feat/bedrock-ci-4095 branch 2 times, most recently from 2d82f01 to 5067cee Compare December 10, 2025 14:16

mergify bot removed the needs-rebase label Dec 10, 2025

skamenan7 force-pushed the feat/bedrock-ci-4095 branch 6 times, most recently from 1d07a5a to 454aefe Compare December 16, 2025 15:32

derekhiggins suggested changes Dec 16, 2025

View reviewed changes

skamenan7 force-pushed the feat/bedrock-ci-4095 branch 4 times, most recently from 1db1c2b to 58fe994 Compare December 18, 2025 16:12

skamenan7 requested a review from derekhiggins December 18, 2025 16:32

skamenan7 force-pushed the feat/bedrock-ci-4095 branch 2 times, most recently from 1d92f48 to f3b96b6 Compare December 19, 2025 18:19

skamenan7 force-pushed the feat/bedrock-ci-4095 branch 2 times, most recently from 4fb5b72 to a0e27f0 Compare December 22, 2025 18:24

skamenan7 force-pushed the feat/bedrock-ci-4095 branch 3 times, most recently from 2b6aaa2 to 4a14dae Compare January 8, 2026 13:57

skamenan7 and others added 7 commits January 9, 2026 09:45

test: update bedrock config test to match new defaults

e9c561c

test: update bedrock config test to match new defaults

062e81b

Changed default region from us-east-2 to us-west-2 to reflect where GPT-OSS model is available. Update test expectation to match.

skamenan7 force-pushed the feat/bedrock-ci-4095 branch from 4a14dae to 61dcea1 Compare January 9, 2026 14:45

leseb requested changes Jan 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(ci): add Bedrock integration tests with record/replay #4292

feat(ci): add Bedrock integration tests with record/replay #4292

Uh oh!

skamenan7 commented Dec 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Dec 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

skamenan7 commented Dec 19, 2025

Uh oh!

skamenan7 commented Dec 22, 2025

Uh oh!

leseb left a comment

Uh oh!

leseb Jan 9, 2026

Uh oh!

leseb Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat(ci): add Bedrock integration tests with record/replay #4292

Are you sure you want to change the base?

feat(ci): add Bedrock integration tests with record/replay #4292

Uh oh!

Conversation

skamenan7 commented Dec 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Dec 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

skamenan7 commented Dec 19, 2025

Uh oh!

skamenan7 commented Dec 22, 2025

Uh oh!

leseb left a comment

Choose a reason for hiding this comment

Uh oh!

leseb Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

leseb Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants