feat(config): Add support for v1.1 protocol of secrets exec backend

[graphcareful]

Summary

The exec backend within the secrets framework works to fetch secrets by execing a provided path to a binary and communicating with it over a simple (but established) protocol. This protocol is versioned and currently Vector only supports 1 version.

This PR introduces support for v1.1. An example of a program that is compliant with these new changes is the datadog-secrets-backend specifically the changes in the v1 branch which will eventually become part of the new standard.

The main changes to Vector were to include the backend configuration with each request to this binary. This means opening up new configuration options to the secrets global option.

Vector configuration

secret:
  exec_backend:
    type: "exec"
    command: [/usr/bin/datadog-secret-backend]
    protocol:
      version: v1_1
      backend_type: file.json
      backend_config:
        file_path: ~/secrets.json

sources:
  agent:
    type: "datadog_agent"
    address: 0.0.0.0:8282
    log_namespace: true

sinks:
  out:
    inputs:
      - "agent"
    type: "datadog_logs"
    default_api_key: SECRET[exec_backend.dd_logs_api_key]

How did you test this PR?

Using the config about and building the datadog-secrets-backend from source.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

[emilychendd]

LGTM

[pront]

thanks!

[graphcareful]

@pront looks like the unit tests failed on windows, possibly because theres no python installed?

      left: "%1 is not a valid Win32 application. (os error 193)"

[graphcareful]

Change in force-push

• Undo previous accidental force push

[thomasqueirozb]

I think the issue is that make_test_backend is trying to run mock_secrets_exec, which in unix is fine since shebangs are recognized. I don't think Windows deals nicely with shebangs - probably need to add something like #[cfg(windows)] and add python3 to the command

[bruceg]

probably need to add something like #[cfg(windows)] and add python3 to the command

If it is needed for Windows and works on Linux, I'd suggest to make it unconditional for uniformity.

[pront]

I would revert this change: #23655 (comment)

I don't see any real benefit to it. And explicitly calling it will probably fix the the Windows failure.