audit followup: ensure secretmanager secrets and bindings are managed by scripts

[spiffxp]

Caught via #1718 (comment)

I've done the following pattern a few times now (most recently #1696 (comment)):

• someone needs to pass a secret to me, wonders how to transfer it
• I setup a secretmanager secret in the relevant project
  • intended to be k8s secrets in a cluster: the project hosting the cluster
    • kubernetes-public for apps running on aaa
    • k8s-infra-prow-build-trusted for prow jobs or components runnong on prow-build-trusted
    • etc.
• Setup appropriate permissions on the secret (need to figure out a consistent pattern here)
  • Org admins should implicitly get access via their role/owner on the organization
  • Give some sort of "oncall" team ownership of the secret and its versions for break-glass cases
  • Give whomever is handing the secret to us write access to the secret version
• Setup labels on the secret
  • app: foo if its for an app in the foo dir running on aaa
  • group: sig-foo if the app is owned by sig-foo
  • ??? maybe the namespace the secret is destined for?
• Request that for things destined to be k8s secrets, people store an actual secret manifest yaml in the secret

Put it all together and this allows for relatively simple / safe deployment: https://github.com/kubernetes/k8s.io/tree/main/slack-infra#how-to-deploy

Problems with the above:

• it needs to be documented instead of me just happening to kinda/sorta do something consistent
• the secret creation should be scripted

[spiffxp]

/kind cleanup
/kind documentation
/priority important-soon
/area cluster-mgmt
/area access

Labels and areas for relevant sigs:

/sig testing
/area prow
I'd like to set this up as the standard for managing secrets for prow.k8s.io (with possible exception of kubeconfigs that speak to google-internal clusters)

/sig release
/area release-eng
triage-party, publishing-bot, and possibly other secrets

/sig contributor-experience
slack-infra, groups management

/sig scalability
perf-dash

/sig node
node-perf-dash