Invariant Signal Collection for Kubernetes Testing #5197
Conversation
Change-Id: Ic5d1b1f8c0e33bf1ef9ad320f46982f1c5587451
k8s-ci-robot
added
the
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: aojea The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
kind/kep
|
@aojea: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
|
||
| ## Summary | ||
|
|
||
| This proposal defines a system to gather and analyze invariant signals about the Kubernetes cluster during test execution. We want to see how key parts of the system are behaving in a way that's similar to how real users would see it. This will help us find problems that normal tests might miss, and give us a better picture of how stable and reliable Kubernetes is. |
There was a problem hiding this comment.
Nit: please wrap lines.
There was a problem hiding this comment.
Commented in #5085.
You can set your browser width to control wrapping client side, or review the rendered markdown which will flow independently of any manual line wrap anyhow.
There was a problem hiding this comment.
(This also allows you to control the width, and avoids having the lines awkwardly broken up if I choose a different width)
There was a problem hiding this comment.
Viewing wasn't the reason - see #5085 (comment)
There was a problem hiding this comment.
hahha, I just put the PR as placeholder to not forget and copied pasted from my google doc, sorry about that.
My editor wraps the lines, is that this time I didn't use it
There was a problem hiding this comment.
to be clear, this is a mistake and I agree with Patrick on line wrapping
|
|
||
| 3. Storing and Processing Invariant Data: | ||
|
|
||
| We will use existing tools to store the invariant data in a database that is easy to search. |
There was a problem hiding this comment.
I don't think that's sufficient, we can't tell people "go pay to run bigquery to search for results". (and do so periodically)
This is why I'm still hesitant to abandon the approach of reporting a pass/fail, which we already disseminate in various forms including email alerts.
If we're going to commit to building a frontend for these results instead, we should be discussing that here.
There was a problem hiding this comment.
reporting of tests is making this reporting a problem of everyone, and when is a problem of everyone then is a problem of noone ... if there is interest on this I expect people to participate on creating and maintaining it , otherwise we'll add more complexity and flakiness to an under maintained area
|
|
||
| 4. (Future) Looking at the Invariant Data: | ||
|
|
||
| We will create dashboards to show the invariant data in a way that's easy to understand. |
There was a problem hiding this comment.
Which look like ..? And use what?
metrics/ is ~dead. There are no dashboards available for this. It lacked maintainers. There are ... some questionable JSON files, that nobody uses, that are not rendered anywhere.
There was a problem hiding this comment.
and why there will be the reason that adding it to the test we'll not end in the same situation?
I don't find sustainable to add new failures modes to the CI without a plan of how to maintain it
There was a problem hiding this comment.
we should have alternatives considered, e.g. an opt-in "runs last" test that reports when the metric is tripped
there's a substantial tradeoff in latency of reporting this result, and complexity of implementing the system, versus attempting to silently gather from all jobs using barely maintained systems and having people poll for results
|
/close @BenTheElder I'm going to close this and reassign to you, since you have a better proposal |
|
@aojea: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
One-line PR description: Create and define a system for reporting and tracking invariant violations detected during Kubernetes testing.
Issue link: Invariant Signal Collection for Kubernetes Testing #5196
Other comments: