Skip to content

Conversation

jngz-es
Copy link
Collaborator

@jngz-es jngz-es commented Jun 20, 2025

Description

Refresh ML key from the index to keep the consistence between memory and index which is under a flag. In some edge cases, we could get divergence between the key cached in memory and in the system index. Currently we only have to restart the node to re-fetch the key from the index to mitigate the issue. With the fix, we can only use a flag to control.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@jngz-es
Copy link
Collaborator Author

jngz-es commented Jun 20, 2025

I am adding more IT for cover the fix.

@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 18:58 — with GitHub Actions Error
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 18:58 — with GitHub Actions Failure
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 18:58 — with GitHub Actions Failure
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 18:58 — with GitHub Actions Error
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 19:13 — with GitHub Actions Error
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 19:13 — with GitHub Actions Failure
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 19:13 — with GitHub Actions Failure
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 19:13 — with GitHub Actions Error
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 19:20 — with GitHub Actions Failure
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 19:20 — with GitHub Actions Error
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 19:20 — with GitHub Actions Error
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env June 20, 2025 19:20 — with GitHub Actions Failure
@jngz-es jngz-es temporarily deployed to ml-commons-cicd-env June 20, 2025 20:24 — with GitHub Actions Inactive
@jngz-es jngz-es temporarily deployed to ml-commons-cicd-env June 20, 2025 20:24 — with GitHub Actions Inactive
@jngz-es jngz-es temporarily deployed to ml-commons-cicd-env July 1, 2025 18:04 — with GitHub Actions Inactive
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env July 1, 2025 18:04 — with GitHub Actions Failure
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env July 1, 2025 18:04 — with GitHub Actions Error
@jngz-es jngz-es temporarily deployed to ml-commons-cicd-env July 1, 2025 19:08 — with GitHub Actions Inactive
@jngz-es jngz-es temporarily deployed to ml-commons-cicd-env July 1, 2025 19:08 — with GitHub Actions Inactive
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env July 1, 2025 20:09 — with GitHub Actions Failure
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env July 1, 2025 20:09 — with GitHub Actions Error
@jngz-es jngz-es temporarily deployed to ml-commons-cicd-env July 1, 2025 21:00 — with GitHub Actions Inactive
@jngz-es jngz-es temporarily deployed to ml-commons-cicd-env July 1, 2025 21:01 — with GitHub Actions Inactive
@jngz-es jngz-es temporarily deployed to ml-commons-cicd-env July 3, 2025 20:12 — with GitHub Actions Inactive
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env July 3, 2025 20:12 — with GitHub Actions Error
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env July 3, 2025 20:12 — with GitHub Actions Failure
@jngz-es jngz-es temporarily deployed to ml-commons-cicd-env July 3, 2025 20:12 — with GitHub Actions Inactive
Zhangxunmt
Zhangxunmt previously approved these changes Jul 3, 2025
@dhrubo-os
Copy link
Collaborator

In some edge cases, we could get divergence between the key cached in memory and in the system index.

Is this possible to explain this more?

Does that mean node has a stale cached key? We create key only one time. How node's key information can be different?

Also how important is it to refresh in a regular interval? Seems like one time refresh is required, why do we need to keep refreshing?

@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env July 3, 2025 23:36 — with GitHub Actions Error
@jngz-es jngz-es had a problem deploying to ml-commons-cicd-env July 3, 2025 23:36 — with GitHub Actions Failure
Signed-off-by: Jing Zhang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants