Skip to content

Conversation

@jminor
Copy link
Collaborator

@jminor jminor commented Oct 28, 2025

OTIO's code coverage reports have been stuck in a stale state for a while now. Pull requests report coverage based on a commit from Apr 14, 2024 instead of the current main. For example:

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.10%. Comparing base (c0e97b0) to head (646eaec).
⚠️ Report is 80 commits behind head on main.
❌ Your changes status has failed because you have indirect coverage changes. Learn more about Unexpected Coverage Changes and reasons for indirect coverage changes.
The "base" commit that codecov compares against seems to be this one: c0e97b0 for reasons that are unclear.

In an attempt to get this fixed I have just re-read the instructions https://github.com/codecov/codecov-action, re-entered the repo's CODECOV token in the "secrets" settings area, and in this PR, updated the codecov action to v5.

Let's see if that helps...

Signed-off-by: Joshua Minor <[email protected]>
@github-actions github-actions bot added the ci label Oct 28, 2025
@jminor
Copy link
Collaborator Author

jminor commented Oct 28, 2025

See also: #1838

@codecov-commenter
Copy link

codecov-commenter commented Oct 28, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.11%. Comparing base (829a913) to head (c6e235f).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1961      +/-   ##
==========================================
+ Coverage   85.10%   85.11%   +0.01%     
==========================================
  Files         181      181              
  Lines       12768    12768              
  Branches     1206     1206              
==========================================
+ Hits        10866    10868       +2     
+ Misses       1719     1717       -2     
  Partials      183      183              
Flag Coverage Δ
py-unittests 85.11% <ø> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 829a913...c6e235f. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jminor
Copy link
Collaborator Author

jminor commented Oct 28, 2025

Reading a bit more about how this works, I suspect that the coverage for PRs is working okay, but the coverage upload after a PR lands in the main branch is not. That's why codecov is able to compare this PR's head but not to main (since it hasn't received an upload from main since 80 commits ago). Maybe?

If I look at a recent codecov task on the main branch, I see some suspicious warnings: https://github.com/AcademySoftwareFoundation/OpenTimelineIO/actions/runs/18852215970/job/53791315602

Warning 1:

gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.

Warning 2:

Could not pull latest version information: SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON

Warning 3:

warning - 2025-10-27 18:44:12,891 -- xcrun is not installed or can't be found.
warning - 2025-10-27 18:44:12,925 -- Running gcov on the following list of files:
<warnings about lots of files>

Despite these warnings, it says "Process Upload complete" at the end:

info - 2025-10-27 18:44:15,105 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/ssteinbach/OpenTimelineIO/commit/60171a4a3f4fc102c609c77743ffe6efae68bc54
info - 2025-10-27 18:44:15,669 -- Process Upload complete

Of all those, the one I'm most suspicious of is the last one. Why is it uploading to an URL with ssteinbach in it? That PR isn't from that user. Maybe the reports are successfully uploading but to the wrong place? Or maybe this is just an artifact left over from the fact that @ssteinbach set this up from the beginning?

@jminor
Copy link
Collaborator Author

jminor commented Oct 28, 2025

The verbose logs from this PR look slightly better:
https://github.com/AcademySoftwareFoundation/OpenTimelineIO/actions/runs/18865561345/job/53832413869?pr=1961

I still see Warning 1:

gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.

I do not see Warning 2 (something about DOCTYPE).

I still see Warning 3 (maybe trying to detect macOS?)

warning - 2025-10-28 06:00:33,566 -- xcrun is not installed or can't be found.
...
warning - 2025-10-28 06:00:33,601 -- Running gcov on the following list of files:
<warnings about lots of files>

The ending looks more like I expected:

debug - 2025-10-28 06:00:35,037 -- Sending upload request to Codecov
info - 2025-10-28 06:00:35,357 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/AcademySoftwareFoundation/OpenTimelineIO/commit/c9f03d10c9ea17bf8132570301aaf53c9d42cb72
debug - 2025-10-28 06:00:35,357 -- Upload request to Codecov complete. --- {"response": {"raw_upload_location": "https://storage.googleapis.com/codecov-production/shelter/github/AcademySoftwareFoundation%3A%3A%3A%3AOpenTimelineIO/c9f03d10c9ea17bf8132570301aaf53c9d42cb72/79cecc1f-12fe-4110-8715-c8b5f0b7e459.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJWQHUGIBILH4J7Q6ZUSCIFNEOLYSNDS7L3B4N5SIBQ2J4YLYE5CRFCD%2F20251028%2Fus%2Fs3%2Faws4_request&X-Amz-Date=20251028T060035Z&X-Amz-Expires=30&X-Amz-SignedHeaders=host&X-Amz-Signature=4ecc58aaf64ab1a1eeac276bcc212e684e14ef1dd0a8f7dadfc11d59c4dcca48", "url": "https://app.codecov.io/github/AcademySoftwareFoundation/OpenTimelineIO/commit/c9f03d10c9ea17bf8132570301aaf53c9d42cb72"}}
debug - 2025-10-28 06:00:35,358 -- Sending upload to storage
info - 2025-10-28 06:00:35,639 -- Process Upload complete
debug - 2025-10-28 06:00:35,639 -- Upload result --- {"result": "RequestResult(error=None, warnings=[], status_code=200, text='')"}

Notably it says AcademySoftwareFoundation instead of a username.

Copy link

@jfpanisset jfpanisset left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When using third party GitHub Actions, it is recommended to pin to a specific version using the commit SHA of the corresponding commit.

So instead of:

uses: codecov/codecov-action@v5

it is preferred to use:

uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1

where you can find out the commit SHA on the release page for the action

@jfpanisset
Copy link

Also there may be codecov (possibly expired?) secrets at the AcademySoftwareFoundation GitHub organization level that are getting in the way of what you are trying to do?

I'd suggest opening a LF helpdesk ticket to ask what Codecov secrets may be installed at the org level?

@jminor
Copy link
Collaborator Author

jminor commented Oct 28, 2025

A more recent PR now also says says AcademySoftwareFoundation instead of a username in the upload confirmation. I think that means that the updated token I set in this repo's settings is aiming those uploads to a better place.
https://github.com/AcademySoftwareFoundation/OpenTimelineIO/actions/runs/18879771529/job/53879313553

Also, when I navigate here: https://app.codecov.io/github/AcademySoftwareFoundation/OpenTimelineIO now Codecov's dashboard shows "Source: latest commit 829a913" from a few hours ago, where it used to show the commit from a year ago.

However, the most recent PR (also from a few hours ago) still shows the "80 commits behind" stale base: #1964 Maybe that was generated before 829a913 landed?

Let's see what the next PR reports. It might be fixed now? 🙏

@jminor
Copy link
Collaborator Author

jminor commented Oct 28, 2025

When using third party GitHub Actions, it is recommended to pin to a specific version using the commit SHA of the corresponding commit.

Thanks for the suggestion @jfpanisset. Once we get it working properly, we can pin it (and our other actions).

@jminor
Copy link
Collaborator Author

jminor commented Oct 28, 2025

I rebased this PR onto latest main and the coverage report looks to be fixed!!

@darbyjohnston
Copy link
Contributor

Nice, It definitely looks better. Should we check this in and test it with a PR that has some C++ changes? I did notice this warning is still there, but I don't know how important it is:

gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.

@jminor
Copy link
Collaborator Author

jminor commented Oct 29, 2025

So my conclusion is that the problem was related to the Codecov token stored in this repo's secrets. Updating that token fixed the problem.

The changes in this PR are helpful/harmless, but were not needed to fix the issue. Specifically it updates to v5 of the action and moves the coverage report to a newer version of Python.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants