CI: tags releases #589
CI: tags releases #589
Conversation
you have clearly gone through CI hell for us 😅
Was a fun read, and good to know the pain you went through for this.
I'll have a good at that in a bit 👍 No idea what exactly my rights in the repo are, if I cant do it ill ensure someone who is autorized does it. |
|
@est31 I do not have access to the settings page, can you set up the workflow token? |
|
I have added a
|
|
lets see if it works |
|
nope :( logs give me: it seems to fail on: any idea @romainpi ? no rush btw, I'm about to go to sleep anyways, have a good night all! 🌙 |
Hmm yes I have a few ideas as to why it might be failing. I'm on mobile now with no access to my computer but I'll be able to look into it within 18 hours. Also I'm not able to see the details of the failing job (access rights perhaps?) . Could I ask for a full log of the |
|
The raw log: its pretty large2024-07-05T23:26:03.4752184Z Current runner version: '2.317.0' |
|
Hello, @dvdsk thanks for the log. It appear you may have sent me the logs for the It looks like est31 may have created a "New fine-grained personal access token (beta)" not a "New personal access token (classic)" as specified in the PR description. I'm saying this because the screenshot here looks like the UI for a newly created fine-grained access token. @est31 if you follow exactly these steps (you may not have read them as they were included deep inside my wall-of-text PR description) the workflow should hopefully work correctly.
Once this is done, rerunning the failed jobs should work. I'm not 100% sure if est31 added the If you would like me to find a solution using the new fine grained tokens I'm sure it can be made to work. Currently I'm not sure that only the "Workflow RW rights" + "Metadata read" rights (as they appear in the screenshot) are enough for it to work - it could possibly require the "Contents RW" rights for it to work. I can definitely look into it if you think the classic personal access tokens are going to be phased out by Github or if you're not comfortable having that wide personal access token exposed by a Github secrets leak. I would recommend setting the expiration date of the token to "No expiration" since the token will only live in est31's clipboard and in the Github secrets - it's not like they are being shared between people or put in some temporary line of codes (if you trust the Github secrets...). Otherwise we may forget about the token expiring and when publishing a new release the
I made a mistake with this statement, I had of course tested on I apologize for this seemingly simple PR blowing up into something more complex than it should be. PS: Maybe a bit ambitious but I may later suggest revamps to the Rodio CI / release process. As it stands every commit to PPS: Perhaps we could set up a beta release line on crates.io and a Thanks! |
yes, since the failed for the same reason I send only one, to me the create-git-tag one seemed more important as the other one I can apply locally. Though I forgot to mention any of this to you 😅
No worries, I put all the blame on Microsoft where it belongs (since they build a terrible CI thing, anyway back to topic).
Valid those on the other hand two haven't coincided yet and if they do its not that big a deal (just wait till crates.io is back up).
To be honest that seems like it would increase the burden on the few maintainers rodio has. Semi off-topic: |
The latter please, I don't want to give this repository an access token that is valid for all repositories my account has access to.
it's not hard to extract the token once you have repository access to the rodio account. The UI limitation only keeps away the weakly determined hackers. There is a million ways to extract it, only the most obvious ones got protection from github. |
This PR is a solution to issue #587.
This PR contains two new jobs in
.github/workflows/ci.yml:create-tags-for-past-releasesthat will create and push tags for previous releases ofRodio.create-git-tagthat will detect when theversionfield in the[package]section of the./Cargo.tomlfile is modified on themasterbranch, create a tag and push it.The ephemeral job
create-tags-for-past-releasescontains a JSON array storing release version numbers and their corresponding git commit hash. Usingjqit loops through the JSON array and tags the commits and then finally pushes the tags.I have tested these changes and they are working correctly on the
self-hostedworkers.The next step will be to remove the ephemeral job
create-tags-for-past-releasesfrom.github/workflows/ci.yml.Important !!!
For some reason, Github is not letting the script push some tags with the following error message:
! [remote rejected] v0.17.3 -> v0.17.3 (refusing to allow a GitHub App to create or update workflow `.github/workflows/ci.yml` without `workflows` permission)It's a little puzzling since the action is not touching the
.github/workflows/ci.yml. The only clue is the error first occurs with the tagv0.13.1which is when the CI was migrated from Travis to Github actions.In order to have this work, one of the maintainers needs to create a Personal Access Token with workflow rights, they will need to:
workflowcheckbox, save and copy the token to clipboard.WORKFLOW_TOKENand paste the token that should be in the clipboard.I would recommend setting the expiration date of the token to "No expiration" since the token will only be put into Github secrets. The script will check for the existence of this token and will fail if not set.
Notes
For the
Rodioproject the logic that defines a new release version is as follow:Looking at the
.github/workflows/ci.ymlfile where the logic is implemented we can see that thecargo-publishjob runs for all commits that are pushed to themasterbranch. The job will try to publish to thecrates.ioregistry; it should succeed when trying to publish with a new version number and it should fail when trying to publish with a version number that was already present on thecrates.ioregistry.Here comes a long useless explanation. TLDR: I thought I could parse the git history, realized I couldn't then cheated by doing it manually.
At first I tried building a bash script that would go through the commit history detecting changes to the
versionfield in the[package]section of the./Cargo.tomlfile and tag those with the value in theversionfield. That worked with some of the releases (the ones that had been pushed directly to themasterbranch) but failed by flagging commits that were parts of a PR merge; these new versions would only be published to the registry when the PR was merged to themasterbranch, and sometimes the PR branches contained posterior changes to the code so tagging those commits where the version changed was sometimes incorrect - even in the case that the commit changing the version was the last commit in the PR the tag would point to the correct codebase corresponding to that release but not the actual commit and that would be inaceptable;).I then tried many different techniques such as having the bash script go through all the descending commits of these version changing commits and try to detect PR merge commits with
git log --reverse --merges --ancestry-path <hash>..HEAD(and other commands) and reading the commit subject looking for "Merge pull request #". I will spare you the details but it got messy and I'm starting to forget the myriad of problems I ran into - for a little while it looked like I was making progress but I got issues with merges rebasingmasterto a PR branch, I did things like counting the number of children and parents commits and making assumptions. The bash script was starting to grow in size, it didn't look good. In the end I had something that would sometimes mislabel release commits that had been pushed directly to themasterbranch, would sometimes tag the same commit with multiple versions, etc.The
0.0.1release was impossible to detect since this was the starting value there was no way of discerning which commit had triggered the publish, and the subsequent commits working on0.0.2were not easily detectable. That's when I thought of scraping the data off of https://crates.io/api/v1/crates/rodio which doesn't include commit hashes but has acreated_atvalue. I thought I could perhaps compare datetimes of the published versions with commits but it wasn't the solution - firstly nothing guarantees that the commiter's machine date can be wrongly set; we can't tell how much time the commiter waited to push his commits to github; we don't know how long it takes the workers to start working, etc. But at least we had an idea of the day that the release was published.In the end I gave up on the bash script that tries to detect version commits. It started to be too much code and since it's ideally meant to be run once there was no reason to keep going down that route. I' sure a git savant might be able to solve this with 3 lines of code. Anyways, with all the almost-correct info I had generated and scraping
https://crates.io/api/v1/crates/rodioto get theversionsandcreated_atvalues, I ended up manually identifying and confirming some releases. Nevermind the fact I finally read the message from @est31 stating most of the commit hashes were available from https://lib.rs/crates/rodio/versions but that was already too late!