Pants: Add GHA workflow to regenerate lockfiles #6356
Conversation
the srcset/src of the picture will need to be adjusted once we know how to get the artifacts.
pull-request-size
bot
added
the
size/L
It needs to handle new PR and exiting PR. The existing PR can be from a branch in a fork or in this repo.
It needs to handle new PR and exiting PR. The existing PR can be from a branch in a fork or in this repo.
bs4 improved their typing allowing pylint to catch a possible coding error in this example. Ultimately it would not have been an issue because of the broad try/except block, but this makes pylint happy. The result set can be None, so handle that case explicitly.
…pylint, twine __________________________________________________________________ Lockfile diff: lockfiles/st2.lock [st2] __________________________________________________________________ == Upgraded dependencies == ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ bcrypt 4.3.0 --> 5.0.0 beautifulsoup4 4.13.5 --> 4.14.2 httplib2 0.30.0 --> 0.31.0 psutil 7.0.0 --> 7.1.0 pycparser 2.22 --> 2.23 pynacl 1.5.0 --> 1.6.0 pyyaml 6.0.2 --> 6.0.3 simplejson 3.20.1 --> 3.20.2 st2-auth-backend-flat-file 0.3.0 --> 0.4.1 wcwidth 0.2.13 --> 0.2.14 __________________________________________________________________ == Removed dependencies == ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ passlib 1.7.4 __________________________________________________________________ Lockfile diff: lockfiles/bandit.lock [bandit] __________________________________________________________________ == Upgraded dependencies == ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ pyyaml 6.0.2 --> 6.0.3 __________________________________________________________________ Lockfile diff: lockfiles/pants-plugins.lock [pants-plugins] __________________________________________________________________ == Upgraded dependencies == ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ pyparsing 3.2.3 --> 3.2.5 pyyaml 6.0.2 --> 6.0.3 __________________________________________________________________ Lockfile diff: lockfiles/twine.lock [twine] __________________________________________________________________ == Upgraded dependencies == ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ pycparser 2.22 --> 2.23 No changes to lockfiles/black.lock No changes to lockfiles/flake8.lock No changes to lockfiles/pylint.lock
cognifloyd
force-pushed
the
gha-lockfile-regen
branch
from
3703673 to
6567d3a
Compare
There was a problem hiding this comment.
If you want to run this script, the easiest way is pants run scripts/lockfiles_to_reqs.py
This pulls the locked versions from lockfiles, and replaces the pins in both fixed-requirements.txt and test-requirements.txt with those locked versions. Then it runs the fixate-requirements logic to update the rest of the requirements.txt files.
| lockfile_bytes = strip_comments_from_pex_json_lockfile( | ||
| Path(lockfile).read_bytes() | ||
| ) | ||
| pex_lock = json.loads(lockfile_bytes.decode("utf-8")) | ||
| locked_requirements = pex_lock["locked_resolves"][0]["locked_requirements"] | ||
| locked_reqs_name_version_map = { | ||
| req["project_name"]: req["version"] for req in locked_requirements | ||
| } |
There was a problem hiding this comment.
This is ugly. It relies on implementation details from pants and pex; details that should normally be opaque to us. I look forward to deleting this script and all of the other legacy requirements machinery at some point.
| with: | ||
| aws-region: ${{ env.AWS_REGION }} | ||
| role-session-name: GitHubActions-${{ github.run_id }} | ||
| role-to-assume: arn:aws:iam::${{ env.AWS_ACCOUNT_ID }}:role/GitHubActions@org=${{ github.repository_owner }},repo=${{ github.event.repository.name }} |
There was a problem hiding this comment.
I ended up with this @org=...,repo=... because role names cannot contain / characters.
| # the next one adds a line of overline chars to replace the ansi overline | ||
| /^== .* ==$/{ # heading text line (the matched line goes in pattern space) | ||
| h; # save copy of heading line in "hold" space | ||
| s/./‾/g; # make an line of overline chars to replace the heading's ansi underline |
There was a problem hiding this comment.
The ‾ (unicode "overline" character) is visible in the github UI when I'm using a desktop browser. And it shows up nicely in git log output on my linux machine. On my android phone, it shows up as a box, so the fonts apparently don't have that character. A line of boxes still basically underlines the previous header line, so ‾ was the best I could do with raw unicode chars.
There was a problem hiding this comment.
The changes in this file should improve debugging when auth backends like flat_file aren't working (eg because the wheel on pypi was broken). This is not strictly related to this PR, but it was something I stumbled over while developing it.
I can split it into a separate PR if needed.
|
CI is green! |
This PR does several things:
pants generate-lockfileswith several output processing steps:pants run scripts/lockfiles_to_reqs.py. This is a new script that:make requirementsto update the requirements filesfixed-requirements.txtandtest-requirements.txtpipandsetuptoolsintoMakefilescripts/fixate-requirements.pylogic to update therequirements.txtfilestools/config_gen.pyto ease debugging this issue and hopefully catch errors like this more quickly in the future.examples.forloop_parse_github_reposto resolve an issue revealed by typing updates in the latest versions of BeautifulSoup4.st2clientinstall fixes by @guzzijones from delete queues for integration #6354 that preventModuleNotFoundError: No module named 'ply'Note
I tested in another repo (st2sandbox/st2), because
workflow_dispatchworkflows like this have to be on the default branch to use them.Lockfile Diffs
st2 Lockfile Diff
Lockfile diff: lockfiles/st2.lock (plain text)
(diff from master)
bandit Lockfile Diff
Lockfile diff: lockfiles/bandit.lock (plain text)
(diff from master)
black Lockfile Diff
No changes required for lockfiles/black.lock (from master)
flake8 Lockfile Diff
No changes required for lockfiles/flake8.lock (from master)
pants-plugins Lockfile Diff
Lockfile diff: lockfiles/pants-plugins.lock (plain text)
(diff from master)
pylint Lockfile Diff
No changes required for lockfiles/pylint.lock (from master)
twine Lockfile Diff
Lockfile diff: lockfiles/twine.lock (plain text)
(diff from master)
🤖 GitHub Actions Workflow Run