Adding a wheel management API

[agronholm]

I'm the maintainer of wheel. I'm wondering if you would be interested in receiving an API for reading and writing wheels in this project. I've been wondering for a while if it makes any sense for PyPA to have so many packaging projects.

This is what I had in mind:

1. The new "wheel public API" would go into this project instead of wheel
2. setuptools would adopt this API and ditch wheel as a dependency for building wheels
3. wheel would depend on packaging at run time (currently it vendors this project) and become nothing but a CLI tool
4. (optional) auditwheel and wheel would be merged (preferably into wheel) (optional, to be discussed later)

The goal is to get rid of at least one PyPA project.

[brettcannon]

Do you have an issue or something that shows what you're proposing as a "wheel public API"?

And for reading or writing wheel files, we have purposefully taken a sans-I/O approach, so it would be more like "processing and creating zipfile.ZipFile instances that represent wheel files".

[agronholm]

There's a long time draft PR here: pypa/wheel#472

And for reading or writing wheel files, we have purposefully taken a sans-I/O approach, so it would be more like "processing and creating zipfile.ZipFile instances that represent wheel files".

But ZipFile does I/O, so how does that work? And is there ongoing work towards this in packaging already?

[brettcannon]

There's a long time draft PR here: pypa/wheel#472

It looks like https://github.com/pypa/wheel/pull/472/files#diff-8ca627810a744afb59bbb1d28a71f6ee66c9878506cfbbf2b6b980053a13c2b8 is the key bit. We already deal with wheel file names, so the key bit seems to be the parts that parse/create all the details of a wheel file.

But ZipFile does I/O, so how does that work?

ZipFile can do whatever it wants, but the point is we as a project wouldn't directly be doing any I/O and would be defining more of an interface that we would work with.

And is there ongoing work towards this in packaging already?

Nope. Closest thing we're working on right now is parsing metadata.

[agronholm]

ZipFile can do whatever it wants, but the point is we as a project wouldn't directly be doing any I/O and would be defining more of an interface that we would work with.

Ok, so to clarify, does that mean you don't want to have an I/O implementation in this project? I was hoping to reduce the amount of separate projects, and having wheel only host a thin I/O implementation + a CLI tool that few people use doesn't seem sensible to me.

[pradyunsg]

FWIW, an abstracted interface for interacting with wheels was created as part of the installer project (https://installer.pypa.io/en/stable/concepts/).

[pradyunsg]

I am doubtful that we can create a sans-io implementation of the wheel format -- wheels can and do contain files that are GBs in size, which would make operating in-memory with such wheels memory-intensive, if we insist on operating with only complete strings/bytes. If we're doing things incrementally, it's better to rely on existing I/O patterns.

Notably, that's basically what we have with https://github.com/pypa/installer/blob/5f281b8e312bac4da00670b643dd4499e1c66a9a/src/installer/sources.py#L139 and even the wheel public API PR ~basically implements that interface (thanks @agronholm for doing that!).

[brettcannon]

@pradyunsg if installer already has an API then I think it more sense to rely on that API then us moving it here and duplicating it or going through the hassle of deprecating in installer. Thoughts?

[dstufft]

packaging should probably not depend on installer?

I suspect wheel also shouldn't depend on installer?

[abravalheri]

Probably installer also does not cover the "creating" part of the API thar wheel covers, just the "reading" part.

[brettcannon]

packaging should probably not depend on installer?

Correct, and I wasn't proposing that. We actually can't have any dependencies at this point.

[pradyunsg]

Quoting https://installer.pypa.io/en/stable/concepts/ (about WheelSource):

This protocol/abstraction is designed to be implementable without a direct dependency on this library. This allows for other libraries in the Python packaging ecosystem to provide implementations of the protocol, allowing for more code reuse opportunities.

Do I get a cookie for this not being needless overengineering? 🙃

---

Probably installer also does not cover the "creating" part of the API thar wheel covers, just the "reading" part.

That's correct. The installation usecase only cares about reading, so that was the scoping there.

[pradyunsg]

I'm wondering if you would be interested in receiving an API for reading and writing wheels in this project.

I just realised that this was never answered directly -- yes, I think that's a good idea. If @brettcannon and @dstufft agree (or at least are OK with going down this route), we'd have concensus among that packaging maintainers. :)

[brettcannon]

I'm wondering if you would be interested in receiving an API for reading and writing wheels in this project.

I just realised that this was never answered directly -- yes, I think that's a good idea. If @brettcannon and @dstufft agree (or at least are OK with going down this route), we'd have concensus among that packaging maintainers. :)

Sorry, missed the notification for this comment!

My only real concern is starting to have to care about file I/O which we have avoided thus far. If we can manage to keep a sans-I/O approach then I'm fine with it. If we can't then I think we need to discuss what that change in stance means and how we are going to keep it under control (i.e., I don't want to take on any networking, for instance).