Skip to content

Conversation

FlipperPA
Copy link

@FlipperPA FlipperPA commented Jul 14, 2025

Work-in-progress to fix #53

Large result sets that grow beyond the 32-bit 4gb limit currently error with IO Error: Failed to close entry. This sets the minizip-ng 64-bit flag to MZ_ZIP64_AUTO, which will switch to 64-bit ZIP only when necessary.

Compiled and tested on Ubuntu 24.04 and Rocky 9.

@FlipperPA FlipperPA changed the title Add 64-bit Support for Large xlsx Files WIP: Add 64-bit Support for Large xlsx Files Jul 14, 2025
@FlipperPA
Copy link
Author

FlipperPA commented Jul 14, 2025

I've gotten this compiled, and successfully writing files, but it seems to be writing ZIP64 even when it isn't necessary, despite using MZ_ZIP64_AUTO. This causes certain apps to be able to open it (Google Sheets, polars, etc) and others to reject it (O365 Web Excel):

$ head -c 64 new_export.xlsx | xxd
00000000: 504b 0304 2d00 0800 0800 0000 0000 0000 PK..-...........
00000010: 0000 ffff ffff ffff ffff 1800 1400 786c ..............xl
00000020: 2f77 6f72 6b73 6865 6574 732f 7368 6565 /worksheets/shee
00000030: 7431 2e78 6d6c 0100 1000 0000 0000 0000 t1.xml..........

$ head -c 64 v1.3.0_export.xlsx | xxd
00000000: 504b 0304 1400 0800 0800 0000 0000 0000 PK..............
00000010: 0000 0000 0000 0000 0000 1800 0000 786c ..............xl
00000020: 2f77 6f72 6b73 6865 6574 732f 7368 6565 /worksheets/shee
00000030: 7431 2e78 6d6c bc9d 5f93 1c37 72ed 9faf t1.xml.._..7r...

Any thoughts or suggestions would be greatly appreciated!

@FlipperPA FlipperPA force-pushed the feature/ai/zip64-auto branch from c6e5162 to ef6707c Compare July 14, 2025 19:47
@FlipperPA
Copy link
Author

I've opened up an issue with minizip-ng to see if I'm missing anything: zlib-ng/minizip-ng#894 (comment)

@FlipperPA FlipperPA force-pushed the feature/ai/zip64-auto branch from ef6707c to 9f5be73 Compare July 15, 2025 20:51
@Maxxen
Copy link
Member

Maxxen commented Jul 16, 2025

Hey! Thanks again for digging into this. Perhaps we could also add an option to the COPY function to control this behavior, similar to how other libraries seem to solve this. E.g. `COPY table TO 'file.xlsx' (FORMAT XLSX, ZIP64 [AUTO / NO / YES]). In which case if its set to NO (or FALSE) we throw an error if it exceeds the size (or catch and format the minigzip-ng error)

Although It doesn't have to be in this PR, having it default to AUTO might makes sense already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

"IO Error: Failed to close entry" when COPYing from PostgreSQL to EXCEL
2 participants