GH-47728: [Python] Check the source argument in parquet.read_table #48008
+27
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
raulcd
reviewed
Oct 31, 2025
| if isinstance(source, list): | ||
| raise ValueError( | ||
| "the 'source' argument cannot be a list of files " | ||
| "when the pyarrow.dataset is not available" |
There was a problem hiding this comment.
Suggested change
| "when the pyarrow.dataset is not available" | |
| "when the pyarrow.dataset module is not available" |
| # TODO test that source is not a directory or a list | ||
| try: | ||
| source = filesystem.open_input_file(path) | ||
| except (OSError, FileNotFoundError) as e: |
There was a problem hiding this comment.
Will filesystem.open_input_file raise OSError and/or FileNotFoundError only when source is not a file and is a dir? Could for example raise OSError when the process doesn't have permission to read the file or others?
If one of those exceptions can raise on other cases, we might be misleading users as we will always raise a ValueError with the description pointing to the dir vs file issue.
We might want to check with something like (I haven't tested just checked at the filesystem API):
filesystem.get_file_info(path).is_file()
github-actions
bot
added
awaiting changes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
See #47728. Check
sourceargument inpyarrow.parquet.read_tableifpyarrow.datasetis not available.What changes are included in this PR?
Check the
sourceargument, raiseValueErrorif thesourceargument is either a list of.parquetfiles or a directory.Are these changes tested?
Yes
Are there any user-facing changes?
No
In case if the
sourceargument is a directory, I decided not to check it directly, but to catch the exceptions coming from thefs.open_input_file, since it already checks for it, and add extra exception on top of the stack that explains the actual reason.