Skip to content

ENH: improve support for datetime columns #486

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 65 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
aaf8818
ENH: deal properly with naive datetimes with arrow
theroggy Oct 17, 2024
3e463a1
Add more testcases, also for tz datetimes
theroggy Oct 18, 2024
afdd0c1
Merge remote-tracking branch 'upstream/main' into ENH-deal-properly-w…
theroggy Jan 16, 2025
c18ab22
Use datetime_as_string for reading with arrow
theroggy Jan 17, 2025
597855f
Update _io.pyx
theroggy Jan 17, 2025
fa4b86e
Skip tests where appropriate
theroggy Jan 17, 2025
0e41ae4
Improve support for mixed and naive datetimes
theroggy Jan 17, 2025
1378ace
Skip use_arrow tests with old gdal versions
theroggy Jan 17, 2025
0f1ab27
Take in account pandas version
theroggy Jan 17, 2025
6f78c68
Update test_geopandas_io.py
theroggy Jan 17, 2025
336d0d8
Also support columns with datetime objects
theroggy Jan 18, 2025
3035a11
Rename some test functions for consistency
theroggy Jan 18, 2025
9efdc09
Avoid warning in test
theroggy Jan 18, 2025
eb80e08
Improve inline comment
theroggy Jan 18, 2025
d50b2d0
Update CHANGES.md
theroggy Jan 18, 2025
47aa298
Merge remote-tracking branch 'upstream/main' into ENH-deal-properly-w…
theroggy Jan 19, 2025
1efa5bf
Symplify code
theroggy Jan 20, 2025
0032839
Don't cast UTC data to string when writing
theroggy Jan 20, 2025
9d2bfce
Various improvements to tests
theroggy Jan 20, 2025
ca9a8ae
Smal fixes to tests
theroggy Jan 20, 2025
deb862c
Xfail some tests where needed
theroggy Jan 20, 2025
e35c356
Make UTC assert more specific
theroggy Jan 22, 2025
593b282
Revert "Make UTC assert more specific"
theroggy Jan 22, 2025
35d8d87
Update test_geopandas_io.py
theroggy Jan 22, 2025
41c9da6
Use astype("string") instead of apply
theroggy Jan 23, 2025
f53af87
Improve tests
theroggy Jan 23, 2025
a8c85b7
Fix tests for older versions
theroggy Jan 23, 2025
40ca1a5
Update test_geopandas_io.py
theroggy Jan 23, 2025
fc53d44
Merge remote-tracking branch 'upstream/main' into ENH-deal-properly-w…
theroggy Jan 30, 2025
458d75b
Merge
theroggy Apr 23, 2025
8a38961
Merge remote-tracking branch 'upstream/main' into ENH-deal-properly-w…
theroggy May 13, 2025
9c764eb
Merge remote-tracking branch 'upstream/main' into ENH-deal-properly-w…
theroggy Aug 2, 2025
e0273b5
Add parameter to specify if dates need to be read as UTC or not
theroggy Aug 2, 2025
91027d1
Fix tests for old gdal versions
theroggy Aug 3, 2025
46f7847
Treat Obkect column as datetime if the first non-null value is a date…
theroggy Aug 3, 2025
8f7d853
Add support to return datetimes as string
theroggy Aug 3, 2025
9398ae4
Convert to string for older GDAL versions
theroggy Aug 3, 2025
3d3e3da
Fixes to tests for old gdal versions
theroggy Aug 3, 2025
097d85a
Fix tests for older GDAL versions
theroggy Aug 3, 2025
41be45c
Fixes in tests for old GDAL versions
theroggy Aug 3, 2025
4f27049
Fix tests for older gdal versions
theroggy Aug 3, 2025
cf9eba9
Fix tests
theroggy Aug 3, 2025
86239e2
Fix tests for gdal < 3.7
theroggy Aug 3, 2025
7d28e3d
Test fixes for gdal <3.7
theroggy Aug 3, 2025
201c01e
Fix linter error
theroggy Aug 3, 2025
4923178
Several textual improvements
theroggy Aug 3, 2025
f0e7409
Add test for dates long ago
theroggy Aug 3, 2025
8117e96
Merge remote-tracking branch 'upstream/main' into ENH-deal-properly-w…
theroggy Aug 3, 2025
538b95f
Improve docstring of test with dates from long ago
theroggy Aug 3, 2025
f23bd38
Fix long ago test for old versions of gdal
theroggy Aug 3, 2025
b55cc2f
xfail dates of long ago with arrow and gdal <3.11
theroggy Aug 3, 2025
616a144
Fix some errors with pandas 3
theroggy Aug 3, 2025
93dbc6e
Improve error
theroggy Aug 4, 2025
6af6d63
Support for pandas 3.0
theroggy Aug 4, 2025
066ec42
Update geopandas.py
theroggy Aug 4, 2025
5c9efa1
Support pandas 3.0
theroggy Aug 4, 2025
b421a06
Support pandas 3.0
theroggy Aug 4, 2025
4486c9e
Update test_geopandas_io.py
theroggy Aug 5, 2025
a3a0393
Update test_geopandas_io.py
theroggy Aug 5, 2025
69dcc6a
Merge branch 'ENH-deal-properly-with-naive-datetimes-with-arrow' of h…
theroggy Aug 5, 2025
7248d1b
Small textual improvements
theroggy Aug 5, 2025
86529a2
Typo in changelog
theroggy Aug 5, 2025
8303c05
Fix UTC mode so it is backwards compatible
theroggy Aug 7, 2025
ac5f20f
Fix for pandas 3.0
theroggy Aug 7, 2025
23bf348
Improve inline doc.
theroggy Aug 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# CHANGELOG

## 0.12.0 (????-??-??)

### Improvements

- Add `datetimes` parameter to `read_dataframe` to choose the way datetime columns are
returned + several fixes when reading and writing datetimes (#486).

## 0.11.1 (2025-08-02)

### Bug fixes
Expand Down
22 changes: 18 additions & 4 deletions pyogrio/_io.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -936,10 +936,16 @@ cdef process_fields(

if datetime_as_string:
# defer datetime parsing to user/ pandas layer
# Update to OGR_F_GetFieldAsISO8601DateTime when GDAL 3.7+ only
data[i] = get_string(
OGR_F_GetFieldAsString(ogr_feature, field_index), encoding=encoding
)
IF CTE_GDAL_VERSION >= (3, 7, 0):
data[i] = get_string(
OGR_F_GetFieldAsISO8601DateTime(ogr_feature, field_index, NULL),
encoding=encoding,
)
ELSE:
data[i] = get_string(
OGR_F_GetFieldAsString(ogr_feature, field_index),
encoding=encoding,
)
else:
success = OGR_F_GetFieldAsDateTimeEx(
ogr_feature,
Expand Down Expand Up @@ -1502,6 +1508,7 @@ def ogr_open_arrow(
int return_fids=False,
int batch_size=0,
use_pyarrow=False,
datetime_as_string=False,
):

cdef int err = 0
Expand Down Expand Up @@ -1722,6 +1729,12 @@ def ogr_open_arrow(
"GEOARROW".encode("UTF-8")
)

# Read DateTime fields as strings, as the Arrow DateTime column type is
# quite limited regarding support for mixed timezones,...
IF CTE_GDAL_VERSION >= (3, 11, 0):
if datetime_as_string:
options = CSLSetNameValue(options, "DATETIME_AS_STRING", "YES")

# make sure layer is read from beginning
OGR_L_ResetReading(ogr_layer)

Expand Down Expand Up @@ -1749,6 +1762,7 @@ def ogr_open_arrow(
"crs": crs,
"encoding": encoding,
"fields": fields[:, 2],
"dtypes": fields[:, 3],
"geometry_type": geometry_type,
"geometry_name": geometry_name,
"fid_column": fid_column,
Expand Down
8 changes: 8 additions & 0 deletions pyogrio/_ogr.pxd
Original file line number Diff line number Diff line change
Expand Up @@ -415,6 +415,14 @@ IF CTE_GDAL_VERSION >= (3, 6, 0):
)


IF CTE_GDAL_VERSION >= (3, 7, 0):

cdef extern from "ogr_api.h":
const char* OGR_F_GetFieldAsISO8601DateTime(
OGRFeatureH feature, int n, char** papszOptions
)


IF CTE_GDAL_VERSION >= (3, 8, 0):

cdef extern from "ogr_api.h":
Expand Down
Loading
Loading