String coords broken in pandas git tip

[crusaderky]

DataArray.expand_dims with the new coords passed as keyword arguments recently broke in pandas nightly.

import xarray

ab = xarray.DataArray([1, 2], dims=['x'], coords={'x': ['a', 'b']})
c = xarray.DataArray(3).expand_dims(x=["c"])
# c.coords["x"] = c.coords["x"].values
arr = xarray.concat([ab, c], dim="x")

crashes with

TypeError: Cannot interpret '<StringDtype(storage='python', na_value=nan)>' as a data type

Uncommenting the third line, which overwrites the coord with a numpy array copy of itself, works around the problem.

It broke somewhere between 3.0.0.dev0+2231 and 3.0.0.dev0+2248; I did not bisect it exactly.

[crusaderky]

>>> ab._coords['x']._data
# both pandas 2.3.1 and 3.0.0.dev0+2248:
PandasIndexingAdapter(array=Index(['a', 'b'], dtype='str', name='x'), dtype=dtype('<U1'))
>>> c._coords['x']._data
# pandas 2.3.1:
PandasIndexingAdapter(array=Index(['c'], dtype='object', name='x'), dtype=dtype('O'))
# pandas 3.0.0.dev0+2248:
PandasIndexingAdapter(array=Index(['c'], dtype='str', name='x'), dtype=<StringDtype(storage='python', na_value=nan)>)

[crusaderky]

This does not seem to be limited to expand_dims.
I'm observing more use cases where a coord created from a pandas MultiIndex and then unstacked fails with the same error.

[crusaderky]

dtype=<StringDtype(storage='python', na_value=nan)> is a pandas.StringDtype, not to be confused with numpy.dtypes.StringDType (note capitalization difference 🙄 )

[dcherian]

@ilan-gold do you have bandwidth to look at this? I think you have most context around the pandas dtypes.

[ilan-gold]

Sure @dcherian I can take a look.

[ilan-gold]

@crusaderky So the crux of the issue is what pd.Index does with an input that is of type <U1 or object (or likely other similar things). It appears that it now yields a str dtype (which seems good?) but previously was just an object dtype.

It would appear that this line of code is where this change occurred and is then percolating outward to our various extension array handling locations:

xarray/xarray/core/indexes.py

Line 538 in fb49a3b

index = pd.Index(to_numpy(array), **kwargs)

What we do about this issue is a different story. I think the best thing is probably to first make a PR ensuring the old behavior is maintained.

Going forward, though, I am not sure how much of an essential method casting lists to indexes is so not sure we want something deeper.

I would guess that that deeper thing would involve removing coord_dtype but @dcherian could probably give more insight into why it is there in the first place. It seems mostly internal and only used for real in normalize_label within PandasIndex.sel - I can't find any instance of my_indexing_object.coord_dtype in the codebase except in tests for asserting that it exists and matches something else. If we could remove that, we probably could rely on the machinery in #10423 to handle concatenating sensibly and other operations while really preserving dtypes.

I think there are other issues with pandas>=3 but at least this issue has a solution.

Also, it appears xarray has nightly tests, but is this not covered? I see other failures locally but the latest CI on main appears to have passed?