Skip to content

Conversation

gmertes
Copy link
Member

@gmertes gmertes commented Oct 1, 2025

Description

Fixing an edge case in the external_graph runner when used with a LAM model. When using this runner with the graph_dataset option, it will patch the dataloader.dataset entry:

graph_ds = open_dataset(graph_dataset)
LOG.info(
"The external graph was built using a different anemoi-dataset than that in the checkpoint. "
"Patching metadata to ensure correct data loading."
)
self.checkpoint._metadata.patch(
{
"config": {"dataloader": {"dataset": graph_dataset}},
"dataset": {"shape": graph_ds.shape},
}
)

But if the existing dataloader.dataset entry in the metadata happens to be a list of datasets, the patching crashes. When patching an existing entry, the patch function did not check if the receiving entry is also a dictionary. In the case of a list on the receiving side, it would try to index the list with the incoming dictionary keys. This PR fixes that.

Also adds a test for this function.

As a contributor to the Anemoi framework, please ensure that your changes include unit tests, updates to any affected dependencies and documentation, and have been tested in a parallel setting (i.e., with multiple GPUs). As a reviewer, you are also responsible for verifying these aspects and requesting changes if they are not adequately addressed. For guidelines about those please refer to https://anemoi.readthedocs.io/en/latest/

By opening this pull request, I affirm that all authors agree to the Contributor License Agreement.

@gmertes gmertes merged commit ff222e2 into main Oct 8, 2025
85 of 91 checks passed
@github-project-automation github-project-automation bot moved this from To be triaged to Done in Anemoi-dev Oct 8, 2025
@gmertes gmertes deleted the fix/metadata-patch-list branch October 8, 2025 09:34
gmertes pushed a commit that referenced this pull request Oct 15, 2025
🤖 Automated Release PR

This PR was created by `release-please` to prepare the next release.
Once merged:

1. A new version tag will be created
2. A GitHub release will be published
3. The changelog will be updated

Changes to be included in the next release:
---


##
[0.8.0](0.7.3...0.8.0)
(2025-10-15)


### Features

* Add requirements subcommand
([#245](#245))
([fd0877c](fd0877c))
* Better support for dynamic forcings and coupling in prepml
([#262](#262))
([fd0877c](fd0877c))
* Lazy torch loading
([#277](#277))
([fd0877c](fd0877c))
* Replace no grad with inference mode
([#331](#331))
([4b7c495](4b7c495))
* Support empty input files
([#342](#342))
([8078a89](8078a89))


### Bug Fixes

* Consolidate path behaviour on outputs
([#339](#339))
([ba8faf1](ba8faf1))
* **metadata:** Patching when receiving entry is not a dict
([#338](#338))
([ff222e2](ff222e2))
* Netcdf incorrect shape when using extract post-processors
([#346](#346))
([95c23ae](95c23ae)),
closes [#341](#341)
* **parallel runner:** Support base class with main_argument decorator
([#333](#333))
([7ec5cce](7ec5cce))
* Update cutout due to
[#262](#262)
([#334](#334))
([35c1cdb](35c1cdb))


### Miscellaneous Chores

* Bump 0.8.0
([ec25c99](ec25c99))

---
> [!IMPORTANT]
> Please do not change the PR title, manifest file, or any other
automatically generated content in this PR unless you understand the
implications. Changes here can break the release process.
> ⚠️ Merging this PR will:
> - Create a new release
> - Trigger deployment pipelines
> - Update package versions

 **Before merging:**
 - Ensure all tests pass
 - Review the changelog carefully
 - Get required approvals

[Release-please
documentation](https://github.com/googleapis/release-please)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants