Incremental parquet writing

[michel-aractingi]

What this does

Branched from #1894, fixed tests and added lazy loading

[sandhawalia]

any opposition to walrus operator := ?

if (writer := getattr(self, "writer", None)):

[michel-aractingi]

Not at all. I just think we don't use it anywhere else in the code and for readability its a bit better

[Copilot]

Pull Request Overview

This PR implements incremental parquet writing functionality for LeRobot datasets, improving performance during data collection by avoiding expensive file reloads and memory operations between episodes.

Key changes include:

• Implements PyArrow-based incremental parquet writing with persistent writers
• Adds lazy loading mechanism to defer expensive dataset loading until actually needed for reading
• Introduces finalize() method to properly close parquet writers and complete dataset creation

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/lerobot/datasets/lerobot_dataset.py	Core implementation of incremental parquet writing, lazy loading, and writer management
src/lerobot/datasets/video_utils.py	Adds finalize() call to properly close writers in video recording context
tests/datasets/test_datasets.py	Updates test cases to call finalize() method after dataset creation

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}