fix: cap max_workers in insert_documents

[Robbe-Superlinear]

Changes:

1. Cap default ingestion worker threads with a heuristic: when max_workers is None, with a cap of 4.

[Copilot]

Pull Request Overview

This PR adds a default value heuristic for the max_workers parameter in the insert_documents function to prevent resource oversubscription. When max_workers is not explicitly provided, it now defaults to the minimum of CPU count, document count, and a cap of 4 workers.

Key Changes:

• Introduced MAX_DEFAULT_WORKERS constant set to 4 to cap thread pool workers
• Added logic to automatically set max_workers based on CPU count, document count, and the defined cap when not explicitly provided
• Added os import to support os.cpu_count() call

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.