From e5c6fa483faa2ab09c05a0339d3658db03689dce Mon Sep 17 00:00:00 2001 From: Copilot <198982749+Copilot@users.noreply.github.com> Date: Tue, 12 Aug 2025 10:41:16 +0000 Subject: [PATCH 1/4] Initial plan From 01d4f4b81e8a417bccc8dea0aa3432a5a909da7f Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 12 Aug 2025 10:51:00 +0000 Subject: [PATCH 2/4] Add comprehensive .github/copilot-instructions.md with build, test, and development guidance Co-authored-by: ggorman <5394691+ggorman@users.noreply.github.com> --- .github/copilot-instructions.md | 201 ++++++++++++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100644 .github/copilot-instructions.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000000..27846d1816 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,201 @@ +# Devito Copilot Instructions + +## Repository Overview + +Devito is a Python package for implementing optimized stencil computation (finite differences, image processing, machine learning) from high-level symbolic problem definitions. It builds on SymPy and employs automated code generation and just-in-time compilation to execute optimized computational kernels on CPUs, GPUs, and clusters. + +**Key Facts:** +- Production-stable package (Development Status 5) for scientific computing +- Python 3.10-3.13 support (current development uses 3.12.3) +- ~50MB repository with extensive test coverage +- Primary languages: Python (core), C/C++ (generated code) +- Main dependencies: NumPy, SymPy, cgen, codepy, psutil +- Supports multiple compilation backends: C, OpenMP, OpenACC, MPI + +## Build and Development Environment + +### Prerequisites and Installation + +**System Dependencies (Ubuntu/Debian):** +```bash +sudo apt update +sudo apt install -y python3-numpy python3-sympy python3-psutil python3-pip +sudo apt install -y gcc g++ flake8 +``` + +**Development Installation (preferred method):** +```bash +# Install with all dependencies - requires network access +pip install -e .[tests,extras] --break-system-packages + +# Alternative: conda environment (recommended for full development) +conda env create -f environment-dev.yml +conda activate devito +``` + +**Docker Development (if dependencies fail):** +```bash +docker-compose up devito # Starts Jupyter on port 8888 +# OR +docker build . --file docker/Dockerfile.devito --tag devito_img +``` + +### Environment Variables (Critical for Compilation) +```bash +export DEVITO_ARCH="gcc" # gcc, clang, icx, custom +export DEVITO_LANGUAGE="openmp" # C, openmp, CXX, CXXopenmp +export OMP_NUM_THREADS=2 # For parallel tests +``` + +### Build and Test Commands + +**Linting (ALWAYS run before committing):** +```bash +flake8 --builtins=ArgumentError . +# Configuration in pyproject.toml: max-line-length=90, specific ignore rules +``` + +**Testing:** +```bash +# Core tests (required, ~10-15 minutes) +pytest -k "not adjoint" -m "not parallel" --maxfail=5 tests/ + +# Adjoint tests (advanced feature tests) +pytest -k "adjoint" -m "not parallel" tests/ + +# Example tests (integration validation) +pytest examples/ + +# Specific test file +pytest tests/test_operator.py -v +``` + +**Configuration Check:** +```bash +python3 -c "from devito import configuration; print(''.join(['%s: %s \n' % (k, v) for (k, v) in configuration.items()]))" +``` + +### Common Build Issues and Solutions + +1. **Import errors for cgen/codepy:** Install via pip or use conda environment +2. **Compiler not found:** Set DEVITO_ARCH to match installed compiler +3. **OpenMP issues on macOS:** Install with `brew install llvm libomp` +4. **Network timeouts during pip install:** Use conda or Docker alternatives +5. **Permission issues:** Use `--user` or `--break-system-packages` flags + +## Project Architecture and Layout + +### Core Directory Structure +``` +devito/ # Main source package +├── __init__.py # Package initialization, configuration setup +├── arch/ # Architecture-specific compilation backends +├── builtins/ # Built-in mathematical functions +├── core/ # Core IR (Intermediate Representation) classes +├── data/ # Data structures and memory management +├── finite_differences/# Finite difference operators and stencils +├── ir/ # Intermediate representation passes +├── mpi/ # MPI parallelization support +├── operator/ # Operator compilation and generation +├── passes/ # Compiler optimization passes +├── symbolics/ # Symbolic computation utilities +├── tools/ # Utility functions and helpers +└── types/ # Core types (Grid, Function, Equation, etc.) + +tests/ # Test suite (comprehensive coverage) +examples/ # Tutorials and demonstrations +├── seismic/ # Seismic modeling examples +├── cfd/ # Computational fluid dynamics +├── misc/ # Miscellaneous examples +└── userapi/ # User API demonstrations + +scripts/ # Utility scripts +docker/ # Docker configurations +benchmarks/ # Performance benchmarks +``` + +### Key Configuration Files +- `pyproject.toml`: Build configuration, dependencies, flake8 settings +- `pytest.ini`: Test configuration and options +- `requirements*.txt`: Dependency specifications for different environments +- `environment-dev.yml`: Conda environment specification +- `.github/workflows/`: CI/CD pipelines + +## Continuous Integration and Validation + +### GitHub Workflows (All Must Pass) +1. **pytest-core-nompi.yml**: Core functionality tests across Python 3.10-3.13, multiple GCC versions +2. **pytest-core-mpi.yml**: MPI parallel execution tests +3. **pytest-gpu.yml**: GPU computation tests (OpenACC/CUDA) +4. **flake8.yml**: Code style validation +5. **examples.yml**: Integration tests via example execution +6. **tutorials.yml**: Jupyter notebook validation + +### Pre-commit Validation Steps +```bash +# 1. Lint code +flake8 --builtins=ArgumentError . + +# 2. Run core tests +pytest -k "not adjoint" -m "not parallel" --maxfail=5 tests/ + +# 3. Check configuration +python3 -c "from devito import configuration; print(configuration)" + +# 4. Test example (optional but recommended) +python examples/seismic/acoustic/acoustic_example.py +``` + +### Test Execution Times +- Core tests: 10-15 minutes +- Full test suite: 30-45 minutes +- Individual test files: 1-5 minutes +- Linting: <1 minute +- Examples: 5-10 minutes each + +## Development Guidelines + +### Code Style (Enforced by CI) +- **Line length**: 90 characters maximum +- **Indentation**: 4 spaces (no tabs) +- **Imports**: Grouped (stdlib, third-party, devito) and alphabetically sorted +- **Docstrings**: NumPy-style documentation required for public APIs +- **Naming**: CamelCase classes, snake_case methods/variables + +### Making Changes +1. **Always run linting first**: `flake8 --builtins=ArgumentError .` +2. **Test specific areas**: Run relevant tests before committing +3. **Environment setup**: Ensure DEVITO_ARCH and DEVITO_LANGUAGE are set +4. **Documentation**: Update docstrings for API changes +5. **Examples**: Add/update examples for new features + +### Critical Dependencies +- **SymPy**: Symbolic mathematics (versions 1.12-1.14 supported) +- **NumPy**: Numerical arrays (version 2.x) +- **cgen/codepy**: C code generation and compilation +- **psutil**: System monitoring and resource management + +### Common Workarounds +- **Star imports**: F403/F405 warnings are expected in `__init__.py` due to API design +- **Lambda expressions**: E731 warnings are sometimes acceptable for configuration +- **Import order**: Follow the pattern in `devito/__init__.py` + +## Agent-Specific Instructions + +**Trust these instructions** - they are validated and current. Only search/explore if: +1. Instructions appear outdated or incorrect +2. New functionality is not covered +3. Specific error messages require investigation + +**Time-saving tips:** +1. Use Docker if pip installation fails due to network issues +2. Set environment variables before running any compilation-dependent tests +3. Start with small test files when debugging +4. Check configuration output to verify environment setup +5. Use `--maxfail=5` to stop test runs early on systematic failures + +**File modification priorities:** +1. Core types and operators: `devito/types/`, `devito/operator/` +2. Mathematical operations: `devito/finite_differences/`, `devito/symbolics/` +3. Compilation backends: `devito/arch/`, `devito/passes/` +4. Always update corresponding tests in `tests/` \ No newline at end of file From ca1e196290b73abd1e30365c2e5480ba0676576b Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 12 Aug 2025 11:23:18 +0000 Subject: [PATCH 3/4] docs: Refresh .github/copilot-instructions.md with comprehensive repository review Co-authored-by: ggorman <5394691+ggorman@users.noreply.github.com> --- .github/copilot-instructions.md | 97 +++++++++++++++++++++++++-------- 1 file changed, 74 insertions(+), 23 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 27846d1816..ae9e9c2197 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -5,12 +5,13 @@ Devito is a Python package for implementing optimized stencil computation (finite differences, image processing, machine learning) from high-level symbolic problem definitions. It builds on SymPy and employs automated code generation and just-in-time compilation to execute optimized computational kernels on CPUs, GPUs, and clusters. **Key Facts:** -- Production-stable package (Development Status 5) for scientific computing -- Python 3.10-3.13 support (current development uses 3.12.3) -- ~50MB repository with extensive test coverage +- Production-stable package (Development Status :: 5 - Production/Stable) +- Python 3.10-3.13 support with comprehensive CI testing +- ~290MB repository with extensive test coverage and benchmarking - Primary languages: Python (core), C/C++ (generated code) - Main dependencies: NumPy, SymPy, cgen, codepy, psutil - Supports multiple compilation backends: C, OpenMP, OpenACC, MPI +- Active development with 600+ stars, 238+ forks, and regular releases ## Build and Development Environment @@ -91,16 +92,21 @@ devito/ # Main source package ├── __init__.py # Package initialization, configuration setup ├── arch/ # Architecture-specific compilation backends ├── builtins/ # Built-in mathematical functions +├── checkpointing/ # Checkpointing and reverse-mode functionality ├── core/ # Core IR (Intermediate Representation) classes ├── data/ # Data structures and memory management ├── finite_differences/# Finite difference operators and stencils ├── ir/ # Intermediate representation passes +├── mpatches/ # Memory patches and optimizations ├── mpi/ # MPI parallelization support +├── operations/ # Mathematical operations and transformations ├── operator/ # Operator compilation and generation ├── passes/ # Compiler optimization passes ├── symbolics/ # Symbolic computation utilities ├── tools/ # Utility functions and helpers -└── types/ # Core types (Grid, Function, Equation, etc.) +├── types/ # Core types (Grid, Function, Equation, etc.) +├── deprecations.py # Deprecation warnings and legacy support +└── warnings.py # Warning system and configuration tests/ # Test suite (comprehensive coverage) examples/ # Tutorials and demonstrations @@ -110,26 +116,35 @@ examples/ # Tutorials and demonstrations └── userapi/ # User API demonstrations scripts/ # Utility scripts -docker/ # Docker configurations -benchmarks/ # Performance benchmarks +docker/ # Docker configurations +benchmarks/ # Performance benchmarks and ASV configuration +binder/ # Jupyter Binder configuration ``` ### Key Configuration Files -- `pyproject.toml`: Build configuration, dependencies, flake8 settings +- `pyproject.toml`: Build configuration, dependencies, flake8 settings (max-line-length=90) - `pytest.ini`: Test configuration and options -- `requirements*.txt`: Dependency specifications for different environments -- `environment-dev.yml`: Conda environment specification -- `.github/workflows/`: CI/CD pipelines +- `requirements*.txt`: Dependency specifications for different environments (mpi, nvidia, testing, optional) +- `environment-dev.yml`: Conda environment specification (recommended for development) +- `.github/workflows/`: CI/CD pipelines (13 comprehensive workflows) +- `MANIFEST.in`: Package distribution file inclusion/exclusion rules ## Continuous Integration and Validation ### GitHub Workflows (All Must Pass) 1. **pytest-core-nompi.yml**: Core functionality tests across Python 3.10-3.13, multiple GCC versions -2. **pytest-core-mpi.yml**: MPI parallel execution tests +2. **pytest-core-mpi.yml**: MPI parallel execution tests 3. **pytest-gpu.yml**: GPU computation tests (OpenACC/CUDA) -4. **flake8.yml**: Code style validation +4. **flake8.yml**: Code style validation (90-char line limit, specific ignore rules) 5. **examples.yml**: Integration tests via example execution -6. **tutorials.yml**: Jupyter notebook validation +6. **examples-mpi.yml**: MPI-based example validation +7. **tutorials.yml**: Jupyter notebook validation +8. **asv.yml**: Performance benchmarking and regression detection +9. **docker-bases.yml**: Docker base image builds for testing environments +10. **docker-devito.yml**: Devito-specific Docker image builds +11. **pythonpublish.yml**: Package publishing to PyPI +12. **release-notes.yml**: Automated release documentation +13. **triggers.yml**: Workflow orchestration and triggers ### Pre-commit Validation Steps ```bash @@ -156,12 +171,38 @@ python examples/seismic/acoustic/acoustic_example.py ## Development Guidelines ### Code Style (Enforced by CI) -- **Line length**: 90 characters maximum +- **Line length**: 90 characters maximum (configured in pyproject.toml) - **Indentation**: 4 spaces (no tabs) - **Imports**: Grouped (stdlib, third-party, devito) and alphabetically sorted - **Docstrings**: NumPy-style documentation required for public APIs - **Naming**: CamelCase classes, snake_case methods/variables +### Commit Messages and PR Titles +Use descriptive prefixes to categorize changes (based on recent repository patterns): +- **compiler:** Changes to compilation system, IR passes, code generation +- **mpi:** MPI-related functionality and parallel execution +- **misc:** General improvements, cleanup, modernization +- **docs:** Documentation updates, docstring fixes +- **install:** Installation, packaging, dependencies +- **bug-py:** Python code bug fixes +- **bug-C:** Generated C code bug fixes + +**Examples:** +- `compiler: Hotfix estimate memory for certain devices` +- `misc: switch to f-string throughout` +- `docs: fix mmin/mmax docstring` +- `install: Remove files and directories not needed for install` + +### Common Labels (for PR categorization) +- **bug-py**, **bug-py-minor**, **bug-C**: Bug fixes by category +- **compiler**: Compilation system changes +- **MPI**: MPI and parallel execution +- **documentation**: Documentation improvements +- **installation**: Package installation and setup +- **misc**: General improvements +- **dependencies**: Dependency updates +- **python**: Python code changes + ### Making Changes 1. **Always run linting first**: `flake8 --builtins=ArgumentError .` 2. **Test specific areas**: Run relevant tests before committing @@ -170,15 +211,21 @@ python examples/seismic/acoustic/acoustic_example.py 5. **Examples**: Add/update examples for new features ### Critical Dependencies -- **SymPy**: Symbolic mathematics (versions 1.12-1.14 supported) -- **NumPy**: Numerical arrays (version 2.x) +- **SymPy**: Symbolic mathematics (versions 1.12+ supported) +- **NumPy**: Numerical arrays (version 2.x compatible) - **cgen/codepy**: C code generation and compilation - **psutil**: System monitoring and resource management +- **setuptools-scm**: Dynamic versioning from git tags + +### Flake8 Configuration and Common Workarounds +The project uses flake8 with specific ignore rules (configured in pyproject.toml): +- **F403/F405**: Star imports are expected in `__init__.py` files due to API design +- **E731**: Lambda expressions are acceptable for configuration and simple cases +- **W503**: Line breaks before binary operators (older style preference) +- **E722**: Bare except clauses are sometimes necessary for compatibility +- **W605**: Invalid escape sequences in docstrings (legacy patterns) -### Common Workarounds -- **Star imports**: F403/F405 warnings are expected in `__init__.py` due to API design -- **Lambda expressions**: E731 warnings are sometimes acceptable for configuration -- **Import order**: Follow the pattern in `devito/__init__.py` +**Import order**: Follow the pattern in `devito/__init__.py` for consistency ## Agent-Specific Instructions @@ -188,14 +235,18 @@ python examples/seismic/acoustic/acoustic_example.py 3. Specific error messages require investigation **Time-saving tips:** -1. Use Docker if pip installation fails due to network issues +1. Use Docker if pip installation fails due to network issues 2. Set environment variables before running any compilation-dependent tests -3. Start with small test files when debugging +3. Start with small test files when debugging (e.g., `pytest tests/test_operator.py::test_specific`) 4. Check configuration output to verify environment setup 5. Use `--maxfail=5` to stop test runs early on systematic failures +6. Use conda environment for full development workflow when possible +7. Run flake8 first before making commits to catch style issues early **File modification priorities:** 1. Core types and operators: `devito/types/`, `devito/operator/` 2. Mathematical operations: `devito/finite_differences/`, `devito/symbolics/` 3. Compilation backends: `devito/arch/`, `devito/passes/` -4. Always update corresponding tests in `tests/` \ No newline at end of file +4. MPI functionality: `devito/mpi/` +5. Always update corresponding tests in `tests/` +6. Update documentation in examples/ if adding new features \ No newline at end of file From 0e0124a2abad2db3115ed9f2802fd8fc2729e372 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 12 Aug 2025 12:13:11 +0000 Subject: [PATCH 4/4] docs: Fix copilot-instructions.md based on review feedback Co-authored-by: ggorman <5394691+ggorman@users.noreply.github.com> --- .github/copilot-instructions.md | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index ae9e9c2197..28199fa2c2 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -20,14 +20,17 @@ Devito is a Python package for implementing optimized stencil computation (finit **System Dependencies (Ubuntu/Debian):** ```bash sudo apt update -sudo apt install -y python3-numpy python3-sympy python3-psutil python3-pip -sudo apt install -y gcc g++ flake8 +sudo apt install -y python3-dev python3-pip python3-venv build-essential ``` **Development Installation (preferred method):** ```bash +# Create and activate virtual environment +python3 -m venv devito-env +source devito-env/bin/activate + # Install with all dependencies - requires network access -pip install -e .[tests,extras] --break-system-packages +pip install -e .[tests,extras] # Alternative: conda environment (recommended for full development) conda env create -f environment-dev.yml @@ -178,7 +181,13 @@ python examples/seismic/acoustic/acoustic_example.py - **Naming**: CamelCase classes, snake_case methods/variables ### Commit Messages and PR Titles -Use descriptive prefixes to categorize changes (based on recent repository patterns): +Use descriptive prefixes to categorize changes (based on recent repository patterns). + +**Guidelines:** +- Capitalize the title after the prefix (e.g., `misc: Patch` not `misc: patch`) +- Follow the official guide: https://github.com/devitocodes/devito/wiki/Tags-for-commit-messages-and-PR-titles + +**Common prefixes:** - **compiler:** Changes to compilation system, IR passes, code generation - **mpi:** MPI-related functionality and parallel execution - **misc:** General improvements, cleanup, modernization @@ -201,7 +210,7 @@ Use descriptive prefixes to categorize changes (based on recent repository patte - **installation**: Package installation and setup - **misc**: General improvements - **dependencies**: Dependency updates -- **python**: Python code changes +- **API**: Application Programming Interface changes ### Making Changes 1. **Always run linting first**: `flake8 --builtins=ArgumentError .` @@ -240,7 +249,7 @@ The project uses flake8 with specific ignore rules (configured in pyproject.toml 3. Start with small test files when debugging (e.g., `pytest tests/test_operator.py::test_specific`) 4. Check configuration output to verify environment setup 5. Use `--maxfail=5` to stop test runs early on systematic failures -6. Use conda environment for full development workflow when possible +6. Use python venv for full development workflow when possible 7. Run flake8 first before making commits to catch style issues early **File modification priorities:**