-
Notifications
You must be signed in to change notification settings - Fork 240
docs: Update .github/copilot-instructions.md system dependencies and development setup #2713
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
Copilot
wants to merge
4
commits into
main
Choose a base branch
from
copilot/fix-2e1d5240-bd37-45d1-a5b6-a17a678505b5
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
e5c6fa4
Initial plan
Copilot 01d4f4b
Add comprehensive .github/copilot-instructions.md with build, test, a…
Copilot ca1e196
docs: Refresh .github/copilot-instructions.md with comprehensive repo…
Copilot 0e0124a
docs: Fix copilot-instructions.md based on review feedback
Copilot File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,261 @@ | ||
# Devito Copilot Instructions | ||
|
||
## Repository Overview | ||
|
||
Devito is a Python package for implementing optimized stencil computation (finite differences, image processing, machine learning) from high-level symbolic problem definitions. It builds on SymPy and employs automated code generation and just-in-time compilation to execute optimized computational kernels on CPUs, GPUs, and clusters. | ||
|
||
**Key Facts:** | ||
- Production-stable package (Development Status :: 5 - Production/Stable) | ||
- Python 3.10-3.13 support with comprehensive CI testing | ||
- ~290MB repository with extensive test coverage and benchmarking | ||
- Primary languages: Python (core), C/C++ (generated code) | ||
- Main dependencies: NumPy, SymPy, cgen, codepy, psutil | ||
- Supports multiple compilation backends: C, OpenMP, OpenACC, MPI | ||
- Active development with 600+ stars, 238+ forks, and regular releases | ||
|
||
## Build and Development Environment | ||
|
||
### Prerequisites and Installation | ||
|
||
**System Dependencies (Ubuntu/Debian):** | ||
```bash | ||
sudo apt update | ||
sudo apt install -y python3-dev python3-pip python3-venv build-essential | ||
``` | ||
|
||
**Development Installation (preferred method):** | ||
```bash | ||
# Create and activate virtual environment | ||
python3 -m venv devito-env | ||
source devito-env/bin/activate | ||
|
||
# Install with all dependencies - requires network access | ||
pip install -e .[tests,extras] | ||
|
||
# Alternative: conda environment (recommended for full development) | ||
conda env create -f environment-dev.yml | ||
conda activate devito | ||
``` | ||
|
||
**Docker Development (if dependencies fail):** | ||
```bash | ||
docker-compose up devito # Starts Jupyter on port 8888 | ||
# OR | ||
docker build . --file docker/Dockerfile.devito --tag devito_img | ||
``` | ||
|
||
### Environment Variables (Critical for Compilation) | ||
```bash | ||
export DEVITO_ARCH="gcc" # gcc, clang, icx, custom | ||
export DEVITO_LANGUAGE="openmp" # C, openmp, CXX, CXXopenmp | ||
export OMP_NUM_THREADS=2 # For parallel tests | ||
``` | ||
|
||
### Build and Test Commands | ||
|
||
**Linting (ALWAYS run before committing):** | ||
```bash | ||
flake8 --builtins=ArgumentError . | ||
# Configuration in pyproject.toml: max-line-length=90, specific ignore rules | ||
``` | ||
|
||
**Testing:** | ||
```bash | ||
# Core tests (required, ~10-15 minutes) | ||
pytest -k "not adjoint" -m "not parallel" --maxfail=5 tests/ | ||
|
||
# Adjoint tests (advanced feature tests) | ||
pytest -k "adjoint" -m "not parallel" tests/ | ||
|
||
# Example tests (integration validation) | ||
pytest examples/ | ||
|
||
# Specific test file | ||
pytest tests/test_operator.py -v | ||
``` | ||
|
||
**Configuration Check:** | ||
```bash | ||
python3 -c "from devito import configuration; print(''.join(['%s: %s \n' % (k, v) for (k, v) in configuration.items()]))" | ||
``` | ||
|
||
### Common Build Issues and Solutions | ||
|
||
1. **Import errors for cgen/codepy:** Install via pip or use conda environment | ||
2. **Compiler not found:** Set DEVITO_ARCH to match installed compiler | ||
3. **OpenMP issues on macOS:** Install with `brew install llvm libomp` | ||
4. **Network timeouts during pip install:** Use conda or Docker alternatives | ||
5. **Permission issues:** Use `--user` or `--break-system-packages` flags | ||
|
||
## Project Architecture and Layout | ||
|
||
### Core Directory Structure | ||
``` | ||
devito/ # Main source package | ||
├── __init__.py # Package initialization, configuration setup | ||
├── arch/ # Architecture-specific compilation backends | ||
├── builtins/ # Built-in mathematical functions | ||
├── checkpointing/ # Checkpointing and reverse-mode functionality | ||
├── core/ # Core IR (Intermediate Representation) classes | ||
├── data/ # Data structures and memory management | ||
├── finite_differences/# Finite difference operators and stencils | ||
├── ir/ # Intermediate representation passes | ||
├── mpatches/ # Memory patches and optimizations | ||
├── mpi/ # MPI parallelization support | ||
├── operations/ # Mathematical operations and transformations | ||
├── operator/ # Operator compilation and generation | ||
├── passes/ # Compiler optimization passes | ||
├── symbolics/ # Symbolic computation utilities | ||
├── tools/ # Utility functions and helpers | ||
├── types/ # Core types (Grid, Function, Equation, etc.) | ||
├── deprecations.py # Deprecation warnings and legacy support | ||
└── warnings.py # Warning system and configuration | ||
|
||
tests/ # Test suite (comprehensive coverage) | ||
examples/ # Tutorials and demonstrations | ||
├── seismic/ # Seismic modeling examples | ||
├── cfd/ # Computational fluid dynamics | ||
├── misc/ # Miscellaneous examples | ||
└── userapi/ # User API demonstrations | ||
|
||
scripts/ # Utility scripts | ||
docker/ # Docker configurations | ||
benchmarks/ # Performance benchmarks and ASV configuration | ||
binder/ # Jupyter Binder configuration | ||
``` | ||
|
||
### Key Configuration Files | ||
- `pyproject.toml`: Build configuration, dependencies, flake8 settings (max-line-length=90) | ||
- `pytest.ini`: Test configuration and options | ||
- `requirements*.txt`: Dependency specifications for different environments (mpi, nvidia, testing, optional) | ||
- `environment-dev.yml`: Conda environment specification (recommended for development) | ||
- `.github/workflows/`: CI/CD pipelines (13 comprehensive workflows) | ||
- `MANIFEST.in`: Package distribution file inclusion/exclusion rules | ||
|
||
## Continuous Integration and Validation | ||
|
||
### GitHub Workflows (All Must Pass) | ||
1. **pytest-core-nompi.yml**: Core functionality tests across Python 3.10-3.13, multiple GCC versions | ||
2. **pytest-core-mpi.yml**: MPI parallel execution tests | ||
3. **pytest-gpu.yml**: GPU computation tests (OpenACC/CUDA) | ||
4. **flake8.yml**: Code style validation (90-char line limit, specific ignore rules) | ||
5. **examples.yml**: Integration tests via example execution | ||
6. **examples-mpi.yml**: MPI-based example validation | ||
7. **tutorials.yml**: Jupyter notebook validation | ||
8. **asv.yml**: Performance benchmarking and regression detection | ||
9. **docker-bases.yml**: Docker base image builds for testing environments | ||
10. **docker-devito.yml**: Devito-specific Docker image builds | ||
11. **pythonpublish.yml**: Package publishing to PyPI | ||
12. **release-notes.yml**: Automated release documentation | ||
13. **triggers.yml**: Workflow orchestration and triggers | ||
|
||
### Pre-commit Validation Steps | ||
```bash | ||
# 1. Lint code | ||
flake8 --builtins=ArgumentError . | ||
|
||
# 2. Run core tests | ||
pytest -k "not adjoint" -m "not parallel" --maxfail=5 tests/ | ||
|
||
# 3. Check configuration | ||
python3 -c "from devito import configuration; print(configuration)" | ||
|
||
# 4. Test example (optional but recommended) | ||
python examples/seismic/acoustic/acoustic_example.py | ||
``` | ||
|
||
### Test Execution Times | ||
- Core tests: 10-15 minutes | ||
- Full test suite: 30-45 minutes | ||
- Individual test files: 1-5 minutes | ||
- Linting: <1 minute | ||
- Examples: 5-10 minutes each | ||
|
||
## Development Guidelines | ||
|
||
### Code Style (Enforced by CI) | ||
- **Line length**: 90 characters maximum (configured in pyproject.toml) | ||
- **Indentation**: 4 spaces (no tabs) | ||
- **Imports**: Grouped (stdlib, third-party, devito) and alphabetically sorted | ||
- **Docstrings**: NumPy-style documentation required for public APIs | ||
- **Naming**: CamelCase classes, snake_case methods/variables | ||
|
||
### Commit Messages and PR Titles | ||
Use descriptive prefixes to categorize changes (based on recent repository patterns). | ||
|
||
**Guidelines:** | ||
- Capitalize the title after the prefix (e.g., `misc: Patch` not `misc: patch`) | ||
- Follow the official guide: https://github.com/devitocodes/devito/wiki/Tags-for-commit-messages-and-PR-titles | ||
|
||
**Common prefixes:** | ||
- **compiler:** Changes to compilation system, IR passes, code generation | ||
- **mpi:** MPI-related functionality and parallel execution | ||
- **misc:** General improvements, cleanup, modernization | ||
- **docs:** Documentation updates, docstring fixes | ||
- **install:** Installation, packaging, dependencies | ||
- **bug-py:** Python code bug fixes | ||
- **bug-C:** Generated C code bug fixes | ||
|
||
**Examples:** | ||
- `compiler: Hotfix estimate memory for certain devices` | ||
- `misc: switch to f-string throughout` | ||
- `docs: fix mmin/mmax docstring` | ||
- `install: Remove files and directories not needed for install` | ||
|
||
### Common Labels (for PR categorization) | ||
- **bug-py**, **bug-py-minor**, **bug-C**: Bug fixes by category | ||
ggorman marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- **compiler**: Compilation system changes | ||
- **MPI**: MPI and parallel execution | ||
- **documentation**: Documentation improvements | ||
- **installation**: Package installation and setup | ||
- **misc**: General improvements | ||
- **dependencies**: Dependency updates | ||
- **API**: Application Programming Interface changes | ||
|
||
### Making Changes | ||
1. **Always run linting first**: `flake8 --builtins=ArgumentError .` | ||
2. **Test specific areas**: Run relevant tests before committing | ||
3. **Environment setup**: Ensure DEVITO_ARCH and DEVITO_LANGUAGE are set | ||
4. **Documentation**: Update docstrings for API changes | ||
5. **Examples**: Add/update examples for new features | ||
|
||
### Critical Dependencies | ||
- **SymPy**: Symbolic mathematics (versions 1.12+ supported) | ||
- **NumPy**: Numerical arrays (version 2.x compatible) | ||
- **cgen/codepy**: C code generation and compilation | ||
- **psutil**: System monitoring and resource management | ||
- **setuptools-scm**: Dynamic versioning from git tags | ||
|
||
### Flake8 Configuration and Common Workarounds | ||
The project uses flake8 with specific ignore rules (configured in pyproject.toml): | ||
- **F403/F405**: Star imports are expected in `__init__.py` files due to API design | ||
- **E731**: Lambda expressions are acceptable for configuration and simple cases | ||
- **W503**: Line breaks before binary operators (older style preference) | ||
- **E722**: Bare except clauses are sometimes necessary for compatibility | ||
- **W605**: Invalid escape sequences in docstrings (legacy patterns) | ||
|
||
**Import order**: Follow the pattern in `devito/__init__.py` for consistency | ||
|
||
## Agent-Specific Instructions | ||
|
||
**Trust these instructions** - they are validated and current. Only search/explore if: | ||
1. Instructions appear outdated or incorrect | ||
2. New functionality is not covered | ||
3. Specific error messages require investigation | ||
|
||
**Time-saving tips:** | ||
1. Use Docker if pip installation fails due to network issues | ||
2. Set environment variables before running any compilation-dependent tests | ||
3. Start with small test files when debugging (e.g., `pytest tests/test_operator.py::test_specific`) | ||
4. Check configuration output to verify environment setup | ||
5. Use `--maxfail=5` to stop test runs early on systematic failures | ||
6. Use python venv for full development workflow when possible | ||
7. Run flake8 first before making commits to catch style issues early | ||
|
||
**File modification priorities:** | ||
1. Core types and operators: `devito/types/`, `devito/operator/` | ||
2. Mathematical operations: `devito/finite_differences/`, `devito/symbolics/` | ||
3. Compilation backends: `devito/arch/`, `devito/passes/` | ||
4. MPI functionality: `devito/mpi/` | ||
5. Always update corresponding tests in `tests/` | ||
6. Update documentation in examples/ if adding new features |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.