Packer tests, updated format, .env

* Updated tests
* Added fetching a scenario directly in the Jupyter notebook to the example
* Convert from yml settings to .env
* Separated inputs and outputs into their own folders at root

Author: louispt1

.gitignore

@@ -18,5 +18,6 @@ import_graph.svg
 tmp/
 
 *.xlsx
-# Keep the example input
-!examples/example_input_excel.xlsx
+# Keep the examples
+!inputs/example_input_excel.xlsx
+!examples/example.config.env

README.md

@@ -80,7 +80,7 @@ If you want development dependencies (testing, linting, etc.) then append the
 
 #### How to use the environment:
 You can either:
-- Run commands inside Poetry’s environment:
+- Run commands inside Poetry's environment:
   ```bash
   poetry run pytest
   poetry run pyetm
@@ -97,25 +97,40 @@ You can either:
 
 ## Configuring Your Settings
 
-You can configure your API token and base URL either with a **config.yml** file or environment variables. You can now simply set an `environment` and the base URL will be inferred for you.
+You can configure your API token and base URL either with a **config.env** file or environment variables. You can simply set an `environment` and the base URL will be inferred for you.
 
-### Option 1: `config.yml`
-1. Duplicate the example file (`examples/example.config.yml`) and rename it to `config.yml`.
-2. Edit `config.yml`:
-   - **etm_api_token**: Your ETM API token (overridden by `$ETM_API_TOKEN` if set).
-   - **environment**: pro (default), beta, local, or a stable tag like `2025-01`. When set, `base_url` is inferred automatically.
-   - (optional) **base_url**: API base URL (overridden by `$BASE_URL` if set). If both `environment` and `base_url` are set, `base_url` wins.
-     Examples if you need a direct override:
-       - `https://engine.energytransitionmodel.com/api/v3` (pro)
-       - `https://beta.engine.energytransitionmodel.com/api/v3` (beta)
-       - `https://2025-01.engine.energytransitionmodel.com/api/v3` (stable tag)
-   - **proxy_servers**: (Optional) HTTP/HTTPS proxy URLs.
-   - **csv_separator** and **decimal_separator**: Defaults are `,` and `.`.
+### Option 1: `config.env` (Recommended)
+1. Copy the example file (`example.config.env`) and rename it to `config.env`.
+2. Edit `config.env`:
+   ```bash
+   # Your ETM API token (required)
+   ETM_API_TOKEN=your.token.here
 
-Place `config.yml` in the project root (`pyetm/` folder).
+   # Environment (default: pro)
+   ENVIRONMENT=pro
+
+   # Optional: Override base URL directly
+   # BASE_URL=https://engine.energytransitionmodel.com/api/v3
+
+   # Optional: Proxy settings
+   # PROXY_SERVERS_HTTP=http://user:[email protected]:8080
+   # PROXY_SERVERS_HTTPS=http://user:[email protected]:8080
+
+   # CSV settings (optional)
+   CSV_SEPARATOR=,
+   DECIMAL_SEPARATOR=.
+   ```
+
+Place `config.env` in the project root (`pyetm/` folder).
+
+**Environment Options:**
+- `pro` (default): Production environment
+- `beta`: Staging environment
+- `local`: Local development environment
+- `YYYY-MM`: Stable tagged environment (e.g., `2025-01`)
 
 ### Option 2: Environment Variables
-If you prefer, set these environment variables:
+If you prefer, set these environment variables directly:
 ```bash
 ETM_API_TOKEN=<your token>
 ENVIRONMENT=<pro|beta|local|YYYY-MM>

examples/create_or_query_scenarios.ipynb

@@ -31,7 +31,7 @@
     "\n",
     "In the example, there are two scenarios, with short names scen_a and scen_b. You can use short names in the slider_settings sheet to specify which inputs belong to which scenario. Because scen_b has no scenario_id, it is being created. It will be created with all the metadata included in the sheet, plus any of the inputs under the column with its short_name and any sortables and curves specified in the sheets named beside the sortables and custom_curves rows. The same goes for scen_a, but because it has a scenario_id (1357395) that scenario will be loaded, and then updated with anything as set in the excel.\n",
     "\n",
-    "**TODO**: Figure out how to manage the fact that for this example whatever scenario 1357395 is will be constantly updated etc by everyone who wants to try running this script on pro/beta. At the moment its just a local scenario."
+    "The example scenario ids are scenarios on pro. It's recommended to change the scenario_ids in the example and experiment with scenarios you own."
    ]
   },
   {
@@ -42,9 +42,14 @@
    "outputs": [],
    "source": [
     "from pyetm.models.scenarios import Scenarios\n",
+    "from pyetm.models.scenario import Scenario\n",
+    "\n",
     "\n",
     "scenarios = Scenarios.from_excel(\"example_input_excel.xlsx\")\n",
-    "#scenario_a = Scenario.load(123456789) #TODO: Also load a scenario and include it in the array"
+    "\n",
+    "# Here we're also loading a scenario directly from the API and adding it to the scenarios loaded/created via the excel\n",
+    "scenario_a = Scenario.load(1357691)\n",
+    "scenarios.add(scenario_a)"
    ]
   },
   {
@@ -84,7 +89,7 @@
    "source": [
     "# Inputs\n",
     "for scenario in scenarios:\n",
-    "    inputs = scenario.inputs.to_dataframe(columns=[\"user\", \"default\", \"min\", \"max\"]).head(20)\n",
+    "    inputs = scenario.inputs.to_dataframe(columns=[\"user\", \"default\", \"min\", \"max\"]).head(15)\n",
     "    print(inputs)\n",
     "    print(\"\")"
    ]

examples/example.config.env

@@ -0,0 +1,36 @@
+# ETM API Configuration
+# Copy this file to .env and update with your personal settings
+# Never commit your .env file to version control!
+
+# Your personal ETM API Token (REQUIRED)
+# Get your token from: https://docs.energytransitionmodel.com/api/authentication
+# Format: etm_<JWT> or etm_beta_<JWT>
+ETM_API_TOKEN=your.token.here
+
+# ETM Environment (default: pro)
+# Options: pro, beta, local, or stable tags like 2025-01
+ENVIRONMENT=pro
+
+# Override API base URL (optional - will be inferred from ENVIRONMENT if not set)
+# Examples:
+# BASE_URL=https://engine.energytransitionmodel.com/api/v3
+# BASE_URL=https://beta.engine.energytransitionmodel.com/api/v3
+# BASE_URL=http://localhost:3000/api/v3
+# BASE_URL=https://2025-01.engine.energytransitionmodel.com/api/v3
+# BASE_URL=
+
+# Logging level (default: INFO)
+# Options: DEBUG, INFO, WARNING, ERROR, CRITICAL
+LOG_LEVEL=INFO
+
+# Proxy Settings (optional)
+# Never commit authenticated proxy URLs to version control!
+# PROXY_SERVERS_HTTP=http://user:[email protected]:8080
+# PROXY_SERVERS_HTTPS=http://user:[email protected]:8080
+
+# CSV File Settings
+# CSV separator character (default: ,)
+CSV_SEPARATOR=,
+
+# Decimal separator character (default: .)
+DECIMAL_SEPARATOR=.

examples/example.config.yml

@@ -1,32 +1,31 @@
 from pathlib import Path
 import re
-import yaml, os
 from typing import Optional, ClassVar, List, Annotated
 from pydantic import Field, ValidationError, HttpUrl, field_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
 PROJECT_ROOT = Path(__file__).resolve().parents[3]
-CONFIG_FILE = PROJECT_ROOT / "config.yml"
+ENV_FILE = PROJECT_ROOT / "config.env"
 
 
 class AppConfig(BaseSettings):
     """
-    Application configuration loaded from YAML.
+    Application configuration loaded from .env file and environment variables.
     """
 
     etm_api_token: Annotated[
         str,
         Field(
             ...,
-            description="Your ETM API token: must be either `etm_<JWT>` or `etm_beta_<JWT>`. If not set please set $ETM_API_TOKEN or config.yml:etm_api_token",
+            description="Your ETM API token: must be either `etm_<JWT>` or `etm_beta_<JWT>`",
         ),
     ]
-    base_url: HttpUrl = Field(
-        "https://engine.energytransitionmodel.com/api/v3",
-        description="Base URL for the ETM API",
+    base_url: Optional[HttpUrl] = Field(
+        None,
+        description="Base URL for the ETM API (will be inferred from environment if not provided)",
     )
     environment: Optional[str] = Field(
-        None,
+        "pro",
         description=(
             "ETM environment to target. One of: 'pro' (default), 'beta', 'local', or a stable tag 'YYYY-MM'. "
             "When set and base_url is not provided, base_url will be inferred."
@@ -37,12 +36,40 @@ class AppConfig(BaseSettings):
         description="App logging level",
     )
 
+    proxy_servers_http: Optional[str] = Field(
+        None,
+        description="HTTP proxy server URL",
+    )
+    proxy_servers_https: Optional[str] = Field(
+        None,
+        description="HTTPS proxy server URL",
+    )
+    csv_separator: str = Field(
+        ",",
+        description="CSV file separator character",
+    )
+    decimal_separator: str = Field(
+        ".",
+        description="Decimal separator character",
+    )
+
     model_config: ClassVar[SettingsConfigDict] = SettingsConfigDict(
-        env_file=None, extra="ignore", case_sensitive=False
+        case_sensitive=False,
+        extra="ignore",
     )
 
     temp_folder: Optional[Path] = PROJECT_ROOT / "tmp"
 
+    def __init__(self, **values):
+        """
+        This ensures tests can monkeypatch `pyetm.config.settings.ENV_FILE`
+        """
+        super().__init__(
+            _env_file=ENV_FILE,
+            _env_file_encoding="utf-8",
+            **values,
+        )
+
     @field_validator("etm_api_token")
     @classmethod
     def check_jwt(cls, v: str) -> str:
@@ -77,43 +104,33 @@ def check_jwt(cls, v: str) -> str:
 
         return v
 
+    def model_post_init(self, __context) -> None:
+        """Post-initialization to handle base_url inference."""
+        if not self.base_url:
+            self.base_url = HttpUrl(_infer_base_url_from_env(self.environment))
+
     def path_to_tmp(self, subfolder: str):
         folder = self.temp_folder / subfolder
         folder.mkdir(parents=True, exist_ok=True)
         return folder
 
-    @classmethod
-    def from_yaml(cls, path: Path) -> "AppConfig":
-        raw = {}
-        if path.is_file():
-            try:
-                raw = yaml.safe_load(path.read_text()) or {}
-            except yaml.YAMLError:
-                raw = {}
-
-        data = {k.lower(): v for k, v in raw.items()}
-
-        # Collect environment variables overriding YAML
-        for field in ("etm_api_token", "base_url", "log_level", "environment"):
-            if val := os.getenv(field.upper()):
-                data[field] = val
-
-        # If base_url wasn't explicitly provided, infer it from environment if present
-        if "base_url" not in data or not data["base_url"]:
-            env = (data.get("environment") or "").strip().lower()
-            if env:
-                data["base_url"] = _infer_base_url_from_env(env)
-
-        return cls(**data)
+    @property
+    def proxy_servers(self) -> dict[str, str]:
+        """Return proxy servers as a dictionary for backward compatibility."""
+        proxies = {}
+        if self.proxy_servers_http:
+            proxies["http"] = self.proxy_servers_http
+        if self.proxy_servers_https:
+            proxies["https"] = self.proxy_servers_https
+        return proxies
 
 
 def get_settings() -> AppConfig:
     """
-    Always re-load AppConfig from disk and ENV on each call,
-    and raise a clear, aggregated message if anything required is missing.
+    Load AppConfig from .env file and environment variables.
     """
     try:
-        return AppConfig.from_yaml(CONFIG_FILE)
+        return AppConfig()
     except ValidationError as exc:
         missing_or_invalid: List[str] = []
         for err in exc.errors():
@@ -125,7 +142,7 @@ def get_settings() -> AppConfig:
         raise RuntimeError(
             f"\nConfiguration error: one or more required settings are missing or invalid:\n\n"
             f"{detail}\n\n"
-            f"Please set them via environment variables or in `{CONFIG_FILE}`."
+            f"Please set them via environment variables or in `{ENV_FILE}`."
         ) from exc
 
 

examples/example_input_excel.xlsx renamed to inputs/example_input_excel.xlsx

@@ -120,7 +120,14 @@ def from_json(cls, data: dict) -> CustomCurve:
         Initialize a CustomCurve from JSON data
         """
         try:
-            curve = cls.model_validate(data)
+            curve = cls(**data)
+            missing = [k for k in ("key", "type") if k not in data]
+            if missing:
+                curve.add_warning(
+                    "base",
+                    f"Failed to create curve from data: missing required fields: {', '.join(missing)}",
+                )
+
             return curve
         except Exception as e:
             basic_data = {

src/pyetm/config/settings.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 import pandas as pd
 from datetime import datetime
+from pathlib import Path
 from typing import Any, Dict, List, Optional, Set, Union
 from urllib.parse import urlparse
 from pydantic import Field, PrivateAttr
@@ -108,8 +109,12 @@ def from_excel(cls, xlsx_path: PathLike | str) -> List["Scenario"]:
         Load or create one or more scenarios from an Excel workbook.
         """
         from pyetm.models.scenario_packer import ScenarioPacker
+        from pyetm.utils.paths import PyetmPaths
 
-        packer = ScenarioPacker.from_excel(xlsx_path)
+        resolver = PyetmPaths()
+        path = resolver.resolve_for_read(xlsx_path, default_dir="inputs")
+
+        packer = ScenarioPacker.from_excel(str(path))
         scenarios = list(packer._scenarios())
         scenarios.sort(key=lambda s: s.id)
         return scenarios
@@ -130,10 +135,15 @@ def to_excel(
         Output curves are exported to a separate workbook only when enabled, with one
         sheet per carrier. Use carriers to filter which carriers to include when exporting.
         """
+
         from pyetm.models.scenarios import Scenarios
+        from pyetm.utils.paths import PyetmPaths
+
+        resolver = PyetmPaths()
+        out_path = resolver.resolve_for_write(path, default_dir="outputs")
 
         Scenarios(items=[self, *others]).to_excel(
-            path,
+            str(out_path),
             carriers=carriers,
             include_inputs=include_inputs,
             include_sortables=include_sortables,