Packer tests, updated format, .env

* Updated tests
* Added fetching a scenario directly in the Jupyter notebook to the example
* Convert from yml settings to .env
* Separated inputs and outputs into their own folders at root

Author: louispt1

.gitignore

@@ -18,5 +18,6 @@ import_graph.svg
 tmp/
 
 *.xlsx
-# Keep the example input
-!examples/example_input_excel.xlsx
+# Keep the examples
+!inputs/example_input_excel.xlsx
+!examples/example.config.env

README.md

@@ -80,7 +80,7 @@ If you want development dependencies (testing, linting, etc.) then append the
 
 #### How to use the environment:
 You can either:
-- Run commands inside Poetry’s environment:
+- Run commands inside Poetry's environment:
   ```bash
   poetry run pytest
   poetry run pyetm
@@ -97,25 +97,40 @@ You can either:
 
 ## Configuring Your Settings
 
-You can configure your API token and base URL either with a **config.yml** file or environment variables. You can now simply set an `environment` and the base URL will be inferred for you.
+You can configure your API token and base URL either with a **config.env** file or environment variables. You can simply set an `environment` and the base URL will be inferred for you.
 
-### Option 1: `config.yml`
-1. Duplicate the example file (`examples/example.config.yml`) and rename it to `config.yml`.
-2. Edit `config.yml`:
-   - **etm_api_token**: Your ETM API token (overridden by `$ETM_API_TOKEN` if set).
-   - **environment**: pro (default), beta, local, or a stable tag like `2025-01`. When set, `base_url` is inferred automatically.
-   - (optional) **base_url**: API base URL (overridden by `$BASE_URL` if set). If both `environment` and `base_url` are set, `base_url` wins.
-     Examples if you need a direct override:
-       - `https://engine.energytransitionmodel.com/api/v3` (pro)
-       - `https://beta.engine.energytransitionmodel.com/api/v3` (beta)
-       - `https://2025-01.engine.energytransitionmodel.com/api/v3` (stable tag)
-   - **proxy_servers**: (Optional) HTTP/HTTPS proxy URLs.
-   - **csv_separator** and **decimal_separator**: Defaults are `,` and `.`.
+### Option 1: `config.env` (Recommended)
+1. Copy the example file (`example.config.env`) and rename it to `config.env`.
+2. Edit `config.env`:
+   ```bash
+   # Your ETM API token (required)
+   ETM_API_TOKEN=your.token.here
 
-Place `config.yml` in the project root (`pyetm/` folder).
+   # Environment (default: pro)
+   ENVIRONMENT=pro
+
+   # Optional: Override base URL directly
+   # BASE_URL=https://engine.energytransitionmodel.com/api/v3
+
+   # Optional: Proxy settings
+   # PROXY_SERVERS_HTTP=http://user:[email protected]:8080
+   # PROXY_SERVERS_HTTPS=http://user:[email protected]:8080
+
+   # CSV settings (optional)
+   CSV_SEPARATOR=,
+   DECIMAL_SEPARATOR=.
+   ```
+
+Place `config.env` in the project root (`pyetm/` folder).
+
+**Environment Options:**
+- `pro` (default): Production environment
+- `beta`: Staging environment
+- `local`: Local development environment
+- `YYYY-MM`: Stable tagged environment (e.g., `2025-01`)
 
 ### Option 2: Environment Variables
-If you prefer, set these environment variables:
+If you prefer, set these environment variables directly:
 ```bash
 ETM_API_TOKEN=<your token>
 ENVIRONMENT=<pro|beta|local|YYYY-MM>

examples/create_or_query_scenarios.ipynb

@@ -27,11 +27,11 @@
    "id": "2884d014",
    "metadata": {},
    "source": [
-    "For the purposes of this demonstration workbook, we will use the pre-filled template 'my_input_excel.xlsx' which demonstrates a few of the possibilities afforded by the pyetm package. If it's your first time using the tool, have a look at the excel to get a sense of the structure.\n",
+    "For the purposes of this demonstration workbook, we will use the pre-filled template 'example_input_excel.xlsx' which demonstrates a few of the possibilities afforded by the pyetm package. If it's your first time using the tool, have a look at the excel to get a sense of the structure. The input excel is available in the /inputs folder. In pyetm, by default files will be read from the /inputs folder and written to the /outputs folder.\n",
     "\n",
     "In the example, there are two scenarios, with short names scen_a and scen_b. You can use short names in the slider_settings sheet to specify which inputs belong to which scenario. Because scen_b has no scenario_id, it is being created. It will be created with all the metadata included in the sheet, plus any of the inputs under the column with its short_name and any sortables and curves specified in the sheets named beside the sortables and custom_curves rows. The same goes for scen_a, but because it has a scenario_id (1357395) that scenario will be loaded, and then updated with anything as set in the excel.\n",
     "\n",
-    "**TODO**: Figure out how to manage the fact that for this example whatever scenario 1357395 is will be constantly updated etc by everyone who wants to try running this script on pro/beta. At the moment its just a local scenario."
+    "The example scenario ids are scenarios on pro. It's recommended to change the scenario_ids in the example and experiment with scenarios you own."
    ]
   },
   {
@@ -42,9 +42,14 @@
    "outputs": [],
    "source": [
     "from pyetm.models.scenarios import Scenarios\n",
+    "from pyetm.models.scenario import Scenario\n",
+    "\n",
     "\n",
     "scenarios = Scenarios.from_excel(\"example_input_excel.xlsx\")\n",
-    "#scenario_a = Scenario.load(123456789) #TODO: Also load a scenario and include it in the array"
+    "\n",
+    "# Here we're also loading a scenario directly from the API and adding it to the scenarios loaded/created via the excel\n",
+    "scenario_a = Scenario.load(1357691)\n",
+    "scenarios.add(scenario_a)"
    ]
   },
   {
@@ -84,7 +89,7 @@
    "source": [
     "# Inputs\n",
     "for scenario in scenarios:\n",
-    "    inputs = scenario.inputs.to_dataframe(columns=[\"user\", \"default\", \"min\", \"max\"]).head(20)\n",
+    "    inputs = scenario.inputs.to_dataframe(columns=[\"user\", \"default\", \"min\", \"max\"]).head(15)\n",
     "    print(inputs)\n",
     "    print(\"\")"
    ]

examples/example.config.env

@@ -0,0 +1,36 @@
+# ETM API Configuration
+# Copy this file to .env and update with your personal settings
+# Never commit your .env file to version control!
+
+# Your personal ETM API Token (REQUIRED)
+# Get your token from: https://docs.energytransitionmodel.com/api/authentication
+# Format: etm_<JWT> or etm_beta_<JWT>
+ETM_API_TOKEN=your.token.here
+
+# ETM Environment (default: pro)
+# Options: pro, beta, local, or stable tags like 2025-01
+ENVIRONMENT=pro
+
+# Override API base URL (optional - will be inferred from ENVIRONMENT if not set)
+# Examples:
+# BASE_URL=https://engine.energytransitionmodel.com/api/v3
+# BASE_URL=https://beta.engine.energytransitionmodel.com/api/v3
+# BASE_URL=http://localhost:3000/api/v3
+# BASE_URL=https://2025-01.engine.energytransitionmodel.com/api/v3
+# BASE_URL=
+
+# Logging level (default: INFO)
+# Options: DEBUG, INFO, WARNING, ERROR, CRITICAL
+LOG_LEVEL=INFO
+
+# Proxy Settings (optional)
+# Never commit authenticated proxy URLs to version control!
+# PROXY_SERVERS_HTTP=http://user:[email protected]:8080
+# PROXY_SERVERS_HTTPS=http://user:[email protected]:8080
+
+# CSV File Settings
+# CSV separator character (default: ,)
+CSV_SEPARATOR=,
+
+# Decimal separator character (default: .)
+DECIMAL_SEPARATOR=.

examples/example.config.yml

@@ -2,29 +2,43 @@
 requires = ["poetry-core>=1.0.0"]
 build-backend = "poetry.core.masonry.api"
 
-[tool.poetry]
+
+[project]
 name = "pyetm"
 version = "2.0"
-description = "A Python package for energy transition modeling."
-authors = ["Your Name <[email protected]>"]
-license = "MIT"
+description = "Python-ETM Connector"
 readme = "README.md"
-homepage = "https://github.com/quintel/pyetm"
-repository = "https://github.com/quintel/pyetm"
-documentation = "https://docs.energytransitionmodel.com/main/pyetm/introduction"
-keywords = ["energy", "transition", "modeling", "api"]
+license = { text = "MIT" }
+authors = [
+  { name = "Nora Schinkel", email = "[email protected]" },
+  { name = "Louis Parkes-Talbo", email = "[email protected]" }
+]
+keywords = ["energy", "transition", "modeling", "api", "ETM"]
+requires-python = ">=3.12,<4.0"
+
+dependencies = [
+  "pydantic",
+  "pandas",
+  "requests",
+  "python-dotenv",
+  "pyyaml",
+  "pydantic-settings",
+  "xlsxwriter",
+  "openpyxl"
+]
+
+[project.scripts]
+pyetm = "pyetm.__main__:main"
+
+[project.urls]
+"Repository"    = "https://github.com/quintel/pyetm"
+"Documentation" = "https://docs.energytransitionmodel.com/main/pyetm/introduction"
+"Issue tracker" = "https://github.com/quintel/pyetm/issues"
+
+[tool.poetry]
 packages = [{ include = "pyetm", from = "src" }]
+include = ["LICENSE"]
 
-[tool.poetry.dependencies]
-python = ">=3.12,<4.0"
-pydantic = "*"
-pandas = "*"
-requests = "*"
-python-dotenv = "*"
-pyyaml = "*"
-pydantic-settings = "*"
-xlsxwriter = "*"
-openpyxl = "*"
 
 [tool.poetry.group.dev.dependencies]
 pytest = "*"
@@ -35,12 +49,7 @@ notebook = "*"
 pytest-cov = "*"
 pydeps = "*"
 
-[tool.poetry.scripts]
-pyetm = "pyetm.__main__:main"
 
-[tool.poetry.urls]
-"Issue tracker" = "https://github.com/quintel/pyetm/issues"
-"Documentation" = "https://docs.energytransitionmodel.com/main/pyetm/introduction"
 
 [tool.pytest.ini_options]
 addopts = "--cov=pyetm --cov-report=term-missing"