Trace Archive Tool

.gitignore

@@ -38,4 +38,7 @@
 [Ff]ast[Dd]ebug
 
 # Backup files
-*~
+*~
+
+# Python cache file
+__pycache__/

traces/stf_trace_archive/README.md

@@ -1,113 +1,182 @@
 # Trace Archive Tool
 
-A Python command-line interface (CLI) tool to manage shared trace files,such as uploding, searching and downloading traces.
+## Table of Contents
+
+1. [Quickstart](#quickstart)
+2. [Introduction](#introduction)
+3. [Dependencies](#dependencies)
+4. [Project Structure](#project-structure)
+5. [Usage](#usage)
+    1. [Initial Setup](#initial-setup)
+    2. [Upload Command](#upload-command)
+    3. [List Command](#list-command)
+    4. [Get Command](#get-command)
+6. [Examples](#examples)
+    1. [Uploading a Trace](#uploading-a-trace)
+    2. [Downloading a Trace](#downloading-a-trace)
+    3. [Downloading a Workload](#downloading-a-workload)
+    4. [Creating and using second storage source](#creating-and-using-second-storage-source)
+7. [Trace ID](#trace-id)
+    1. [Example Trace IDs](#example-trace-ids)
+8. [Storage Folder Structure](#storage-folder-structure)
+
+## Quickstart
 
-## Usage
+```bash
+# Install dependencies
+pip install -r requirements.txt
 
-Run the script using:
+# Configure initial storage (local)
+python trace_archive.py setup
 
-```bash
-python trace_archive.py <command> [options]
+# Upload a workload and trace
+python trace_archive.py upload --workload ../../stf_metadata/example/dhrystone --trace ../../stf_metadata/example/dhrystone.zstf
 ```
 
-To view all available commands and options use `--help` or `-h`:
+## Introduction
 
-```bash
-$ python trace_archive.py --help
-Usage: python trace_archive.py COMMAND [OPTIONS]
+A Python CLI tool for uploading, organizing, and sharing trace and workload files.
+Currently supports local storage, with planned extensions for cloud sources (e.g., Google Drive).
+
+## Dependencies
 
-CLI tool for Olympia traces exploration
+To use the trace archive tool, ensure you have the following installed:
 
-Commands:
-  connect    Connect to the system or database.
-  upload     Upload workload and trace.
-  search     Search traces by specified expression.
-  list       List items by category.
-  get        Download a specified trace file.
+- **Python 3** (recommended: Python 3.8 or newer)
+- **Required Python packages**: Install dependencies with:
+    ```bash
+    pip install -r requirements.txt
+    ```
+    The main requirements are:
+    - `pandas`
+    - `PyYAML`
 
-Run 'trace_archive COMMAND --help' for more information on a command.
+## Project Structure
 
-For more help on how to use trace_archive, head to GITHUB_README_LINK
+```text
+src/
+├── data/                  # Core data models and classes
+│   └── storage/           # Storage backend implementations (local, cloud, etc.)
+├── handlers/              # Command handlers (upload, get, list, setup)
+├── utils/                 # Utility functions and helpers
+└── trace_archive.py       # Main CLI entry point
 ```
 
----
 
-## Available Commands
+## Usage
+
+The tool can be used with the following commands:
+
+* **[Upload](#upload-command).** Upload workload and/or trace.
+* **[List](#list-command).** List items by category.
+* **[Get](#get-command).** Download a specified trace file.
+* **[Setup](#setup-command).** Create or edit current tool configurations.
 
-### `upload`
 
-Uploads a trace file along with its associated workload and metadata.
+### Initial Setup
+
+To set up the trace archive tool, run the `setup` command to configure the inital storage type and it's path. For example, to set up a local storage type, with the name `local` and path to the storage folder `/home/user/trace_archive`, run:
 
 ```bash
-$ python trace_archive.py upload --help
-Usage:  python trace_archive.py upload [OPTIONS]
+$ python trace_archive.py setup
+Creating a new storage source.
+Registred storage type options: local-storage
+Select your storage type: local-storage
+Enter your storage name: local
+Enter the storage folder path: /home/user/trace_archive
+```
+
+All storage sources contains a type and a name. The type is used to identify the storage source, like `local-storage` or `google-drive`, while the name is used to identify the storage configuration in the tools commands.
 
-Upload a workload, trace and metadata to the database
+With the initial setup done, you can add new storage sources or change the default storage source using the `setup` command again, with the commands `--add-storage` and `--set-default-storage`, respectively.
 
-Options:
-      --workload  Path to the workload file.
-      --trace     Path to the trace file.
-      --it        Iteractive files selection mode.
+```bash
+$ python trace_archive.py setup --add-storage
+$ python trace_archive.py setup --set-default-storage
 ```
 
-> Requires a metadata file located at `<trace>.metadata.yaml`.
+All configurations are stored in the `config.yaml` file, which is created in the current working directory when the `setup` command is run for the first time.
 
-> For every upload, a unqiue [trace id](#trace-id) will be generated
+Checkout the [Creating and using second storage source](#creating-and-using-second-storage-source) section for more details on how to create and use a second storage source.
 
----
+### Upload Command
 
-### `search`
+The `upload` command is for upload a trace and it's workload. The a trace file, and if not presented in the storage yet, hte workload file. The tools also expects a metadata file, which is a YAML file with the name `<trace>.metadata.yaml`, where `<trace>` is the name of the trace file. Multiple traces can be uploaded at once, as long as they are from the same trace attempt.
 
-Search can be used to search for the given regex term in the list of available traces and metadata matches
+The `upload` command options are:
 
-```bash
-$ python trace_archive.py search --help
-Usage:  python trace_archive.py search [OPTIONS] [REGEX]
+* `--workload`: Path to the workload file.
+* `--trace`: Path to the trace file.
+* `--it`: Interactive files selection mode. If this option is used, the tool will prompt the user to select the workload and trace files 
 
-Search for traces and metadata using a regular expression.
+For every upload, a unique [`trace-id`](#trace-id) will be generated and filled into the metadata file.
 
-Arguments:
-      REGEX             Regex expression to search with.
+### List Command
 
-Options:
-      --names-only      Search only by trace id (ignore metadata).
-```
+The `list` command is used to list the available traces or workloads in the archive.
 
----
+### Get Command
 
-### `list`
+The `get` command is used to download a specified trace, workload or metadata file from the archive. The command options are:
 
-```bash
-$ python trace_archive.py list --help
-Usage:  python trace_archive.py list [OPTIONS]
+* `--trace`: Id of the trace to download.
+* `--workload`: Id of the workload to download. 
+* `--metadata`: Id of the metadata (same as the trace id) to download.
+* `-o, --output`: Output file path. If not specified, the file will be downloaded to the current working directory.
 
-List database traces or related entities.
+## Examples
 
-Options:
-      --traces        Lists available traces (default)
-      --workloads     Lists available workloads
+Assuming the trace archive tool is set up with a local storage type named `local`, you can use the following commands:
+
+### Uploading a Trace
+
+To upload a trace file named `dhrystone.zstf` and its workload `dhrystone`, present in the metadata example folder, you can run:
+
+```bash
+$ python trace_archive.py --storage-name local upload --workload ../../stf_metadata/example/dhrystone --trace ../../stf_metadata/example/dhrystone.zstf
+
+Uploading workload: ../../stf_metadata/example/dhrystone with id: 0
+Uploading trace: ../../stf_metadata/example/dhrystone.zstf with id: 0.0.0000_dhrystone
 ```
 
----
+### Downloading a Trace
+
+To download the trace file `000.000.000_dhrystone.zstf` and its metadata, you can run:
+
+```bash
+$ python trace_archive.py get --trace 000.000.000_dhrystone.zstf
 
-### `get`
+Trace 0.0.0000_dhrystone saved on ./0.0.0000_dhrystone.zstf
+Metadata 0.0.0000_dhrystone saved on ./0.0.0000_dhrystone.zstf.metadata.yaml
+```
+
+### Downloading a Workload
 
-Downloads a specified trace file.
+To download the workload `0` (dhrystone), you can run:
 
 ```bash
-$ python trace_archive.py get --help
-Usage:  python trace_archive.py get [OPTIONS] TRACE
+$ python trace_archive.py get --workload 0
 
-Download a specified trace file.
+Workload 0 saved on ./dhrystone
+```
 
-Arguments:
-      TRACE             Name of the trace to download.
+### Creating and using second storage source
 
-Options:
-      --revision        Revision number. If not specified, the latest revision is used.
-      --company         Filter by associated company.
-      --author          Filter by author.
-      -o, --output      Output file path.
+To create a second storage source you can run the `setup` command with the `--add-storage` option:
+
+```bash
+$ python trace_archive.py setup --add-storage
+Creating a new storage source.
+Registred storage type options: local-storage
+Select your storage type: local-storage
+Enter your storage name: private-storage
+Enter the storage folder path: ./private
+```
+
+This will create a new storage source named `private-storage` with the path `./private`. You can then use this storage source in the `upload` command by specifying the `--storage` option:
+
+```bash
+$ python trace_archive.py --storage-name private-storage upload --workload ../../stf_metadata/example/dhrystone --trace ../../stf_metadata/example/dhrystone.zstf
 ```
 
 ## Trace ID
@@ -135,13 +204,13 @@ Where:
 
 | Upload # | Description                                               | Trace ID                |
 | -------- | --------------------------------------------------------- | ----------------------- |
-| 1st      | `dhrystone` compiled with `-O3`, fully traced             | `000.000.000_dhrystone` |
-| 2nd      | `dhrystone` `-O3`, traced from instruction 0 to 1,000,000 | `000.001.000_dhrystone` |
-| 3rd      | `dhrystone` `-O3`, traced from 1,000,000 to 2,000,000     | `000.001.001_dhrystone` |
-| 4th      | `dhrystone` `-O3`, traced from 2,000,000 to 3,000,000     | `000.001.002_dhrystone` |
-| 5th      | Same trace as 1st (re-uploaded)                           | `000.002.000_dhrystone` |
-| 6th      | `dhrystone` compiled with `-O2`, fully traced             | `001.000.000_dhrystone` |
-| 7th      | `embench` compiled with `-O3`, fully traced               | `002.000.000_embench`   |
+| 1st      | `dhrystone` compiled with `-O3`, fully traced             | `0.0.0000_dhrystone` |
+| 2nd      | `dhrystone` `-O3`, traced from instruction 0 to 1,000,000 | `0.1.0000_dhrystone` |
+| 3rd      | `dhrystone` `-O3`, traced from 1,000,000 to 2,000,000     | `0.1.0001_dhrystone` |
+| 4th      | `dhrystone` `-O3`, traced from 2,000,000 to 3,000,000     | `0.1.0002_dhrystone` |
+| 5th      | Same trace as 1st (re-uploaded)                           | `0.2.0000_dhrystone` |
+| 6th      | `dhrystone` compiled with `-O2`, fully traced             | `0.0.0000_dhrystone` |
+| 7th      | `embench` compiled with `-O3`, fully traced               | `0.0.0000_embench`   |
 
 ---
 
@@ -152,33 +221,33 @@ For the trace archive structure, each workload is stored in its own folder, iden
 The tree graph below illustrates a setup of the [Trace Id Example](#example-trace-ids):
 
 ```text
-000/
+0000_dhrystone/
 ├── dhrystone
 ├── dhrystone.objdump
 ├── dhrystone.stdout
-├── 000/
-│   ├── 000.000.000_dhrystone.zstf
-│   └── 000.000.000_dhrystone.zstf.metadata.yaml
-├── 001/
-│   ├── 000.001.000_dhrystone.zstf
-│   ├── 000.001.000_dhrystone.zstf.metadata.yaml
-│   ├── 000.001.001_dhrystone.zstf
-│   ├── 000.001.001_dhrystone.zstf.metadata.yaml
-│   ├── 000.001.002_dhrystone.zstf
-│   └── 000.001.002_dhrystone.zstf.metadata.yaml
-001/
+├── attempt_0000/
+│   ├── 0.0.0000_dhrystone.zstf
+│   └── 0.0.0000_dhrystone.zstf.metadata.yaml
+├── attempt_0001/
+│   ├── 0.1.0000_dhrystone.zstf
+│   ├── 0.1.0000_dhrystone.zstf.metadata.yaml
+│   ├── 0.1.0001_dhrystone.zstf
+│   ├── 0.1.0001_dhrystone.zstf.metadata.yaml
+│   ├── 0.1.0002_dhrystone.zstf
+│   └── 0.1.0002_dhrystone.zstf.metadata.yaml
+0001_dhrystone/
 ├── dhrystone
 ├── dhrystone.objdump
 ├── dhrystone.stdout
-└── 000/
-    ├── 001.000.000_dhrystone.zstf
-    └── 001.000.000_dhrystone.zstf.metadata.yaml
-002/
+└── attempt_0000/
+    ├── 1.0.0000_dhrystone.zstf
+    └── 1.0.0000_dhrystone.zstf.metadata.yaml
+0002_embench/
 ├── embench.zip
 ├── embench.objdump
 ├── embench.stdout
-└── 000/
-    ├── 002.000.000_embench.zstf
-    └── 002.000.000_embench.zstf.metadata.yaml
+└── attempt_0000/
+    ├── 2.0.0000_embench.zstf
+    └── 2.0.0000_embench.zstf.metadata.yaml
 
 ```

traces/stf_trace_archive/src/data/config.py

@@ -0,0 +1,46 @@
+from dataclasses import dataclass
+from typing import Dict, Optional, Type, Union, List
+
+
+@dataclass
+class LocalStorageConfig:
+    path: str
+
+
+CONFIG_TYPE_MAP: Dict[str, Type] = {
+    "local-storage": LocalStorageConfig,
+}
+
+
+@dataclass
+class StorageConfig:
+    type: str
+    name: str
+    config: Union[LocalStorageConfig]
+
+    @staticmethod
+    def from_dict(data: dict):
+        specific_config_type = data['type']
+        if specific_config_type not in CONFIG_TYPE_MAP:
+            raise ValueError(f"Unknown storage type: {specific_config_type}")
+
+        specific_config_class = CONFIG_TYPE_MAP.get(specific_config_type)
+        specific_config = specific_config_class(**data['config'])
+        return StorageConfig(type=data['type'], name=data['name'], config=specific_config)
+
+
+@dataclass
+class Config:
+    storages: List[StorageConfig]
+    default_storage: Optional[str]
+
+    @staticmethod
+    def from_dict(data: dict):
+        if not data:
+            return None
+
+        storages = []
+        if 'storages' in data:
+            storages = [StorageConfig.from_dict(s) for s in data['storages']]
+
+        return Config(storages=storages, default_storage=data.get('default_storage'))

traces/stf_trace_archive/src/data/consts.py

@@ -0,0 +1,6 @@
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class Const():
+    PAD_LENGHT = 4