Skip to content

Conversation

minhancao
Copy link
Contributor

@minhancao minhancao commented Aug 21, 2025

Improved organization of pbench output directory to navigate easier for streams runs.

Example 3 streams run:

pbench command ran:

../pbench_mcao run -n testing_output_dir_change_%t -s http://127.0.0.1:8081 -o /root/oss_pbench/pbench/benchmarks/benchmark_base native_oss.json save_output.json save_json.json
 tpch/sf1_tpchstandard.json tpch/3streams.json > pbench_output2.txt 2>&
1

3streams.json:

{
    "description": "Launch 3 clients, each running from tpch.",
    "next": [
      "tpch_stream1.json",
      "tpch_stream2.json",
      "tpch_stream3.json"
    ],
    "query_files": [
      "queries_tpchstandard/query_01.sql"
    ]
}


tpch_stream1.json:

{
    "description": "Sequentially run 3 TPCH queries.",
    "next": [
      "tpch_stream1_child.json"
    ],
    "query_files": [
      "queries_tpchstandard/query_01.sql"
    ],
    "expected_row_counts": {
      "tpch_sf1000_": [
        4,
        100,
        10
      ]
    }
}


tpch_stream2.json, tpch_stream3.json:

{
    "description": "Sequentially run 3 TPCH queries.",
    "query_files": [
      "queries_tpchstandard/query_01.sql"
    ],
    "expected_row_counts": {
      "tpch_sf1000_": [
        4,
        100,
        10
      ]
    }
}


tpch_stream1_child.json:

{
    "description": "Sequentially run 3 TPCH queries.",
    "query_files": [
      "queries_tpchstandard/query_01.sql"
    ],
    "expected_row_counts": {
      "tpch_sf1000_": [
        4,
        100,
        10
      ]
    }
}

image

@minhancao minhancao requested a review from allenshen13 August 21, 2025 00:01
@minhancao minhancao self-assigned this Aug 21, 2025
@minhancao minhancao requested a review from ethanyzhang as a code owner August 21, 2025 00:01
@minhancao minhancao added the enhancement New feature or request label Aug 21, 2025
@minhancao minhancao force-pushed the organize_pbench_output_directory branch from 9ef664f to f0b961c Compare August 21, 2025 23:01
@minhancao minhancao force-pushed the organize_pbench_output_directory branch 3 times, most recently from 0c29bc2 to 6b49b12 Compare September 3, 2025 23:15
@minhancao minhancao changed the title Improved organization of pbench output directory for easier navigation/viewing Improved organization of pbench output directory and detection of duplicate stages in nested stages Sep 3, 2025
@minhancao
Copy link
Contributor Author

@ethanyzhang I have changed the implementation to the way that we discussed and also improved on the checkStageLinks() to detect for duplicate stages in nested stages. Please review when you can, thanks!

@minhancao minhancao force-pushed the organize_pbench_output_directory branch from 6b49b12 to fdb5e72 Compare September 3, 2025 23:21
nextStageMap := make(map[string]bool)
for _, nextStage := range stage.NextStages {
if nextStage == nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it is nil, you probably want to error out right?

utils.PrepareOutputDirectory(stageOutputPath)

// Recursively call the function on the child stage
nextStage.createNextStagesOutputDirectories()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? when the next stage is run, it will call this again right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants