Add support for manually running the real-time benchmark on PRs #19265
+149
−36
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,13 +2,44 @@ name: Real-time Benchmark | |
| on: | ||
| schedule: | ||
| - cron: "30 0 * * *" | ||
| workflow_dispatch: | ||
| inputs: | ||
| repository: | ||
| description: 'Repository name that is going to be benchmarked (e.g. "johndoe/php-src")' | ||
| required: true | ||
| type: string | ||
| branch: | ||
| description: 'Branch name that is going to be benchmarked (e.g. "my-branch")' | ||
| required: true | ||
| type: string | ||
| commit: | ||
| description: 'Commit SHA that is going to be benchmarked (e.g. "123456789a")' | ||
| required: true | ||
| type: string | ||
| baseline_commit: | ||
| description: 'A commit SHA that is compared against the one provided by the "commit" input (e.g. "abcdef123456")' | ||
| required: true | ||
| type: string | ||
| jit: | ||
| description: 'Whether JIT is benchmarked' | ||
| required: false | ||
| type: choice | ||
| options: | ||
| - "0" | ||
| - "1" | ||
| permissions: | ||
| contents: read | ||
| jobs: | ||
| REAL_TIME_BENCHMARK: | ||
| name: REAL_TIME_BENCHMARK | ||
| if: github.repository == 'php/php-src' || github.event_name == 'workflow_dispatch' | ||
| runs-on: ubuntu-22.04 | ||
| env: | ||
| REPOSITORY: ${{ inputs.repository || 'php/php-src' }} | ||
|
There was a problem hiding this comment. These fallbacks seem useless if all inputs are required? |
||
| BRANCH: ${{ inputs.branch || 'master' }} | ||
| COMMIT: ${{ inputs.commit || github.sha }} | ||
| BASELINE_COMMIT: ${{ inputs.baseline_commit || 'd5f6e56610c729710073350af318c4ea1b292cfe' }} | ||
| JIT: ${{ inputs.jit || '1' }} | ||
| steps: | ||
| - name: Install dependencies | ||
| run: | | ||
|
|
@@ -29,73 +60,143 @@ jobs: | |
| ref: 'main' | ||
| fetch-depth: 1 | ||
| path: 'php-version-benchmarks' | ||
| - name: Checkout php-src (benchmarked version) | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: '${{ env.REPOSITORY }}' | ||
| ref: '${{ env.COMMIT }}' | ||
| fetch-depth: 100 | ||
| path: 'php-version-benchmarks/tmp/php_${{ env.BRANCH }}' | ||
|
There was a problem hiding this comment. Could probably drop the dynamic path suffix |
||
| - name: Checkout php-src (baseline version) | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: '${{ env.REPOSITORY }}' | ||
| ref: '${{ env.BASELINE_COMMIT }}' | ||
| fetch-depth: 100 | ||
| path: 'php-version-benchmarks/tmp/php_baseline' | ||
| - name: Setup benchmark results | ||
| run: | | ||
| git config --global user.name "Benchmark" | ||
| git config --global user.email "[email protected]" | ||
| rm -rf ./php-version-benchmarks/docs/results | ||
| - name: Checkout benchmark data | ||
| if: github.repository == 'php/php-src' && github.event_name != 'workflow_dispatch' | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: php/real-time-benchmark-data | ||
| ssh-key: ${{ secrets.PHP_VERSION_BENCHMARK_RESULTS_DEPLOY_KEY }} | ||
| path: 'php-version-benchmarks/docs/results' | ||
| - name: Setup infra config | ||
| run: | | ||
| set -e | ||
| cp ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini.dist ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini | ||
| ESCAPED_DOCKER_REGISTRY=$(printf '%s\n' "${{ secrets.PHP_VERSION_BENCHMARK_DOCKER_REGISTRY }}" | sed -e 's/[\/&]/\\&/g') | ||
| sed -i "s/INFRA_DOCKER_REGISTRY=public.ecr.aws\/abcdefgh/INFRA_DOCKER_REGISTRY=$ESCAPED_DOCKER_REGISTRY/g" ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini | ||
| cp ./php-version-benchmarks/build/infrastructure/config/aws.tfvars.dist ./php-version-benchmarks/build/infrastructure/config/aws.tfvars | ||
| sed -i 's/access_key = ""/access_key = "${{ secrets.PHP_VERSION_BENCHMARK_AWS_ACCESS_KEY }}"/g' ./php-version-benchmarks/build/infrastructure/config/aws.tfvars | ||
| sed -i 's/secret_key = ""/secret_key = "${{ secrets.PHP_VERSION_BENCHMARK_AWS_SECRET_KEY }}"/g' ./php-version-benchmarks/build/infrastructure/config/aws.tfvars | ||
| - name: Setup PHP config - baseline PHP version | ||
| if: github.repository == 'php/php-src' && github.event_name != 'workflow_dispatch' | ||
| run: | | ||
| set -e | ||
| BASELINE_SHORT_SHA="$(echo "${{ env.BASELINE_COMMIT }}" | cut -c1-4)" | ||
| cat << EOF > /php-version-benchmarks/config/php/baseline.ini | ||
| PHP_NAME="PHP - baseline@$BASELINE_SHORT_SHA" | ||
| PHP_ID=php_baseline | ||
| PHP_REPO=https://github.com/${{ env.REPOSITORY }}.git | ||
| PHP_BRANCH=${{ env.BRANCH }} | ||
| PHP_COMMIT=${{ env.BASELINE_COMMIT }} | ||
| PHP_OPCACHE=2 | ||
| PHP_JIT=0 | ||
| EOF | ||
| - name: Setup PHP config - baseline PHP version with JIT (manual only) | ||
| if: github.repository == 'php/php-src' && github.event_name == 'workflow_dispatch' && inputs.jit == '1' | ||
| run: | | ||
| set -e | ||
| BASELINE_SHORT_SHA="$(echo "${{ env.BASELINE_COMMIT }}" | cut -c1-4)" | ||
| cat << EOF > /php-version-benchmarks/config/php/baseline.ini | ||
| PHP_NAME="PHP - baseline@$BASELINE_SHORT_SHA (JIT)" | ||
| PHP_ID=php_baseline_jit | ||
| PHP_REPO=https://github.com/${{ env.REPOSITORY }}.git | ||
| PHP_BRANCH=${{ env.BRANCH }} | ||
| PHP_COMMIT=${{ env.BASELINE_COMMIT }} | ||
| PHP_OPCACHE=2 | ||
| PHP_JIT=${{ env.JIT }} | ||
| EOF | ||
| git clone ./php-version-benchmarks/tmp/php_baseline/ ./php-version-benchmarks/tmp/php_baseline_jit | ||
| - name: Setup PHP config - previous PHP version (scheduled only) | ||
| if: github.repository == 'php/php-src' && github.event_name != 'workflow_dispatch' | ||
| run: | | ||
| set -e | ||
| YEAR="$(date '+%Y')" | ||
| DATABASE="./php-version-benchmarks/docs/results/$YEAR/database.tsv" | ||
| if [ -f "$DATABASE" ]; then | ||
| LAST_RESULT_SHA="$(tail -n 2 "$DATABASE" | head -n 1 | cut -f 6)" | ||
| else | ||
| YESTERDAY="$(date -d "-2 day 23:59:59" '+%Y-%m-%d %H:%M:%S')" | ||
| LAST_RESULT_SHA="$(cd ./php-version-benchmarks/tmp/php_${{ env.BRANCH }}/ && git --no-pager log --until="$YESTERDAY" -n 1 --pretty='%H')" | ||
| fi | ||
| cat << EOF > /php-version-benchmarks/config/php/previous.ini | ||
| PHP_NAME="PHP - previous ${{ env.BRANCH }}" | ||
| PHP_ID=php_${{ env.BRANCH }}_previous | ||
| PHP_REPO=https://github.com/${{ env.REPOSITORY }}.git | ||
| PHP_BRANCH=${{ env.BRANCH }} | ||
| PHP_COMMIT=$LAST_RESULT_SHA | ||
| PHP_OPCACHE=1 | ||
| PHP_JIT=0 | ||
| EOF | ||
| - name: Setup PHP config - benchmarked PHP version | ||
| run: | | ||
| set -e | ||
| cat << EOF > /php-version-benchmarks/config/php/this.ini | ||
| PHP_NAME="PHP - ${{ env.BRANCH }}" | ||
| PHP_ID=php_${{ env.BRANCH }} | ||
| PHP_REPO=https://github.com/${{ env.REPOSITORY }}.git | ||
| PHP_BRANCH=${{ env.BRANCH }} | ||
| PHP_COMMIT=${{ env.COMMIT }} | ||
| PHP_OPCACHE=1 | ||
| PHP_JIT=0 | ||
| EOF | ||
| - name: Setup PHP config - benchmarked PHP version with JIT | ||
| if: env.JIT == '1' | ||
| run: | | ||
| set -e | ||
| cat << EOF > /php-version-benchmarks/config/php/this_jit.ini | ||
| PHP_NAME="PHP - ${{ env.BRANCH }} (JIT)" | ||
| PHP_ID=php_${{ env.BRANCH }}_jit | ||
| PHP_REPO=https://github.com/${{ env.REPOSITORY }}.git | ||
| PHP_BRANCH=${{ env.BRANCH }} | ||
| PHP_COMMIT=${{ env.COMMIT }} | ||
| PHP_OPCACHE=1 | ||
| PHP_JIT=${{ env.JIT }} | ||
| EOF | ||
| git clone ./php-version-benchmarks/tmp/php_${{ env.BRANCH }}/ ./php-version-benchmarks/tmp/php_${{ env.BRANCH }}_jit | ||
| - name: Setup test config | ||
| run: | | ||
| set -e | ||
| cp ./php-version-benchmarks/config/test/1_laravel.ini.dist ./php-version-benchmarks/config/test/1_laravel.ini | ||
| cp ./php-version-benchmarks/config/test/2_symfony_main.ini.dist ./php-version-benchmarks/config/test/2_symfony_main.ini | ||
| cp ./php-version-benchmarks/config/test/4_wordpress.ini.dist ./php-version-benchmarks/config/test/4_wordpress.ini | ||
|
|
@@ -104,6 +205,7 @@ jobs: | |
| - name: Run benchmark | ||
| run: ./php-version-benchmarks/benchmark.sh run aws | ||
| - name: Store results | ||
| if: github.repository == 'php/php-src' && github.event_name != 'workflow_dispatch' | ||
| run: | | ||
| set -ex | ||
|
|
@@ -119,6 +221,17 @@ jobs: | |
| fi | ||
| git commit -m "Add result for ${{ github.repository }}@${{ github.sha }}" | ||
| git push | ||
| - name: Comment results | ||
| if: github.repository != 'php/php-src' && github.event_name == 'workflow_dispatch' | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| run: | | ||
| PR_NUMBER=$(gh pr list --head "${{ inputs.branch }}" --state open --json number --jq '.[0].number') | ||
|
There was a problem hiding this comment. You could request the PR number as input directly, especially if you switch to ref / baseline_ref. |
||
| if [ ! -z "$PR_NUMBER" ]; then | ||
| YEAR="$(date '+%Y')" | ||
| NEWEST_RESULT_DIRECTORY=$(ls -td ./php-version-benchmarks/docs/results/$YEAR/*/ | head -1) | ||
| gh pr comment $PR_NUMBER --body-file "$NEWEST_RESULT_DIRECTORY/result.md" | ||
| fi | ||
| - name: Cleanup | ||
| if: always() | ||
| run: | | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just use ref and baseline_ref? This can be either a branch, tag, commit, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Possibly, this could even request just the PR number instead of repository, branch, commit, baseline_commit; as we can determine all of these from the PR.