-
Notifications
You must be signed in to change notification settings - Fork 134
[Baseline Profiles] Test Generation & Benchmarking via FTL (Without Login) #14256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
c9e1627
Build: File new module with baseline profile generator
ParaskP7 1178c79
Build: Update auto-generated new module with baseline profile generator
ParaskP7 177697b
Build: Enable baseline profile dex layout optimization for woocommerce
ParaskP7 358f021
Logs: Use profile verifier compilation status to debug baseline profiles
ParaskP7 e2d6782
Build: Use gradle managed device via firebase test lab to generate bp
ParaskP7 c2b88cb
CI: Test generate baseline profile step on ci
ParaskP7 36ea8af
CI: Test benchmark baseline profile step on ci
ParaskP7 e65f0e4
CI: Test generate and benchmark baseline profile on same ci job
ParaskP7 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,108 +1,25 @@ | ||
| # yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json | ||
| --- | ||
|
|
||
| # Nodes with values to reuse in the pipeline. | ||
| common_params: | ||
| # Common plugin settings to use with the `plugins` key. | ||
| - &test_collector_common_params | ||
| files: "WooCommerce/build/buildkite-test-analytics/*.xml" | ||
| format: "junit" | ||
|
|
||
| agents: | ||
| queue: "android" | ||
|
|
||
| steps: | ||
| - label: Gradle Wrapper Validation | ||
| command: validate_gradle_wrapper | ||
| agents: | ||
| queue: linter | ||
|
|
||
| # Wait for Gradle Wrapper to be validated before running any other jobs | ||
| - wait | ||
|
|
||
| ######################################## | ||
| - group: "🕵️ Linters" | ||
| steps: | ||
|
|
||
| - label: "☢️ Danger - PR Check" | ||
| command: danger | ||
| key: danger | ||
| if: "build.pull_request.id != null" | ||
| retry: | ||
| manual: | ||
| permit_on_passed: true | ||
| agents: | ||
| queue: "linter" | ||
|
|
||
| - label: "detekt" | ||
| command: | | ||
| if .buildkite/commands/should-skip-job.sh --job-type validation; then | ||
| exit 0 | ||
| fi | ||
| ./gradlew detektAll | ||
| plugins: [$CI_TOOLKIT] | ||
| artifact_paths: | ||
| - "**/build/reports/detekt/detekt.html" | ||
|
|
||
| - label: "lint" | ||
| command: .buildkite/commands/lint.sh | ||
| plugins: [$CI_TOOLKIT] | ||
| artifact_paths: | ||
| - "**/build/reports/lint-results*.*" | ||
|
|
||
| - label: "Dependency Tree Diff" | ||
| command: comment_with_dependency_diff 'woocommerce' 'vanillaReleaseRuntimeClasspath' | ||
| if: build.pull_request.id != null | ||
| plugins: [$CI_TOOLKIT] | ||
| artifact_paths: | ||
| - "**/build/reports/diff/*" | ||
|
|
||
| - label: "Merged Manifest Diff" | ||
| command: ".buildkite/commands/diff-merged-manifest.sh vanillaRelease" | ||
| if: build.pull_request.id != null | ||
| plugins: [$CI_TOOLKIT] | ||
| artifact_paths: | ||
| - "**/build/reports/diff_manifest/**/**/*" | ||
|
|
||
| ######################################## | ||
| - group: "🛠 Prototype Builds" | ||
| steps: | ||
|
|
||
| - label: "🛠 Prototype Build: Mobile App" | ||
| command: ".buildkite/commands/prototype-build.sh WooCommerce" | ||
| if: build.pull_request.id != null | ||
| plugins: [$CI_TOOLKIT] | ||
|
|
||
| - label: "🛠 Prototype Build: Wear App" | ||
| command: ".buildkite/commands/prototype-build.sh WooCommerce-Wear" | ||
| if: build.pull_request.id != null | ||
| plugins: [$CI_TOOLKIT] | ||
|
|
||
| ######################################## | ||
| - group: "🔬 Tests" | ||
| steps: | ||
|
|
||
| - label: "Unit tests" | ||
| command: .buildkite/commands/run-unit-tests.sh | ||
| plugins: | ||
| - $CI_TOOLKIT | ||
| - $TEST_COLLECTOR : | ||
| <<: *test_collector_common_params | ||
| api-token-env-name: "BUILDKITE_ANALYTICS_TOKEN_UNIT_TESTS" | ||
| artifact_paths: | ||
| - "**/build/test-results/merged-test-results.xml" | ||
|
|
||
| - label: "Instrumented tests" | ||
| command: .buildkite/commands/run-instrumented-tests.sh | ||
| plugins: | ||
| - $CI_TOOLKIT | ||
| - $TEST_COLLECTOR : | ||
| <<: *test_collector_common_params | ||
| api-token-env-name: "BUILDKITE_ANALYTICS_TOKEN_INSTRUMENTED_TESTS" | ||
| artifact_paths: | ||
| - "**/build/instrumented-tests/**/*" | ||
|
|
||
|
|
||
| - label: "🐘 Populate Gradle build cache" | ||
| command: .buildkite/commands/gradle-cache-build.sh | ||
| plugins: [$CI_TOOLKIT] | ||
| - label: "Generate & Benchmark Baseline Profile" | ||
| key: "generate-baseline-profile" | ||
| command: | | ||
| echo "--- :rubygems: Setting up Gems" | ||
| install_gems | ||
|
|
||
| echo "--- :closed_lock_with_key: Installing Secrets" | ||
| bundle exec fastlane run configure_apply | ||
|
|
||
| echo "--- :hammer_and_wrench: Generate Baseline Profile" | ||
| ./gradlew :WooCommerce:generateVanillaReleaseBaselineProfile | ||
|
|
||
| echo "--- :hammer_and_wrench: Benchmark Baseline Profile" | ||
| ./gradlew :baselineprofile:ftlDeviceTokay35VanillaBenchmarkReleaseAndroidTest | ||
| plugins: [ $CI_TOOLKIT ] | ||
| artifact_paths: | ||
| - "**/src/vanillaRelease/generated/baselineProfiles/*" | ||
| - "**/build/outputs/androidTest-results/managedDevice/benchmarkrelease/flavors/vanilla/**/results/**/artifacts/storage/emulated/0/Android/media/**/com.woocommerce.android.baselineprofile-benchmarkData.json" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /build |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| import com.android.build.api.dsl.ManagedVirtualDevice | ||
|
|
||
| plugins { | ||
| alias(libs.plugins.android.test) | ||
| alias(libs.plugins.kotlin.android) | ||
| alias(libs.plugins.androidx.baselineprofile) | ||
| alias(libs.plugins.firebase.testlab) | ||
| } | ||
|
|
||
| android { | ||
| namespace 'com.woocommerce.android.baselineprofile' | ||
| compileSdk gradle.ext.compileSdkVersion | ||
|
|
||
| compileOptions { | ||
| sourceCompatibility = libs.versions.java.get() | ||
| targetCompatibility = libs.versions.java.get() | ||
| } | ||
|
|
||
| defaultConfig { | ||
| minSdk 28 // Cant use current "gradle.ext.minSdkVersion", which points to 26, because various calls requires API level 28. | ||
| targetSdk gradle.ext.targetSdkVersion | ||
|
|
||
| testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner" | ||
| } | ||
|
|
||
| targetProjectPath = ":WooCommerce" | ||
|
|
||
| flavorDimensions += ["buildType"] | ||
| productFlavors { | ||
| vanilla { dimension = "buildType" } | ||
| } | ||
|
|
||
| // This code creates the gradle managed device used to generate baseline profiles. | ||
| // To use GMD please invoke generation through the command line: | ||
| // ./gradlew :WooCommerce:generateVanillaReleaseBaselineProfile | ||
| testOptions.managedDevices.devices { | ||
| pixel6Api34(ManagedVirtualDevice) { | ||
| device = "Pixel 6" | ||
| apiLevel = 34 | ||
| systemImageSource = "google" | ||
| } | ||
| } | ||
| } | ||
|
|
||
| def directory = "/storage/emulated/0/Android/media/${android.namespace}" | ||
| firebaseTestLab { | ||
| serviceAccountCredentials = rootProject.file(".configure-files/firebase.secrets.json") | ||
| managedDevices { | ||
| "ftlDeviceTokay35" { | ||
| device = "tokay" | ||
| apiLevel = 35 | ||
| } | ||
| } | ||
| testOptions { | ||
| results { | ||
| directoriesToPull.addAll(directory.toString()) | ||
| recordVideo = true | ||
| performanceMetrics = true | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // This is the configuration block for the Baseline Profile plugin. | ||
| // You can specify to run the generators on a managed devices or connected devices. | ||
| baselineProfile { | ||
| managedDevices += "ftlDeviceTokay35" | ||
| useConnectedDevices = false | ||
| } | ||
|
|
||
| dependencies { | ||
| implementation libs.androidx.test.ext.junit | ||
| implementation libs.androidx.test.espresso.core | ||
| implementation libs.androidx.test.uiautomator | ||
| implementation libs.androidx.benchmark.macro.junit4 | ||
| } | ||
|
|
||
| androidComponents { | ||
| onVariants(selector().all()) { v -> | ||
| def artifactsLoader = v.artifacts.getBuiltArtifactsLoader() | ||
| v.instrumentationRunnerArguments.put( | ||
| "targetAppId", | ||
| v.testedApks.map { artifactsLoader.load(it)?.applicationId } | ||
| ) | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The log message for RESULT_CODE_ERROR_UNSUPPORTED_API_VERSION appears misleading. It should probably log something like 'Unsupported API version' instead of 'Enqueued for compilation'.
Copilot uses AI. Check for mistakes.