Call start time performance tests

[iAmmar7]

Description

An E2E Test to make sure the call start time does not exceed 5 seconds;

1. CF SDK: Video call test
2. CF SDK: Audio-only call test
3. Video SDK: Room session test

Type of change

• Internal refactoring
• Bug fix (bugfix - non-breaking)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Code snippets

In case of new feature or breaking changes, please include code snippets.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 70d3604

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[jpsantosbh]

This is very interesting!

Unfortunately I don't think we can have this as an e2e test on the SDK CI pipeline.

This will be great tools for monitoring the whole platform performance, and we should keep it for that.

The problem is, If the goal is to verify any performance issues on the SDK implementation, and make sure we don't introduce code that delays the call setup. Then we need a test that excludes the external variables like network, the server, etc... And test only the SDK implementation performance. Otherwise, we risk introducing an even more flakier test on the SDK CI.

You can verify this in the last run... this test passed and production but failed on staging.
Since its the same SDK implementation you can see that other variables(server, runner, network) will influence this more then the SDK implementation.

We either need to isolate only the SDK implementation performance or we should keep this test disable in the CI pipeline and only run it in production health checks.

[iAmmar7]

Well, I agree that ideally the test should exclude external variables like network, region, etc. This was just an initial performance test so we can actually determine whether the PR introduces any delays. The test fails in staging because staging is slow.

Tomorrow, if someone creates a PR and the performance test in production consistently fails, we will know there’s something wrong with the PR.

[jpsantosbh]

Tomorrow, if someone creates a PR and the performance test in production consistently fails, we will know there’s something wrong with the PR.

Unfortunately, that's not true
Most of the negotiation time is network and servers, which means that any failure on these tests is most likely to be because of that and not SDK implementation. Tomorrow, if this test fails in production, we still need to exclude the network and the server performance. Before we can say that's an SDK issue.

I think it's a good test. We should keep it.
But to execute it manually or in the RTC tests.

Just not in the CI pipeline. Just remove them from the pipeline setup.

[jpsantosbh]

Could we run this and not fail? Like, just add it as a report. In a separate GHA workflow?

[LittorWired]

do you want to measure the dial() as well?

[iAmmar7]

I am not sure, dial() is a sync method, which just initiates the JS class and some Saga workers. Maybe we can write a different test for that? Or maybe one more test including both dial and start? 👀

[LittorWired]

same as above comment: just wondering if we also want to have seperate measurements that include dial()?

[LittorWired]

Could we run this and not fail? Like, just add it as a report. In a separate GHA workflow?

You can report it but using a "soft assertion"
https://playwright.dev/docs/test-assertions#soft-assertions

You might want to check what it looks like when it fails in CI otherwise it might go unnoticed if it's just a ✅ in CI anyway.

[LittorWired]

LGTM 👍 , but I'll defer to @jpsantosbh and the feedback provided

[jpsantosbh]

Could we run this and not fail? Like, just add it as a report. In a separate GHA workflow?

You can report it but using a "soft assertion" https://playwright.dev/docs/test-assertions#soft-assertions

You might want to check what it looks like when it fails in CI otherwise it might go unnoticed if it's just a ✅ in CI anyway.

This is interesting to explore and might also be useful in other e2e tests, too.