Cypress Hangs Running Tests With Electron (Headless or Headed)

[nbcarey]

Current behavior:

Our CI/CI pipeline runs our Cypress tests on a schedule. This has been working as expected. However, yesterday (12 September 2019), the behavior changed. Instead of running to completion (with or without errors), the test suite now hangs on one particular test case. The CI/CD pipeline eventually times out after 60 minutes.

Running the tests manually on my machine with Chrome via npx cypress run --browser chrome or npx cypress open --browser chrome works just fine.

Running the tests manually with Electron, headless or headless, does not. It exhibits the same hang that our CI/CD pipeline exhibits.

Digging into it I see that it is hanging on the afterEach() action, trying to revisit the same page, via cy.visit() on which the test started. Playing around with things, It seems to fail the same way, any time cy.visit() is called twice in the context of the same test of describe() block.

Even odder, turning on debug logging via

env DEBUG=cypress:* npx cypress run --browser electron 2>&1 | tee cypress-log.txt

completed successfully.

Desired behavior:

Tests should run to completion and not hang.

Steps to reproduce: (app code and test code)

1. Clone this repo: https://github.com/nbcarey/cypress-io-hangs-with-electron
2. Running the command npm run succeeds-with-chrome gives you the happy path.
3. Running the command npm run hangs-with-electron gives you the unhappy path.

Versions

• MacOS Mojave, v10.14.6

• Chrome, v77.0.3865.75

• Node.js, v10.16.3

• Npm, v6.11.3

• Cypress, v3.4.1 (and Electron 61?)

[hiqqs]

headless electron for us stopped working for also. Noticed that electron is not able to connect to localhost as part of our test setup which is whats causing ours to hang. They are working fine in chrome but will be broken for CI

Can also resolve other domains no problem but cannot use our local web server, noticed after updating to latest webpack

[nbcarey]

I finally got almost all of our tests that were hanging working again.

Adding cy.wait() calls in strategic places seemed to fix some, but not all of the hangs, and breaking up single *.spec.js files into multiple files.

It's weird, because an individual test run by itself would work, but running the entire *.spec.js test file wouldn't. And once I got the test file to work in toto, running it as part of the entire test suite would fail.

My suspicion is some sort of race condition or timing issue in Electron. I did notice that when a test would hang, a process called Cypress Helper on my Mac would spin up to 100% CPU, consuming an entire core. Took a snapshot of it and seemed to be buried deep, deep somewhere within Electron.

Also noticed an Node.js error message getting written to stderr:

(node:16167) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exitEarlyWithErr listeners added. Use emitter.setMaxListeners() to increase limit

Wonder if that is tied to it.

[jennifer-shehane]

@nbcarey Thanks so much for providing a repo. I actually couldn't get the first case in chrome or electron from your repo to pass. Here's a screenshot and description of the failure I'm getting.

  1) product-module Any product "before each" hook for "Has a product title":
     CypressError: Timed out retrying: Expected to find element: 'div[data-element~=product-results-result-set-desktop] article[data-element~=product-results-product-module-desktop]', but never found it.

Because this error occurred during a 'before each' hook we are skipping the remaining tests in the current suite: 'Any product'

The mention of it just beginning to fail on a specific date would never be a failure caused from Cypress. Our releases our packaged complete applications, so nothing will change underneath. I would definitely encourage doing some debugging on your side to determine what changed in your application that could have triggered a change (even minor dep updates that seem innocuous could have created a condition that will fail in Cypress Electron).

[rudolfs]

Also just bumped into the same issue. Test suite passes in chrome 76, but hangs on electron 61 on CI, locally head-less and with a GUI. I could narrow it down to this:

• it only happens if there is more than one testcase in one file, this would be enough to trigger it:

context('session', () => {
  it('first', () => {
    cy.visit('/')
  })

  it('second', () => {
    cy.visit('/')
  })
})

• the app that we're testing makes GraphQL (apollo) requests to a backend; this backend is offline and in chrome we get an exception: OPTIONS http://localhost:8080/graphql net::ERR_CONNECTION_REFUSED

Will try to provide a repo with a test case soon.

[jagregory]

I'm also seeing this issue with Electron 61 in any tests where I need to switch users for a multi-user interaction.

[rudolfs]

Didn't manage to create a test repo yet, but here's the debug output of the test run of the small test-case from #5139 (comment):
https://gist.github.com/rudolfs/25bc14986d4d1451d98cee3c4307fb30

It seems to hang on trying to clean up a request from the first test, which has a timeout: null.

[hiqqs]

@rudolfs are you still having this issue?

[rudolfs]

@johnhiggs yes. I had to pause my investigation because of work priorities. I was trying to create a minimal test case but it's a little tricky because we use ReasonML.

[hiqqs]

We were finally able to figure out our issue today related to electron not working. It was related to our webpack upgrade and failing on a form try catch error block not cooperating with electron. IMO an electron bug

[julian-sf]

Yeah, so our team's been running into this issue as well. A LOT of the problems I've seen relating to this are just coming from old Electron bugs. Electron 61 was released TWO entire years ago on September 5, 2017. So yeah, this is going to be an issue until we can get a newer version of Electron running with Cypress.

In a way this IS a Cypress issue, because we need to be able to run on a more recent version of electron. The Cypress team is currently working on an upgrade, so it looks like we probably just have to wait until then.

• Roadmap
• Upgrade to the latest version of Electron #3568
• Electron v5.0.10 #4720

[flotwig]

Hey @julian-sf, Electron 5.0.10 upgrade (Chrome 73 - PR: #4720) is releasing with 3.5.0, which should come out later this week. So hopefully you won't need to wait much longer.

We're also working on upgrading to Electron 6 and keeping it up to date so this big gap doesn't happen again.

[julian-sf]

@flotwig Awesome! I know you guys have been working hard on it. I appreciate the update and I'm very much looking forward to the new version!

[fr0]

Hey @julian-sf, Electron 5.0.10 upgrade (Chrome 73 - PR: #4720) is releasing with 3.5.0, which should come out later this week. So hopefully you won't need to wait much longer.

I assume this got delayed for some reason?