profile
viewpoint

Ask questionsAllow asynchronously skipping tests

🚀 Feature Proposal

Other testing frameworks allow tests to asynchronously decide whether they should skip themselves.

For example, in Mocha:

it('tests a remote service', async function() {
  if (!(await remoteService.isActive())) {
    return this.skip()
  }
  … test the remote service …
})

Currently, however, it is impossible to asynchronously decide whether a test should be skipped in Jest.

See also: discussion here: https://github.com/facebook/jest/issues/7245

Motivation

Some tests either depend on - or are explicitly testing - remote services which may or may not be available.

Without being able to programatically and asynchronously decide whether tests can be skipped, there are only three options for writing these sorts of tests:

  1. Decide that they will either always pass or always fail if the service is unavailable. In either case the result can be misleading (ie, because in many cases "failure" indicates "the service is wrong", not merely "the service is unavailable", and "passing" suggests that everything is okay, which is also not necessarily true).

  2. Keep them in a separate suite, one per remote service, which can be run with (for example), npm run test:service-a).

  3. Use a regular expression (or similar) to include / exclude these tests from a test run.

Example

A complete, real-world (but anonymized) example from a Mocha-based test suite:

describe('example.com', () => {
  beforeAll(async function() {
    try {
      await fetch('https://example.com/api/whoami')
    } catch (e) {
      return this.skip(`Skipping tests for example.com: ${e}`)
    }
  })

  it('returns the current user', async () => { … })
  it('does the other thing', async () => { … })
})

Pitch

This belongs in Jest core because:

  1. It's currently supported by Mocha: https://mochajs.org/ (search for this.skip)
  2. It's impossible to implement outside of core (see elaboration below)
  3. The related discussion on #7245 suggests that it's important to a number of people (see, ex, this comment, which as of this writing has 16 👍 : https://github.com/facebook/jest/issues/7245#issuecomment-432240159)

FAQ

Why can't you use an if-statement?

A common suggestion in #7245 is to use an if-statement (or similar) to skip tests:

describe('example.com', () => {
  const isActive = remoteService.isActive()
  if (isActive) it('returns the current user', async () => { … })
})

However, this will not work for asynchronous tests, as tests must be declared synchronously, but the "is a remote service active?" check is necessarily asynchronous.

Wouldn't it be better if the tests failed/succeeded/retried/did something else?

There are situations when this is true, but (as evidenced by discussion on #7245) there are also situations where "skip tests when a remote service is not available" is a reasonable business decision (ex: https://github.com/facebook/jest/issues/7245#issuecomment-474587219)

facebook/jest

Answer questions ClaytonSmith

@mattphillips What would your solution be for platform specific tests?

It would be wildly inappropriate to falsely mark tests as passed or failed when it doesn't even make sense to run the test.

The questions we want our test frameworks to answer are: Can I run this test? Did the test pass? A boolean Pass/Fail cannot answer both of those question.

useful!

Related questions

"Received: serializes to the same string" on object equality checking hot 451
The module factory of `jest.mock()` is not allowed to reference any out-of-scope variables hot 230
jest-worker and typescript unexpected behavior hot 224
"Received: serializes to the same string" on object equality checking hot 190
Async callback was not invoked within the 5000ms timeout - Error Message Improvement hot 177
spyOn getter requires explicit type on getter after Jest 24 hot 173
Cannot spyOn on a primitive value; undefined given hot 171
Upgrade to fsevent v2 - Node 13 support hot 161
window.location.href can't be changed in tests. hot 153
Jest failing with "Call retries were exceeded", using --runInBand works-around issue hot 149
jest-resolve cannot resolve location in network mapped folder on Windows hot 148
24.1.0: TypeError: Cannot read property 'cwd' of undefined hot 135
Error: jest-haste-map: Haste module naming collision hot 133
Mocking read-only properties for a class hot 125
TypeError: environment.teardown is not a function at node_modules/jest-runner/build/run_test.js hot 109
Github User Rank List