profile
viewpoint

Ask questionsAllow asynchronously skipping tests

🚀 Feature Proposal

Other testing frameworks allow tests to asynchronously decide whether they should skip themselves.

For example, in Mocha:

it('tests a remote service', async function() {
  if (!(await remoteService.isActive())) {
    return this.skip()
  }
  … test the remote service …
})

Currently, however, it is impossible to asynchronously decide whether a test should be skipped in Jest.

See also: discussion here: https://github.com/facebook/jest/issues/7245

Motivation

Some tests either depend on - or are explicitly testing - remote services which may or may not be available.

Without being able to programatically and asynchronously decide whether tests can be skipped, there are only three options for writing these sorts of tests:

  1. Decide that they will either always pass or always fail if the service is unavailable. In either case the result can be misleading (ie, because in many cases "failure" indicates "the service is wrong", not merely "the service is unavailable", and "passing" suggests that everything is okay, which is also not necessarily true).

  2. Keep them in a separate suite, one per remote service, which can be run with (for example), npm run test:service-a).

  3. Use a regular expression (or similar) to include / exclude these tests from a test run.

Example

A complete, real-world (but anonymized) example from a Mocha-based test suite:

describe('example.com', () => {
  beforeAll(async function() {
    try {
      await fetch('https://example.com/api/whoami')
    } catch (e) {
      return this.skip(`Skipping tests for example.com: ${e}`)
    }
  })

  it('returns the current user', async () => { … })
  it('does the other thing', async () => { … })
})

Pitch

This belongs in Jest core because:

  1. It's currently supported by Mocha: https://mochajs.org/ (search for this.skip)
  2. It's impossible to implement outside of core (see elaboration below)
  3. The related discussion on #7245 suggests that it's important to a number of people (see, ex, this comment, which as of this writing has 16 👍 : https://github.com/facebook/jest/issues/7245#issuecomment-432240159)

FAQ

Why can't you use an if-statement?

A common suggestion in #7245 is to use an if-statement (or similar) to skip tests:

describe('example.com', () => {
  const isActive = remoteService.isActive()
  if (isActive) it('returns the current user', async () => { … })
})

However, this will not work for asynchronous tests, as tests must be declared synchronously, but the "is a remote service active?" check is necessarily asynchronous.

Wouldn't it be better if the tests failed/succeeded/retried/did something else?

There are situations when this is true, but (as evidenced by discussion on #7245) there are also situations where "skip tests when a remote service is not available" is a reasonable business decision (ex: https://github.com/facebook/jest/issues/7245#issuecomment-474587219)

facebook/jest

Answer questions ClaytonSmith

@mattphillips What would your solution be for platform specific tests?

It would be wildly inappropriate to falsely mark tests as passed or failed when it doesn't even make sense to run the test.

The questions we want our test frameworks to answer are: Can I run this test? Did the test pass? A boolean Pass/Fail cannot answer both of those question.

useful!

Related questions

Lots of handlebars warnings with text+coverage reporter hot 4
Error: SyntaxError: Cannot use import statement outside a module hot 4
SyntaxError: Cannot use import statement outside a module hot 4
Upgrade to fsevent v2 - Node 13 support hot 3
Improve description of "Jest did not exit one second after the test run has completed" hot 3
ReferenceError: You are trying to `import` a file after the Jest environment has been torn down. hot 3
Jest prints superfluous warning when ran with '--passWithNoTests' and there are no tests hot 2
Error: jest-haste-map: Haste module naming collision hot 2
"Syntax Error: Invalid or unexpected token" with .png hot 2
babel-jest issues with monorepo and jest multi project runner hot 2
"Received: serializes to the same string" on object equality checking hot 2
Make test name available in beforeEach() and afterEach() hot 2
Jest encountered unexpected token with React app hot 2
Unexpected end of JSON input while parsing near '...,"semver":"^5.0.1","u' hot 2
jest-resolve fails on network drives and RAM-drives on Windows hot 2
Github User Rank List