Skip to content

Refactor tests to use test cases #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 15, 2023
Merged

Refactor tests to use test cases #7

merged 2 commits into from
Jun 15, 2023

Conversation

remcohaszing
Copy link
Member

Initial checklist

  • I read the support docs
  • I read the contributing guide
  • I agree to follow the code of conduct
  • I searched issues and couldn’t find anything (or linked relevant results below)
  • If applicable, I’ve added docs and tests

Description of changes

Using test cases provides a better DX.

  • It allows running multiple tests
  • It shows which failed / succeeded
  • It enables test filtering
  • Assertion messages hid diffs in node:test

A describe block was added for the function that’s being tested. I tend to like this if multiple functions are exported. In this case, where only one function is exported, it’s a bit less useful IMO.

No actual test logic was changed. Most changes are related to indentation.

Using test cases provides a better DX.

- It allows running multiple tests
- It shows which failed / succeeded
- It enables test filtering
- Assertion messages hid diffs in `node:test`

A describe block was added for the function that’s being tested.
@remcohaszing remcohaszing added 🕸️ area/tests This affects tests 🤞 phase/open Post is being triaged manually labels Jun 15, 2023
@wooorm
Copy link
Member

wooorm commented Jun 15, 2023

Why describe? Why not nest tests?
This looks buggy: if a test case would be async, I think it would break?
This works: https://github.com/vfile/vfile/blob/86677e2a739992fc6d7a2bd13a8b7c318f3e3361/test.js#L53-L68

Assertion messages hid diffs in node:test

I don’t understand this? Can you repeat in different words?

@remcohaszing
Copy link
Member Author

describe just groups tests together into one test suite. Since test() just schedules a test, there’s no need to await anything.

Tests with nested tests are still tests. Nested tests need to finish before their parent, so the parent must await the t.test() call.

import assert from 'node:assert/strict'
import {describe, test} from 'node:test'
import {setTimeout} from 'node:timers/promises'

describe('describe', () => {
  let value

  test('test 1', async () => {
    value = 1
    await setTimeout(1000)
    assert.equal(value, 1)
  })

  test('tes 2', async () => {
    value = 2
    assert.equal(value, 2)
  })
})

test('nested test', (t) => {
  let value

  t.test('t.test 1', async () => {
    value = 1
    await setTimeout(1000)
    assert.equal(value, 1)
  })

  t.test('t.test2', async () => {
    value = 2
    assert.equal(value, 2)
  })
})

yields

▶ describe
  ✔ test 1 (1005.279917ms)
  ✔ tes 2 (0.516756ms)
▶ describe (1012.74256ms)

▶ nested test
  ✖ t.test 1 (2.110934ms)
    'test did not finish before its parent and was cancelled'

  ✖ t.test2
    'test did not finish before its parent and was cancelled'

▶ nested test (9.796304ms)

ℹ tests 5
ℹ suites 1
ℹ pass 2
ℹ fail 1
ℹ cancelled 2
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 2023.04679

✖ failing tests:

✖ t.test 1 (2.110934ms)
  'test did not finish before its parent and was cancelled'

✖ t.test2
  'test did not finish before its parent and was cancelled'

✖ nested test (9.796304ms)
  '2 subtests failed'

Assertion messages hid diffs in node:test

I don’t understand this? Can you repeat in different words?

This assertion without a message

import assert from 'node:assert/strict'
import {test} from 'node:test'

test('test', () => {
  assert.deepEqual({foo: 1}, {bar: 2})
})

yields

✖ test (2.74485ms)
  AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
  + actual - expected
  
    {
  +   foo: 1
  -   bar: 2
    }
      at TestContext.<anonymous> (file:///home/remco/Projects/estree-util-build-jsx/bar.test.js:5:10)
      at Test.runInAsyncScope (node:async_hooks:203:9)
      at Test.run (node:internal/test_runner/test:550:25)
      at Test.start (node:internal/test_runner/test:466:17)
      at startSubtest (node:internal/test_runner/harness:203:17) {
    generatedMessage: true,
    code: 'ERR_ASSERTION',
    actual: [Object],
    expected: [Object],
    operator: 'deepStrictEqual'
  }

ℹ tests 1
ℹ suites 0
ℹ pass 0
ℹ fail 1
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 9.405037

✖ failing tests:

✖ test (2.74485ms)
  AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
  + actual - expected
  
    {
  +   foo: 1
  -   bar: 2
    }
      at TestContext.<anonymous> (file:///home/remco/Projects/estree-util-build-jsx/bar.test.js:5:10)
      at Test.runInAsyncScope (node:async_hooks:203:9)
      at Test.run (node:internal/test_runner/test:550:25)
      at Test.start (node:internal/test_runner/test:466:17)
      at startSubtest (node:internal/test_runner/harness:203:17) {
    generatedMessage: true,
    code: 'ERR_ASSERTION',
    actual: [Object],
    expected: [Object],
    operator: 'deepStrictEqual'
  }

This assertion with a message

import assert from 'node:assert/strict'
import {test} from 'node:test'

test('test', () => {
  assert.deepEqual({foo: 1}, {bar: 2}, 'objects should be equal')
})

yields (the diff is gone)

✖ test (1.631264ms)
  AssertionError [ERR_ASSERTION]: objects should be equal
      at TestContext.<anonymous> (file:///home/remco/Projects/estree-util-build-jsx/bar.test.js:5:10)
      at Test.runInAsyncScope (node:async_hooks:203:9)
      at Test.run (node:internal/test_runner/test:550:25)
      at Test.start (node:internal/test_runner/test:466:17)
      at startSubtest (node:internal/test_runner/harness:203:17) {
    generatedMessage: false,
    code: 'ERR_ASSERTION',
    actual: [Object],
    expected: [Object],
    operator: 'deepStrictEqual'
  }

ℹ tests 1
ℹ suites 0
ℹ pass 0
ℹ fail 1
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 9.309917

✖ failing tests:

✖ test (1.631264ms)
  AssertionError [ERR_ASSERTION]: objects should be equal
      at TestContext.<anonymous> (file:///home/remco/Projects/estree-util-build-jsx/bar.test.js:5:10)
      at Test.runInAsyncScope (node:async_hooks:203:9)
      at Test.run (node:internal/test_runner/test:550:25)
      at Test.start (node:internal/test_runner/test:466:17)
      at startSubtest (node:internal/test_runner/harness:203:17) {
    generatedMessage: false,
    code: 'ERR_ASSERTION',
    actual: [Object],
    expected: [Object],
    operator: 'deepStrictEqual'
  }

IMO the former is much more useful.

@wooorm
Copy link
Member

wooorm commented Jun 15, 2023

Since test() just schedules a test

My point is that test doesn’t just schedule tests. Or at least: it has the exact same report printed that you get with describe. Here’s the test suite from vfile:

✔ core (2.229159ms)
▶ new VFile(options?)
  ✔ should accept missing options (0.246972ms)
  ✔ should accept a string (0.133248ms)
  ✔ should accept a vfile (0.187441ms)
  // …
  ✔ #message(reason[, position][, origin]) (0.705131ms)
  ✔ #fail(reason[, position][, origin]) (0.234225ms)
  ✔ #info(reason[, position][, origin]) (0.204965ms)
▶ new VFile(options?) (10.694484ms)

▶ p (POSIX path for browsers)
  ✔ basename (0.515743ms)
  ✔ dirname (0.245249ms)
  ✔ extname (0.3752ms)
  ✔ join (1.175445ms)
▶ p (POSIX path for browsers) (2.561059ms)

I dunno:
a) I don‘t really like describe, it feels old mocha-style describe/it/etc
b) it feels a bit useless to do the same as test but slightly different? we can use test too?


IMO the former is much more useful.

The output of Node’s test runner is TAP. That means that you can use formatters, such as for example tap-difflet to print how you want, and in this case a diff, here’s what I get:

  Subtest: test
    ⨯ test

      {
        "foo" : 1 // != undefined
        // "bar" : 2
      }
Screenshot 2023-06-15 at 11 49 21

So it’s possible to access the diff.

As for moving the messages from assert(a, b, x) to test(x), that’s fine. It sounded like you were removing them first.
Most of the messages in the test case are pretty good IMO, which is different from your example of “objects should be equal

@wooorm
Copy link
Member

wooorm commented Jun 15, 2023

So:

  • I’m still meh about describe vs test
  • I do think it‘s OK to wrap every assert in a test here, use good messages on the test and not have them on the assert
  • I do wonder if that’s needed everywhere: this project is complex and has a complex test suite so all this helps here, but other projects are simple and have 8 assertions and I don’t think we need to wrap every one of them in a test case

@remcohaszing
Copy link
Member Author

  • I’m still meh about describe vs test

I don’t think describe is old school. Most modern test frameworks support it (node:test, Vitest, Jest, Jasmine, Mocha, uvu (calls it suite)). Anyway, subtests are very similar, but not the same. Either works. I mostly wish they supported one clear way to do it instead of two.

I could:

  1. Keep the describe
  2. Switch to subtests
  3. Not group tests (not super useful anyway for packages that export one function IMO)

Whatever you prefer.

  • I do wonder if that’s needed everywhere: this project is complex and has a complex test suite so all this helps here, but other projects are simple and have 8 assertions and I don’t think we need to wrap every one of them in a test case

I think it’s useful to always keep tests focused to testing one thing. This way one failure doesn’t obscure other test results.

The output of Node’s test runner is TAP

It was TAP. Now it’s TAP for non-interactive prompts, but a more human-readable format for interactive prompts. I imagine you might have missed this, because you always pipe the output? :)

@wooorm
Copy link
Member

wooorm commented Jun 15, 2023

Anyway, subtests are very similar, but not the same. Either works. I mostly wish they supported one clear way to do it instead of two.

Why did you not go with test? For me that’s one good way to go with. Unless there’s a good benefit for describe, I’d prefer test

This way one failure doesn’t obscure other test results.

And I think that’s very useful here, but for utilities that are simple and have 8 assertions, I feel like it doesn‘t add much, feels a bit like premature optimization, it’s nice but it’s also a lot of lines. I’m noting it more because I don’t think this style introduced here needs to be introduced everywhere. Or do you think that’s the case?

I imagine you might have missed this, because you always pipe the output? :)

Potato potato for me, I do not always pipe, I pipe when I want to diff things!

@remcohaszing
Copy link
Member Author

Why did you not go with test? For me that’s one good way to go with. Unless there’s a good benefit for describe, I’d prefer test

Going with test just doesn’t come to mind. AFAIK most frameworks use describe, and that looks familiar to me. Also there’s not need to await tests then.

But I’m not trying to convince you. I can switch to t.test if you like. The question remains: Should I use nested tests or no nesting at all?

And I think that’s very useful here, but for utilities that are simple and have 8 assertions, I feel like it doesn‘t add much, feels a bit like premature optimization, it’s nice but it’s also a lot of lines. I’m noting it more because I don’t think this style introduced here needs to be introduced everywhere. Or do you think that’s the case?

I do think it’s nice to apply this pattern everywhere. It just takes 2 extra lines, one to call test, another to close it. However, I don’t think it’s worth going over everything to change this, or to block a PR over this.

@wooorm
Copy link
Member

wooorm commented Jun 15, 2023

  • No nesting is 👍 with me!

  • I prefer test over describe: can you change that?

  • However, I don’t think it’s worth going over everything to change this, or to block a PR over this.

    yeah that’s like the point I’m thinking of with every change like this: changing everything everywhere takes a lot of time

@remcohaszing
Copy link
Member Author

  • No nesting is 👍 with me!
  • I prefer test over describe: can you change that?

Those two conflict 😅

I went with no nesting.

  • However, I don’t think it’s worth going over everything to change this, or to block a PR over this.

    yeah that’s like the point I’m thinking of with every change like this: changing everything everywhere takes a lot of time

Definitely! I only changed this, because it bugged my while implementing something else. Same thing goes for example for updating dev dependencies.

@wooorm
Copy link
Member

wooorm commented Jun 15, 2023

Those two conflict 😅

I don‘t see how, you don‘t have describe anymore, which is what I like :)

@wooorm wooorm changed the title Use test cases Refactor tests to use test cases Jun 15, 2023
@wooorm wooorm merged commit 6c35b60 into main Jun 15, 2023
@wooorm wooorm deleted the test-cases branch June 15, 2023 13:40
@wooorm wooorm added the 💪 phase/solved Post is done label Jun 15, 2023
@github-actions

This comment has been minimized.

@github-actions github-actions bot removed the 🤞 phase/open Post is being triaged manually label Jun 15, 2023
@wooorm
Copy link
Member

wooorm commented Jun 15, 2023

Thanks! Merged quickly because it sounds like you want to do more work on this project

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🕸️ area/tests This affects tests 💪 phase/solved Post is done
Development

Successfully merging this pull request may close these issues.

2 participants