I am having non deterministic error after migrate to v12.4.0

[CagriUysal]

Recently we have migrated react-native-testing-library from 9.1.0 to 12.4.0.
After the migration, we started receiving the following error when we ran the tests.

/builds/seller/mobile-core/microapps/gilgamesh-promotion/node_modules/@testing-library/react-native/build/screen.js:11
  throw new Error(SCREEN_ERROR);
  ^
Error: `render` method has not been called
    at Object.notImplemented (/builds/seller/mobile-core/microapps/gilgamesh-promotion/node_modules/@testing-library/react-native/src/screen.ts:7:9)
    at toJSON (/builds/seller/mobile-core/microapps/gilgamesh-promotion/node_modules/@testing-library/react-native/src/queries/make-queries.ts:94:23)
    at formatErrorMessage (/builds/seller/mobile-core/microapps/gilgamesh-promotion/node_modules/@testing-library/react-native/src/queries/make-queries.ts:106:19)
    at appendElementTreeToError (/builds/seller/mobile-core/microapps/gilgamesh-promotion/node_modules/@testing-library/react-native/src/queries/make-queries.ts:223:32)
    at Timeout.onTimeout [as _onTimeout] (/builds/seller/mobile-core/microapps/gilgamesh-promotion/node_modules/@testing-library/react-native/src/wait-for.ts:1[83](https://gitlab.trendyol.com/seller/mobile-core/microapps/gilgamesh-promotion/-/jobs/30696203#L83):24)
    at listOnTimeout (node:internal/timers:569:17)
    at processTimers (node:internal/timers:512:7)

The strange part is, that error occurs randomly in one of the test suits among 70. Failed suite is non-deterministic, error appears randomly in one of the suites. If I run tests for failed suit in isolation, they pass.

I've come across this StackOverflow question, it seems like other people came across this. I haven't opened this as a bug report since I can't reproduce it deterministically. Have you got any insight into why this might happen?

[mdjastrzebski]

Such error occurs when you call any of screen methods without calling render first. It's hard to give any recommendation without more details, but since you are mentioning non-deterministic behavior that would suggest that you have some race condition that is occurring in some case. Do you have any non-awaited async operation that happen before render call in failing tests?

[there]

I also ran into the same error. For us it seemed to be related to using new Date() in some of our tests. I added this code to those tests which resolved the issue:

beforeEach(() => {
  jest
    .useFakeTimers({ doNotFake: ["nextTick", "setImmediate"] })
    .setSystemTime(new Date("2024-01-01"));
});

afterEach(() => {
  jest.useRealTimers();
});

Originally the tests were timing out when I tried to use jest.useFakeTimers() without the config, but I referenced this comment to resolve that issue:

• Timeout when using jest "useFakeTimers" functionality nock/nock#2200 (comment)

[Marc-Cilliers]

We've just migrated from v11 to v12 and are experiencing the same thing. Out of ~4K tests, around 3-5 random tests fail on every test run, even though they pass in isolation.

I've tried the above solution of using fake timers as well as downgrading to v12.2.0 (earliest version that contains userEvent) but no luck.

[pierrezimmermannbam]

That's interesting, the stack trace you provided suggest the error comes from the fact that the component tree is now printed along with the error message when a findBy query fails. I guess what happens here is that this is somehow called after the screen has been unmounted? This should not happen though because waitFor won't reject before this is done. It could happen if you're not waiting for a findBy query somewhere and it rejects after your test is done and the cleanup has already occured? Could you look at the tests that fail and make sure every findBy call is correctly awaited?

[mdjastrzebski]

@CagriUysal, @pierrezimmermannbam seems like a good trail. The stack trace and your analysis is indeed helpful.

I think we can improve the situation by:

1. Make sure that find* queries access screen in a safe way
2. Print warning/error to the console that your async queries have not been closed properly

I'll try to prepare a fix for that soon.

[mdjastrzebski]

I've done some research and here's what I've found as most probable scenario:

1. The presented error occurs when findBy* times out (not being able to find relevant element) after the test env has been cleaned up.
2. The findBy* would throw either way, it should throw "Unable to find an element with ..." error, as it timed out, but it will throw "render method has not been called" instead.
3. The root cause seems to be with the user test, as it did not await some findBy*/waitFor calls before the test ended
4. The non-deterministic aspect might be that there is more than one async call being made, and this could cause race condition under which the awaited call sometimes "wins" and sometimes not.

What would be of great value is a repro scenario, even if it would be non-deterministic. Otherwise, we're just doing our best guess.

@pierrezimmermannbam, @CagriUysal, @there pls check improved error message in #1576.

[Marc-Cilliers]

I've just taken a look again. I managed to find one of the simpler tests that failed:

it('should show box with failure message', () => {
    const { getByText } = render(
      <MockThemeProvider>
        <NotificationBox type="failure" />
      </MockThemeProvider>
    );

    expect(getByText('Sorry, there’s a problem')).toBeDefined();
    expect(
      getByText('Process failed. Please try again later.')
    ).toBeDefined();
  });

with the stack trace:

`render` method has not been called

    at Object.notImplemented (../../node_modules/.pnpm/@testing-library+react-native@12.4.4_jest@29.7.0_react-native@0.72.5_react-test-renderer@18.2.0_react@18.2.0/node_modules/@testing-library/react-native/src/screen.ts:7:9)
    at toJSON (../../node_modules/.pnpm/@testing-library+react-native@12.4.4_jest@29.7.0_react-native@0.72.5_react-test-renderer@18.2.0_react@18.2.0/node_modules/@testing-library/react-native/src/queries/make-queries.ts:90:23)
    at formatErrorMessage (../../node_modules/.pnpm/@testing-library+react-native@12.4.4_jest@29.7.0_react-native@0.72.5_react-test-renderer@18.2.0_react@18.2.0/node_modules/@testing-library/react-native/src/queries/make-queries.ts:102:19)
    at appendElementTreeToError (../../node_modules/.pnpm/@testing-library+react-native@12.4.4_jest@29.7.0_react-native@0.72.5_react-test-renderer@18.2.0_react@18.2.0/node_modules/@testing-library/react-native/src/queries/make-queries.ts:199:32)
    at Timeout.onTimeout [as _onTimeout] (../../node_modules/.pnpm/@testing-library+react-native@12.4.4_jest@29.7.0_react-native@0.72.5_react-test-renderer@18.2.0_react@18.2.0/node_modules/@testing-library/react-native/src/wait-for.ts:177:24)

As far as I can tell, there's no floating promise present as getByText isn't asynchronous.
MockThemeProvider is also simple and has no async code;

export const MockThemeProvider = ({ children, theme = {} }: Props) => (
  <ThemeProvider theme={{ ...defaultTheme, ...theme }}>
    {children}
  </ThemeProvider>
);

Not sure if this helps or just muddies the waters further.
Let me know if there's anything I can provide from my side that might help

[pierrezimmermannbam]

@Marc-Cilliers ideally we'd need a reproduction repo to be able to debug properly, I can't tell what could be going wrong just looking at this test. You did mention that tests pass in isolation though, is that correct? If so it could indicate that the issue lies in another test and the error occurring during the teardown from that other test caused probably by a non resolved promise may affect this particular test. I would suggest looking for non awaited promises by using the eslint plugin testing library which has rules to make sure waitFor and findBy are correctly awaited

[Marc-Cilliers]

@Marc-Cilliers ideally we'd need a reproduction repo to be able to debug properly, I can't tell what could be going wrong just looking at this test. You did mention that tests pass in isolation though, is that correct? If so it could indicate that the issue lies in another test and the error occurring during the teardown from that other test caused probably by a non resolved promise may affect this particular test. I would suggest looking for non awaited promises by using the eslint plugin testing library which has rules to make sure waitFor and findBy are correctly awaited

You're right! I added the eslint plugin, adjusted all tests where there were hanging promises and everything passing now! 🙌

I never suspected that a failure of 1 test might be the result of another test not correctly handling its promises.

[pierrezimmermannbam]

That's good news! That eslint plugin is very useful but not very frequently used I feel like, @mdjastrzebski maybe we should mention it on the readme? it is in the docs but it's probably missed by a lot of users

[mdjastrzebski]

@pierrezimmermannbam Which ESLint plugin? The testing library one? https://callstack.github.io/react-native-testing-library/docs/eslint-plugin-testing-library

[pierrezimmermannbam]

Yeah exactly, there is a section in the docs about it, https://callstack.github.io/react-native-testing-library/docs/eslint-plugin-testing-library, the part about userEvent may not be accurate anymore by the way

[mdjastrzebski]

@CagriUysal @there could you verify that installing eslint plugin and fixing indicated errors fixed this for you?