Why do some Flutter widget tests fail if executed together but pass individually?

Question

When having multiple tests in the same file, and running one test after the other. Some kinds of tests are failing, but the same tests pass when run individually.

This is my test file at the moment, it's kinda long though:

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:rec/Components/Inputs/text_fields/DniTextField.dart';
import 'package:rec/Helpers/Validators.dart';

import '../../test_utils.dart';

void main() {
  testWidgets('DniTextField works with invalid DNI', (WidgetTester tester) async {
    var key = GlobalKey<DniTextFieldState>();
    var formKey = GlobalKey<FormState>();
    var onChangedResult;

    var widget = Form(
      key: formKey,
      child: DniTextField(
        key: key,
        onChange: (String value) {
          onChangedResult = value;
        },
        validator: (s) => Validators.verifyIdentityDocument(s),
      ),
    );
    await tester.pumpWidget(
      TestUtils.wrapPublicRoute(widget),
    );
    await tester.pumpAndSettle();

    // Test that widget has at least rendered
    TestUtils.widgetExists(widget);

    // Enter text into field with
    var widgetFinder = find.byWidget(widget);
    await tester.tap(widgetFinder);
    await tester.showKeyboard(widgetFinder);
    await tester.enterText(widgetFinder, 'invaliddni');

    await tester.pumpAndSettle();

    expect(onChangedResult, 'invaliddni');

    expect(formKey.currentState.validate(), false);
  });

  testWidgets('DniTextField works with valid DNI', (WidgetTester tester) async {
    var key = GlobalKey<DniTextFieldState>();
    var formKey = GlobalKey<FormState>();
    var onChangedResult;

    var widget = Form(
      key: formKey,
      child: DniTextField(
        key: key,
        onChange: (String value) {
          onChangedResult = value;
        },
        validator: (s) => Validators.verifyIdentityDocument(s),
      ),
    );
    await tester.pumpWidget(
      TestUtils.wrapPublicRoute(widget),
    );
    await tester.pumpAndSettle();

    // Enter text into field
    var widgetFinder = find.byType(DniTextField);
    await tester.tap(widgetFinder);
    await tester.showKeyboard(widgetFinder);
    await tester.enterText(widgetFinder, '80008000k');

    await tester.pumpAndSettle();

    expect(onChangedResult, '80008000k');

    expect(formKey.currentState.validate(), true);
  });

  testWidgets('DniTextField works with valid DNI with trailing space', (WidgetTester tester) async {
    var key = GlobalKey<DniTextFieldState>();
    var formKey = GlobalKey<FormState>();
    var onChangedResult;

    var widget = Form(
      key: formKey,
      child: DniTextField(
        key: key,
        onChange: (String value) {
          onChangedResult = value;
        },
        validator: (s) => Validators.verifyIdentityDocument(s),
      ),
    );
    await tester.pumpWidget(TestUtils.wrapPublicRoute(widget));
    await tester.pumpAndSettle();

    // Enter text into field
    var widgetFinder = find.byWidget(widget);
    await tester.tap(widgetFinder);
    await tester.showKeyboard(widgetFinder);
    await tester.enterText(widgetFinder, '80008000k ');

    await tester.pumpAndSettle();

    // The value emitted by the field, should be free of trailing whitespace
    expect(onChangedResult, '80008000k');

    expect(formKey.currentState.validate(), true);
  });
}

If I run this like this:

$ flutter test test/Components/inputs/dni_text_field_test.dart

The first test passes, but the next 2 don't, spitting out the following error:

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown running a test:
The finder "zero widgets with the given widget
(DniTextField-[LabeledGlobalKey<DniTextFieldState>#0fb86]) (ignoring offstage widgets)" (used in a
call to "tap()") could not find any matching widgets.

When the exception was thrown, this was the stack:
#0      WidgetController._getElementPoint (package:flutter_test/src/controller.dart:902:7)
#1      WidgetController.getCenter (package:flutter_test/src/controller.dart:841:12)
#2      WidgetController.tap (package:flutter_test/src/controller.dart:273:18)
#3      main.<anonymous closure> (file:///[masked]/test/Components/inputs/dni_text_field_test.dart:99:18)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)

If I then comment out the first test, then the second one passes but the third one does not. As I said the test pass if I run them individually.

I can't find any information about this, not even one thing. Maybe there is someone here that can guide me in the right direction. I'm a bit of a noob with Flutter widget testing, so I might be missing some important things.

The only solution I've found for now is having each test in a separate file. It's not ideal though, would be much better to contain related tests in the same file. Flutter has examples showing that multiple tests per file is allowed as it should be.

Flutter Version: 2.4.0-5.0.pre.87

Answer 1

Okay I found the solution and reason, fixed thanks to this issue and this SO question.

The main issue was the use of Localizations that loaded from JSON files. The solution was to wrap each test with tester.runAsync()

testWidgets('widget test 2', (WidgetTester tester) async {
  await tester.runAsync(() async {
    // tests
  });
});

Answer 2

I suppose this trouble occure when different test are not craeting different instances of widgets (or they are conflicting in some other way), but there is a turnaround that might help you.

If you want to create a single button running all your tests - you can create a bash (or cmd on windows) file, that is running all your tests sequentially.

Example all_tests.cmd

dart test_1.dart
dart test_2.dart
dart test_3.dart

It is not the clearest solution, but it might work for first time.