Conformance: Make "# E[tag]" exactly once, add support for at least once (#1982)

Author: JelleZijlstra

conformance/README.md

@@ -46,12 +46,26 @@ Test cases use the following conventions:
 * Lines that are expected to produce a type checker error should have a comment starting with # E",
   either by itself or followed by an explanation after a colon (e.g., "# E: int is not a subtype
   of str"). Such explanatory comments are purely for human understanding, but type checkers are not
-  expected to use their exact wording.
+  expected to use their exact wording. There are several syntactic variations; see "Test Case Syntax"
+  below.
 * Lines that may produce an error (e.g., because the spec allows multiple behaviors) should be
   marked with "# E?" instead of "# E".
 * If a test case tests conformance with a specific passage in the spec, that passage should be
   quoted in a comment prefixed with "# > ".
 
+## Test Case Syntax
+
+Test cases support the following special comments for declaring where errors should be raised:
+
+* `# E`: an error must be raised on this line
+* `# E?`: an error may be raised on this line
+* `# E[tag]`, where `tag` is an arbitrary string: must appear multiple times in a file with the same tag.
+  Exactly one line with this tag must raise an error.
+* `# E[tag+]`: like `# E[tag]`, but errors may be raised on multiple lines.
+
+Each comment may be followed by a colon plus an explanation of the error; the explanation is ignored
+by the scoring system.
+
 ## Running the Conformance Test Tool
 
 To run the conformance test suite:
@@ -60,7 +74,7 @@ To run the conformance test suite:
 * Switch to the `conformance` subdirectory and install all dependencies (`pip install -r requirements.txt`).
 * Switch to the `src` subdirectory and run `python main.py`.
 
-Note that some type checkers may not run on some platforms. If a type checker fails to install, tests will be skipped for that type checker.  
+Note that some type checkers may not run on some platforms. If a type checker fails to install, tests will be skipped for that type checker.
 Currently, the only unsupported type checker is Pyre on Windows.
 
 ## Reporting Conformance Results

conformance/results/mypy/overloads_definitions.toml

@@ -5,7 +5,7 @@ Does not allow an overload with no implementation in an abstract base class.
 Allows @override to be on all overloads and implementation, instead of just implementation.
 """
 errors_diff = """
-Lines 220, 222, 226: Expected error (tag 'override_impl')
+Lines 220, 221, 222, 225, 226: Expected error (tag 'override_impl')
 Line 47: Unexpected errors ['overloads_definitions.py:47: error: An overloaded function outside a stub file must have an implementation  [no-overload-impl]']
 """
 output = """
@@ -14,8 +14,6 @@ overloads_definitions.py:27: error: An overloaded function outside a stub file m
 overloads_definitions.py:47: error: An overloaded function outside a stub file must have an implementation  [no-overload-impl]
 overloads_definitions.py:58: error: An overloaded function outside a stub file must have an implementation  [no-overload-impl]
 overloads_definitions.py:71: error: Overload does not consistently use the "@staticmethod" decorator on all function signatures.  [misc]
-overloads_definitions.py:81: error: Overloaded function implementation does not accept all possible arguments of signature 1  [misc]
-overloads_definitions.py:81: error: Overloaded function implementation does not accept all possible arguments of signature 2  [misc]
 overloads_definitions.py:84: error: Overload does not consistently use the "@classmethod" decorator on all function signatures.  [misc]
 overloads_definitions.py:121: error: @final should be applied only to overload implementation  [misc]
 overloads_definitions.py:135: error: @final should be applied only to overload implementation  [misc]

conformance/results/mypy/overloads_definitions_stub.toml

@@ -9,7 +9,6 @@ Lines 143, 147, 149: Expected error (tag 'override_impl')
 output = """
 overloads_definitions_stub.pyi:13: error: Single overload definition, multiple required  [misc]
 overloads_definitions_stub.pyi:32: error: Overload does not consistently use the "@staticmethod" decorator on all function signatures.  [misc]
-overloads_definitions_stub.pyi:33: error: Self argument missing for a non-static method (or an invalid type for self)  [misc]
 overloads_definitions_stub.pyi:39: error: Overload does not consistently use the "@classmethod" decorator on all function signatures.  [misc]
 overloads_definitions_stub.pyi:71: error: In a stub file @final must be applied only to the first overload  [misc]
 overloads_definitions_stub.pyi:84: error: In a stub file @final must be applied only to the first overload  [misc]

conformance/results/mypy/tuples_type_compat.toml

@@ -9,27 +9,29 @@ tuples_type_compat.py:32: error: Incompatible types in assignment (expression ha
 tuples_type_compat.py:33: error: Incompatible types in assignment (expression has type "tuple[int, ...]", variable has type "tuple[int]")  [assignment]
 tuples_type_compat.py:43: error: Incompatible types in assignment (expression has type "tuple[int, ...]", variable has type "tuple[int]")  [assignment]
 tuples_type_compat.py:62: error: Incompatible types in assignment (expression has type "tuple[int, ...]", variable has type "tuple[int, int]")  [assignment]
-tuples_type_compat.py:95: error: Expression is of type "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]", not "tuple[int]"  [assert-type]
-tuples_type_compat.py:99: error: Expression is of type "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]", not "tuple[str, str] | tuple[int, int]"  [assert-type]
-tuples_type_compat.py:103: error: Expression is of type "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]", not "tuple[int, str, int]"  [assert-type]
-tuples_type_compat.py:136: error: Expression is of type "Sequence[object]", not "Sequence[complex | list[int]]"  [assert-type]
-tuples_type_compat.py:139: error: Expression is of type "Sequence[object]", not "Sequence[int | str]"  [assert-type]
-tuples_type_compat.py:144: error: Incompatible types in assignment (expression has type "tuple[int, str, str]", variable has type "tuple[int, str]")  [assignment]
-tuples_type_compat.py:149: error: Incompatible types in assignment (expression has type "tuple[int, int, str]", variable has type "tuple[int, *tuple[str, ...]]")  [assignment]
-tuples_type_compat.py:150: error: Incompatible types in assignment (expression has type "tuple[int, str, int]", variable has type "tuple[int, *tuple[str, ...]]")  [assignment]
-tuples_type_compat.py:156: error: Incompatible types in assignment (expression has type "tuple[int, str, str]", variable has type "tuple[int, *tuple[str, ...], int]")  [assignment]
-tuples_type_compat.py:157: error: Incompatible types in assignment (expression has type "tuple[int, str, str, float]", variable has type "tuple[int, *tuple[str, ...], int]")  [assignment]
-tuples_type_compat.py:162: error: Incompatible types in assignment (expression has type "tuple[int, str, int]", variable has type "tuple[*tuple[str, ...], int]")  [assignment]
-tuples_type_compat.py:163: error: Incompatible types in assignment (expression has type "tuple[str, str, float]", variable has type "tuple[*tuple[str, ...], int]")  [assignment]
-tuples_type_compat.py:168: error: Incompatible types in assignment (expression has type "tuple[str, str]", variable has type "tuple[str, str, int]")  [assignment]
-tuples_type_compat.py:171: error: Incompatible types in assignment (expression has type "tuple[str, str]", variable has type "tuple[str, str, str, *tuple[str, ...]]")  [assignment]
-tuples_type_compat.py:175: error: Incompatible types in assignment (expression has type "tuple[str, str]", variable has type "tuple[*tuple[str, ...], str, str, str]")  [assignment]
+tuples_type_compat.py:76: error: Expression is of type "tuple[int]", not "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]"  [assert-type]
+tuples_type_compat.py:81: error: Expression is of type "tuple[str, str] | tuple[int, int]", not "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]"  [assert-type]
+tuples_type_compat.py:86: error: Expression is of type "tuple[int, str, int]", not "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]"  [assert-type]
+tuples_type_compat.py:101: error: Expression is of type "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]", not "tuple[int]"  [assert-type]
+tuples_type_compat.py:106: error: Expression is of type "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]", not "tuple[str, str] | tuple[int, int]"  [assert-type]
+tuples_type_compat.py:111: error: Expression is of type "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]", not "tuple[int, str, int]"  [assert-type]
+tuples_type_compat.py:127: error: Expression is of type "tuple[int | str, str]", not "tuple[int | str, int | str]"  [assert-type]
+tuples_type_compat.py:130: error: Expression is of type "tuple[int | str, int]", not "tuple[int | str, int | str]"  [assert-type]
+tuples_type_compat.py:149: error: Expression is of type "Sequence[object]", not "Sequence[complex | list[int]]"  [assert-type]
+tuples_type_compat.py:152: error: Expression is of type "Sequence[object]", not "Sequence[int | str]"  [assert-type]
+tuples_type_compat.py:157: error: Incompatible types in assignment (expression has type "tuple[int, str, str]", variable has type "tuple[int, str]")  [assignment]
+tuples_type_compat.py:162: error: Incompatible types in assignment (expression has type "tuple[int, int, str]", variable has type "tuple[int, *tuple[str, ...]]")  [assignment]
+tuples_type_compat.py:163: error: Incompatible types in assignment (expression has type "tuple[int, str, int]", variable has type "tuple[int, *tuple[str, ...]]")  [assignment]
+tuples_type_compat.py:169: error: Incompatible types in assignment (expression has type "tuple[int, str, str]", variable has type "tuple[int, *tuple[str, ...], int]")  [assignment]
+tuples_type_compat.py:170: error: Incompatible types in assignment (expression has type "tuple[int, str, str, float]", variable has type "tuple[int, *tuple[str, ...], int]")  [assignment]
+tuples_type_compat.py:175: error: Incompatible types in assignment (expression has type "tuple[int, str, int]", variable has type "tuple[*tuple[str, ...], int]")  [assignment]
+tuples_type_compat.py:176: error: Incompatible types in assignment (expression has type "tuple[str, str, float]", variable has type "tuple[*tuple[str, ...], int]")  [assignment]
+tuples_type_compat.py:181: error: Incompatible types in assignment (expression has type "tuple[str, str]", variable has type "tuple[str, str, int]")  [assignment]
+tuples_type_compat.py:184: error: Incompatible types in assignment (expression has type "tuple[str, str]", variable has type "tuple[str, str, str, *tuple[str, ...]]")  [assignment]
+tuples_type_compat.py:188: error: Incompatible types in assignment (expression has type "tuple[str, str]", variable has type "tuple[*tuple[str, ...], str, str, str]")  [assignment]
 """
 conformance_automated = "Fail"
 errors_diff = """
-Line 95: Unexpected errors ['tuples_type_compat.py:95: error: Expression is of type "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]", not "tuple[int]"  [assert-type]']
-Line 99: Unexpected errors ['tuples_type_compat.py:99: error: Expression is of type "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]", not "tuple[str, str] | tuple[int, int]"  [assert-type]']
-Line 103: Unexpected errors ['tuples_type_compat.py:103: error: Expression is of type "tuple[int] | tuple[str, str] | tuple[int, *tuple[str, ...], int]", not "tuple[int, str, int]"  [assert-type]']
-Line 136: Unexpected errors ['tuples_type_compat.py:136: error: Expression is of type "Sequence[object]", not "Sequence[complex | list[int]]"  [assert-type]']
-Line 139: Unexpected errors ['tuples_type_compat.py:139: error: Expression is of type "Sequence[object]", not "Sequence[int | str]"  [assert-type]']
+Line 149: Unexpected errors ['tuples_type_compat.py:149: error: Expression is of type "Sequence[object]", not "Sequence[complex | list[int]]"  [assert-type]']
+Line 152: Unexpected errors ['tuples_type_compat.py:152: error: Expression is of type "Sequence[object]", not "Sequence[int | str]"  [assert-type]']
 """

conformance/results/mypy/version.toml

@@ -1,2 +1,2 @@
 version = "mypy 1.15.0"
-test_duration = 2.3
+test_duration = 1.8

conformance/results/pyre/overloads_definitions.toml

@@ -5,10 +5,14 @@ Does not allow an overload with no implementation in a Protocol or an abstract b
 Expects @final/@override on all overloads and implementation, instead of implementation only.
 """
 errors_diff = """
-Lines 135, 137, 142: Expected error (tag 'invalid_final_2')
-Lines 220, 222, 226: Expected error (tag 'override_impl')
+Lines 71, 73, 78, 81: Expected exactly one error (tag 'func5')
+Lines 135, 136, 137, 141, 142: Expected error (tag 'invalid_final_2')
+Lines 220, 221, 222, 225, 226: Expected error (tag 'override_impl')
 Line 40: Unexpected errors ['overloads_definitions.py:40:4 Missing overload implementation [42]: Overloaded function `MyProto.func3` must have an implementation.']
 Line 49: Unexpected errors ['overloads_definitions.py:49:4 Missing overload implementation [42]: Overloaded function `MyAbstractBase.func4` must have an implementation.']
+Line 73: Unexpected errors ['overloads_definitions.py:73:4 Incompatible overload [43]: The implementation of `C.func5` does not accept all possible arguments of overload defined on line `73`.']
+Line 78: Unexpected errors ['overloads_definitions.py:78:4 Incompatible overload [43]: The implementation of `C.func5` does not accept all possible arguments of overload defined on line `78`.']
+Line 81: Unexpected errors ['overloads_definitions.py:81:4 Incompatible overload [43]: This definition does not have the same decorators as the preceding overload(s).']
 Line 117: Unexpected errors ['overloads_definitions.py:117:4 Incompatible overload [43]: This definition does not have the same decorators as the preceding overload(s).']
 Line 215: Unexpected errors ['overloads_definitions.py:215:4 Incompatible overload [43]: This definition does not have the same decorators as the preceding overload(s).']
 """