IdempotencyTests is flaky on Windows

[smarter]

Starting with the merge of #11210 (EDIT: actually long before that, see comments below), it looks like test_windows_full has been failing IdempotencyTests from time to time, from https://github.com/lampepfl/dotty/runs/2178741539:

Test 'CheckPosIdempotency from idempotency/check' failed with output:           
Idempotency test failed between out\idempotency\posIdempotency1\pos\i6507b\Test.tasty and out\idempotency\posIdempotency2\pos\i6507b\Test.tasty
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at dotty.tools.vulpix.ChildJVMMain.runMain(ChildJVMMain.java:40)
	at dotty.tools.vulpix.ChildJVMMain.main(ChildJVMMain.java:47)
Caused by: java.lang.AssertionError: assertion failed: Failed 1 idempotency checks (out of 5116)
	at scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8)
	at IdempotencyCheck$.checkIdempotency(IdempotencyCheck.scala:70)
	at Test$.main(CheckPosIdempotency.scala:4)
	at Test.main(CheckPosIdempotency.scala)
	... 6 more

@bishabosha could you investigate this (I've forwarded you the access information for the Windows machine if you want to investigate there)? /cc @liufengyun

[griggt]

I think this test has been flaky for quite some time (at least many weeks). I've been meaning to trace back and see when it started. I want to say it was sometime around when the "expand non-transparent macros after typer" PR was merged, but my memory may be faulty.

[smarter]

Oh you're right, I didn't go back enough to see that, not @bishabosha's fault then :). /cc @nicolasstucki since he worked on both the PR you mentioned and idempotency tests.

[griggt]

Here's an example of it failing during the nightly build of Feb 9 (the night after #9984 was merged):

https://github.com/lampepfl/dotty/runs/1868294322?check_suite_focus=true#step:3:784

[griggt]

Incidentally I'll open a PR for #8734 sometime soon (maybe after the RC release is done), which will draw attention to these things right away.

[michelou]

I think this test has been flaky for quite some time (at least many weeks). I've been meaning to trace back and see when it started. I want to say it was sometime around when the "expand non-transparent macros after typer" PR was merged, but my memory may be faulty.

@griggt I confirm your observation. Here are the 3 failing tests I currently have on my Win10 machine :

[...]
[error] Failed tests:
[error]         dotty.tools.dotc.FromTastyTests
[error]         dotty.tools.dotc.IdempotencyTests
[error]         dotty.tools.dotc.classpath.MultiReleaseJarTest

PS. I build at the moment 16 snapshot distributions (tar.gz+zip) on a little used Win10 machine (~30 min build time each) : BellSoft 8+11, Corretto 8+11, DCEVM 11, OpenJ9 8+11, OpenJDK 8+11+17, RedHat 8+11, SapMachine 11+17 and Zulu 8+11.

[griggt]

The discrepancy seems to be in the positions section:

--- out/idempotency/posIdempotency1/debug-idemp/i6507b/tasty.txt	2021-03-25 12:16:34.354135344 -0700
+++ out/idempotency/posIdempotency2/debug-idemp/i6507b/tasty.txt	2021-03-25 12:16:53.166135344 -0700
@@ -264,7 +264,7 @@
    436:         STRINGconst 56 [tests/debug-idemp/i6507b.scala]
    438:
 
- 291 position bytes:
+ 288 position bytes:
    lines: 11
    line sizes: 26, 0, 13, 79, 36, 56, 3, 0, 28, 1, 0
    positions:
@@ -337,7 +337,7 @@
        339: 197 .. 215
        341: 193 .. 193
        347: 183 .. 196
-       350: 200 .. 215
+       350: 183 .. 183
        353: 183 .. 196
        357: 200 .. 215
        364: 197 .. 215

   284:       INLINED(130)
   287:         BLOCK(124)
   289:           TYPED(112)
   291:             APPLY(107)
   293:               TYPEAPPLY(102)
   295:                 SELECTin(95) 41 [*:[Signed Signature(List(2, java.lang.Object),scala.Product) @*:]]
   298:                   INLINED(89)
   300:                     BLOCK(83)
   302:                       TYPED(71)
   304:                         APPLY(66)
   306:                           TYPEAPPLY(61)
   308:                             SELECTin(54) 41 [*:[Signed Signature(List(2, java.lang.Object),scala.Product) @*:]]
   311:                               INLINED(48)
   313:                                 BLOCK(42)
   315:                                   TYPED(30)
   317:                                     APPLY(25)
   319:                                       TYPEAPPLY(20)
   321:                                         SELECTin(13) 41 [*:[Signed Signature(List(2, java.lang.Object),scala.Product) @*:]]
   324:                                           INLINED(7)
   326:                                             SHAREDterm 151
   329:                                             IDENTtpt 6 [Test[ModuleClass]]
   331:                                               SHAREDtype 18
   333:                                           SHAREDtype 241
   336:                                         INTconst 98
   339:                                         SHAREDtype 112
   341:                                       TERMREFdirect 347
   344:                                     SHAREDterm 253
   347:                                   VALDEF(8) 44 [x[Unique $ 1]]
   350:                                     SHAREDtype 336
   353:                                     INTconst 98
   356:                                     SYNTHETIC

The output files from this test run: i11885-output.zip