Transient failure: "hash.same as universal hash / scala hash" (ScalaCheck-based) in cats community project

[mbovel]

Got that one 3 times in unrelated PRs.

It seems these laws have been deprecated upstream since cats 2.9.0: https://github.com/typelevel/cats/blob/726be059c8f6815889e47e6b5030a56b11402307/kernel-laws/shared/src/main/scala/cats/kernel/laws/HashLaws.scala#L33-L41.

Note: these laws are used from https://github.com/typelevel/cats/blob/726be059c8f6815889e47e6b5030a56b11402307/alleycats-laws/shared/src/main/scala/alleycats/laws/discipline/SystemIdentityHashTests.scala#L40-L41.

So maybe we could disable testing those in our community build fork? Or just update cats?

2025-04-01 (https://github.com/scala/scala3/actions/runs/14197500021/job/39776053542?pr=22893#step:9:6342):

failing seed for hash.same as universal hash is 8LwMrAMZ7Oy9MkMite7cPKyMMoDTwZRWz7pPXlSCZKE=
[error] ==> X cats.tests.ChainSuite.Chain[ListWrapper[Int]]: hash.same as universal hash  0.048s munit.FailException: /__w/scala3/scala3/community-build/community-projects/cats/tests/shared/src/test/scala/cats/tests/ChainSuite.scala:86
[error] 85:    implicit val hash: Hash[ListWrapper[Int]] = ListWrapper.hash[Int]
[error] 86:    checkAll("Chain[ListWrapper[Int]]", HashTests[Chain[ListWrapper[Int]]].hash)
[error] 87:    checkAll("Hash[Chain[ListWrapper[Int]]", SerializableTests.serializable(Hash[Chain[ListWrapper[Int]]]))
[error] 
[error] Failing seed: zf7oGPXX4XH1bi_L50espZ-AAfoZmCkE8HLOPGjZqEJ=
[error] You can reproduce this failure by adding the following override to your suite:
[error] 
[error]   override val scalaCheckInitialSeed = "zf7oGPXX4XH1bi_L50espZ-AAfoZmCkE8HLOPGjZqEJ="
[error] 
[error] Falsified after 40 passed tests.
[error] > Labels of failing property: 
[error] Expected: true
[error] Received: false
[error] > ARG_0: Chain(ListWrapper(List(2147483647, 1883865195, -1286053713, -524834978, -87489302, 763091306)), ListWrapper(List(0, 2147483647, -1233314808)), ListWrapper(List(-714612646)), ListWrapper(List(2147483647, 1, 2147483647)), ListWrapper(List(1, 1, 1)), ListWrapper(List()))
[error] > ARG_1: Chain(ListWrapper(List(1439354466, 2147483647, -1203917048, 1622240348, 475667921)), ListWrapper(List(92326472)), ListWrapper(List()), ListWrapper(List(1470867167, 2147483647, 1954645151, 2147483647, -1698744174, 1035291033)), ListWrapper(List(-1421952691, 1)), ListWrapper(List(0, 1, 0)))
[error]     at munit.Assertions.fail(Assertions.scala:283)
[error]     at munit.Assertions.fail$(Assertions.scala:15)
[error]     at munit.FunSuite.fail(FunSuite.scala:11)
[error]     at munit.ScalaCheckSuite.propToTry$$anonfun$1(ScalaCheckSuite.scala:104)
[error]     at scala.util.Try$.apply(Try.scala:217)
[error]     at munit.ScalaCheckSuite.propToTry(ScalaCheckSuite.scala:104)
[error]     at munit.ScalaCheckSuite.$init$$$anonfun$1$$anonfun$1$$anonfun$1(ScalaCheckSuite.scala:46)
[error]     at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:475)
[error]     at munit.Suite$$anon$1.execute(Suite.scala:26)
[error]     at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:429)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:338)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:312)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.transform(Promise.scala:128)
[error]     at munit.internal.FutureCompat$ExtensionFuture.transformCompat(FutureCompat.scala:16)
[error]     at munit.ScalaCheckSuite.$init$$$anonfun$1$$anonfun$1(ScalaCheckSuite.scala:48)
[error]     at munit.GenericTest.withBodyMap$$anonfun$1(GenericTest.scala:33)
[error]     at munit.MUnitRunner.$anonfun$10(MUnitRunner.scala:296)
failing seed for hash.same as scala hashing is YmvfArGFfSiiwI0yAGQ5BQ1e877pWdz9_tLCp6e2tGM=
[error] ==> X cats.tests.ArraySeqSuite.ArraySeq[Int]: hash.same as scala hashing  0.005s munit.FailException: /__w/scala3/scala3/community-build/community-projects/cats/tests/shared/src/test/scala-2.13+/cats/tests/ArraySeqSuite.scala:81
[error] 80:    implicit val hash: Hash[ListWrapper[Int]] = ListWrapper.hash[Int]
[error] 81:    checkAll("ArraySeq[Int]", HashTests[ArraySeq[ListWrapper[Int]]].hash)
[error] 82:    checkAll("Hash[ArraySeq]", SerializableTests.serializable(Hash[ArraySeq[ListWrapper[Int]]]))
[error] 
[error] Failing seed: 8IjQ2gkp6cnWXYLqteLwPYLVio28UDz35QSJGnaMEAK=
[error] You can reproduce this failure by adding the following override to your suite:
[error] 
[error]   override val scalaCheckInitialSeed = "8IjQ2gkp6cnWXYLqteLwPYLVio28UDz35QSJGnaMEAK="
[error] 
[error] Falsified after 25 passed tests.
[error] > Labels of failing property: 
[error] Expected: true
[error] Received: false
[error] > ARG_0: ArraySeq(ListWrapper(List()), ListWrapper(List()), ListWrapper(List()))
[error] > ARG_1: ArraySeq(ListWrapper(List()), ListWrapper(List()), ListWrapper(List()))
[error]     at munit.Assertions.fail(Assertions.scala:283)
[error]     at munit.Assertions.fail$(Assertions.scala:15)
[error]     at munit.FunSuite.fail(FunSuite.scala:11)
[error]     at munit.ScalaCheckSuite.propToTry$$anonfun$1(ScalaCheckSuite.scala:104)
[error]     at scala.util.Try$.apply(Try.scala:217)
[error]     at munit.ScalaCheckSuite.propToTry(ScalaCheckSuite.scala:104)
[error]     at munit.ScalaCheckSuite.$init$$$anonfun$1$$anonfun$1$$anonfun$1(ScalaCheckSuite.scala:46)
[error]     at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:475)
[error]     at munit.Suite$$anon$1.execute(Suite.scala:26)
[error]     at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:429)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:338)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:312)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.transform(Promise.scala:128)
[error]     at munit.internal.FutureCompat$ExtensionFuture.transformCompat(FutureCompat.scala:16)
[error]     at munit.ScalaCheckSuite.$init$$$anonfun$1$$anonfun$1(ScalaCheckSuite.scala:48)
[error]     at munit.GenericTest.withBodyMap$$anonfun$1(GenericTest.scala:33)
[error]     at munit.MUnitRunner.$anonfun$10(MUnitRunner.scala:296)
[error] Failed: Total 12846, Failed 2, Errors 0, Passed 12844
[error] Failed tests:
[error] 	cats.tests.ChainSuite
[error] 	cats.tests.ArraySeqSuite
[error] (testsJVM / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 256 s (04:16), completed Apr 1, 2025 4:45:10 PM

2024-01-30 (https://github.com/scala/scala3/actions/runs/13059513292/job/36438758376#step:9:9328):

failing seed for hash.same as scala hashing failing seed for hash.same as scala hashing is 6WOpVBqfScSGIOwkKMgoZEq-ooaMrSvbrHILevjE2MK=
[error] ==> X cats.tests.ArraySeqSuite.ArraySeq[Int]: hash.same as scala hashing  0.053s munit.FailException: /__w/scala3/scala3/community-build/community-projects/cats/tests/shared/src/test/scala-2.13+/cats/tests/ArraySeqSuite.scala:81
[error] 80:    implicit val hash: Hash[ListWrapper[Int]] = ListWrapper.hash[Int]
[error] 81:    checkAll("ArraySeq[Int]", HashTests[ArraySeq[ListWrapper[Int]]].hash)
[error] 82:    checkAll("Hash[ArraySeq]", SerializableTests.serializable(Hash[ArraySeq[ListWrapper[Int]]]))
[error] 
[error] Failing seed: wj92InV2YLooKRdmylJtz_EBOC6YjpOBBrXe-1_y4BO=
[error] You can reproduce this failure by adding the following override to your suite:
[error] 
[error]   override val scalaCheckInitialSeed = "wj92InV2YLooKRdmylJtz_EBOC6YjpOBBrXe-1_y4BO="
[error] 
[error] Falsified after 33 passed tests.
[error] > Labels of failing property: 
[error] Expected: true
[error] Received: false
[error] > ARG_0: ArraySeq(ListWrapper(List()), ListWrapper(List()), ListWrapper(List()))
[error] > ARG_1: ArraySeq(ListWrapper(List(0, -671202680)), ListWrapper(List(-255645408, -618800360)))
[error]     at munit.Assertions.fail(Assertions.scala:283)
[error]     at munit.Assertions.fail$(Assertions.scala:15)
[error]     at munit.FunSuite.fail(FunSuite.scala:11)
[error]     at munit.ScalaCheckSuite.propToTry$$anonfun$1(ScalaCheckSuite.scala:104)
[error]     at scala.util.Try$.apply(Try.scala:217)
[error]     at munit.ScalaCheckSuite.propToTry(ScalaCheckSuite.scala:104)
[error]     at munit.ScalaCheckSuite.$init$$$anonfun$1$$anonfun$1$$anonfun$1(ScalaCheckSuite.scala:46)
[error]     at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:475)
[error]     at munit.Suite$$anon$1.execute(Suite.scala:26)
[error]     at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:429)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:338)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:312)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.transform(Promise.scala:128)
[error]     at munit.internal.FutureCompat$ExtensionFuture.transformCompat(FutureCompat.scala:16)
[error]     at munit.ScalaCheckSuite.$init$$$anonfun$1$$anonfun$1(ScalaCheckSuite.scala:48)
[error]     at munit.GenericTest.withBodyMap$$anonfun$1(GenericTest.scala:33)
[error]     at munit.MUnitRunner.$anonfun$10(MUnitRunner.scala:296)
[error] Failed: Total 12846, Failed 1, Errors 0, Passed 12845
[error] Failed tests:
[error] 	cats.tests.ArraySeqSuite
[error] (testsJVM / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 269 s (04:29), completed Jan 30, 2025 9:42:54 PM
Building coop with dotty-bootstrapped 3.7.0-RC1-bin-SNAPSHOT...
sbt -sbt-version 1.10.5 -Dsbt.supershell=false -Ddotty.communitybuild.dir=/__w/scala3/scala3/community-build --addPluginSbtFile=/__w/scala3/scala3/community-build/sbt-injected-plugins set Global/testOptions += Tests.Argument(TestFramework("munit.Framework"), "+l"); clean; set Global/logLevel := Level.Error; set Global/updateOptions ~= (_.withLatestSnapshots(false)); set Global/scalacOptions ++= List("-Xcheck-macros","-Wsafe-init");++3.7.0-RC1-bin-SNAPSHOT!; test
Error:  Test dotty.communitybuild.CommunityBuildTestB.cats failed: java.lang.AssertionError: 
Error:  
Error:  sbt exited with an error code. To reproduce without JUnit, use:
Error:  
Error:      sbt community-build/prepareCommunityBuild
Error:      cd community-build/community-projects/cats
Error:      sbt -sbt-version 1.10.5 -Dsbt.supershell=false -Ddotty.communitybuild.dir=/__w/scala3/scala3/community-build --addPluginSbtFile=/__w/scala3/scala3/community-build/sbt-injected-plugins "set Global/testOptions += Tests.Argument(TestFramework("munit.Framework"), "+l"); clean; set Global/logLevel := Level.Error; set Global/updateOptions ~= (_.withLatestSnapshots(false)); set Global/scalacOptions ++= List("-Xcheck-macros");++3.7.0-RC1-bin-SNAPSHOT!; set Global/scalaJSStage := FastOptStage;rootJVM/test;rootJS/test"
Error:  
Error:  For a faster feedback loop on SBT projects, one can try to extract a direct call to dotc
Error:  using the sbt export command. For instance, for scalacheck, use
Error:      sbt export jvm/test:compileIncremental
Error:  
Error:  , took 1313.263 sec
Error:      at dotty.communitybuild.CommunityBuildTest$package$testRunner$.failWith(CommunityBuildTest.scala:15)
Error:      at dotty.communitybuild.CommunityBuildRunner.runProject(CommunityBuildRunner.scala:81)
Error:      at dotty.communitybuild.CommunityBuildRunner.runProject$(CommunityBuildRunner.scala:26)
Error:      at dotty.communitybuild.CommunityBuildTest$package$testRunner$.runProject(CommunityBuildTest.scala:14)
Error:      at dotty.communitybuild.CommunityBuildRunner$.run(CommunityBuildRunner.scala:23)
Error:      at dotty.communitybuild.CommunityBuildTestB.cats(CommunityBuildTest.scala:38)
Error:      ...

[mbovel]

Mmh, and this is often silenced because community build tests are automatically retried up to 3 times 🤔

scala3/community-build/src/scala/dotty/communitybuild/CommunityBuildRunner.scala

Line 71 in 72d0026

val testsCompletedSuccessfully = execTimes(projectDef.build, 3)

[mbovel]

2025-04-01 on main (https://github.com/scala/scala3/actions/runs/14199976771/job/39806817179#step:9:6793):

[error] ==> X cats.tests.NonEmptyLazyListSuite.NonEmptyLazyList[Int]: hash.same as scala hashing  0.046s munit.FailException: /__w/scala3/scala3/community-build/community-projects/cats/tests/shared/src/test/scala-2.13+/cats/tests/NonEmptyLazyListSuite.scala:52
[error] 51:
[error] 52:  checkAll(s"NonEmptyLazyList[Int]", HashTests[NonEmptyLazyList[Int]].hash)
[error] 53:  checkAll(s"Hash[NonEmptyLazyList[Int]]", SerializableTests.serializable(Hash[NonEmptyLazyList[Int]]))
[error] 
[error] Failing seed: 1acNLauHyWMStpjIPKrK1iMgII69CY5ROgZSe9QXaNN=
[error] You can reproduce this failure by adding the following override to your suite:
[error] 
[error]   override val scalaCheckInitialSeed = "1acNLauHyWMStpjIPKrK1iMgII69CY5ROgZSe9QXaNN="
[error] 
[error] Falsified after 49 passed tests.
[error] > Labels of failing property: 
[error] Expected: true
[error] Received: false
[error] > ARG_0: LazyList(0, 0, 0)
[error] > ARG_1: LazyList(<not computed>)
[error]     at munit.Assertions.fail(Assertions.scala:283)
[error]     at munit.Assertions.fail$(Assertions.scala:15)
[error]     at munit.FunSuite.fail(FunSuite.scala:11)
[error]     at munit.ScalaCheckSuite.propToTry$$anonfun$1(ScalaCheckSuite.scala:104)
[error]     at scala.util.Try$.apply(Try.scala:217)
[error]     at munit.ScalaCheckSuite.propToTry(ScalaCheckSuite.scala:104)
[error]     at munit.ScalaCheckSuite.$init$$$anonfun$1$$anonfun$1$$anonfun$1(ScalaCheckSuite.scala:46)
[error]     at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:475)
[error]     at munit.Suite$$anon$1.execute(Suite.scala:26)
[error]     at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:429)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:338)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:312)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.transform(Promise.scala:128)
[error]     at munit.internal.FutureCompat$ExtensionFuture.transformCompat(FutureCompat.scala:16)
[error]     at munit.ScalaCheckSuite.$init$$$anonfun$1$$anonfun$1(ScalaCheckSuite.scala:48)
[error]     at munit.GenericTest.withBodyMap$$anonfun$1(GenericTest.scala:33)
[error]     at munit.MUnitRunner.$anonfun$10(MUnitRunner.scala:296)
failing seed for hash.same as scala hashing is o9ts7Pad7qrzf8YEVEgEHbVf4BW_Q050OeAT5wERvFJ=
[error] ==> X cats.tests.ChainSuite.Chain[ListWrapper[Int]]: hash.same as scala hashing  0.011s munit.FailException: /__w/scala3/scala3/community-build/community-projects/cats/tests/shared/src/test/scala/cats/tests/ChainSuite.scala:86
[error] 85:    implicit val hash: Hash[ListWrapper[Int]] = ListWrapper.hash[Int]
[error] 86:    checkAll("Chain[ListWrapper[Int]]", HashTests[Chain[ListWrapper[Int]]].hash)
[error] 87:    checkAll("Hash[Chain[ListWrapper[Int]]", SerializableTests.serializable(Hash[Chain[ListWrapper[Int]]]))
[error] 
[error] Failing seed: RXDXoio43BvGvzZHkkM6beLOpxXiBVZHry7BvaAfnrE=
[error] You can reproduce this failure by adding the following override to your suite:
[error] 
[error]   override val scalaCheckInitialSeed = "RXDXoio43BvGvzZHkkM6beLOpxXiBVZHry7BvaAfnrE="
[error] 
[error] Falsified after 43 passed tests.
[error] > Labels of failing property: 
[error] Expected: true
[error] Received: false
[error] > ARG_0: Chain(ListWrapper(List(-2147483648, 0, 1, 1)), ListWrapper(List(2147483647, 1, -837828676, 1, 0, -1218981389, 1)), ListWrapper(List(1264078750, 0)), ListWrapper(List(2147483647, 0, -1311984484, 1991506457, 0, 0, 1228787768)), ListWrapper(List(1, 1, 1)), ListWrapper(List(-2147483648, -1, -1138217589, -1)), ListWrapper(List(235219411, -235533261, -2147483648, 0, -1255376156, -1615013631)), ListWrapper(List(-1983960940, 1, -279977609)))
[error] > ARG_1: Chain(ListWrapper(List(-393313933, -1, -931760875, 805542481, 2147483647, 0, -2147483648)), ListWrapper(List(-2147483648)), ListWrapper(List(-922595982, 0, -2147483648, 1, 1157150347, -217687762, -1)), ListWrapper(List(-1, 131[6824](https://github.com/scala/scala3/actions/runs/14199976771/job/39806817179#step:9:6825)960, 2147483647, 860925070, -1407239740, -2147483648, 1178942938, 2147483647)), ListWrapper(List(-1286135301, 1450394857, -2147483648, 1, 0, -1403540053, 765006984, 819725793)), ListWrapper(List(-2147483648, 1883988516, 132907317, -2027462182, 1458282527, -2147483648, 1, -645712917)), ListWrapper(List(0, -1603092190)), ListWrapper(List(1, -812187897, 0, 0, 0, 175060663, -2147483648)))
[error]     at munit.Assertions.fail(Assertions.scala:283)
[error]     at munit.Assertions.fail$(Assertions.scala:15)
[error]     at munit.FunSuite.fail(FunSuite.scala:11)
[error]     at munit.ScalaCheckSuite.propToTry$$anonfun$1(ScalaCheckSuite.scala:104)
[error]     at scala.util.Try$.apply(Try.scala:217)
[error]     at munit.ScalaCheckSuite.propToTry(ScalaCheckSuite.scala:104)
[error]     at munit.ScalaCheckSuite.$init$$$anonfun$1$$anonfun$1$$anonfun$1(ScalaCheckSuite.scala:46)
[error]     at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:475)
[error]     at munit.Suite$$anon$1.execute(Suite.scala:26)
[error]     at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:429)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:338)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:312)
[error]     at scala.concurrent.impl.Promise$DefaultPromise.transform(Promise.scala:128)
[error]     at munit.internal.FutureCompat$ExtensionFuture.transformCompat(FutureCompat.scala:16)
[error]     at munit.ScalaCheckSuite.$init$$$anonfun$1$$anonfun$1(ScalaCheckSuite.scala:48)
[error]     at munit.GenericTest.withBodyMap$$anonfun$1(GenericTest.scala:33)
[error]     at munit.MUnitRunner.$anonfun$10(MUnitRunner.scala:296)
[error] Failed: Total 12846, Failed 2, Errors 0, Passed 12844
[error] Failed tests:
[error] 	cats.tests.NonEmptyLazyListSuite
[error] 	cats.tests.ChainSuite
[error] (testsJVM / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 244 s (04:04), completed Apr 1, 2025 8:28:35 PM
[error] Test dotty.communitybuild.CommunityBuildTestB.cats failed: java.lang.AssertionError: 
Error:  
Error:  sbt exited with an error code. To reproduce without JUnit, use:
Error:  
Error:      sbt community-build/prepareCommunityBuild
Error:      cd community-build/community-projects/cats
Error:      sbt -sbt-version 1.10.7 -Dsbt.supershell=false -Ddotty.communitybuild.dir=/__w/scala3/scala3/community-build --addPluginSbtFile=/__w/scala3/scala3/community-build/sbt-injected-plugins "set Global/testOptions += Tests.Argument(TestFramework("munit.Framework"), "+l"); clean; set Global/logLevel := Level.Error; set Global/updateOptions ~= (_.withLatestSnapshots(false)); set Global/scalacOptions ++= List("-Xcheck-macros");++3.7.1-RC1-bin-SNAPSHOT!; set Global/scalaJSStage := FastOptStage;rootJVM/test;rootJS/test"

[mbovel]

Discussed today during the LAMP meeting; I should try to update the cats dependency, and otherwise to disable the failing test suite.

And we're not sure why community project tests are systematically retried 3 times; probably to account for some flackiness.

[mbovel]

Discussed today during the LAMP meeting; I should try to update the cats dependency, and otherwise to disable the failing test suite.

As updating Cats and its dependencies (#22949) in the community build takes time, it was decided during today's core meeting to we should go with the second solution instead: just disabling the flaky tests in our Cats community build fork, without updating it (#23007).

cc @bracevac