v1.21 nondeterministic linting due to `ctrlflow: failed prerequisites`

[howardjohn]

Thank you for creating the issue!

• Yes, I'm using a binary release within 2 latest major releases. Only such installations are supported.
• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've included all information below (version, config, etc).

Please include the following information:

Version of golangci-lint

$ golangci-lint --version
v1.21

Config file

$ cat .golangci.yml
# WARNING: DO NOT EDIT, THIS FILE IS PROBABLY A COPY
#
# The original version of this file is located in the https://github.com/istio/common-files repo.
# If you're looking at this file in a different repo and want to make a change, please go to the
# common-files repo, make the change there and check it in. Then come back to this repo and run
# "make update-common".

service:
  # When updating this, also update the version stored in docker/build-tools/Dockerfile in the istio/tools repo.
  golangci-lint-version: 1.21.x # use the fixed version to not introduce new linters unexpectedly
run:
  # timeout for analysis, e.g. 30s, 5m, default is 1m
  deadline: 20m

  # which dirs to skip: they won't be analyzed;
  # can use regexp here: generated.*, regexp is applied on full path;
  # default value is empty list, but next dirs are always skipped independently
  # from this option's value:
  #   	vendor$, third_party$, testdata$, examples$, Godeps$, builtin$
  skip-dirs:
    - genfiles$
    - vendor$

  # which files to skip: they will be analyzed, but issues from them
  # won't be reported. Default value is empty list, but there is
  # no need to include all autogenerated files, we confidently recognize
  # autogenerated files. If it's not please let us know.
  skip-files:
    - ".*\\.pb\\.go"
    - ".*\\.gen\\.go"

linters:
  enable-all: true
  disable:
    - bodyclose
    - depguard
    - dogsled
    - dupl
    - funlen
    - gochecknoglobals
    - gochecknoinits
    - gocognit
    - goconst
    - gocyclo
    - godox
    - gosec
    - nakedret
    - prealloc
    - scopelint
    - whitespace
    - wsl
  fast: false

linters-settings:
  errcheck:
    # report about not checking of errors in type assetions: `a := b.(MyStruct)`;
    # default is false: such cases aren't reported by default.
    check-type-assertions: false

    # report about assignment of errors to blank identifier: `num, _ := strconv.Atoi(numStr)`;
    # default is false: such cases aren't reported by default.
    check-blank: false
  govet:
    # report about shadowed variables
    check-shadowing: false
  golint:
    # minimal confidence for issues, default is 0.8
    min-confidence: 0.0
  gofmt:
    # simplify code: gofmt with `-s` option, true by default
    simplify: true
  goimports:
    # put imports beginning with prefix after 3rd-party packages;
    # it's a comma-separated list of prefixes
    local-prefixes: istio.io/
  maligned:
    # print struct with more effective memory layout or not, false by default
    suggest-new: true
  misspell:
    # Correct spellings using locale preferences for US or UK.
    # Default is to use a neutral variety of English.
    # Setting locale to US will correct the British spelling of 'colour' to 'color'.
    locale: US
    ignore-words:
    - cancelled
  lll:
    # max line length, lines longer will be reported. Default is 120.
    # '\t' is counted as 1 character by default, and can be changed with the tab-width option
    line-length: 160
    # tab width in spaces. Default to 1.
    tab-width: 1
  unused:
    # treat code as a program (not a library) and report unused exported identifiers; default is false.
    # XXX: if you enable this setting, unused will report a lot of false-positives in text editors:
    # if it's called for subdir of a project it can't find funcs usages. All text editor integrations
    # with golangci-lint call it on a directory with the changed file.
    check-exported: false
  unparam:
    # call graph construction algorithm (cha, rta). In general, use cha for libraries,
    # and rta for programs with main packages. Default is cha.
    algo: cha

    # Inspect exported functions, default is false. Set to true if no external program/library imports your code.
    # XXX: if you enable this setting, unparam will report a lot of false-positives in text editors:
    # if it's called for subdir of a project it can't find external interfaces. All text editor integrations
    # with golangci-lint call it on a directory with the changed file.
    check-exported: false
  gocritic:
    enabled-checks:
      - appendCombine
      - argOrder
      - assignOp
      - badCond
      - boolExprSimplify
      - builtinShadow
      - captLocal
      - caseOrder
      - codegenComment
      - commentedOutCode
      - commentedOutImport
      - defaultCaseOrder
      - deprecatedComment
      - docStub
      - dupArg
      - dupBranchBody
      - dupCase
      - dupSubExpr
      - elseif
      - emptyFallthrough
      - equalFold
      - flagDeref
      - flagName
      - hexLiteral
      - indexAlloc
      - initClause
      - methodExprCall
      - nilValReturn
      - octalLiteral
      - offBy1
      - rangeExprCopy
      - regexpMust
      - sloppyLen
      - stringXbytes
      - switchTrue
      - typeAssertChain
      - typeSwitchVar
      - typeUnparen
      - underef
      - unlambda
      - unnecessaryBlock
      - unslice
      - valSwap
      - weakCond

      # Unused
      # - yodaStyleExpr
      # - appendAssign
      # - commentFormatting
      # - emptyStringTest
      # - exitAfterDefer
      # - ifElseChain
      # - hugeParam
      # - importShadow
      # - nestingReduce
      # - paramTypeCombine
      # - ptrToRefParam
      # - rangeValCopy
      # - singleCaseSwitch
      # - sloppyReassign
      # - unlabelStmt
      # - unnamedResult
      # - wrapperFunc

issues:
  # List of regexps of issue texts to exclude, empty list by default.
  # But independently from this option we use default exclude patterns,
  # it can be disabled by `exclude-use-default: false`. To list all
  # excluded by default patterns execute `golangci-lint run --help`
  exclude:
    - composite literal uses unkeyed fields

  exclude-rules:
    # Exclude some linters from running on test files.
    - path: _test\.go$|^tests/|^samples/
      linters:
        - errcheck
        - maligned

  # Independently from option `exclude` we use default exclude patterns,
  # it can be disabled by this option. To list all
  # excluded by default patterns execute `golangci-lint run --help`.
  # Default value for this option is true.
  exclude-use-default: true

  # Maximum issues count per one linter. Set to 0 to disable. Default is 50.
  max-per-linter: 0

  # Maximum count of issues with the same text. Set to 0 to disable. Default is 3.
  max-same-issues: 0

Go environment

$ go version && go env
1.13

In our CI, we ran the same commit multiple times, and got different results

1. Failed https://prow.istio.io/view/gcs/istio-prow/pr-logs/pull/istio_istio/18520/lint_istio/3431
2. Passed https://prow.istio.io/view/gcs/istio-prow/pr-logs/pull/istio_istio/18520/lint_istio/3435
3. Passed https://prow.istio.io/view/gcs/istio-prow/pr-logs/pull/istio_istio/18520/lint_istio/3432

Since it merged, some others have passed:

https://prow.istio.io/view/gcs/istio-prow/logs/lint_istio_postsubmit/422
https://prow.istio.io/view/gcs/istio-prow/logs/lint_istio_postsubmit/423

and some have failed

https://prow.istio.io/view/gcs/istio-prow/pr-logs/pull/istio_istio/18571/lint_istio/3452
https://prow.istio.io/view/gcs/istio-prow/pr-logs/pull/istio_istio/18512/lint_istio/3453

All of the ones that pass warn that they are skipping some linter:

level=warning msg="[runner] Can't run linter goanalysis_metalinter: fact_purity: failed prerequisites: buildssa@istio.io/istio/pilot/pkg/config/memory [istio.io/istio/pilot/pkg/model.test]"

level=warning msg="[runner] Can't run linter goanalysis_metalinter: fact_deprecated: failed prerequisites: fact_deprecated@istio.io/istio/pilot/pkg/config/memory [istio.io/istio/pilot/pkg/model.test], fact_deprecated@istio.io/istio/pilot/pkg/serviceregistry/memory [istio.io/istio/pilot/pkg/model.test], fact_deprecated@istio.io/istio/pilot/test/mock [istio.io/istio/pilot/pkg/model.test]"

level=warning msg="[runner] Can't run linter goanalysis_metalinter: ctrlflow: failed prerequisites: inspect@istio.io/istio/pilot/pkg/config/memory [istio.io/istio/pilot/pkg/model.test]"

And all the ones that fail do not have these messages

[howardjohn]

I am not certain but I think the root cause is #866

[howardjohn]

I am pretty sure #866 is not exactly the same, although they may be related. We are still seeing this even though our build passes. Another example, two CI runs of the same code:

https://prow.istio.io/view/gcs/istio-prow/pr-logs/pull/istio_istio/19217/lint_istio/4405
https://prow.istio.io/view/gcs/istio-prow/pr-logs/pull/istio_istio/19217/lint_istio/4406

[howardjohn]

Originally I could only reproduce in our CI as well, now I can do so locally.

$ docker run -t -i --sig-proxy=true -u 576696:996 --rm -e IN_BUILD_CONTAINER=1 -e TZ=America/Los_Angeles -e TARGET_ARCH=amd64 -e TARGET_OS=linux -e TARGET_OUT=/work/out/linux_amd64 -e USER=howardjohn -v /etc/passwd:/etc/passwd:ro -v /var/run/docker.sock:/var/run/docker.sock --mount type=bind,source=/usr/local/google/home/howardjohn/go/src/istio.io/istio,destination=/work --mount type=volume,source=go,destination=/go --mount type=volume,source=gocache,destination=/gocache --mount type=bind,source=/usr/local/google/home/howardjohn/.docker,destination=/config/.docker,readonly --mount type=bind,source=/usr/local/google/home/howardjohn/.config/gcloud,destination=/config/.config/gcloud,readonly -w /work gcr.io/istio-testing/build-tools:master-2019-11-14T12-01-13 make --no-print-directory -e -f Makefile.core.mk lint-go
$ docker run -t -i --sig-proxy=true -u 576696:996 --rm -e IN_BUILD_CONTAINER=1 -e TZ=America/Los_Angeles -e TARGET_ARCH=amd64 -e TARGET_OS=linux -e TARGET_OUT=/work/out/linux_amd64 -e USER=howardjohn -v /etc/passwd:/etc/passwd:ro -v /var/run/docker.sock:/var/run/docker.sock --mount type=bind,source=/usr/local/google/home/howardjohn/go/src/istio.io/istio,destination=/work --mount type=volume,source=go,destination=/go --mount type=volume,source=gocache,destination=/gocache --mount type=bind,source=/usr/local/google/home/howardjohn/.docker,destination=/config/.docker,readonly --mount type=bind,source=/usr/local/google/home/howardjohn/.config/gcloud,destination=/config/.config/gcloud,readonly -w /work gcr.io/istio-testing/build-tools:master-2019-11-14T12-01-13 make --no-print-directory -e -f Makefile.core.mk lint-go
WARN [runner] Can't run linter goanalysis_metalinter: ctrlflow: failed prerequisites: inspect@istio.io/istio/pilot/pkg/config/memory [istio.io/istio/pilot/pkg/model.test]

All the noise with the docker run command was just to more closely simulate what is going on in CI

[howardjohn]

$ for i in {1..100}; do docker run -v "${PWD}":"${PWD}" -w "${PWD}" -e GOPATH=$GOPATH golangci/golangci-lint:v1.21 golangci-lint run -c ./common/config/.golangci.yml ./pilot/...; echo "done $i"; done
level=warning msg="[runner] Can't run linter goanalysis_metalinter: fact_deprecated: failed prerequisites: fact_deprecated@istio.io/istio/pkg/kube/inject"
done 1
done 2
level=warning msg="[runner] Can't run linter goanalysis_metalinter: fact_deprecated: failed prerequisites: fact_deprecated@istio.io/istio/pkg/kube/inject"
done 3
level=warning msg="[runner] Can't run linter goanalysis_metalinter: ctrlflow: failed prerequisites: inspect@istio.io/istio/pkg/kube/inject"
done 4
level=warning msg="[runner] Can't run linter goanalysis_metalinter: fact_deprecated: failed prerequisites: fact_deprecated@istio.io/istio/pkg/kube/inject"

Can be reproduced on master branch of https://github.com/istio/istio. Specifically I ran at 5803182856ec3a831529586e8ac265c17f417bf4 but probably similar regardless of cmmit