Failed Pester 5 tests are not shown in Problems

[fflaten]

Issue Description

When using Pester 5.0.3 to execute a test task, the failed tests are not shown in Problems-view like they used to do with Pester 3.4. This seems to be caused by the new output-format in Pester 5 which doesn't fit with the $pester-problemMatcher regex.

Replacing the current matcher ^\s*(?:\[-\]\s+)(.*?)(?:\s+\d+\.?\d*\s*m?s)\s*$ with ^\s*(?:\[-\]\s+)(.*?)(?:\s+\d+\.?\d*\s*m?s)(?:\s+\(.*?\))?\s*$ in a regex online tester seems to detect both Pester 3.4 and Pester 5 output, but I had problems making it work by editing directly in package.json (remember to escape \), so might need something more.

Proof-of-concept(Click to Expand)

demo.tests.ps1

Describe "VSCode Problem demo" {
    Context "MyContext" {
        if((Get-Module Pester).Version -ge "5.0") {
            It "fails Pester5" {
                1 | Should -Be 2
            }
        } else {
            It "fails Pester" {
                1 | Should Be 2
            }
        }
    }
}

tasks.json

{
    "version": "2.0.0",
    "windows": {
        "options": {
            "shell": {
                "executable": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
                "args": [
                    "-NoProfile",
                    "-ExecutionPolicy",
                    "Bypass",
                    "-Command"
                ]
            }
        }
    },
    "tasks": [
        {
            "label": "Test",
            "type": "shell",
            "command": "Import-Module 'C:\\Program Files\\WindowsPowerShell\\Modules\\Pester\\3.4.0\\Pester.psd1'; Invoke-Pester -PesterOption @{IncludeVSCodeMarker=$true}",
            "group": {
                "kind": "test",
                "isDefault": true
            },
            "problemMatcher": [
                "$pester"
            ]
        },
        {
            "label": "TestPester5",
            "type": "shell",
            "command": "Import-Module 'C:\\Program Files\\WindowsPowerShell\\Modules\\Pester\\5.0.3\\Pester.psd1'; Invoke-Pester -Configuration ([PesterConfiguration]@{Output=@{Verbosity = 'Detailed'}; Debug = @{ ShowNavigationMarkers = $true } })",
            "group": {
                "kind": "test",
                "isDefault": true
            },
            "problemMatcher": [
                "$pester"
            ]
        }
    ]
}

Output Pester 3.4 (shown in Problems-tab):

Describing VSCode Problem demo
   Context MyContext
    [-] fails Pester 398ms
      at <ScriptBlock>, C:\Sandbox\demo.tests.ps1: line 9
      Expected: {2}
      But was:  {1}
      9:                 1 | Should Be 2
      at <ScriptBlock>, C:\Sandbox\demo.tests.ps1: line 9
Tests completed in 398ms
Passed: 0 Failed: 1 Skipped: 0 Pending: 0 Inconclusive: 0

Output Pester 5.0.3 (not shown in Problems-tab):

Starting discovery in 1 files.
Discovering in demo.tests.ps1.
Found 1 tests. 95ms
Discovery finished in 192ms.

Running tests from 'demo.tests.ps1'
Describing VSCode Problem demo, C:\Sandbox\demo.tests.ps1:1
 Context MyContext, C:\Sandbox\demo.tests.ps1:2
   [-] fails Pester5, C:\Sandbox\demo.tests.ps1:4 124ms (94ms|30ms)
    Expected 2, but got 1.
    at 1 | Should -Be 2, C:\Sandbox\demo.tests.ps1:5
    at <ScriptBlock>, C:\Sandbox\demo.tests.ps1:5
Tests completed in 727ms
Tests Passed: 0, Failed: 1, Skipped: 0 NotRun: 0

Attached Logs

Follow the instructions in the README about
capturing and sending logs.

Environment Information

Visual Studio Code

Name	Version
Operating System	Windows_NT x64 10.0.19041
VSCode	1.48.2
PowerShell Extension Version	2020.6.0

PowerShell Information

Name	Value
PSVersion	7.0.3
PSEdition	Core
GitCommitId	7.0.3
OS	Microsoft Windows 10.0.19041
Platform	Win32NT
PSCompatibleVersions	1.0 2.0 3.0 4.0 5.0 5.1.10032.0 6.0.0 6.1.0 6.2.0 7.0.3
PSRemotingProtocolVersion	2.3
SerializationVersion	1.1.0.1
WSManStackVersion	3.0

Visual Studio Code Extensions

Visual Studio Code Extensions(Click to Expand)

Extension	Author	Version
azure-account	ms-vscode	0.9.2
azure-pipelines	ms-azure-devops	1.174.2
gc-excelviewer	GrapeCity	3.0.40
githistory	donjayamanne	0.6.9
gitlens	eamodio	10.2.2
hexeditor	ms-vscode	1.2.1
ilspy-vscode	icsharpcode	0.9.0
material-icon-theme	PKief	4.2.0
material-theme	zhuangtongfa	3.8.7
open-in-browser	techer	2.0.0
powershell	ms-vscode	2020.6.0
rainbow-csv	mechatroner	1.7.1
reg	ionutvmi	1.0.1
remote-containers	ms-vscode-remote	0.134.1
remote-ssh	ms-vscode-remote	0.51.0
remote-ssh-edit	ms-vscode-remote	0.51.0
remote-wsl	ms-vscode-remote	0.44.5
vscode-azurefunctions	ms-azuretools	0.24.0
vscode-azurestorage	ms-azuretools	0.9.0
vscode-docker	ms-azuretools	1.5.0
vscode-remote-extensionpack	ms-vscode-remote	0.20.0
vscode-test-explorer	hbenl	2.19.1
vsliveshare	ms-vsliveshare	1.0.2731
xml	DotJoshJohnson	2.5.1

[SydneyhSmith]

Thanks @fflaten I believe @nohwnd has started on a new problems matcher for Pester 5 output, can you confirm? Thanks!

[fflaten]

Thanks for the update. Seems to be related to pester/Pester#1452 and pester/Pester#1577. Should still track the issue here I guess considering the problemmatcher is a feature of the extension.

[nohwnd]

There was this PR #2447 where we talked about this, but I actually did not do anything. @fflaten if you want to implement this PRs are more than welcome. On Pester side you'd need to use the mentioned environment variables to detect if we are running in vscode to enable the appropriate output (if needed). I There already is an option on the configuration object but that one is ignored as described in pester/Pester#1577.

[SydneyhSmith]

Thanks @nohwnd for the update, on the vscode side looks like the change would happen here https://github.com/powershell/vscode-powershell/blob/master/package.json#L456-L474 I will go ahead and mark this issue as up for grabs

[fflaten]

@nohwnd I'll take a look when I get a chance.

  /**
   * The name of a predefined problem pattern, the inline definition
   * of a problem pattern or an array of problem patterns to match
   * problems spread over multiple lines.
   */
  pattern?: string | ProblemPattern | ProblemPattern[];

Quote from: https://code.visualstudio.com/docs/editor/tasks-appendix

Do the patterns need to match the line order perfectly when using the ProblemPattern[] option? The pester problemmatcher has two patterns, but Pester 5 doesn't output the file+line pattern on line 2. Might explain why my first attempt to modify the pattern in package.json failed.