[Regression] PS7.3 - Weird network issues inside Session

[phdavis]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

When inside a session to a remote machine with PS7.3+ and attempt to invoke some commands that rely on the network they seem to fail. Not all network commands fail, such as Test-NetConnection, while others such as Invoke-WebRequest, Connect-AzAccount, etc do. Those that do fail seem to reference the same failure condition 'Could not load file or assembly 'System.Net.Quic, Version=7.0.0.0' when I can get enough output from them.

• Reverting the remote machine back to 7.2.6/7.2.7 resolves the problem and the same code runs without issue.
• If ConfigurationName is set to anything lower than 7.3.0 things begin working as well.
• All systems have good internal network access and internet access.
• Account/s used for testing are all local admins.
• The same code works locally on the machines with 7.3.0. This appears to only be occurring within sessions.

Create a session or invoke a command directly on a remote machine with 7.3+ installed

$TestSession = New-PSSession -Name 'TestSession' -ComputerName 'PS73Machine.tld.com' -ConfigurationName PowerShell.7.3.0 -Credential $Creds

Setup your test and invoke the commands. Some commands work, such as Invoke-RestMethod while others fail.

$TestParams = @{
    Session = $TestSession
    ScriptBlock = {
        try {
            Invoke-WebRequest -Uri 'https://www.powershellgallery.com/api/v2' # Fails
            Find-Module -Name Az -Repository PSGallery # Fails
        } catch {
            $_.Exception
        }
    }
}

Invoke-Command @TestParams

Expected behavior

Commands such as Find-Module with session should return modules matching "Az" from PSGallery.

Actual behavior

Commands such as Find-Module, Invoke-WebRequest, etc within session fail. For those that return an error they reference -

Could not load file or assembly 'System.Net.Quic, Version=7.0.0.0, Culture=neutral The system cannot find the file specified.

Error details

Invoke-WebRequest example -

Exception                     : System.IO.FileNotFoundException: Could not load file or assembly 'System.Net.Quic, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
                                File name: 'System.Net.Quic, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
                                at System.Net.Http.HttpConnectionPool..ctor(HttpConnectionPoolManager poolManager, HttpConnectionKind kind, String host, Int32 port, String sslHostName, Uri proxyUri)
                                at System.Net.Http.HttpConnectionPoolManager.SendAsyncCore(HttpRequestMessage request, Uri proxyUri, Boolean async, Boolean doRequestAuth, Boolean isProxyConnect, CancellationToken cancellationToken)
                                at System.Net.Http.HttpConnectionPoolManager.SendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
                                at System.Net.Http.HttpConnectionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
                                at System.Net.Http.HttpMessageHandlerStage.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
                                at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
                                at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
                                at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
                                at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(HttpClient client, HttpRequestMessage request, Boolean keepAuthorization)
                                at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
                                at System.Management.Automation.CommandProcessor.ProcessRecord()
FullyQualifiedErrorId         : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
InvocationInfo                : System.Management.Automation.InvocationInfo
ErrorCategory_Activity        : Invoke-WebRequest
ErrorCategory_Reason          : FileNotFoundException
ErrorCategory_Message         : NotSpecified: (:) [Invoke-WebRequest], FileNotFoundException
ErrorDetails_ScriptStackTrace : at <ScriptBlock>, <No file>: line 3

Environment data

Test Source -
Name                           Value
----                           -----
PSVersion                      7.3.0
PSEdition                      Core
GitCommitId                    7.3.0
OS                             Microsoft Windows 10.0.17763
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Test Destination -
Name                           Value
----                           -----
PSVersion                      7.3.0
PSEdition                      Core
GitCommitId                    7.3.0
OS                             Microsoft Windows 10.0.20348
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

[iSazonov]

/cc @SteveL-MSFT @PaulHigin

[PaulHigin]

It looks like dotNet assembly path probing logic has changed for v7.3, and is preventing the dependent System.Net.Quic.dll binary from loading. When I try to load it directly, it fails from looking in the wrong place (system drive, instead of $PSHOME).

[remote]: PS> Add-Type -AssemblyName 'System.Net.Quic' -PassThru

Error: Cannot find path 'C:\System.Net.Quic.dll' because it does not exist.

However, I am able to get the probing path corrected by running other commands:

[remote]: PS> Find-module -name az -repository psgallery
[remote]: PS> Add-Type -AssemblyName 'System.Net.Quic' -PassThru

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
False    False    Interop                                  System.Object
...

[remote]: PS> Invoke-WebRequest -Uri 'https://www.powershellgallery.com/api/v2'

StatusCode        : 200
StatusDescription : OK
...

@daxian-dbw I suspect this problem is happening on remote sessions because they are hosted as WinRM plugins, and the dotNet default assembly probing location is not correct (should be $PSHOME). For some reason running other cmdlets (Find-Module) fixes the probing path so that the assembly can be loaded. Are you aware of any changes in dotNet v7.3 that would cause this, and is there anything we can do to ensure the probing path for remote sessions starts at $PSHOME?

[daxian-dbw]

This is because the Trusted Platform Assemblies used in our WinRM plugin was not updated. The fix was submitted: PowerShell/PowerShell-Native#83

[dmpe]

Hi, does this require a new patch release for 7.3? Or is there any workaround when PS 7.3 is already installed?

[iSazonov]

@dmpe The fix will be in 7.3.1 (Jan 23).