Proposal: Basic Telemetry for PSSA

[SydneyhSmith]

PSScript Analyzer Proposed Telemetry

Goal

To better understand how PSSA is being used both within PSES and in isolation in order to make informed investments in PSSA 2.0.

Questions

• What versions of PowerShell are being used with PSSA? What versions of PowerShell should we continue to support?
• What tasks are PSSA currently being used to accomplish?
  • To what extent is PSSA being used inside vs. outside of PSES? Which of these experiences is most important to optimize?
  • To what extent are custom rules vs. built in rules being used?
  • What is the proportional use of PSSA as a formatter vs "analyzer"?
• To what extent are the compatibility rules being used? Which platforms are being targeted the most?
• What rules are being turned off? What rules should we strip out?
• Which rules are causing the largest performance delays?

Proposed Metrics

• Version of PowerShell
• Inside vs Outside of PSES
• CI usage/environment type
• Time for each rule to complete
• Count of:
  • Known formatter type rules
  • Known analyzer type rules (tally for each one)
  • Custom rules (not on our known list)
  • Known compatibility profiles (tally for each one)
  • Use of a configuration

[bergmeister]

Which rules are causing the largest performance delays?

This is very easy, by largest it's the 5 rules that call Helper.Instance.GetCommandInfo (which calls Get-Command) (file names are very similar to the rule names):

• AvoidAlias.cs
• UseCmdletCorrectly.cs
• UseCorrectCasing.cs
• UseCmdletCorrectly.cs
• UseShouldProcessCorrectly.cs

I've found that by measuring it against small or big (>1000 lines) scripts. There is a cache, so there is usually just a performance penalty for the first cold run but not for re-runs.

Generally, because custom rules are written in PowerShell and not C# (although I can technically one could write a custom rule in C#), they are much slower than any other rule. I've done a POC once to include 1 custom rule in the default rule set of PSSA and it would've increased the test execution time by a factor of around 3.

I propose to just measure the total time of an invoke-scriptanalyzer/invoke-formatter run because all rules are run in parallel, therefore execution time is always bottle-necked by the slowest rule, if you wanted to measure rules individually, I'd only report the slowest 5 ones.

Looking forward into the future, I'd also report the version of PSSA

Please be aware that PSSA is also used in other editors (VisualStudio, JetBrains, etc.), therefore if you could detect other editor usages, that would be interesting

How about reporting OS type/version?

[iSazonov]

therefore if you could detect other editor usages, that would be interesting

Is PSSA dll loaded in editor? If yes the telemetry could be a root assembly name.

[iSazonov]

In PowerShell repo I shared an idea to inject performance counters in PowerShell engine. Would this help PSSA?