Add Windows Compat Pack 2.0.0 to PSCore6 and update package references to final versions including dotnetcore 2.1 by SteveL-MSFT · Pull Request #6958 · PowerShell/PowerShell

Steve Lee (SteveL-MSFT) · 2018-05-30T16:30:05Z

PR Summary

To support PowerShell Modules built with .Net Windows Compatibility Pack, we decided that it was best to ship the WCP assemblies with PSCore6. This also adds many new .Net apis be default while only adding ~3.5 MB additional disk footprint (to a ~137 MB install currently).

Added ref to WCP 2.0.0 to SDK csproj.

Updated all refs of 4.5.0-RC1 to 4.5.0 final.

Update dotnetcore2.1 to final build

Update WebListener to aspnetcore.app 2.1.0 final
(Microsoft.VisualStudio.Web.CodeGeneration.Tools is not final yet, so no change)

PR Checklist

Dongbo Wang (daxian-dbw) · 2018-05-30T16:43:11Z

    <PackageReference Include="PSDesiredStateConfiguration" Version="6.0.0-beta.8" />
    <PackageReference Include="PowerShellHelpFiles" Version="1.0.0-*" />
    <PackageReference Include="psrp.windows" Version="6.1.0-*" />
+    <PackageReference Include="Microsoft.Windows.Compatibility" Version="2.0.0" />


I think this reference should be put in SDK.csproj with a Condition attribute, like <PackageReference Include="Microsoft.Windows.Compatibility" Version="2.0.0" Condition=" '$(IsWindows)' != 'true' " />. In this way, application that refernce to SDK nuget package will get those extended pack too.

IsWindows is defined in https://github.com/PowerShell/PowerShell/blob/master/PowerShell.Common.props#L113

/cc Aditya Patwardhan (@adityapatwardhan), in case I'm wrong about the SDK reference part.

Actually, I don't know if we should even have the condition part. Some of the compat assemblies work on Unix platforms too.

Yes, this should be part of SDK, so apps hosting PowerShell can use the modules with WCP.

Perhaps we don't need the condition as Dongbo Wang (@daxian-dbw) is correct that some of the assemblies from the WCP meta-package work on non-Windows so we should make those available. I'll make the change.

updated all refs of 4.5.0-RC1 to 4.5.0 final

Dongbo Wang (daxian-dbw) · 2018-05-30T17:59:45Z

Ha, it looks dotnet core 2.1 is officially out: https://www.nuget.org/packages/Microsoft.NETCore.App/2.1.0
Shall we update our build to use the official 2.1?

Mark Kraus (markekraus) · 2018-05-30T19:11:48Z

Steve Lee (@SteveL-MSFT) Can you push a commit with [feature] to ensure that the version bump didn't cause any regressions?

Christoph Bergmeister (bergmeister) · 2018-05-30T19:12:31Z

Steve Lee (@SteveL-MSFT) I believe you also need to update the installer for .Net Core 2.1 RTM and cleanup comments, see my commit here

update cache names for CI update files.wxs to reflect new versions

Ilya (iSazonov) · 2018-05-31T06:50:39Z

We have a issue with System.Text.Encoding.CodePages. Maybe not still published.

Ilya (iSazonov) · 2018-05-31T06:52:51Z

We could use Version="4.5.*" for packages.

Ilya (iSazonov) · 2018-05-31T15:31:44Z

we decided that it was best to ship the WCP assemblies with PSCore6

Can we use it directly - to access registry, perfcounters and so on?

Steve Lee (SteveL-MSFT) · 2018-05-31T16:37:31Z

Ilya (@iSazonov) System.Text.Encoding.CodePages 4.5.0 is published: https://www.nuget.org/packages/System.Text.Encoding.CodePages/

update pwrshplugin.dll trusted assembly list

Dongbo Wang (daxian-dbw) · 2018-05-31T22:01:19Z

-        "Microsoft.CodeAnalysis",
        "Microsoft.CodeAnalysis.CSharp",
+        "Microsoft.CodeAnalysis",
+        "Microsoft.CodeAnalysis.VisualBasic",


Do we want to include this one? I think we will remove VB support before 6.1 RC. If we include it here now, we will have to update the windows.psrp package again when removing the VB support.

Will remove. We can always add it back later if needed.

Dongbo Wang (daxian-dbw) · 2018-05-31T22:05:13Z

  </ItemGroup>

  <ItemGroup>
+    <PackageReference Include="Microsoft.Windows.Compatibility" Version="2.0.0" />


If we don't know the source location of this package, maybe put it at the end, in the section with the comment the source could not be found for the following package(s), along with Microsoft.NETCore.Windows.ApiSets

address Dongbo's feedback

Steve Lee (SteveL-MSFT) · 2018-06-01T14:34:07Z

Can't tell why System.Text.Encoding.CodePages is failing to load as the signature matches the assembly.

Ilya (iSazonov) · 2018-06-01T15:02:00Z

I suspect this is a publishing error. Could you ask .Net team?

Steve Lee (SteveL-MSFT) · 2018-06-01T16:35:44Z

Ilya (@iSazonov) that xunit test passes fine on my desktop machine, just not in AppVeyor though

Dongbo Wang (daxian-dbw) · 2018-06-01T17:36:27Z

I see the following from Start-PSxUnit:

            $requiredDependencies = @(
                $nativeLib,
                "$Content/Microsoft.Management.Infrastructure.dll",
                "$Content/System.Text.Encoding.CodePages.dll"
            )

The codepages.dll line is suspicious to me. But I haven't spent time digging further yet.

Removed unncessary dep ref for psxunit on System.Text.Encoding.CodePages.dll Updated packaging nupkg dep assembly versions

Ilya (iSazonov) · 2018-06-01T18:05:45Z

Errors in Appveyor tests:

[00:21:47]   Describing CmsMessage cmdlets thorough tests
[00:21:48] +++++++++++++++++++?+
[00:21:48] 
[00:21:48] New-Item : The directory specified, 'Modules', is not a subdirectory of 'C:\'.
[00:21:48] Parameter name: path
[00:21:48] At C:\projects\powershell\test\powershell\Modules\Microsoft.PowerShell.Security\ConstrainedLanguageRestriction.Tests.ps1:87 char:21
[00:21:48] + ...          $null = New-Item -ItemType Directory $moduleDirectory -Force
[00:21:48] +                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[00:21:48] + CategoryInfo          : InvalidArgument: (C:\Modules:String) [New-Item], ArgumentException
[00:21:48] + FullyQualifiedErrorId : CreateDirectoryArgumentError,Microsoft.PowerShell.Commands.NewItemCommand

[00:22:22]   Describing Invoke-Item basic tests
[00:22:22] 
[00:22:22]     Context Invoke a text file on Unix
[00:22:22] !!
[00:22:22] Get-Process : Cannot find a process with the name "notepad". Verify the process name and call the cmdlet again.
[00:22:22] At C:\projects\powershell\test\powershell\Modules\Microsoft.PowerShell.Utility\Invoke-Item.Tests.ps1:70 char:17
[00:22:22] +                 Get-Process -Name $notepadProcessName | Stop-Process  ...
[00:22:22] +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[00:22:22] + CategoryInfo          : ObjectNotFound: (notepad:String) [Get-Process], ProcessCommandException
[00:22:22] + FullyQualifiedErrorId : NoProcessFoundForGivenName,Microsoft.PowerShell.Commands.GetProcessCommand

[00:37:57] new file {$(env.ProductSourcePath)\Microsoft.Win32.SystemEvents.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.CodeDom.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.ComponentModel.Composition.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Configuration.ConfigurationManager.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Data.DataSetExtensions.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Data.Odbc.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Diagnostics.PerformanceCounter.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.DirectoryServices.AccountManagement.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.DirectoryServices.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.DirectoryServices.Protocols.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Drawing.Common.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.IO.Ports.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Management.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Runtime.Caching.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Security.Cryptography.ProtectedData.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.Security.Cryptography.Xml.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57] new file {$(env.ProductSourcePath)\System.ServiceModel.Syndication.dll} need to be added to {C:\projects\powershell\tools\packaging\..\..\assets\Files.wxs}
[00:37:57]

Steve Lee (SteveL-MSFT) · 2018-06-01T20:33:08Z

Ok, figured out the problem with System.Text.Encoding.CodePages.dll. The nupkg doesn't contain unix runtime only Windows. Older version has both. Created https://github.com/dotnet/corefx/issues/30059

Moving back to v4.3.0 fixes that issue. Will update files.wxs and submit new commit. (v4.4.0 also doesn't have unix runtime)

revert ref to System.Text.Encoding.CodePages to 4.3.0 as newer versions don't have Unix runtime updated files.wxs

revert CodePages back to 4.5.0-rc1

Dongbo Wang (daxian-dbw) · 2018-06-02T00:05:45Z

PSXunit tests passed on my machine (consistently failing previously) after I update the xunit packages. So push a change to see if that also fixes the issue in CI.

Steve Lee (SteveL-MSFT) · 2018-06-02T01:16:42Z

AppVeyor failure is due to recent PR to migrate the install docs, the aka.ms link now returned 404. I updated the aka.ms link so restarting AppVeyor.

Dongbo Wang (daxian-dbw)

LGTM

Ilya (iSazonov)

Leave a comment

Ilya (iSazonov) · 2018-06-02T09:19:30Z

+    <PackageReference Include="System.Text.Encoding.CodePages" Version="4.5.0" />
    <!-- the following package(s) are from the powershell org -->
    <PackageReference Include="Microsoft.Management.Infrastructure" Version="1.0.0-alpha*" />
    <PackageReference Include="PowerShell.Core.Instrumentation" Version="6.0.0-beta*" />


Have we this APIs in WCP?

These are pkgs PowerShell Team publishes

Should we migrate to WCP assemblies if we have them on long time?

You mean include them in WCP? Probably not.

Ilya (iSazonov) · 2018-06-02T09:24:50Z

        "Microsoft.ApplicationInsights",
-        "Microsoft.CodeAnalysis",
        "Microsoft.CodeAnalysis.CSharp",
+        "Microsoft.CodeAnalysis",


Why was the order changed?

The list was generated rather than trying to insert a large number of new assemblies into the list. Sort-Object seems to prefer the period over end of string.

Ilya (iSazonov) · 2018-06-02T09:26:11Z

    <!-- DotNetCliToolReference element specifies the CLI tool that the user wants to restore in the context of the project. -->
-    <DotNetCliToolReference Include="dotnet-xunit" Version="2.3.1" />
-    <PackageReference Include="Xunit.SkippableFact" Version="1.3.3" />
+    <DotNetCliToolReference Include="dotnet-xunit" Version="2.4.0-beta.1.build3958" />


I think we should open new tracking issue to update the beta to release.
And Microsoft.VisualStudio.Web.CodeGeneration.Tools too.

Created #6977

Ilya (iSazonov) · 2018-06-02T09:29:32Z

                'Microsoft.PowerShell.Commands.Management' {
                    $deps.Add([tuple]::Create([tuple]::Create('id', 'Microsoft.PowerShell.Security'), [tuple]::Create('version', $PackageVersion))) > $null
-                    $deps.Add([tuple]::Create([tuple]::Create('id', 'System.ServiceProcess.ServiceController'), [tuple]::Create('version', '4.4.1'))) > $null
+                    $deps.Add([tuple]::Create([tuple]::Create('id', 'System.ServiceProcess.ServiceController'), [tuple]::Create('version', '4.5.0'))) > $null


Not related the PR - why we set the versions manually and do not take from csproj files?

Ilya (iSazonov) · 2018-06-02T09:32:33Z

        <DirectoryRef Id="$(var.ProductDirectoryName)">
+            <Component Id="cmpCBE3857586454EB3B634B118E8EBD1D1" Guid="{CBE38575-8645-4EB3-B634-B118E8EBD1D1}">
+                <File Id="filCBE3857586454EB3B634B118E8EBD1D1" KeyPath="yes" Source="$(env.ProductSourcePath)\Microsoft.Win32.SystemEvents.dll" />
+            </Component>


I lost understanding - why do we update the file manually? Can we generate it automatically?

The WiX tools will break patching. When you build, I added errors to tell you what to add or remove, but didn't fully automate the update. I updated the issue with the requirements of the update.

Ilya (iSazonov) · 2018-06-02T18:10:37Z

Newtonsoft.Json latest version is 11.0.1
https://github.com/JamesNK/Newtonsoft.Json/releases

NJsonSchema latest version is 9.10.50
https://www.nuget.org/packages/NJsonSchema/

update Newtonsoft.Json and NJsonSchema to latest versions

…l .NET Core 2.1 (PowerShell#6958) To support PowerShell modules built with .NET Windows Compatibility Pack, we decided that it was best to ship the WCP assemblies with PS Core. This also adds many new .NET APIs be default while only adding ~3.5 MB additional disk footprint (to a ~137 MB install currently). Also update the build to adopt the official .NET Core 2.1.

add Windows Compat Pack 2.0.0 to PSCore6

7bb7c0b

Steve Lee (SteveL-MSFT) requested review from Travis Plunk (TravisEz13), Aditya Patwardhan (adityapatwardhan) and Dongbo Wang (daxian-dbw) as code owners May 30, 2018 16:30

Steve Lee (SteveL-MSFT) assigned Dongbo Wang (daxian-dbw) May 30, 2018

Dongbo Wang (daxian-dbw) reviewed May 30, 2018

View reviewed changes

moved ref of WCP to SDK

f4b146f

updated all refs of 4.5.0-RC1 to 4.5.0 final

update dotnetcore to 2.1.300 RTM

24c5bc8

Steve Lee (SteveL-MSFT) changed the title ~~Add Windows Compat Pack 2.0.0 to PSCore6~~ Add Windows Compat Pack 2.0.0 to PSCore6 and update package references to final versions including dotnetcore 2.1 May 30, 2018

[feature]

625f106

update cache names for CI update files.wxs to reflect new versions

[feature]

ce61eb9

update pwrshplugin.dll trusted assembly list

Steve Lee (SteveL-MSFT) force-pushed the add-wcp branch from 5bfdb3c to ce61eb9 Compare May 31, 2018 21:25

Dongbo Wang (daxian-dbw) reviewed May 31, 2018

View reviewed changes

[feature]

0ee8d62

address Dongbo's feedback

[feature]

fdf87e4

Removed unncessary dep ref for psxunit on System.Text.Encoding.CodePages.dll Updated packaging nupkg dep assembly versions

[feature]

6c0fa8b

revert ref to System.Text.Encoding.CodePages to 4.3.0 as newer versions don't have Unix runtime updated files.wxs

Steve Lee (SteveL-MSFT) force-pushed the add-wcp branch from e25f2f2 to 6c0fa8b Compare June 1, 2018 21:07

Steve Lee (POWERSHELL) and others added 2 commits June 1, 2018 15:13

[feature]

39628b7

revert CodePages back to 4.5.0-rc1

[Feature] Update xunit packages

40d3425

Dongbo Wang (daxian-dbw) approved these changes Jun 2, 2018

View reviewed changes

Ilya (iSazonov) approved these changes Jun 2, 2018

View reviewed changes

Steve Lee (SteveL-MSFT) mentioned this pull request Jun 2, 2018

Update PackageReferences to GA releases #6977

Closed

[feature]

ceb25f0

update Newtonsoft.Json and NJsonSchema to latest versions

Steve Lee (SteveL-MSFT) force-pushed the add-wcp branch from f5e1552 to ceb25f0 Compare June 2, 2018 20:20

Dongbo Wang (daxian-dbw) merged commit 5dc7a6c into PowerShell:master Jun 4, 2018

Steve Lee (SteveL-MSFT) deleted the add-wcp branch June 5, 2018 01:15

mi-hol mentioned this pull request Jun 20, 2018

Question: has it been tested with PowerShell Core 6.0.0? tomohulk/WinSCP#65

Closed

Conversation

Steve Lee (SteveL-MSFT) commented May 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dongbo Wang (daxian-dbw) commented May 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mark Kraus (markekraus) commented May 30, 2018

Uh oh!

Christoph Bergmeister (bergmeister) commented May 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ilya (iSazonov) commented May 31, 2018

Uh oh!

Ilya (iSazonov) commented May 31, 2018

Uh oh!

Ilya (iSazonov) commented May 31, 2018

Uh oh!

Steve Lee (SteveL-MSFT) commented May 31, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Steve Lee (SteveL-MSFT) commented Jun 1, 2018

Uh oh!

Ilya (iSazonov) commented Jun 1, 2018

Uh oh!

Steve Lee (SteveL-MSFT) commented Jun 1, 2018

Uh oh!

Dongbo Wang (daxian-dbw) commented Jun 1, 2018

Uh oh!

Ilya (iSazonov) commented Jun 1, 2018

Uh oh!

Steve Lee (SteveL-MSFT) commented Jun 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dongbo Wang (daxian-dbw) commented Jun 2, 2018

Uh oh!

Steve Lee (SteveL-MSFT) commented Jun 2, 2018

Uh oh!

Dongbo Wang (daxian-dbw) left a comment

Choose a reason for hiding this comment

Uh oh!

Ilya (iSazonov) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ilya (iSazonov) Jun 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Steve Lee (SteveL-MSFT) commented May 30, 2018 •

edited

Loading

Dongbo Wang (daxian-dbw) commented May 30, 2018 •

edited

Loading

Christoph Bergmeister (bergmeister) commented May 30, 2018 •

edited

Loading

Steve Lee (SteveL-MSFT) commented Jun 1, 2018 •

edited

Loading

Ilya (iSazonov) Jun 3, 2018 •

edited

Loading