Set-Content/Add-Content/Get-Content use an 8-bit character encoding by default, but the help topics state ASCII; problematic Core default file encoding

Ilya (@iSazonov)

This issue has two distinct aspects:

discussion of an existing documentation bug
discussion of the problematic fixed default file encoding currently (alpha16) chosen for Core.

Steps to reproduce

'ö' | Set-Content -NoNewline -Encoding ASCII tmp.txt 
'ö' | Add-Content -Encoding ASCII -NoNewline tmp.txt 
Get-Content -Encoding ASCII tmp.txt
(Get-Content -Encoding Byte -TotalCount 2 tmp.txt) | % { '0x{0:x}' -f $_ }
'--'
'ö' | Set-Content -NoNewline tmp.txt   # use default encoding
'ö' | Add-Content -NoNewline tmp.txt   # use default encoding
Get-Content tmp.txt                    # use default encoding
(Get-Content -Encoding Byte -TotalCount 2 tmp.txt) | % { '0x{0:x}' -f $_ }

Expected behavior

??
0x3f
0x3f
--
??
0x3f
0x3f

Actual behavior

??
0x3f
0x3f
--
öö
0xf6
0xf6

That is, ASCII encoding turns a non-ASCII character into literal ? (0x3f)

The fact that Set-Content without an -Encoding argument resulted in ö on reading implies that ASCII encoding wasn't used, and the specific byte value of 0xf6 further implies that that a single-byte, extended-ASCII encoding was used:

For Windows PowerShell, it is the respective system's legacy codepage ("ANSI"), such as Windows-1252 on US-English systems, or Windows-1251 on Russian systems. In other words: the specific encoding is, to put it in Unix terms, locale-dependent.
For PowerShell Core, as of alpha 16, it is ISO-8859-1, as Ilya (@iSazonov) helpfully points out (see his comment below for the source-code links).
- Using a fixed encoding that is limited to 256 code points is problematic, however.
- See Ilya (@iSazonov)'s comment below and the discussion of the RFC about default file encodings.

In contrast, Get-Help Set-Content, Get-Help Add-Content, and Get-Help Get-Content state for parameter -Encoding:

Specifies the file encoding. The default is ASCII.

The help-topic sources (branch live) for the relevant cmdlets can be found here.

Additionally:

While these cmdlets accept an encoding identifier Default, as used in other cmdlets, the help only mentions String.
Given that the two appear to result in the same encoding - what is their relationship?
The description for encoding String in the online help is inadequate:

Uses the encoding type for a string.

Environment data

PowerShell Core v6.0.0-alpha (v6.0.0-alpha.16) on Microsoft Windows 10 Pro (64-bit; v10.0.14393)

The default is changed for PowerShell Core to ISO-8859-1
For OEM too ISO-8859-1

File provider
GetContentReader
GetContentWriter

This is real regression (breaking change) and we should fix it.

The Default is changed for PowerShell Core to ISO-8859-1
For OEM ISO-8859-1 too.

Lets see PowerShell FullCRL.
Default use function GetACP (Retrieves the current Windows ANSI code page identifier for the operating system.) See .Net Framework Reference

OEM is in only PowerShell and use function GetOEMCP (Returns the current original equipment manufacturer (OEM) code page identifier for the operating system.) See PowerShell code

I don't know why the both encodings was added in Powershell. Maybe PowerShell PG comment this.

Default is still not released in CoreCLR - so it is external issue. Maybe PowerShell PG find out this internally with .Net team. And should we use waiting-netstandart20 label?
OEM is not in CoreCLR at all - so it seems is internal issue. Is it make sense to fix it?

Michael Klement (@mklement0)

Michael Klement (@mklement0) See .Net Framework Reference
We can not use Windows-1252 as default.

Here Default is not PowerShell default, it is OS default. Every system can has own default. Original .Net code use GetACP() to get OS default code page. Modern Unix use en_US.UTF-8 as default.

James Truher (@JamesWTruher)

cc James Truher (@JamesWTruher) Bruce Payette (@BrucePay)

Michael Klement (@mklement0)

Michael Klement (@mklement0) Current Windows Powershell behavior is based on .Net Framework and it is dynamic:

System	FileSystemCmdletProviderEncoding.Default (GetACP())	FileSystemCmdletProviderEncoding.OEM (GetOEMCP())
Windows English	1252	437
Windows Russian	1251	866

Current PowerShell Core behavior is hard-coded to ISO-8859-1. If we change it on Windows-1252 we still not get Windows PowerShell behavior:

System	FileSystemCmdletProviderEncoding.Default (GetACP())	FileSystemCmdletProviderEncoding.OEM (GetOEMCP())
Windows English	1252	437
Windows Russian	1252	437

Preferred solution is to fix this in CoreCLR.

As you can see then we can properly read and write files on Windows Russian with 1251 (default system) code page by:
Get-Content -Encoding Default and Set-Content -Encoding Default

As far as Unix locale names mapping Unix uses standard names Table of locales (This is not a complete sample). Therefore, the fix in CoreCLR will not be too difficult.

We should wait for MSFT expert conclusion and then it will be clear that we need to fix in the code and documentation.

Michael Klement (@mklement0)

Michael Klement (@mklement0) We have a RFC draft for this. Welcome to discuss PowerShell/PowerShell-RFC#71

Ilya (@iSazonov)

Ilya (@iSazonov): Good idea. I've cleaned up my comments here, and I've revised the original post to point to the relevant other issues / comments / RFC.

Michael Klement (@mklement0)

Michael Klement (@mklement0)

•Are the help topics open-sourced too?

See full PowerShell repo list https://github.com/PowerShell/

Michael Klement (@mklement0)

Michael Klement (@mklement0) more specifically https://github.com/powershell/powershell-docs

Do we really need to support legacy Windows code pages in PowerShell Core (whether on Unix or Windows)?

There is a huge amount of software that only works with legacy code pages. Although the number has decreased since 2003 year.

Ilya (@iSazonov)

Ilya (@iSazonov): I hear you, but do users who rely on this kind of software need this legacy support in PowerShell Core? Aren't users who rely on legacy Windows applications likely to stay in the realm of Windows PowerShell?

I see the following fundamental options (from a purely conceptual standpoint - ease of implementation is not being considered here).

I think it's important to get clarity with respect to what should be implemented - as opposed to what can be.
Based on the appropriate way forward, we can assess what aspects should be implemented by the underlying .NET Core rather than in PowerShell Core.

(a) Core-NoWinLegacy:

Forgo all Windows legacy support in PowerShell Core - only implement it in Windows PowerShell.
(b) Core-WinLegacy-onWindows:

Provide Windows legacy support in PowerShell Core too, but only on Windows.
(c) Core-WinLegacy-Everywhere:

Provide Windows legacy support in PowerShell Core on all platforms, including Unix.

(c) may be what James Truher (@JamesWTruher) had in mind in his RFC, though the RFC mistakenly assumes only having to deal with 1, fixed single-byte legacy behavior: ASCII - whereas the truth is that a whole raft of culture-specific "ANSI" codepages must be emulated.

—

As for how these options relate to things needed from .NET Core:

(a) clearly needs no additional effort.

(b) also needs no additional effort - the aforementioned NuGet package can bring in all Windows legacy code pages, and [cultureinfo]::CurrentCulture.TextInfo.ANSICodePage and [cultureinfo]::CurrentCulture.TextInfo.OEMCodePage on Windows already tell us what code pages to use as the legacy defaults.

(c) Will be implemented in .NET CoreCLR v2: The current pre-v2 version already reports values for [cultureinfo]::CurrentCulture.TextInfo.ANSICodePage and [cultureinfo]::CurrentCulture.TextInfo.OEMCodePage on Unix too, performing the aforementioned mapping of Unix locale identifiers to legacy Windows code pages.
There are still edge cases - see below.
That said, if the consensus is to implement (c), these edge cases (see bottom) should be considered an acceptable price to pay for Windows legacy support on Unix.

Note that currently, Unix legacy support is not even part of the debate - while UTF-8-based locales are near-ubiquitous nowadays, Unix platforms still support legacy locales based on single-byte charmaps comparable to Windows legacy code pages.
To be legacy-friendly on Unix platforms, these charmaps would have to be respected too.

.NET Core, with the limited set of code pages it comes with, is not capable of this legacy Unix support, and, as of this writing, the preview version of v2 doesn't change that.

Edge cases when mapping Unix locales to Windows legacy code pages:

As mentioned before, some culture uses more than one script (alphabet); gotta love Wikipedia:

Serbian is practically the only European standard language with complete synchronic digraphia,[15] using both Cyrillic and Latin alphabets;

Thus, when mapping a Serbian Unix locale to a legacy Windows code page, one or the other alphabet must be chosen, because no single legacy code page can represent both alphabets.

This choice isn't unambiguous.

Here is the list of Serbian (sr_*) locales available on two sample Unix platforms:

macOS 10.12:

sr_YU
sr_YU.ISO8859-2
sr_YU.ISO8859-5
sr_YU.UTF-8

Ubuntu 16.04:

sr_ME UTF-8
sr_RS UTF-8
sr_RS@latin UTF-8

macOS is stuck in the past (YU representing the long-defunct former Yugoslavia); the .ISO8859-2 and .ISO8859-5 unambiguously imply the Latin and Cyrillic alphabet respectively, but what sr_YU.UTF-8 should map to - given that UTF-8 is capable of encoding both alphabets - is open to interpretation.
If there's always a preferred alphabet for such cultures, perhaps that's a non-issue - I don't know the answer to that.

Ubuntu faces the same issues, with only sr_RS@latin UTF-8 implying the alphabet via the embedded @latin script/variant identifier.

In practice, as of .NET Core v2.0.0-beta-001836-00, all of the above cases default to the Latin alpabet.

Joey Aiello (@joeyaiello)

Joey Aiello (@joeyaiello) Steve Lee (@SteveL-MSFT) Michael Klement (@mklement0) I opened issue-question in .Net Core repo and get great comments about defaults dotnet/standard#260 (comment)

Ilya (@iSazonov)

First just want to say, thanks for the extended explanations, everyone. This has been thoroughly useful. And thanks, Ilya (@iSazonov), for looping in the dotnet/standard folks.

Now, next steps:

I want to tease the encoding issue into as many smaller chunks as possible. This thread was originally about the OEM/Default codepage issue. If we can fix that in Windows with a platform-specific guard today, and fix it later on Linux with .NET Standard and System.Text.Encoding.CodePages, we should do that and close out this issue.
As I said in the other thread, I'm not concerned about the legacy Linux encoding scenario. It's uncommon, and we don't have baggage there.
The other questions should all be addressed in RFC0020 Default File Encoding PowerShell-RFC#71 and fixed in the RFC so that everyone can have a central view of how we're addressing encodings and so the @PowerShell/powershell-committee can sign off on that view.

Sound reasonable?

I pushed PR with fix for Default/OEM.

Joey Aiello (@joeyaiello)

Joey Aiello (@joeyaiello): Thanks, that indeed sounds reasonable (and I hope it's OK that, as an outsider, I'm being vocal here - I love PowerShell and I think getting the encoding right is crucial to PowerShell's cross-platform success).

Ilya (@iSazonov)

Ilya (@iSazonov) Sweet!

Michael Klement (@mklement0) I don't consider you an outsider, you're here constructively voicing opinions. Keep it coming 👍

And for what it's worth, I absolutely agree. We need to nail encodings.

Fix Default/OEM encoding behavior PowerShell Core (#3467)

#3467 merged and we would close the Issue or leave it open in waiting our Encoding RFC.

Per the plan I posted above, let's close this particular as resolved, and drive the RFC to completion via the normal process. If other issues arise after that, we should open new issues.

Fix Default/OEM encoding behavior PowerShell Core (PowerShell#3467)

Navigation Menu

Repository navigation

Set-Content/Add-Content/Get-Content use an 8-bit character encoding by default, but the help topics state ASCII; problematic Core default file encoding #3248

Steps to reproduce

Expected behavior

Actual behavior

Environment data

9 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Set-Content/Add-Content/Get-Content use an 8-bit character encoding by default, but the help topics state ASCII; problematic Core default file encoding #3248

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment data

Activity

iSazonov commented on Mar 3, 2017

iSazonov commented on Mar 4, 2017

iSazonov commented on Mar 4, 2017

iSazonov commented on Mar 4, 2017

iSazonov commented on Mar 5, 2017

SteveL-MSFT commented on Mar 5, 2017

iSazonov commented on Mar 5, 2017

iSazonov commented on Mar 6, 2017

mklement0 commented on Mar 6, 2017

iSazonov commented on Mar 6, 2017

SteveL-MSFT commented on Mar 6, 2017

9 remaining items

iSazonov commented on Mar 24, 2017

mklement0 commented on Mar 26, 2017

iSazonov commented on Mar 28, 2017

joeyaiello commented on Mar 30, 2017

iSazonov commented on Mar 31, 2017

mklement0 commented on Apr 1, 2017

joeyaiello commented on Apr 5, 2017

iSazonov commented on Apr 6, 2017

joeyaiello commented on Apr 14, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions