RFC0020 Default File Encoding

Committe has voted and accepted this RFC: https://github.com/PowerShell/PowerShell-RFC/blob/master/2-Draft-Accepted/RFC0020-DefaultFileEncoding.md

Feedback for https://github.com/PowerShell/PowerShell-RFC/blob/master/1-Draft/RFC0020-DefaultFileEncoding.md

James Truher (@JamesWTruher)

I am so in favor of this James Truher (@JamesWTruher)! Dealing with BOM just makes life sad in a cross-platform world.

Keith Hill (@rkeithhill)

If we use UTF8NoBOM for Linux then we should use UTF8 (=UTF8-With-BOM) for Windows.
So it seems the Legacy is unneeded.

And what will be the standard in future? If it is UTF8 as UTF8 w/o BOM we should interpret our UTF8 as UTF8 w/o BOM, add UTF8BOM and migrate Windows Powershell scripts to UTF8BOM. This can be a breaking change but justifiable change as bringing to the standard.

It would be nice to even consider a scenario with a target system when a Windows script works with files on another system - Linux - and vice versa.
Also interesting how it will affect writing portable scripts.

Keith Hill (@rkeithhill) Welcome to comment the RFC.

At first glance I do like the idea of PS v6 on Windows defaulting to UTF8 (or UTF8NoBOM) but I think you would still need a way to set the old Legacy behavior for scripts. In fact, it would be nice if PowerShell could interpret #requires -Version <anything less than 6> as a request to set this variable to Legacy for the script. That said, this is a big change ... need time to think through the ramifications.

Ilya (@iSazonov)

Ilya (@iSazonov) I've talked about this at length with the @PowerShell/powershell-committee and James Truher (@JamesWTruher): we don't want to break existing scripts on the Windows side, hence the need for the Legacy value. As you can see from the RFC, the smattering of different default encodings across different cmdlets is virtually random, so the likelihood of breaking people's existing scripts where they've worked around that behavior is high.

Add onto to that the problem that the market still hasn't decided on a default encoding--MS and Windows still heavily uses UTF-16 and UTF-8 with BOMs, and Linux is almost entirely UTF8NoBOM. We thought about pushing for the latter as a new default, but Windows apps (e.g. Notepad) will default to rendering ASCII in the absence of a BOM, and it turns out that there's a not a 1:1 fallback from UTF-8 to ASCII.

Keith Hill (@rkeithhill) that's absolutely something we need to start solving soon rather than later. There's a ton of platform differences for which we should have some configurable mechanisms--e.g. file encodings, path delimiters, case sensitivity, line endings, etc.--and it's clear to me that we need some way to flip all of them between the Windows and Linux defaults. I'd rather not hide that all behind the #requires -Version as it's not immediately clear to the user what's going on, but we've floated the idea of "modes", an environment variable, a cmdlet or two (coupled maybe with some namespace for the $PS* variables), etc. If you don't see an RFC from me on the matter in the next week or two, yell at me. :)

Joey Aiello (@joeyaiello)

Joey Aiello (@joeyaiello) Thanks for clarify! I agree with Legacy.

I have more questions.

Why do we say only about "file create" cmdlets? If I set $PSDefaultFileEncoding ="UTF32" I expect that is default for Get-Content too not only for Set-Content because the variable name is not "DefaultEncodingForFileCreation".
"UTF8" and "Unicode" can confuse Unix users. Perhaps it make sense add "UTF8BOM" and "UTF16" in FileSystemCmdletProviderEncoding. Moreover Windows write BOM for all Unicode files so we should have UTF8BOM/UTF8NoBOM, UTF16BOM/UTF16NoBOM, UTF32BOM/UTF32NoBOM, UTF7BOM/UTF7NoBOM.
For discussion. I thought about portable scripts: if I use Set-Content -Encoding "UTF8" on Windows then UTF8 is interpreted as UTF8BOM, if I use Set-Content -Encoding "UTF8" on Unix then UTF8 is interpreted as UTF8NoBOM. Such script will create a "correct" file on both platforms.

I think for this proposal to be implemented correctly, PowerShell should first offer a way to treat existing enums as subsets of one another e.g.

enum encoding {
    Unknown
    String
    Unicode
    BigEndianUnicode
    UTF8
    UTF7
    UTF32
    Ascii
    Default
    Oem
    BigEndianUTF32
}

enum fsencoding : encoding {
	Byte
}

should be as if you did:

enum fsencoding : [enum]::GetUnderlyingType([encoding]) {
// [encoding] members pasted in:
	Unknown
	String
	...
	Oem
	BigEndianUTF32
// and now [fsencoding] members:
	Byte
}

# And the following should also work:
[fsencoding]$foo = [encoding]::Ascii

The reason is that:

it only makes sense for a subset of cmdlets which take -Encoding to accept Byte
The Unknown option doesn't make sense to be assgined to $PSDefaultFileEncoding

Ilya (@iSazonov)

Ilya (@iSazonov):

I believe that for Get/Set-Content, we try to be intelligent by reading the BOM and using the existing encoding. If we were to allow $PSDefaultFileEncoding to override that behavior, you'd end up with a bunch of file encodings getting unintentionally changed (or worse, you'd end up with mixed encodings in the same file).
Unicode as UTF-16 is a holdover from a choice (in my opinion, a mistake) that .NET made at some point in the System.Text.Encoding class. We certainly shouldn't remove Unicode, but I think it's worth considering adding UTF-16 for discoverability purposes (even though it would do the same thing as Unicode. As for other encodings without a BOM, they don't really exist out in the wild. With UTF-16, you need to have a BOM to know whether your characters are big or little endian. (Side note, I highly recommend this article on Unicode, and I still refer to this chart on Wikipedia all the time.)
I understand where you're going, but I think it's more important that explicitly specifying a parameter is deterministic no matter which platform you're on. For those who don't specify a parameter, we should try and "do the right thing" based on platform (which I believe this RFC will address), but if you specify a parameter, that behavior should be well-known and not try to do any black magic behind the scenes. (Furthermore, UTF-8, by definition, has a BOM even on Linux. It's just that most utilities on Linux write UTF-8 without a BOM as a matter of practice, so the enum values here are still accurate.)

Derp McDerp (@DerpMcDerp): that would be useful to have, but given that this is implemented in C#, is that actually necessary to implement for this RFC (I'm not a C# wiz, but my gut tells me no).

Joey Aiello (@joeyaiello)

Joey Aiello (@joeyaiello)

we try to be intelligent by reading the BOM and using the existing encoding.

Yes. I believe we use $PSDefaultFileEncoding to set defaults but not override the current behavior ( remove " to be intelligent by reading the BOM"). If so we should set defaults in File provider GetContentReader too.

As for other encodings without a BOM, they don't really exist out in the wild

I used the article:

A full featured character encoding converter will have to provide the following 13 encoding variants of Unicode and UCS:
UCS-2, UCS-2BE, UCS-2LE, UCS-4, UCS-4LE, UCS-4BE, UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, UTF-32LE

From Unicode FAQ:

Q: Why do some of the UTFs have a BE or LE in their label, such as UTF-16LE?
A: UTF-16 and UTF-32 use code units that are two and four bytes long respectively. For these UTFs, there are three sub-flavors: BE, LE and unmarked. The BE form uses big-endian byte serialization (most significant byte first), the LE form uses little-endian byte serialization (least significant byte first) and the unmarked form uses big-endian byte serialization by default, but may include a byte order mark at the beginning to indicate the actual byte serialization used. [AF]

From the same FAQ:

Q: Which of the UTFs do I need to support?
A: UTF-8 is most common on the web. UTF-16 is used by Java and Windows. UTF-8 and UTF-32 are used by Linux and various Unix systems. The conversions between all of them are algorithmically based, fast and lossless. This makes it easy to support data input or output in multiple formats, while using a particular UTF for internal storage or processing. [AF]

As non-English user I often do re-encoding in my scripts. Sometimes it's a real headache. Definitely we should get rid of diversity using encodings. So I would prefer that PoweShell Core used Unicode everywhere.

Many programs are sensitive to system/session locale. I believe we must remove the regression with Default and OEM. All the more so because Linux is also based on locale.

Based on this it would be great if PowerShell Core use UTF16 (WithBOM) as default on Windows with Legacy option for backward compatibility with Windows PowerShell and UTF8 (NoBOM) as default on Unix.

I just skimmed at how PowerShell currently handles encoding and it's an inconsistent clutter. Here is an approximation of how PowerShell currently works if PowerShell used explicit enums instead of [ValidateSet()] everywhere.

// we need the following existing enum
public enum FileSystemCmdletProviderEncoding {
	Unknown,
	String,
	Unicode,
	Byte,
	BigEndianUnicode,
	UTF8,
	UTF7,
	UTF32,
	Ascii,
	Default,
	Oem,
	BigEndianUTF32
}

// and the following 2 synthesized enums:
enum FakeEncodingEnum1 {
	Unicode,
	BigEndianUnicode,
	UTF8,
	UTF7,
	UTF32,
	Ascii,
	Default,
	Oem
}

enum FakeEncodingEnum2 {
	Unknown,
	String,
	Unicode,
	BigEndianUnicode,
	UTF8,
	UTF7,
	UTF32,
	Ascii,
	Default,
	Oem
}

And then here is how the existing cmdlets operate:

Cmdlet	-Encoding	Notes
Add-Content	FileSystemCmdletProviderEncoding	`BigEndianUnicode` doesn't use a BOM. Invalid encodings default to `Unicode`
Export-Clixml	FakeEncodingEnum1	`-Encoding` defaults to `Unicode`. `BigEndianUnicode` uses a BOM. Invalid encodings throw an exception
Export-Csv	FakeEncodingEnum1	With `-Append: $false`, `-Encoding` defaults to `ASCII`. When `-Append: $true`, `-Encoding` defaults to encoding detection backing off to `UTF8NoBOM` if no preamble detected. `BigEndianUnicode` uses a BOM
Export-PSSession	FakeEncodingEnum1	`-Encoding` defaults to `UTF8`. `BigEndianUnicode` uses a BOM
Get-Content	FileSystemCmdletProviderEncoding	`BigEndianUnicode` doesn't use a BOM. Invalid encodings default to `Unicode`
Import-Csv	FakeEncodingEnum1	`-Encoding` defaults to encoding detection backing off to `UTF8NoBOM` if no preamble detected. `BigEndianUnicode` uses a BOM
Out-File	FakeEncodingEnum2	`-Encoding` defaults to `Unicode`. `BigEndianUnicode` uses a BOM
Select-String	FakeEncodingEnum1	`-Encoding` defaults to `UTF8`. `BigEndianUnicode` uses a BOM
Send-MailMessage	FakeEncodingEnum2	`-Encoding` defaults to `ASCII`. `BigEndianUnicode` uses a BOM
Set-Content	FileSystemCmdletProviderEncoding	`BigEndianUnicode` doesn't use a BOM. Invalid encodings default to `Unicode`

The cmdlets that take FileSystemCmdletProviderEncoding use Default if String is specified. Other cmdlets use Unicode if String is specified somehow.
BigEndianUnicode is cmdlet dependant on whether or not it write a BOM.
String and Unknown really should be gotten rid of. If they can't then just make them aliases for Unicode to simplify.
Also, it looks like PathUtils.MasterStreamOpen could be simplified to look like:

enum FakeEncodingEnum3 {
	Unknown,
	String,
	Unicode,
	BigEndianUnicode,
	UTF8,
	UTF7,
	UTF32,
	Ascii,
	Default,
	Oem,
	UTF8NoBom
}

MasterStreamOpen(
	PSCmdlet cmdlet,
	string filePath,
//	string encoding,
//	bool defaultEncoding,
	FakeEncodingEnum3 encoding, /* new */
	bool Append,
	bool Force,
	bool NoClobber,
	out FileStream fileStream,
	out StreamWriter streamWriter,
	out FileInfo readOnlyFileInfo,
	bool isLiteralPath)

We should pay attention to what Default and OEM do not work exactly as in Windows PowerShell. See PowerShell/PowerShell#3248

It should be noted that CoreFX use UTF8NoBOM as default for Streams.

Derp McDerp (@DerpMcDerp)

Derp McDerp (@DerpMcDerp): Great sleuthing. Two points (not sure if I'm misinterpreting your table):

Set-Content and Add-Content (if the file doesn't already exist) do create a BOM with -Encoding BigEndianUnicode - using any of the standard Unicode encoding schemes explicitly (with the exception of the nonstandard UTF-7 encoding) results in a BOM.
- (Set-Content -NoNewline -Value '' -Encoding BigEndianUnicode t.txt); (Get-Item t.txt).Length yields 2, the length of the BOM in bytes.
From what I can tell, all standard cmdlets that read text - including Get-Content - only recognize BigEndianUnicode encoding with a BOM.
- (In line with your table) BOM-less files are blindly interpreted as either "ANSI"-encoded (Get-Content, Import-PowerShellDataFile) or UTF-8-encoded (Import-Csv, Import-CliXml, Select-String).

There are already significant breaking changes between PS v5 and PS Core v6 on Windows. Why not take this chance bring some consistency to the default file encoding on Windows? And since you are giving us the ability to set $PSDefaultFileEncoding (you are, aren't you?), I will change it to UTF8. That means other folks' scripts could break on my system. So I need some sort of mechanism in which to run these script in Legacy mode which is easy:

& { $PSDefaultFileEncoding='Legacy'; ./old-script.ps1 }

Or maybe I pester the script author to set the $PSDefaultFileEncoding to Legacy for their script. So this can be fixed within and without.

Perhaps this could be reason to have a .ps2 file? PS1 files would run with Legacy file encoding on Windows and UT8NoBOM on Linux. PS2 files would run with UTF8 on Windows and UTF8NoBOM on Linux. Are there other PS Core changes that could be handled by .PS2? Maybe PSSA could recognize PS2 files as explicitly portable and warn about use of COM, WMI and other non-portable features?

It's a "ps2" beautiful idea. But it will make our code too complex to support (practically each subsystem will ensure compatibility mode: ex., for parameter bindings it will be a big headache).

I think it unnecessarily support "ps2" because we can directly use Windows PowerShell for old scripts and PowerShell Core for new scripts and we already have #Requires -Version. I believe it is much easier than support "ps2". Therefore, I also agree with:

There are already significant breaking changes between PS v5 and PS Core v6 on Windows. Why not take this chance bring some consistency to the default file encoding on Windows?

Keith Hill (@rkeithhill)

From a Linux-aware developer standpoint, I have long been frustrated by PowerShell's handling of encoding (for > in particular), so I am thrilled to find this discussion. To clarify PowerShell/PowerShell#707 (comment) now that I have been directed to the appropriate place for encoding commentary...

$PSDefaultFileEncoding

👍 If the only change to come from this RFC is the ability to change > encoding, I would still be thrilled.

We should take this opportunity to rationalize our use of the Encoding parameter...

👍 Even better.

Naturally, specific use of the -encoding parameter when invoking the cmdlet shall override $PSDefaultFileEncoding.

👍 I love the thought of my functions with -Encoding just setting $PSDefaultFileEncoding = $Encoding and getting consistency for free.

When $PSDefaultFileEncoding is set to Legacy...the irregular file encoding on non-Windows platforms [is persisted]..."

👍 Important to keep this as an option for legacy scripts.

The default on Windows systems shall remain unchanged (the value for $PSDefaultFileEncoding shall be set to Legacy)...

👎 I'm with Keith Hill (@rkeithhill): it's already unreasonable to expect v5 scripts to work unaltered with Core v6. Make a big deal about encoding on the migration checklist, and use this as an opportunity to educate people about the perils of not using Unicode.

And since you are giving us the ability to set $PSDefaultFileEncoding (you are, aren't you?), I will change it to UTF8. That means other folks' scripts could break on my system.

👍 More generally, this is a chance to educate people about the perils of not being explicit about file encoding. Once this is supported, scripts that expect files to be written with a specific encoding will need to be explicit regardless of system defaults.

Since scripts targeting v5 are expected to require alteration, and since scripts targeting Core are reasonably expected to set $PSDefaultFileEncoding if they need to be deterministic, the choice of default encoding seems relatively unconstrained. Especially with WSL, I vote for aligning with Linux. And really, everything gets easier if you can assume any BOM-less byte stream of text is UTF-8. That's not a fair assumption today on Windows, but we'll never get to the point where you can make any reasonable assumption about text encoding without taking a stand at some point. .NET Core and PS Core seem like great places to start.

Elle (@BatmanAMA)

Elle (@BatmanAMA) They can start Powershell in some "Version 5" mode...

Joey Aiello (@joeyaiello)

Thanks, Joey Aiello (@joeyaiello); that is great news indeed.

A few more thoughts:

If PS will become BOM-less UTF8 by default, the only "meta" $PSDefaultFileEncoding value needed will be Legacy (or perhaps WindowsLegacy, for clarity), which, I presume, will be the preset in the Desktop edition, and unset in Core.
In addition to defaulting all file-producing cmdlets (including those that don't support a specifiable encoding, such as New-Item and New-ModuleManifest) to BOM-less UTF-8, it is vital that:
- anywhere a file without a BOM is read, it be interpreted as UTF-8.
- the engine itself interpret source code, manifests, data files and the like as BOM-less UTF-8 by default (in the absence of a BOM).
- $OutputEncoding default to BOM-less UTF-8
- console windows be effectively initialized as follows: $OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = [System.Text.UTF8Encoding]::new(), where on Windows setting [console]::InputEncoding to UTF-8 makes chcp report 65001, the UTF-8 code page.
- (Ironically, New-Item -Type File, unbeknownst to many, already - and invariably - creates BOM-less UTF-8 files.)
The (only) encoding forms mandated by the Unicode standard are UTF-8, UTF-16, and UTF-32 and their byte-order variants; currently, support for UTF-32BE is missing.
As has been suggested, being able to pass instances of System.Text.Encoding directly to -Encoding would be a nice addition.
Terminology (encoding names):
- Unambiguous encoding names UTF8NoBOM and UTF8BOM should be introduced.
  - Arguably, UTF8 should be repurposed to BOM-less by default, and would only refer to a with-BOM encoding if $PSDefaultFileEncoding is set to Legacy.
- Encoding name Unicode should be deprecated in favor of UTF16LE.
- BigEndianUnicode should be deprecated in favor of UTF16BE
- Default should be deprecated in favor of ANSI.
  - Perhaps Default can then reflect the effective default, i.e., BOM-less UTF-8 by default, and when $PSDefaultFileEncoding is set to Legacy, ANSI (system locale code page).
- UTF32 should be deprecated in favor of UTF32LE, and UTF32BE introduced (along with underlying support for it).
- As an aside: The term "BOM" is firmly entrenched by now (and wonderfully short), but the proper name for a sequence of bytes at the start of a file identifying an encoding scheme is Unicode signature.
It's understandable not to want to bother with Unix legacy encodings, but I wonder if a courtesy warning could be issued when PowerShell starts up with an LC_CTYPE (as reported by locale) value other than UTF-8.
Note that the standard Unix utilities still do support legacy encodings.
The Unix model is alluringly simple: all utilities (except special purpose re-encoding ones) blindly assume that all byte streams that represent text use the encoding defined in the locale (LC_CTYPE) - they neither understand nor produce any other encoding (and have no concept of BOMs).
- PowerShell's seamless ability to recognize BOMs - i.e., the ability to read encodings other than the default one - distinguishes it from other shells, but it does introduce the problem of preserving the specific input encoding on output; somehow making information about the input encoding available to user code would be helpful, as Keith Hill (@rkeithhill) has suggested before.
- As an aside: using cat to illustrate BOM problems on Unix (in the current version of the RFC) is perhaps not the best choice: cat is not text-aware and blindly copies bytes from stdin to stdout, which includes not just the BOM, but also the NULs that result from the UTF-16LE-encoded ASCII-range characters; the terminal just happens not to render the NULs; you can verify their existence as follows: PS> 'hi'>t.txt; cat -v t.txt.

Elle (@BatmanAMA)

Elle (@BatmanAMA): The RFC being discussed here proposes a new preference variable, $PSDefaultEncoding that, when set to Legacy, will continue to exhibit v5 behavior.

Technically, with .NET Core 2.0, I think this legacy behavior could now even be made available on Unix platforms, though I don't know why someone would need it there (the ability to selectively use the same encodings as available on Windows is helpful, however).

Asad Saeeduddin (@masaeedu): Binary (raw-byte) pipelines and output redirection are being discussed here.

Joey Aiello (@joeyaiello)

Joey Aiello (@joeyaiello) sounds great! Do you plan to address PowerShell/PowerShell#3466 for Authenticode signature verification?

Rob Reynolds (@ferventcoder)

Rob Reynolds (@ferventcoder):

If I understand the plan correctly, v6 will read everything that doesn't have a BOM as UTF-8 by default (unless $PSDefaultEncoding is set to Legacy), which will make the linked problem go away.

However, do note that using BOM-less UTF-8 script files is currently not just broken with respect to signature verification, but more fundamentally: your UTF-8-encoded non-ASCII characters in that script are misinterpreted as "ANSI"-encoded characters (the culture-specific, extended-ASCII, single-byte legacy encoding; typically, Windows-1252) - which may or may not surface as a problem, depending on whether these chars. are part of data in the script.

In short: with the current encoding behavior, UTF-8 scripts / files only work reliably as expected with a BOM.

This brings up an interesting migration pain point:

Say you switch to v6 and do not opt for the legacy encoding behavior: any "ANSI"-encoded legacy scripts would then have to be converted to UTF-8 in order to be read correctly (and given that there's no BOM for "ANSI" encoding, there's no other choice).
That said, given that UTF-8 is a superset of ASCII, only scripts that contain non-ASCII chars. are affected.

I was more on the lean that you could have a UTF8 file that works with authenticode signature verification in PowerShell v6, but not in any earlier versions if it has unicode characters and no BOM. I'm fine with this, I just wanted to be sure this will be added to known incompatibilities in the documentation when the change is made.

Michael Klement (@mklement0)

From Michael Klement (@mklement0):

If I understand the plan correctly, v6 will read everything that doesn't have a BOM as UTF-8 by default (unless $PSDefaultEncoding is set to Legacy ), which will make the linked problem go away.

The RFC says only about file creation cmdlets.

We're almost done Add dynamically generated set in ValidateSetAttribute for binary cmdlets. This opens up enormous opportunities in different areas. My main motivation was encoding parameters. By means of valid values generators, we can implement encoding parameters with any flexibility.
This will allow to support all encodings from System.Text.Encoding.
This will allow to easily switch to WindowsLegacy. We shouldn't enhance FileSystemCmdletProviderEncoding but use it as legacy. Instead, we should introduce the modern FileSystemCmdletProviderEncoding - PS6FileSystemCmdletProviderEncoding with members:

ISO-8559-1
UTF32BE
UTF32LE
UTF16BE
UTF16LE
UTF8

Also Byte member in the enum looks as legacy too. It is not encoding - it is a stream type. We should add it as separate parameter. We have some requests to support byte streams (ex., for native commands). This approach paves the way for this.

Set-Content -Encoding UTF8 -NoBOM -Byte

This is more intuitive for users, consistent with .Net Standard, and it is easier for us to manage the default settings.

The RFC says only about file creation cmdlets.

Yes, but it was clearly just a first draft, and the discussion here has shown us that the defaults that matter in the Unix world - and therefore are crucial to cross-platform success - do not just cover file creation. To recap, the defaults that matter are:

what encoding to use when writing without a BOM.
what encoding to use when reading without a BOM.

I think not ubiquitously reading and writing (BOM-less) UTF-8 (by default, unless overridden by configuration, or in situations where an established standard for a specific file format / protocol prescribes a different encoding) would prove very problematic in the long run.

Being able to dynamically generate the set of valid parameter values is a great addition.
How, specifically, do you envision their use with -Encoding? Have the single, existing parameter accept the previous enumeration values and instances of [System.Text.Encoding] / their names?

Ascii encoding shouldn't be replaced by ISO-8559-1, as there are cases when you truly only want 7-bit characters.

If we include ISO-8559-1 as an explicit enumeration value, we need to make it very clear that it is NOT the same encoding as Windows-1252 - even though the two are frequently conflated (to a point where HTML5 basically prescribes: treat ISO-8559-1 as if Windows-1252 had been specified).
ISO-8559-1 is a subset of Windows-1252 and both .NET and Unix locales define them properly, as distinct encodings. A notable absence is €, so that if you read a Windows-1252-encoded file that has € chars. in it as ISO-8559-1, they are transliterated to literal ? chars. (default behavior in .NET) or de/encoding breaks (iconv Unix utility); try
[System.Text.Encoding]::GetEncoding(28591).GetBytes('€') | Format-Hex

Enumeration name pairs such as BigEndianUTF16 / UTF16 to represent endianness are still Windows-centric and imprecise. In Unicode speak, UTF16 (UTF-16) is an abstract encoding form that is endianness-agnostic. Adding an endianness to an encoding form makes it a concrete encoding scheme, requiring an endianness suffix: LE or BE. In other words: To be standards-conformant (leaving punctuation aside), UTF16 should be UTF16LE, and BigEndianUnicode should be UTF16BE; while we need to preserve the old names for backward compatibility, this is an opportunity to introduce the proper names.

True, Byte is not an encoding - it is the opposite of one, if you will. That said, given that -Encoding, loosely speaking, tells the cmdlet how to interpret the data, including Byte as the do-not-interpret value makes sense to me, and then we needn't deal with a separate parameter that is mutually exclusive with -Encoding.

Joey Aiello (@joeyaiello)

Joey Aiello (@joeyaiello) Was there a special opinion on Out-File cmdlet? Why is it placed in an optional section while it is a file creation cmdlet?

The RFC is only aimed at file creation cmdlet. I guess we'll have other RFCs for file and web cmdlets.

Michael Klement (@mklement0)

To be standards-conformant (leaving punctuation aside), UTF16 should be UTF16LE , and BigEndianUnicode should be UTF16BE

Fixed.

Ascii encoding shouldn't be replaced by ISO-8559-1 , as there are cases when you truly only want 7-bit characters.

I'm sorry I didn't explain that ASCII should stay in Legacy sets, in modern set it is more good use ISO-8559-1. If anybody want 7-bit characters he should use System.Text.Encoding.
And we could continue to consider ISO-8559-1 as Windows-1252. Or introduce a new name.

How, specifically, do you envision their use with -Encoding ? Have the single, existing parameter accept the previous enumeration values and instances of [System.Text.Encoding] / their names?

Encoding should be [String]. A valid value generator is a method which can accept all defaults (machine/user/powershell/platform/session/script/module/cmdlet) as parameters and generate the correct list of required encodings.
We can even add an option to accept (-ExtendedEncoding AcceptOnly|Suggest/Enable|Disable) extended encoding values (from System.Text.Encoding), but not to suggest them in IntelliSense.

Byte as the do-not-interpret value makes sense to me

My thoughts were about new requests that we received: binary streams, preserving encoding/re-encoding in pipes, redirections to/from native commands.

Michael Klement (@mklement0)

Michael Klement (@mklement0)

Enumeration name pairs such as BigEndianUTF16 / UTF16 to represent endianness are still Windows-centric and imprecise. In Unicode speak, UTF16 (UTF-16) is an abstract encoding form that is endianness-agnostic. Adding an endianness to an encoding form makes it a concrete encoding scheme, requiring an endianness suffix: LE or BE. In other words: To be standards-conformant (leaving punctuation aside), UTF16 should be UTF16LE, and BigEndianUnicode should be UTF16BE; while we need to preserve the old names for backward compatibility, this is an opportunity to introduce the proper names.

This is wrong. An encoding scheme does not require an endianness suffix in its name. The UTF-16 encoding scheme is a perfectly valid encoding scheme that can be either big-endian or little-endian, and may or may not have a BOM. The presence of a BOM is agnostic of the endianness. If a BOM is not present, the encoding scheme is big-endian unless some higher-level protocol says otherwise.

The UTF-16BE/LE encoding schemes do not permit a BOM. If U+FEFF is present at the beginning of the stream, it is interpreted as ZERO WIDTH NO-BREAK SPACE.

The same goes for the UTF-32 encoding schemes.

This is all according to section 3.10 in Unicode 10.

An encoding scheme does not require an endianness suffix in its name.

Yes, it does. Without an endianness suffix, it is an abstract encoding form that does not prescribe byte order.

The UTF-16 encoding scheme is a perfectly valid encoding scheme that can be either big-endian or little-endian,

No: A scheme implies a specific byte order.

The UTF-16BE/LE encoding schemes do not permit a BOM

Indeed: a scheme - by virtue of implying the byte order - does not require or support a BOM.

That said, if you want to create a file, you do need a BOM, whose byte sequence the -Encoding scheme name determines; that BOM (Unicode signature), in the absence of encoding metadata, will tell readers of that file what encoding scheme was used to write the file.

For the distinction between encoding forms and schemes, see the relevant part of the Glossary of Unicode Terms.

The section 3.10 in chapter 3 of the v10 standard you mention deals with encoding schemes exclusively.

Michael Klement (@mklement0)

Michael Klement (@mklement0)

No: A scheme implies a specific byte order.

I'm sorry that I have to disagree, but this quote from your link chapter 3 of the v10 standard says otherwise:

D98: UTF-16 encoding scheme: The Unicode encoding scheme that serializes a UTF-16
code unit sequence as a byte sequence in either big-endian or little-endian format.
• In the UTF-16 encoding scheme, the UTF-16 code unit sequence <004D 0430
4E8C D800 DF02> is serialized as <FE FF 00 4D 04 30 4E 8C D8 00 DF 02> or
<FF FE 4D 00 30 04 8C 4E 00 D8 02 DF> or <00 4D 04 30 4E 8C D8 00 DF 02>.
• In the UTF-16 encoding scheme, an initial byte sequence corresponding to
U+FEFF is interpreted as a byte order mark; it is used to distinguish between
the two byte orders. An initial byte sequence <FE FF> indicates big-endian
order, and an initial byte sequence <FF FE> indicates little-endian order. The
BOM is not considered part of the content of the text.
• The UTF-16 encoding scheme may or may not begin with a BOM. However,
when there is no BOM, and in the absence of a higher-level protocol, the byte
order of the UTF-16 encoding scheme is big-endian.

PDF-Page: 61/89; Page in Headline: 132

There's clearly an inconsistency in the standard with respect to terminology, as this passage from the start of section 3.9 illustrates (emphasis added):

The Unicode Standard supports three character encoding forms: UTF-32, UTF-16, and
UTF-8.

Clearly, calling something (UTF-16) both a form and a scheme, when these two words are clearly mean different things (as my earlier quote from the glossary shows) is problematic.

Pragmatically speaking, there are only UTF-16 encoding schemes (plural!) - big-endian and little-endian - that the absence of a BOM falls back to big-endian only goes to show that one or the other must be chosen.

So, yes, you could say that UTF-16 without a BOM should effectively be UTF-16 BE (whether that is heeded in practice is a different matter - see bottom).

However, relying on this in encoding names seems ill-advised, when UTF16LE and UTF16BE communicate the intent unambiguously.

The fact that PowerShell's Unicode encoding name creates UTF-16 LE files (and has a contrasting BigEndianUtf16 name) shows that its view of UTF-16 is little-endian-centric.

Similarly, when you (artificially) create a BOM-less file that is LE-encoded, Notepad still recognizes it as such, as opposed to blindly assuming BE, as the standard prescribes.

Navigation Menu

Repository navigation

RFC0020 Default File Encoding #71

Committe has voted and accepted this RFC: https://github.com/PowerShell/PowerShell-RFC/blob/master/2-Draft-Accepted/RFC0020-DefaultFileEncoding.md

26 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RFC0020 Default File Encoding #71

Description

Committe has voted and accepted this RFC: https://github.com/PowerShell/PowerShell-RFC/blob/master/2-Draft-Accepted/RFC0020-DefaultFileEncoding.md

Activity

usepowershell commented on Feb 24, 2017

iSazonov commented on Feb 27, 2017

rkeithhill commented on Feb 27, 2017

joeyaiello commented on Mar 2, 2017

iSazonov commented on Mar 3, 2017

DerpMcDerp commented on Mar 3, 2017

joeyaiello commented on Mar 3, 2017

iSazonov commented on Mar 4, 2017

DerpMcDerp commented on Mar 4, 2017

iSazonov commented on Mar 13, 2017

iSazonov commented on Mar 15, 2017

mklement0 commented on Mar 16, 2017

rkeithhill commented on Mar 21, 2017

iSazonov commented on Mar 21, 2017

dahlbyk commented on Mar 21, 2017

26 remaining items

be5invis commented on Jun 7, 2017

mklement0 commented on Jun 7, 2017

mklement0 commented on Jun 7, 2017

ferventcoder commented on Jun 7, 2017

mklement0 commented on Jun 7, 2017

ferventcoder commented on Jun 7, 2017

iSazonov commented on Jun 18, 2017

mklement0 commented on Jun 18, 2017

iSazonov commented on Jun 19, 2017

amckinlay commented on Nov 23, 2017

mklement0 commented on Nov 23, 2017

TSlivede commented on Nov 23, 2017

mklement0 commented on Nov 23, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions