Make console windows fully UTF-8 by default on Windows, in line with the behavior on Unix-like platforms - character encoding, code page

PowerShell Core now commendably defaults to UTF-8 encoding, including when sending strings to external programs, as reflected in $OutputEncoding's default value.

However, because the console-window shortcut file / taskbar entry still defaults to the OEM code page implied by the legacy system locale (e.g. 437 on US-English systems), it misinterprets strings from external programs; e.g., with Node.js installed:

PSCoreOnWin> $captured = '€' | node -pe "require('fs').readFileSync(0).toString().trim()"; $captured
Γé¼    # !! node's UTF-8 output was misinterpreted.

This currently requires the following workaround (in addition to requiring the console window to use a TrueType font (true by default on Windows 10)):

[console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding

^{Prepend $OutputEncoding = to make a Windows PowerShell console fully UTF-8-aware.}

The above implicitly switches to the UTF-8 code page (65001), as then reflected in chcp.

This obscure workaround shouldn't be necessary, and I think it would make sense for PowerShell to automatically set [console]::InputEncoding and [console]::OutputEncoding to (BOM-less) UTF-8 on startup.

Update: When this issue was originally created, there was no mechanism for presetting code page 65001 (UTF-8) system-wide, which necessitated the awkward workaround. In recent versions of Windows 10 it is now possible to switch to code page 65001 as the system locale and therefore system-wide, although as of Windows 10 version 1909 that feature is still in beta - see this SO answer.

Caveat: In addition to defaulting the OEM code page to 65001 in all console windows (including cmd.exe windows), this invariably also makes Windows PowerShell's ANSI-encoding-default cmdlets default to UTF-8, notably Get-Content and Set-Content, which can be problematic from a backward-compatibility perspective.
Additionally, there is a bug - see below.

The change, which can also be made programmatically (see below), requires administrative privileges and a reboot.

Environment data

PowerShell Core 7.1.0-preview.3 on Windows 10

Michael Klement (@mklement0)

It is a platform default:
https://source.dot.net/#System.Console/System/Console.cs,a570cd79bd33ceab
https://source.dot.net/#System.Console/System/ConsolePal.Windows.cs,c997db0e94f0d1cc
https://source.dot.net/#System.Console/Common/Interop/Windows/Interop.GetConsoleOutputCP.cs,f028312cfc964730

So we need do [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding at PowerShell Core startup. Michael Klement (@mklement0) Right fix for all platforms and Windows versions (Windows 7?) ?

Ilya (@iSazonov)

Thanks for the sleuthing, Ilya (@iSazonov).

Yes, I think the fix is also appropriate for Windows 7:

While you're more likely to run into problems with standard console programs there that can even break with UTF-8 input, I think it's more important for PowerShell Core to exhibit consistent encoding behavior and to support modern, cross-platform utilities that natively speak UTF-8 by default.

Ilya (@iSazonov)

Ilya (@iSazonov): Forgot to clarify: It is only the right fix for Windows - on Unix-like platforms the CoreFx default should be used, as discussed in #7634 (even though there's a CoreFx fix pending).

James Truher (@JamesWTruher)

I hope James Truher (@JamesWTruher) could comment. I think he considered this in time writing and implementing Encoding RFC.

Steve Lee (@SteveL-MSFT)

Since Windows 7 EOL and community are migrating to Windows 10 it seems a time to switch a console default to UTF8 on WIndows.

/cc Steve Lee (@SteveL-MSFT)

Steven Penny (@nu8)

Steven Penny (@nu8), are you using Windows PowerShell? In PowerShell Core, the default encoding for Get-Content on files has been UTF8NoBOM since #5080.

Steven Penny (@nu8)

Let me try to summarize, now that we (hopefully) have the full picture:

^{I've hidden my previous comments in favor of this one, Steven Penny (@nu8) - I encourage you to do the same, as appropriate. This comment also corrects my incorrect earlier claim that you cannot set the ANSI code page to 65001.}

This issue is about making UTF-8 support in PowerShell on Windows complete, by making sure that PowerShell also uses UTF-8 when communicating with external programs (the built-in cmdlets already default to UTF-8, invariably so), which requires setting [console]::InputEncoding and [console]::OutputEncoding to (BOM-less) UTF-8 (possibly indirectly).

Currently, in the absence of PowerShell doing that itself, there are two workarounds:

Option 1: Put the following statement in your `$PROFILE`:

# In *Windows PowerShell*, prepend `$OutputEncoding = `
[console]::InputEncoding = [console]::OutputEncoding = [System.Text.UTF8Encoding]::new()

Pros and cons:

Doesn't require administrative privileges and takes effect in new windows without the need for a reboot.
Requires modifying $PROFILE
Is bypassed if the CLI is used as pwsh -noprofile ...

Note: In Windows PowerShell, you must prepend $OutputEncoding = to the above command, in order to also make Windows PowerShell send UTF-8 to external programs. (In PowerShell [Core], this preference variably commendably defaults to (BOM-less) UTF-8.)

Option 2: Change the active code pages to `65001` system-wide (W10+):

GUI method: via intl.cpl (Control Panel), tab Administrative, Change system locale...); as previously noted, this is still labeled as Beta: as of Windows 10 release 1909, though I suspect it will work fine as long as you use only modern command-line utilities.
Equivalent programmatic method, based on @nu8's approach:

# Requires ELEVATION and a REBOOT
'ACP', 'OEMCP', 'MACCP' | Set-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Control\Nls\CodePage -Name { $_ } 65001
# Restart-Computer

Pros and cons:

Due to a .NET bug still present in the .NET version underlying PowerShell Core 7.1.0-preview.3, [console]::InputEncoding and [console]::OutputEncoding are mistakenly set to UTF-8 encoding with BOM, which causes follow-on bugs; notably, it breaks Start-Job in PowerShell. See System.Console unexpectedly uses a UTF-8 encoding *with BOM* on Windows dotnet/runtime#28929. Option 1 above doesn't have this problem.
- Curiously, by contrast, the [System.Text.Encoding]::Default encoding that reflects the active ANSI code page contains a BOM-less UTF-8 encoding after the system-wide change (see below).
- Note that the bug can also manifest without the system-wide change, namely if you manually run chcp 65001 from cmd.exe, for instance, before invoking PowerShell (running chcp from inside PowerShell isn't supported and requires Option 1 instead).
Requires administrative privileges and a reboot.
Takes effect system-wide: it applies to all console / Windows Terminal windows, notably including those running cmd.exe
Invariably also uses UTF-8 as the ANSI code page (not just the OEM code page), as reflected in [System.Text.Encoding]::Default (note that this also applies if you set only the OEMCP registry value to 65001; (Get-Culture).TextInfo.ANSICodePage, by contrast, continues to report the locale-appropriate code page, e.g. 1252).
- If you're (also) running Windows PowerShell, this means that the setting invariably makes Windows PowerShell's ANSI-encoding-default cmdlets default to UTF-8, notably Get-Content and Set-Content, which, depending on your backward-compatibility needs:
  - may be desirable for consistent UTF-8 use across both PowerShell editions.
    - Note: short of placing $PSDefaultParameterValues['*:Encoding'] = 'utf8' in your $PROFILE, this is the only way to get Windows PowerShell to consistently default to UTF-8.
      Curiously, the system-wide change causes Windows PowerShell to then create BOM-less UTF-8 files by default with Set-Content, something that cannot otherwise achieved, except with direct use of .NET.
  - may be undesired, if you have existing code that uses Get-Content and Set-Content without -Encoding and you need to process BOM-less files that are ANSI- rather than UTF-8-encoded.
- Also, in Windows PowerShell only, you must additionally still run
  $OutputEncoding = [System.Text.Utf8Encoding]::new() (via $PROFILE) in order to also make Windows PowerShell send UTF-8 to external programs.

A note on file encoding:

If making Windows PowerShell too default to UTF-8 via the system-wide change is not an option, BOM-less UTF-8 files will only be read correctly under one of the following conditions:

you use -Encoding Utf8 with file-handling cmdlets.
you convert your BOM-less UTF-8 files to have a BOM
- such files can be problematic in cross-platform use; on Unix-like platforms, a UTF-8 BOM can be misinterpreted as data
- conversely, if you write PowerShell code that contains (runtime-relevant) non-ASCII characters and needs to run in both editions, saving your source code files as UTF-8 with BOM is a must (though you could also use UTF-16).
you preset the default encoding via $PSDefaultParameterValues['*:Encoding'] = 'utf8', but you'll have to scope this setting if you don't want all code to use these defaults.

Note that Windows PowerShell - curiously, except if the system-wide change is made - only ever creates UTF-8 files with BOM (whereas PowerShell [Core] defaults to BOM-less UTF-8 and has an -Encoding utf8BOM opt-in); direct use of .NET is required to work around that - see this SO answer.

Is it a good idea to implement this inside PowerShell?

Changing the encoding inside pwsh also changes the code page, which also affects:

The language of Windows console applications like CMD.EXE
The default encoding assumed when opening a file without BOM.

Those changes remain after pwsh ends, until the console is closed.
In other words: if running PWSH changes the encoding, it will impact the console session permanently:

So, (unless there is a way to decouple the encoding and the codepage... Why are they coupled in first place?) should the current code page by changed by a console app? I don't think so.

IMO, it either should be a system-wide setting, or a setting in WindowsTerminal / ConHost. Not a responsibility of a console app or a shell...

Jason Helmick (@theJasonHelmick)

Been 6 months, any progress from the Working Group Jason Helmick (@theJasonHelmick)

Jason Helmick (@theJasonHelmick)

Jason Helmick (@theJasonHelmick) Steve Lee (@SteveL-MSFT) We are now a full year into the working group reviewing and discussing this... and crickets...

I ran into this nonsense when piping to ripgrep on Windows 10 22H2 in PowerShell 7.5.1. Fortunately they document this in their FAQ https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#how-do-i-pipe-non-ascii-content-to-ripgrep-on-windows

How do I pipe non-ASCII content to ripgrep on Windows?

When piping input into native executables in PowerShell, the encoding of the input is controlled by the $OutputEncoding variable. By default, this is set to US-ASCII, and any characters in the pipeline that don't have encodings in US-ASCII are converted to ? (question mark) characters.

To change this setting, set $OutputEncoding to a different encoding, as represented by a .NET encoding object. Some common examples are below. The value of this variable is reset when PowerShell restarts, so to make this change take effect every time PowerShell is started add a line setting the variable into your PowerShell profile.

Example $OutputEncoding settings:

UTF-8 without BOM: $OutputEncoding = [System.Text.UTF8Encoding]::new()
The console's output encoding: $OutputEncoding = [System.Console]::OutputEncoding

If you continue to have encoding problems, you can also force the encoding that the console will use for printing to UTF-8 with [System.Console]::OutputEncoding = [System.Text.Encoding]::UTF8. This will also reset when PowerShell is restarted, so you can add that line to your profile as well if you want to make the setting permanent.

so I could use [System.Console]::OutputEncoding = [System.Text.UTF8Encoding]::new() to set it to UTF-8 (without BOM as I have a hunch BOM could lead to other issues, but I guess no BOM could also lead to issues but I much prefer issues in programs that undestand UTF-8 without a BOM as I only use modern programs anyways. This seems similar to what Michael Klement (@mklement0) already has pointed out). Importantly (but not surprisingly) I also had to set [System.Console]::InputEncoding = [System.Console]::OutputEncoding as otherwise Import-Module would fail with weird errors like the following on restarts of my shell

Import-Clixml: 'Element' is an invalid XmlNodeType.
InvalidOperation: Index operation failed; the array index evaluated to null.

InvalidOperation: Index operation failed; the array index evaluated to null.
InvalidOperation: Index operation failed; the array index evaluated to null.

Import-Clixml: Unexpected end of file has occurred. The following elements are not closed: S, En, DCT, Obj, En, DCT, Obj, En, DCT, Obj, En, DCT, Obj, Objs. Line 1517, position 46.
InvalidOperation: Index operation failed; the array index evaluated to null.

Import-Clixml: Name attribute for dictionary key is incorrectly specified. Line 800, position 33.
InvalidOperation: Index operation failed; the array index evaluated to null.
InvalidOperation: Index operation failed; the array index evaluated to null.

Documenting is a good first step, obviously I would prefer it be fixed but this should be easier to handle in the meantime maybe a developer setting in Windows if the defaults can't be changed? But preferably a new major release where this can be a breaking change? Also even if this is documented not everyone will read the documents so if the issue can be automatically identified somehow and the user notified that would also improve things.

I worry this issue will be closed as wontfix after it's just documented as "by design".

GitHub
ripgrep/FAQ.md at master · BurntSushi/ripgrep
ripgrep recursively searches directories for a regex pattern while respecting your gitignore - BurntSushi/ripgrep

Frulfump

Frulfump You’re right that this issue has been under discussion within the working group for quite a while, and we have documented the behavior and provided guidance (as you’ve noted). The core challenge is that we haven’t yet reached a consensus across the working group on the right path forward. There are tradeoffs around default behavior, breaking changes, and cross-platform consistency that we’re still trying to balance.

That said, your experience and the clarity of your examples are incredibly helpful. I’ll raise this again in the next working group discussion and push for us to land on a resolution.

Why not make it an experimental feature in 7.7 Preview1?

Michael Klement (@mklement0)

Thank you Michael Klement (@mklement0) and everyone else for the detailed discussion. This issue has been around for a long time - and I deeply appreciate the level of discussion. The Working Group has reviewed this in depth.

After looking into it, we’ve decided not to move forward with this change. The risk is too high, and there are too many unknowns for us to safely make this update. Also, this is something that would likely need to be addressed by the Windows team directly, since they’d need to ensure compatibility across the entire system. Fo these reasons we are closing this issue as "Won't fix".

So it took you 7 years to figure that out, it was kind of looking like it leaned towards that from the get go anyways. So we essentially wasted 7 years where the Windows team could've had this on their plate then? Will this be raised internally or can we track that somewhere?

Compared with 7 years ago, Windows Terminal is the new default terminal now.

Even if MS cannot change the default codepage for console, MS would be able to add UTF-8 Power Shell profile to Windows Terminal, and make it default finally.

This issue has been marked as won't fix and has not had any activity for 1 day. It has been closed for housekeeping purposes.

Michael Klement (@mklement0)

📣 Hey @Michael Klement (@mklement0), how did we do? We would love to hear your feedback with the link below! 🗣️

🔗 https://aka.ms/PSRepoFeedback

Microsoft Forms

Navigation Menu

Repository navigation

Make console windows fully UTF-8 by default on Windows, in line with the behavior on Unix-like platforms - character encoding, code page #7233

Environment data

Option 1: Put the following statement in your `$PROFILE`:

Option 2: Change the active code pages to `65001` system-wide (W10+):

Is it a good idea to implement this inside PowerShell?

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Make console windows fully UTF-8 by default on Windows, in line with the behavior on Unix-like platforms - character encoding, code page #7233

Description

Environment data

Activity

iSazonov commented on Aug 27, 2018

mklement0 commented on Aug 28, 2018

mklement0 commented on Aug 28, 2018

iSazonov commented on Aug 29, 2018

iSazonov commented on Mar 12, 2020

KalleOlaviNiemitalo commented on Jun 6, 2020

mklement0 commented on Jun 6, 2020

mklement0 commented on Jun 6, 2020

mklement0 commented on Jun 6, 2020

mklement0 commented on Jun 6, 2020

mklement0 commented on Jun 6, 2020

mklement0 commented on Jun 7, 2020

Option 1: Put the following statement in your $PROFILE:

Option 2: Change the active code pages to 65001 system-wide (W10+):

mklement0 commented on Dec 8, 2020

gerardog commented on Mar 29, 2022

Is it a good idea to implement this inside PowerShell?

7 remaining items

webstean commented on Jan 13, 2025

Frulfump commented on Jun 21, 2025

theJasonHelmick commented on Jun 23, 2025

iSazonov commented on Jun 24, 2025

theJasonHelmick commented on Jul 1, 2025

Frulfump commented on Jul 1, 2025

methane commented on Jul 2, 2025

microsoft-github-policy-service commented on Jul 3, 2025

microsoft-github-policy-service commented on Jul 3, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Option 1: Put the following statement in your `$PROFILE`:

Option 2: Change the active code pages to `65001` system-wide (W10+):