Common encoding pattern for Utility Cmdlets #3628

Activity

1. Ilya (iSazonov)

   added

   Area-Cmdlets-Utility

   Issue-Discussionthe issue may not have a clear classification yet. The issue may generate an RFC or may be reclassifthe issue may not have a clear classification yet. The issue may generate an RFC or may be reclassif

   on Apr 23, 2017
2. iSazonov commented on May 3, 2017

   

   Ilya (iSazonov)

   on May 3, 2017

   CollaboratorAuthor

   More actions

   Jason Shirk (@lzybkr) Could you please comment and approve this change?
3. lzybkr commented on May 3, 2017

   

   Jason Shirk (lzybkr)

   on May 3, 2017

   Contributor

   More actions

   I think I agree in principle regarding removing Unknown and String from Out-File, but don't have enough data to approve the change.

   I believe Out-File uses the encoding when you specify -Append, but you'd need to confirm.

   A search of GitHub for Out-File could help provide some data, but finding uses is the only reliable way of making a a decision - to keep - because lack of usage on GitHub doesn't mean the encoding values aren't in use, just that those uses aren't on GitHub or other public sources of scripts.
4. iSazonov commented on May 4, 2017

   

   Ilya (iSazonov)

   on May 4, 2017

   CollaboratorAuthor

   More actions

   Jason Shirk (@lzybkr) Thanks for comments.

   I did not found using of EncodingConversion.Unknown and EncodingConversion.String in the repo. It seems it's just a mirror of FileProvider types.

   Search of GitHub. Common usage is:
   Out-File -Encoding Ascii
Out-File -Encoding Uft8
   I did not found (manual search only) "Unknown" and "String" with "out-file -encoding" at all.
5. iSazonov commented on May 4, 2017

   

   Ilya (iSazonov)

   on May 4, 2017

   CollaboratorAuthor

   More actions

   Michael Klement (@mklement0) Have you any special thoughts about the Issue?
6. mklement0 commented on May 5, 2017

   

   Michael Klement (mklement0)

   on May 5, 2017

   Last edited by mklement0

   Contributor

   More actions

   Ilya (@iSazonov): I also can't comment on current usage, except to say I've never personally used the Unknown and String encodings.

   • I agree that Unknown doesn't make sense in output cmdlets. In practice, Out-File / Out-File -Append and Set-Content / Add-Content all treat Unknown as Unicode (UTF-16 LE) - cc Jason Shirk (@lzybkr)

     • That is, when appending to an existing file, the behavior does not depend on that file's existing encoding.
       As far as I know, the only cmdlet that changes its default encoding based on preexisting file content is Export-Csv -Append (and does so in a half-broken manner: preexisting "ANSI" content (Default) results in UTF-8 content getting appended).

   • I'm still not sure what the purpose of String is. Its behavior differs between Out-File and Set-Content - unlike with Unknown, String seems to result in the cmdlets' respective defaults (Unicode for Out-File, Default ("ANSI" on Windows) for Set-Content).

   The following code demonstrates this:

   "==== Set-Content / Add-Content ==== "
   $null, 'Unknown', 'String' | % { $ht=@{}; if ($_) { $ht.Encoding = $_ };  
     'a' | set-content t.txt  @ht; 
     "`nSet-Content $(if ($_) { "-Encoding $_" } else { '(no -Encoding)' })`n----"; od -tx1 t.txt }
   
   $null, 'Unknown', 'String' | % { $ht=@{}; if ($_) { $ht.Encoding = $_ }; 
     'a' | set-content t.txt -Encoding ascii; 'b' | add-content t.txt  @ht; 
     "`nAdd-Content $(if ($_) { "-Encoding $_" } else { '(no -Encoding)' })`n----"; od -tx1 t.txt }
   
   "`n`n==== Out-File / Out-File -Append ==== "
   $null, 'Unknown', 'String' | % { $ht=@{}; if ($_) { $ht.Encoding = $_ };  
     'a' | Out-File t.txt  @ht; 
     "`nOut-File $(if ($_) { "-Encoding $_" } else { '(no -Encoding)' })`n----"; od -tx1 t.txt }
   
   $null, 'Unknown', 'String' | % { $ht=@{}; if ($_) { $ht.Encoding = $_ }; 
     'a' | Out-File t.txt -Encoding ascii; 'b' | Out-File -Append t.txt  @ht; 
     "`nOut-File -Append $(if ($_) { "-Encoding $_" } else { '(no -Encoding)' })`n----"; od -tx1 t.txt }

   ---

   As for the input side of things, using the example of Get-Content:

   In practice, Unknown is interpreted the same way as with Out-File and Set-Content, Unknown is effectively the same as Unicode (UTF-16 LE).

   Similarly, String seems to be equivalent to the default behavior (as if -Encoding had not been specified): which means:

   • if there's no BOM, use Default
   • if there is a BOM, respect it.

   Purely from a naming standpoint, it is this variable behavior that deserves to be called Unknown.
7. iSazonov commented on May 5, 2017

   

   Ilya (iSazonov)

   on May 5, 2017

   Last edited by iSazonov

   CollaboratorAuthor

   More actions

   Michael Klement (@mklement0) Thanks for great comments!

   In context of the Issue we should ignore difference between Utility cmdlets and FileProvider cmdlets. As result we should conclude that we could safely remove String and Unknown from Out-File.

   But I see another question: why Out-File cmdlet uses the Utility cmdlet code base but not FileProvider code base for encoding? It seems it must be placed together with Get-Content and Set-Content. Nevertheless, even if we agree to do this cmdlet as FileProvider cmdlet then we see that we could safely remove String and Unknown from Out-File.

   Also I reviewed MSDN documentation: there is no exact description of use String and Unknown. If these features were never documented properly then no one can use them correctly. It looks as if the project was not fully realized or abandoned long ago.
8. mklement0 commented on May 5, 2017

   

   Michael Klement (mklement0)

   on May 5, 2017

   Contributor

   More actions

   Ilya (@iSazonov)

   I fully agree re removal of String and Unknown; let's see what the powers that be decide.

   Good point re code base - had never paid attention to that.

   👍React with 👍1Reacted by Ilya
9. mklement0 commented on May 5, 2017

   

   Michael Klement (mklement0)

   on May 5, 2017

   Contributor

   More actions

   P.S.: For symmetry and to support all Unicode-mandated encoding forms, we should support UTF-32 BE (big-endian) as well (the UTF32 value is UTF-32 LE).

   I'll address this and other encoding-related issues in an upcoming post in PowerShell/PowerShell-RFC#71

   👍React with 👍1Reacted by Ilya
10. iSazonov commented on Aug 16, 2018

    

    Ilya (iSazonov)

    on Aug 16, 2018

    CollaboratorAuthor

    More actions

    It was fixed with implementing RFC0020 in #5080
11. Ilya (iSazonov)

    closed this as completedon Aug 16, 2018
12. Ilya (iSazonov)

    added

    Resolution-FixedThe issue is fixed.The issue is fixed.

    on Sep 5, 2018
13. Ryan Yates (kilasuit)

    added

    WG-Cmdletsgeneral cmdlet issuesgeneral cmdlet issues

    and removed

    WG-Cmdlets-Utility

    on Jul 17, 2026