You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
utf-8-sig: UnicodeDecodeError offsets do not count the BOM #159071
When the input starts with a BOM, the utf-8-sig codec reports UnicodeDecodeError offsets relative to the input without the BOM, i.e. error.start is off-by-3:
data=b"\xef\xbb\xbf\xff"# Invalid byte at position 3try:
data.decode("utf-8-sig")
exceptUnicodeDecodeErrorase:
print(e) # 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
The input is being sliced in utf_8_sig.py to remove the BOM, but raised exceptions are not modified to correct the offsets and object attributes. This occurs in a few places, e.g.
The utf-16 and utf-32 codecs also remove the BOM, but it is included in the offsets and object attributes in their errors, so this looks like an oversight.
Activity