Home / Guides
Error fixed

invalid byte sequence for encoding "UTF8": 0xa0, 0x92, 0xff

By · October 31, 2026 · 13 min read

The short version

ERROR: invalid byte sequence for encoding "UTF8": 0xa0 (or 0x92, 0x93, 0xa3, 0xfc, 0xff) means PostgreSQL was told the file is UTF-8 and it is not. 0x92 and 0x93 point to Windows-1252 (curly quotes there, invisible control codes in Latin-1); 0xa0, 0xa3 and 0xfc mean the same in Windows-1252 and Latin-1; 0xff at the very start of the file is the byte order mark of a UTF-16 export. Tell PostgreSQL the real encoding with COPY ... WITH (ENCODING 'WIN1252'), or convert the file first with iconv -f CP1252 -t UTF-8. For UTF-16 there is no COPY option: convert with iconv -f UTF-16 -t UTF-8.

The errors, reproduced

I built three small CSV files in Python, each in a different encoding:

rows = "id,name,note\r\n1,O\u2019Brien,\u201cOrder\u201d shipped\r\n2,M\u00fcller GmbH,\u00a3250 paid\r\n3,Acme\u00a0Ltd,ok\r\n"
open("orders_1252.csv", "wb").write(rows.encode("cp1252"))

lat = "id,name,city\n1,Ren\u00e9e,K\u00f6ln\n2,\u00c0 la carte,Paris\n3,Acme\u00a0Ltd,London\n"
open("customers_latin1.csv", "wb").write(lat.encode("latin-1"))

u = "id,name\r\n1,Acme\r\n2,M\u00fcller\r\n"
open("export_utf16.csv", "wb").write(u.encode("utf-16"))

The Windows-1252 file, byte by byte. The curly apostrophe is one byte, 92; the quotes are 93 and 94; fc is the u-umlaut, a3 the pound sign, a0 the no-break space:

$ xxd orders_1252.csv
00000000: 6964 2c6e 616d 652c 6e6f 7465 0d0a 312c  id,name,note..1,
00000010: 4f92 4272 6965 6e2c 934f 7264 6572 9420  O.Brien,.Order.
00000020: 7368 6970 7065 640d 0a32 2c4d fc6c 6c65  shipped..2,M.lle
00000030: 7220 476d 6248 2ca3 3235 3020 7061 6964  r GmbH,.250 paid
00000040: 0d0a 332c 4163 6d65 a04c 7464 2c6f 6b0d  ..3,Acme.Ltd,ok.

Then I loaded each file into PostgreSQL 16.14, in a throwaway database with server_encoding and client_encoding both UTF8. I used COPY ... FROM STDIN through psycopg2's copy_expert, which is what psql's \copy runs under the hood:

>> COPY orders FROM STDIN WITH (FORMAT csv, HEADER true) < orders_1252.csv
   CharacterNotInRepertoire: invalid byte sequence for encoding "UTF8": 0x92
   CONTEXT:  COPY orders, line 2 | sqlstate 22021
>> COPY customers FROM STDIN WITH (FORMAT csv, HEADER true) < customers_latin1.csv
   CharacterNotInRepertoire: invalid byte sequence for encoding "UTF8": 0xe9 0x65 0x2c
   CONTEXT:  COPY customers, line 2 | sqlstate 22021
>> COPY u16 FROM STDIN WITH (FORMAT csv, HEADER true) < export_utf16.csv
   CharacterNotInRepertoire: invalid byte sequence for encoding "UTF8": 0xff
   CONTEXT:  COPY u16, line 1 | sqlstate 22021

COPY stops at the first bad byte and loads nothing. To see the message for each byte people search for, I fed one value at a time:

b'O\x92Brien'          invalid byte sequence for encoding "UTF8": 0x92
b'\x93Order\x94'       invalid byte sequence for encoding "UTF8": 0x93
b'Acme\xa0Ltd'         invalid byte sequence for encoding "UTF8": 0xa0
b'\xa3250'             invalid byte sequence for encoding "UTF8": 0xa3
b'M\xfcller'           invalid byte sequence for encoding "UTF8": 0xfc
b'\xc0 la carte'       invalid byte sequence for encoding "UTF8": 0xc0 0x20
b'\xff\xfei\x00d\x00'  invalid byte sequence for encoding "UTF8": 0xff
b'Ren\xe9e,'           invalid byte sequence for encoding "UTF8": 0xe9 0x65 0x2c

Sometimes the message names one byte and sometimes two or three. That comes from how UTF-8 is built, below.

Why UTF-8 rejects 0xa0, 0x92 and 0xff: a byte table

In UTF-8 every byte carries marker bits. A byte starting 0 is plain ASCII, a byte starting 10 can only continue a character, and a byte starting 110, 1110 or 11110 opens a sequence of two, three or four bytes (Python codecs: Encodings and Unicode has the table). Single-byte encodings like Windows-1252 use every byte from 0x80 to 0xff for a whole character, so most of them break those rules.

Byte Bits cp1252 latin1 Why UTF-8 rejects it
0x92 10010010 U+2019 curly apostrophe U+0092, a control code starts 10: a continuation byte with nothing to continue
0x93 10010011 U+201C left double quote U+0093, a control code same, continuation byte
0xa0 10100000 U+00A0 no-break space U+00A0 no-break space same, continuation byte
0xa3 10100011 U+00A3 pound sign U+00A3 pound sign same, continuation byte
0xfc 11111100 U+00FC u-umlaut U+00FC u-umlaut six leading 1 bits, no UTF-8 pattern starts like that
0xe9 11101001 U+00E9 e-acute U+00E9 e-acute opens a 3-byte sequence, but e and , are not continuation bytes, so PostgreSQL prints all three: 0xe9 0x65 0x2c
0xc0 11000000 U+00C0 A-grave U+00C0 A-grave opens a 2-byte sequence, the next byte is a space: 0xc0 0x20
0xff 11111111 U+00FF y-umlaut U+00FF y-umlaut never valid in UTF-8; as the first byte of a file it is half of a UTF-16 BOM

The character columns are what Python's own codecs returned when I decoded each byte on its own:

for b in (0x92, 0x93, 0xa0, 0xa3, 0xfc, 0xff, 0x81, 0x80, 0x96):
    raw = bytes([b])
    try:
        cp = ascii(raw.decode("cp1252"))
    except UnicodeDecodeError as e:
        cp = "error: " + e.reason
    print(hex(b), "cp1252:", cp, " latin-1:", ascii(raw.decode("latin-1")))
0x92 cp1252: '\u2019'  latin-1: '\x92'
0x93 cp1252: '\u201c'  latin-1: '\x93'
0xa0 cp1252: '\xa0'  latin-1: '\xa0'
0xa3 cp1252: '\xa3'  latin-1: '\xa3'
0xfc cp1252: '\xfc'  latin-1: '\xfc'
0xff cp1252: '\xff'  latin-1: '\xff'
0x81 cp1252: error: character maps to <undefined>  latin-1: '\x81'
0x80 cp1252: '\u20ac'  latin-1: '\x80'
0x96 cp1252: '\u2013'  latin-1: '\x96'

Why is 0xa0 the same in Latin-1 and cp1252, but 0x92 is not?

Windows-1252 is ISO 8859-1 (Latin-1) with the range 0x80 to 0x9f reused. Latin-1 maps byte N straight to code point N, so its bytes 0x00 to 0xff are code points 0 to 255, and 0x80 to 0x9f land on invisible control codes. Windows-1252 puts printable characters there instead: in the run above 0x80 is the euro sign (U+20AC) and 0x96 a dash (U+2013). I checked every byte against Python's two codecs:

same = lambda rng: all(bytes([b]).decode("cp1252") == bytes([b]).decode("latin-1") for b in rng)
print(same(range(0x00, 0x80)), same(range(0xa0, 0x100)))
True True

In 0x80 to 0x9f not one byte agreed, and five of them (0x81 among them) have no character in cp1252 at all. So 0xa0, 0xa3 and 0xfc decode to the same characters either way, and 0x92 does not.

That has a practical edge. My Latin-1 customer file loaded identically with ENCODING 'LATIN1' and ENCODING 'WIN1252', because it has no bytes in 0x80 to 0x9f. The Windows-1252 file loaded with LATIN1 without any error, and stored the wrong text:

>> COPY orders FROM STDIN WITH (FORMAT csv, HEADER true, ENCODING 'LATIN1') < orders_1252.csv
   (1, 'O\x92Brien', '\x93Order\x94 shipped')
   (2, 'M\xfcller GmbH', '\xa3250 paid')
   (3, 'Acme\xa0Ltd', 'ok')

O\x92Brien is O, the control code U+0092, Brien. Nothing complained, and the apostrophe is gone from the data. So when in doubt between the two, pick WIN1252. The exception is a byte that cp1252 leaves undefined (0x81 above): Python refuses to decode it as cp1252, and a file with one of those probably is not Windows-1252 at all.

Step 1: identify the encoding

file makes a first guess. On macOS (file-5.41):

$ file -I *.csv
customers_latin1.csv: text/csv; charset=iso-8859-1
export_utf16.csv:     text/csv; charset=utf-16le
orders_1252.csv:      text/csv; charset=unknown-8bit

unknown-8bit is the hint for Windows-1252: that file has bytes in 0x80 to 0x9f, which are control codes in Latin-1, and file did not call it ISO 8859-1 the way it did the real Latin-1 file. Plain file without -I said CSV text for all three, which is no help.

Python's codecs make a better test. The docs say it plainly: without external information it is impossible to reliably determine which encoding was used, but UTF-8 has a structure that random bytes do not fit. So try UTF-8 first and work down:

import sys
for path in sys.argv[1:]:
    data = open(path, "rb").read()
    print(path)
    for enc in ("utf-8", "utf-16", "cp1252", "latin-1"):
        try:
            text = data.decode(enc)
            print(f"  {enc:8} ok   {ascii(text.splitlines()[1])}")
        except UnicodeDecodeError as e:
            print(f"  {enc:8} fail {e.reason} at byte {e.start}: {data[e.start:e.start+1]!r}")
orders_1252.csv
  utf-8    fail invalid start byte at byte 17: b'\x92'
  utf-16   fail truncated data at byte 80: b'\n'
  cp1252   ok   '1,O\u2019Brien,\u201cOrder\u201d shipped'
  latin-1  ok   '1,O\x92Brien,\x93Order\x94 shipped'
customers_latin1.csv
  utf-8    fail invalid continuation byte at byte 18: b'\xe9'
  utf-16   fail truncated data at byte 62: b'\n'
  cp1252   ok   '1,Ren\xe9e,K\xf6ln'
  latin-1  ok   '1,Ren\xe9e,K\xf6ln'
export_utf16.csv
  utf-8    fail invalid start byte at byte 0: b'\xff'
  utf-16   ok   '1,Acme'
  cp1252   ok   '\x00'
  latin-1  ok   '\x00'

Read it with the order in mind. Latin-1 said "ok" to every file, including the UTF-16 one, so "it decoded" proves nothing for a single-byte codec. The UTF-16 attempts on the other two files failed only because their lengths were odd; an even-length file would have decoded as nonsense. What decides it is the first codec that succeeds and gives readable text when you look at a real row. The codec names cp1252 and latin_1 are in the Standard Encodings table, with windows-1252 and iso-8859-1 as aliases.

Fix 1: tell COPY the encoding

The COPY ENCODING option names the encoding of the file; without it, the current client encoding is used. PostgreSQL supports both WIN1252 and LATIN1 and converts them to the database encoding on the way in:

>> COPY orders FROM STDIN WITH (FORMAT csv, HEADER true, ENCODING 'WIN1252') < orders_1252.csv
   (1, 'O\u2019Brien', '\u201cOrder\u201d shipped')
   (2, 'M\xfcller GmbH', '\xa3250 paid')
   (3, 'Acme\xa0Ltd', 'ok')
>> COPY customers FROM STDIN WITH (FORMAT csv, HEADER true, ENCODING 'LATIN1') < customers_latin1.csv
   (1, 'Ren\xe9e', 'K\xf6ln')
   (2, '\xc0 la carte', 'Paris')
   (3, 'Acme\xa0Ltd', 'London')

Since \copy sends a COPY ... FROM STDIN, the same option goes on it: \copy orders FROM 'orders_1252.csv' WITH (FORMAT csv, HEADER true, ENCODING 'WIN1252'). Or set the client encoding for the session, which is the route the character set docs describe (SET client_encoding, \encoding, or PGCLIENTENCODING). I ran SET client_encoding TO 'WIN1252', the same COPY without the option, then RESET client_encoding:

(1, 'O\u2019Brien', 7, 9)
(2, 'M\xfcller GmbH', 11, 12)
(3, 'Acme\xa0Ltd', 8, 9)

The last two numbers are length() and octet_length(): 7 characters in 9 bytes, so the apostrophe is now stored as a proper 3-byte UTF-8 character. Reset the client encoding afterwards, or everything else you send in that session is read as WIN1252 too.

UTF-16 has no such option:

>> COPY u16 FROM STDIN WITH (FORMAT csv, HEADER true, ENCODING 'UTF16') < export_utf16.csv
   InvalidParameterValue: argument to option "encoding" must be a valid encoding name

UTF-16 is not among the supported character sets, so that file has to be converted first.

Fix 2: convert the file with iconv

$ iconv -f CP1252 -t UTF-8 orders_1252.csv > orders_utf8.csv
$ file -I orders_utf8.csv
orders_utf8.csv: text/csv; charset=utf-8
$ iconv -f UTF-16 -t UTF-8 export_utf16.csv > export_utf8.csv
$ xxd export_utf8.csv
00000000: 6964 2c6e 616d 650d 0a31 2c41 636d 650d  id,name..1,Acme.
00000010: 0a32 2c4d c3bc 6c6c 6572 0d0a            .2,M..ller..

The apostrophe became e2 80 99, the u-umlaut c3 bc, and iconv dropped the UTF-16 BOM. Both converted files loaded with a plain COPY and no ENCODING option. To check whether a file is valid UTF-8 before loading, convert it from UTF-8 to UTF-8 and look at the exit code:

$ iconv -f UTF-8 -t UTF-8 orders_1252.csv > chk.txt; echo "exit $?"
iconv: iconv(): Illegal byte sequence
exit 1

chk.txt stops right before the bad byte, so its length tells you where the first one is.

Pick the source encoding from step 1. Converting a Latin-1 file with -f CP1252 works for the same reason COPY did; converting a Windows-1252 file with -f ISO-8859-1 gives the silent control-code problem shown above.

If the file starts with ff fe, you have UTF-16 little-endian with a BOM; without the BOM a UTF-16 file fails on 0x00 instead, which is covered in invalid byte sequence for encoding UTF8: 0x00. Microsoft lists code page 1200 as UTF-16 little endian and 1252 as ANSI Latin 1, Western European, and warns on the same page that ANSI code pages can differ between computers. A file from a machine set to another ANSI code page needs that code page in the -f; Old Data, Odd Encodings: CP1251, CP866, Latin-1 goes through the others.

What our CSV to PostgreSQL converter does with these files

I ran all three files, plus the iconv output from fix 2, through the reader and PostgreSQL writer behind the CSV to PostgreSQL converter, from backend/:

for name in sys.argv[1:]:
    ds = READERS["csv"](Path(name))
    table = ds.tables[0]
    print(name)
    for w in ds.warnings:
        print(f"  {w.code}: {w.message}")
    print("  columns:", [c.name for c in table.columns])
    for row in ds.rows_for(table.name):
        print("  ", ascii(row))
    out = Path("out_" + Path(name).stem)
    out.mkdir(exist_ok=True)
    WRITERS["postgresql"](ds, out)

The reader looks at the bytes before it decodes anything. A UTF-16 byte order mark means UTF-16. Otherwise the file is UTF-8 if every byte in it is valid UTF-8, Windows-1252 if every byte decodes as cp1252, and Latin-1 if not. Whenever the answer is not UTF-8, it says which codec it used:

orders_1252.csv
  CSV_ENCODING_GUESSED: The file is not valid UTF-8, so text was decoded as Windows-1252 (cp1252). Output is UTF-8. If accented letters, quotes or currency signs came out wrong, re-save the CSV as UTF-8 and convert it again.
  columns: ['id', 'name', 'note']
   [1, 'O\u2019Brien', '\u201cOrder\u201d shipped']
   [2, 'M\xfcller GmbH', '\xa3250 paid']
   [3, 'Acme\xa0Ltd', 'ok']
customers_latin1.csv
  CSV_ENCODING_GUESSED: The file is not valid UTF-8, so text was decoded as Windows-1252 (cp1252). Output is UTF-8. If accented letters, quotes or currency signs came out wrong, re-save the CSV as UTF-8 and convert it again.
  columns: ['id', 'name', 'city']
   [1, 'Ren\xe9e', 'K\xf6ln']
   [2, '\xc0 la carte', 'Paris']
   [3, 'Acme\xa0Ltd', 'London']
export_utf16.csv
  CSV_ENCODING_DECLARED: The file starts with a UTF-16 byte order mark; text was decoded as UTF-16.
  columns: ['id', 'name']
   [1, 'Acme']
   [2, 'M\xfcller']
orders_utf8.csv
  columns: ['id', 'name', 'note']
   [1, 'O\u2019Brien', '\u201cOrder\u201d shipped']
   [2, 'M\xfcller GmbH', '\xa3250 paid']
   [3, 'Acme\xa0Ltd', 'ok']

The Latin-1 file was read as cp1252. For this file that is the same text, because it has no bytes in 0x80 to 0x9f, the only range where the two differ (see the table above). The UTF-8 file got no warning.

Each dump.sql starts with SET client_encoding = 'UTF8'; and holds the text as UTF-8. The first row of the Windows file, in bytes:

$ grep -a "^(1" out_orders_1252/dump.sql | xxd
00000000: 2831 2c20 274f e280 9942 7269 656e 272c  (1, 'O...Brien',
00000010: 2027 e280 9c4f 7264 6572 e280 9d20 7368   '...Order... sh
00000020: 6970 7065 6427 292c 0a                   ipped'),.

The single byte 92 from the source is now e2 80 99, the same three bytes iconv produced in fix 2.

I loaded the three dumps into a fresh UTF8 database on the same PostgreSQL 16.14 and read the rows back, with length() and octet_length() of name on the end:

== orders_1252
   (1, 'O\u2019Brien', '\u201cOrder\u201d shipped', 7, 9)
   (2, 'M\xfcller GmbH', '\xa3250 paid', 11, 12)
   (3, 'Acme\xa0Ltd', 'ok', 8, 9)
== customers_latin1
   (1, 'Ren\xe9e', 'K\xf6ln', 5, 6)
   (2, '\xc0 la carte', 'Paris', 10, 11)
   (3, 'Acme\xa0Ltd', 'London', 8, 9)
== export_utf16
   (1, 'Acme', 4, 4)
   (2, 'M\xfcller', 6, 7)

No errors, no ENCODING option, and the same rows the WIN1252 COPY in fix 1 produced. A guess is still a guess, so read the warning: if the file came from a machine with another ANSI code page, Cyrillic or Greek for example, the letters will be wrong in a way no decoder can notice, and the fix is iconv with the right -f first.

For loading the converted file yourself, with COPY, \copy or pgAdmin, see CSV to PostgreSQL: COPY, \copy and pgAdmin.

Sources