← Full reference

iconv

Converting text between UTF-8, CP1252, ISO-8859-15 and other encodings; transliteration and how to detect an encoding

iconv converts text from one encoding to another: for example, a file from Windows in CP1252 to UTF-8, so that accented letters like é and ü show up instead of mojibake. The command doesn't guess the encoding itself — you need to know or detect the source encoding beforehand.

Syntax

iconv -f FROM -t TO [options] [file…]

Without a file it reads stdin and writes the result to stdout. If -f or -t is omitted, the current locale's encoding is used.

Options

OptionWhat it does
-f ENCODINGsource encoding (--from-code)
-t ENCODINGtarget encoding (--to-code)
-llist all known encodings and their aliases
-o filewrite the result to a file (may be the same as the input)
-csilently drop characters missing from the target encoding; exit code 0
-sdon't print warnings
--verboseprint the names of the files being processed

Target encoding suffixes

SuffixWhat it does with a character missing from the target encoding
(none)stops: illegal input sequence at position N, code 1
//TRANSLITreplaces it with a similar one: é → e, – → -, ß → ss, € → EUR; if there's no replacement — ?
//IGNOREdrops the character, but still reports an error at the end and returns 1

Common encodings

NameWhere you meet it
UTF-8the standard in Linux, on the web, in JSON
CP1252 (WINDOWS-1252)Western European text in older files from Windows
ISO-8859-15 (LATIN9)Western European text in older Unix systems and emails; like Latin-1, but with €
CP850Western European text in DOS and the Windows console
ISO-8859-1 (LATIN1)Western European languages
UTF-16LE / UTF-16files from Windows programs; UTF-16 writes a BOM at the start
ASCIIonly Latin letters, digits and symbols — 128 characters

Names are case-insensitive, and the hyphen can often be omitted: cp1252, utf8 work too.

Examples

# a file from Windows to UTF-8
iconv -f CP1252 -t UTF-8 report.txt > report.utf8.txt

# the other way round: prepare a file for an old Windows program
iconv -f UTF-8 -t CP1252 data.csv > data-win.csv

# convert a file "in place" with -o
iconv -f CP1252 -t UTF-8 -o notes.txt notes.txt

# is the encoding in the list
iconv -l | grep -i 1252

# plain Latin instead of diacritics; the result depends on the locale
echo 'Café – naïve' | LC_ALL=C.UTF-8 iconv -f UTF-8 -t ASCII//TRANSLIT

# drop everything that doesn't fit into ASCII
echo 'Café' | iconv -c -f UTF-8 -t ASCII

# check that a file is valid UTF-8: exit code 0 — yes
iconv -f UTF-8 -t UTF-8 file.txt > /dev/null && echo 'valid UTF-8'

# convert all .txt files in a directory
for f in *.txt; do iconv -f CP1252 -t UTF-8 -o "$f" "$f"; done

How to detect the encoding

file -i report.txt
# report.txt: text/plain; charset=utf-8

file -b --mime-encoding report.txt

The file utility may be missing on a minimal system (it's installed with the file package). file only guesses: it detects us-ascii and utf-8 reliably, but single-byte encodings such as CP1252 or ISO-8859-15 are usually reported as iso-8859-1 or unknown-8bit. Then try the candidates: iconv -f CP1252 -t UTF-8 file | head and see where the text reads correctly.

Pitfalls

  • > the same file empties it. With iconv … f.txt > f.txt the shell truncates the file before iconv starts. Use -o f.txt or a temporary file.
  • A wrong -f doesn't always give an error. Any bytes are valid text in ISO-8859-1 or CP850, so converting from the wrong encoding "succeeds" and produces nonsense like cafÚ crÞme. Check the result by eye.
  • Running it twice corrupts the file. Converting an already converted file again (CP1252 → UTF-8 over UTF-8) goes through without errors and gives Grüße instead of Grüße. In loops over files, write the result to a new directory or check the file before converting.
  • //TRANSLIT depends on the locale. In the C/POSIX locale many characters turn into ?. Run with LC_ALL=C.UTF-8 or another UTF-8 locale.
  • //IGNORE returns 1. Even after dropping everything extra, iconv reports an error. -c is more reliable for scripts.
  • BOM. UTF-8 files from Windows sometimes start with the bytes EF BB BF. iconv -f UTF-8 -t UTF-8 doesn't remove them; they break the first line of a CSV and a script's shebang. You can strip them with sed '1s/^\xEF\xBB\xBF//'.
  • \r\n line endings aren't an encoding issue, iconv doesn't change them. For those you need tr -d '\r' or dos2unix.

See also

tr · sed · base64 · find