iconv
Converting text between UTF-8, CP1252, ISO-8859-15 and other encodings; transliteration and how to detect an encoding
iconv converts text from one encoding to another: for example, a file from Windows in CP1252 to UTF-8, so that accented letters like é and ü show up instead of mojibake. The command doesn't guess the encoding itself — you need to know or detect the source encoding beforehand.
Syntax
iconv -f FROM -t TO [options] [file…]
Without a file it reads stdin and writes the result to stdout. If -f or -t is omitted, the current locale's encoding is used.
Options
| Option | What it does |
|---|---|
-f ENCODING | source encoding (--from-code) |
-t ENCODING | target encoding (--to-code) |
-l | list all known encodings and their aliases |
-o file | write the result to a file (may be the same as the input) |
-c | silently drop characters missing from the target encoding; exit code 0 |
-s | don't print warnings |
--verbose | print the names of the files being processed |
Target encoding suffixes
| Suffix | What it does with a character missing from the target encoding |
|---|---|
| (none) | stops: illegal input sequence at position N, code 1 |
//TRANSLIT | replaces it with a similar one: é → e, – → -, ß → ss, € → EUR; if there's no replacement — ? |
//IGNORE | drops the character, but still reports an error at the end and returns 1 |
Common encodings
| Name | Where you meet it |
|---|---|
UTF-8 | the standard in Linux, on the web, in JSON |
CP1252 (WINDOWS-1252) | Western European text in older files from Windows |
ISO-8859-15 (LATIN9) | Western European text in older Unix systems and emails; like Latin-1, but with € |
CP850 | Western European text in DOS and the Windows console |
ISO-8859-1 (LATIN1) | Western European languages |
UTF-16LE / UTF-16 | files from Windows programs; UTF-16 writes a BOM at the start |
ASCII | only Latin letters, digits and symbols — 128 characters |
Names are case-insensitive, and the hyphen can often be omitted: cp1252, utf8 work too.
Examples
# a file from Windows to UTF-8
iconv -f CP1252 -t UTF-8 report.txt > report.utf8.txt
# the other way round: prepare a file for an old Windows program
iconv -f UTF-8 -t CP1252 data.csv > data-win.csv
# convert a file "in place" with -o
iconv -f CP1252 -t UTF-8 -o notes.txt notes.txt
# is the encoding in the list
iconv -l | grep -i 1252
# plain Latin instead of diacritics; the result depends on the locale
echo 'Café – naïve' | LC_ALL=C.UTF-8 iconv -f UTF-8 -t ASCII//TRANSLIT
# drop everything that doesn't fit into ASCII
echo 'Café' | iconv -c -f UTF-8 -t ASCII
# check that a file is valid UTF-8: exit code 0 — yes
iconv -f UTF-8 -t UTF-8 file.txt > /dev/null && echo 'valid UTF-8'
# convert all .txt files in a directory
for f in *.txt; do iconv -f CP1252 -t UTF-8 -o "$f" "$f"; done
How to detect the encoding
file -i report.txt
# report.txt: text/plain; charset=utf-8
file -b --mime-encoding report.txt
The file utility may be missing on a minimal system (it's installed with the file package). file only guesses: it detects us-ascii and utf-8 reliably, but single-byte encodings such as CP1252 or ISO-8859-15 are usually reported as iso-8859-1 or unknown-8bit. Then try the candidates: iconv -f CP1252 -t UTF-8 file | head and see where the text reads correctly.
Pitfalls
> the same fileempties it. Withiconv … f.txt > f.txtthe shell truncates the file beforeiconvstarts. Use-o f.txtor a temporary file.- A wrong
-fdoesn't always give an error. Any bytes are valid text inISO-8859-1orCP850, so converting from the wrong encoding "succeeds" and produces nonsense likecafÚ crÞme. Check the result by eye. - Running it twice corrupts the file. Converting an already converted file again (
CP1252 → UTF-8over UTF-8) goes through without errors and givesGrüßeinstead ofGrüße. In loops over files, write the result to a new directory or check the file before converting. //TRANSLITdepends on the locale. In theC/POSIXlocale many characters turn into?. Run withLC_ALL=C.UTF-8or another UTF-8 locale.//IGNOREreturns1. Even after dropping everything extra,iconvreports an error.-cis more reliable for scripts.- BOM. UTF-8 files from Windows sometimes start with the bytes
EF BB BF.iconv -f UTF-8 -t UTF-8doesn't remove them; they break the first line of a CSV and a script's shebang. You can strip them withsed '1s/^\xEF\xBB\xBF//'. \r\nline endings aren't an encoding issue,iconvdoesn't change them. For those you needtr -d '\r'ordos2unix.