Special Characters and Unix Files

Hello,

Some preliminary reminders

To understand what encoding is, it's interesting to draw a parallel with a book. A book can be written in English (in which case ASCII characters are sufficient) or in French, Russian, or Chinese (in which case, an encoding that allows for the storage of special characters is necessary).

This has an impact: in ASCII, each character is encoded on one byte (8 bits), which means 2^8 = 256 possibilities. If you look at

man ascii

, you'll also see that accented characters are not included, and that's what led to the emergence of more complicated encodings. That's how Unicode was born (2 bytes per character) and subsequently more compact formats like ISO or UTF-8 (where the character sizes are variable).

In short, encoding is to a file what a language is to a novel.

The problem is that on a computer, everything is just bytes, so it all depends on the meaning attributed to those bytes. However, since the conventions established by these different encodings are different (and incompatible), this is where encoding problems arise. When you observe garbled characters (either in the result of a shell command or on a poorly configured website), it's actually this famous problem that occurs. The software decoding the text uses encoding A while this file was stored with encoding B.

Finally, in addition to encoding problems, an additional issue may occur. The operating systems Linux, BSD, MacOS, etc. use the LF character ('\n' in programming) to signify a new line, whereas Windows does its own thing and uses CRLF ('\r\n' in programming).

The command

cat

To concatenate multiple files in the shell, rather than using a

for

loop as you proposed, it suffices to write:

cat fichier1.csv fichier2.csv fichier3.csv > resultat.csv

How to avoid encoding problems

The approach is straightforward: it's better to always use UTF-8 everywhere and all the time (and ASCII if the file doesn't contain special characters). I recommend making sure that this is indeed the default locale (see the result of the

locale

command).

On Debian / Ubuntu, supported locales and the default locale are declared as explained on this page).

However, even if your system is configured to use the locale

fr_FR.UTF-8

, it may happen that some files do not conform to UTF-8. This concerns:

• certain files already present on your system, which do not need special characters, and are therefore encoded in ASCII (like

  /etc/motd

  , the message of the day)
• files coming from elsewhere (e.g., from the internet, from a Windows partition, etc.).

What encoding is used in a given file, or when I concatenate two files?

If you use the

cat

command as explained above, the encoding chosen at the time of creating a file will generally be the most appropriate (e.g., ASCII if your file doesn’t contain special characters, UTF-8 otherwise).

What happens if you concatenate two files with different encodings? Well, it depends, but generally, the involved software will try to choose an encoding that takes into account the specifics of each file. The

file

command allows you to see which encoding is used:

Example:

(mando@silk) (~) $ file /etc/motd 
/etc/motd: ASCII text
(mando@silk) (~) $ file toto.txt 
toto.txt: UTF-8 Unicode text
(mando@silk) (~) $ cat /etc/motd toto.txt > resultat.txt
(mando@silk) (~) $ file resultat.txt 
resultat.txt: UTF-8 Unicode text

In this example, since UTF-8 is strictly more tolerant than ASCII and since toto.txt contains special characters, the file resultat.txt is written in UTF-8.

However, be cautious; this does not mean you will necessarily get UTF-8 (even if your locale is properly configured), just the most tolerant encoding.

Example 1:

(mando@silk) (~) $ file /media/windows/Documents\ and\ Settings/mando/ntuser.ini 
/media/windows/Documents and Settings/mando/ntuser.ini: Little-endian UTF-16 Unicode text, with CRLF line terminators
(mando@silk) (~) $ cat /media/windows/Documents\ and\ Settings/mando/ntuser.ini /etc/motd > resultat.txt 
(mando@silk) (~) $ file resultat.txt 
resultat.txt: Little-endian UTF-16 Unicode text, with CRLF line terminators

Example 2:

(mando@silk) (~) $ cat /media/windows/Documents\ and\ Settings/mando/ntuser.ini toto.txt > resultat.txt 
(mando@silk) (~) $ file resultat.txt 
resultat.txt: Little-endian UTF-16 Unicode text, with CRLF line terminators

In short, before concatenating files with different (non-ASCII) encodings, it is wiser to correct their encodings beforehand.

• Method 1: the command

  iconv

  (which you have already mentioned) allows changing a file's encoding. The

  file

  command helps see the encoding used in a text file.
• Method 2: text editors also allow (at the time of saving the file) to choose/change the encoding of the text file. Moreover, if you open a file written in ASCII and introduce a special character, it will be converted to UTF-8 (according to your locales). If the prior encoding of the file already supported special characters, the encoding will not be modified:

Windows line endings

If we look at example 2, we see that the file

resultat.txt

contains Windows line endings (therefore

\r\n

= CRLF). As explained in the introduction, other operating systems (including Linux) use

\n

= LF.

Here too, a prior conversion may be necessary. Personally, I use

dos2unix

, but there are other ways to proceed, see this discussion).

Example:

(mando@silk) (~) $ cp /media/windows/devlist.txt .
(mando@silk) (~) $ file devlist.txt 
devlist.txt: ASCII text, with CRLF, CR line terminators
(mando@silk) (~) $ dos2unix devlist.txt 
dos2unix: converting file devlist.txt to Unix format…
(mando@silk) (~) $ file devlist.txt 
devlist.txt: ASCII text, with CRLF, LF line terminators

We can control in detail how line endings are modified (e.g., with

cat -A devlist.txt

).

Good luck!

In fact, it's a UTF-8 conversion without BOM of my final file (needed for transmission) that causes these special characters:

iconv -f IBM-1252 -t UTF-8 TOTO.TOTAL.csv >TOTO.TOTAL.csv.tmp

The question remains the same: how to avoid these special characters?

Hello mamiemando,

Thank you very much for the great effort you made to respond to me!

- For the "cat", I'm using the "for" loop because I have several, at least 20 files, the "cat file1 file2 file3" doesn't work.
- The files I am concatenating all come from the same script present on the same machine where I have my problem.
- My apologies, I forgot to mention that I am on Unix AIX 7.2 and not Linux, the command "dos2unix" doesn't work "ksh: dos2unix: not found."

So what I found is a "sed": sed 's/é/é/' "my_file", it's not great, but I still have the other characters.

Hello Dino,

For "cat," I use the "for" loop because I have several, at least 20 files, and "cat file1 file2 file3" doesn't work.

For cat: you can pass 20 files if you want, that’s not a problem. If then you find it too tedious to type (which I can understand), keep in mind that you can inject the result of one command into another.

Example:

cat $(ls *csv)

The files I'm concatenating all come from the same script on the same machine where I have my problem.

Okay. That doesn't change anything though :-)

My bad, I forgot to mention that I'm on Unix AIX 7.2 and not Linux; the command "dos2unix" doesn’t work "ksh: dos2unix: not found."

You probably just need to install it (on Ubuntu you'd run:

sudo apt update && sudo apt install dos2unix

). Note that this link, which I previously suggested, offers alternative methods, e.g. using

tr

. Finally, note that this conversion is only necessary if you have issues with line breaks; you may not necessarily be facing this aspect of the problem, it's up to you to check with cat -A.

Example 1: Linux line break (LF = \n = $)

(mando@silk) (~) $ cat -A /etc/motd | head
$
The programs included with the Debian GNU/Linux system are free software;$
the exact distribution terms for each program are described in the$
individual files in /usr/share/doc/*/copyright.$
$
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent$
permitted by applicable law.$ 

Example 2: Windows line breaks (CRLF = \r\n = ^M$)

(mando@silk) (~) $ cat -A /media/windows/devlist.txt | head
USB\ROOT_HUB30\4&108F51C9&0&0^M$
 Name: USB Root Hub (xHCI)^M^M$
 Driver is running.^M^M$
{5D624F94-8850-40C3-A3FA-A4FD2080BAF3}\VWIFIMP_WFD\5&3E95BAB&0&11^M$
 Name: Microsoft Wi-Fi Direct Virtual Adapter^M^M$
 Driver is running.^M^M$
ACPI\PNP0C02\1^M$
 Name: Motherboard resources^M^M$
 Device is currently stopped.^M^M$
ACPI\PNP0C02\2^M$

Good luck

Hello,

Thanks mamiemando.
I don’t have the rights to install anything on the server, not even as the root user, so I opted for "SED," but your link is pretty good.

Thank you very much!

You're welcome, take care :-)