Cfrtjryk

I am developing some training and would like to demonstrate file encoding. What I am trying to accomplish is create a text file with one type of encoding that show up as nonsense when read by Linux.

Then convert the file to UTF8 Encoding and be able to read the file in Linux.

Is this possible?

You can use GNU recode to convert between encodings. It reads from stdin and is called like this:

recode from-encoding..to-encoding

For example:

$ recode ascii..ebcdic < file.txt

Or perhaps more relevant, to convert from the Windows-1252 encoding:

$ recode windows-1252..utf8 < extended-latin.txt

So an example demonstration:

$ cat > universal-declaration-french.txt

Tous les êtres humains naissent libres et égaux en dignité et en droits.

Ils sont doués de raison et de conscience et doivent agir les uns envers

les autres dans un esprit de fraternité.

^D

$ recode utf8..windows-1252 < universal-declaration-french.txt > declaration-1252.txt

$ cat declaration-1252.txt

Tous les �tres humains naissent libres et �gaux en dignit� et en droits.

Ils sont dou�s de raison et de conscience et doivent agir les uns envers

les autres dans un esprit de fraternit�.



$ recode windows-1252..utf8 < declaration-1252.txt

Tous les êtres humains naissent libres et égaux en dignité et en droits.

Ils sont doués de raison et de conscience et doivent agir les uns envers

les autres dans un esprit de fraternité.

You can see a list of encodings it supports with 'recode -l'.

If by "Linux" you mean some program in Linux, assuming your terminal emulator's encoding is set to UTF-8, and your locale is some UTF-8 locale:

$ cat > utf8.txt <<<"This is 日本語。"

$ iconv -f UTF-8 -t UTF-16 utf8.txt > utf16.txt

$ head utf*.txt  

==> utf16.txt <==

��This is �e,g��0



==> utf8.txt <==

This is 日本語。

$ iconv -f UTF-16 -t UTF-8 utf16.txt

This is 日本語。

SHIFT-JIS can encode some pretty indecipherable stuff,

% cat phrase

?k?????? ???????????? ???????? ?????? ?????????????? ????????????????????.

% hexdump -C phrase

00000000  82 6b 82 89 82 8b 82 85  20 82 91 82 95 82 89 82  |.k...... .......|

00000010  8c 82 8c 82 93 20 82 95  82 90 82 8f 82 8e 20 82  |..... ........ .|

00000020  94 82 88 82 85 20 82 86  82 92 82 85 82 94 82 86  |..... ..........|

00000030  82 95 82 8c 20 82 90 82  8f 82 92 82 90 82 85 82  |.... ...........|

00000040  8e 82 94 82 89 82 8e 82  85 2e 0a                 |...........|

0000004b

% iconv -f SHIFT-JIS -t UTF-8 < phrase 

Ｌｉｋｅ ｑｕｉｌｌｓ ｕｐｏｎ ｔｈｅ ｆｒｅｔｆｕｌ ｐｏｒｐｅｎｔｉｎｅ.

Images are also necessary for encoding problems, as some display software will "tofu" the text (white rectangles) while others display it just fine, or there may be various other disagreements about how things are being rendered only an image will help clear up (well, image, and a hexdump...)

These are from the Unicode fullwidth range that starts around U+FF01. Even more fun might be had with The Confusables.

Method to this Madness

First you'll need some means to generate text in non-standard unicode ranges, either with automation or by manually pasting a phrase together. Here's a converter that takes the a-zA-Z range and translates them up into the full width range:

#!/usr/bin/env perl

use 5.24.0;

use warnings;



die "Usage: not-ascii ...n" unless @ARGV;



my $s = '';

for my $c ( split //, "@ARGV" ) {

    if ( $c =~ m/[a-z]/ ) {    # FF41

        $s .= chr( 0xFF41 + ord($c) - 97 );

    } elsif ( $c =~ m/[A-Z]/ ) {    # FF21

        $s .= chr( 0xFF21 + ord($c) - 65 );

    } else {

        $s .= $c;

    }

}

binmode *STDOUT, ':encoding(UTF-8)';

say $s;

we can then take our Shakespeare to full width and encode it in SHIFT-JIS:

% not-ascii 'Like quills upon the fretful porpentine.' 

| iconv -f UTF-8 -t SHIFT-JIS > phrase

SHIFT-JIS was found to be usable for this prurpose by doing a brute-force search converting the UTF-8 input into all the encodings listed by iconf -l. Most other encodings aren't very interesting, or fail to convert the UTF-8:

#!/bin/sh

IFS=' '

iconv -l | while read e unused; do

   printf "$e "

   printf "test phrasen" | iconv -f UTF-8 -t "$e"

done

though you really need a hex viewer to inspect the results:

% ./brutus-iconv > x

iconv: (stdin):1:0: cannot convert

iconv: (stdin):1:0: cannot convert

iconv: (stdin):1:0: cannot convert



iconv: (stdin):1:4: cannot convert

iconv: (stdin):1:0: cannot convert

% hexdump -C x

00000000  41 4e 53 49 5f 58 33 2e  34 2d 31 39 36 38 20 74  |ANSI_X3.4-1968 t|

00000010  65 73 74 20 70 68 72 61  73 65 0a 55 54 46 2d 38  |est phrase.UTF-8|

00000020  20 74 65 73 74 20 70 68  72 61 73 65 0a 55 54 46  | test phrase.UTF|

00000030  2d 38 2d 4d 41 43 20 74  65 73 74 20 70 68 72 61  |-8-MAC test phra|

...