3

The following command does not use UTF-8: head -c 1M </dev/urandom >myfile.txt

  • 1
    As long as each char is <=7F, then you have a UTF-8 character – Alastair McCormack Nov 26 '15 at 10:16
  • After executing head -c 1M </dev/urandom >myfile.txt, if I open mytext.txt with gedit it says that there are problems with UTF-8 character encoding – Message Passing Nov 26 '15 at 10:29
  • Sorry, I meant that to get valid UTF-8, you must only have single bytes of <= \x7F or build valid multi-byte UTF-8 sequences. The former is obviously easier from a random perspective. – Alastair McCormack Nov 26 '15 at 10:39
  • 1
    Do you want it covering \u0 to \U7fffffff or just \u0..\ud7ff+ \ue000..\u10ffff or only those that are currently specified in the latest Unicode spec? – Stéphane Chazelas Nov 26 '15 at 11:44
4

If you want UTF-8 encodings of code points 0 to 0x7FFFFFFF (which the UTF-8 encoding algorithm was originally designed to work on):

< /dev/urandom perl -CO -ne '
    BEGIN{$/=\4}
    no warnings "utf8";
    print chr(unpack("L>",$_) & 0x7fffffff)'

Nowadays, Unicode is restricted to 0..D7FF, E000..10FFFF (though some of those characters are not assigned, some of which will never be (are defined as non-characters)).

< /dev/urandom perl -CO -ne '
    BEGIN{$/=\3}
    no warnings "utf8";
    $c = unpack("L>","\0$_") * 0x10f800 >> 24;
    $c += 0x800 if $c >= 0xd800;
    print chr($c)'

If you only want assigned characters, you can pipe that to:

uconv -x '[:unassigned:]>;'

Or change that to:

< /dev/urandom perl -CO -ne '
    BEGIN{$/=\3}
    no warnings "utf8";
    $c = unpack("L>","\0$_") * 0x10f800 >> 24;
    $c += 0x800 if $c >= 0xd800;
    $c = chr $c;
    print $c if $c =~ /\P{unassigned}/'

You may prefer:

             if $c =~ /[\p{Space}\p{Graph}]/ && $c !~ /\p{Co}/

To only get graphical and spacing ones (exclude those from the private-use sections).

Now, to get 1GiB of that, you can pipe it to head -c1G (assuming GNU head), but beware the last character may be cut in the middle.

2

The most efficient way to create a text file with size 10 MB and UTF-8 character encoding is base64 /dev/urandom | head -c 10000000 | egrep -ao "\w" | tr -d '\n' > file10MB.txt

  • Is the egrep and tr necessary? – Alastair McCormack Nov 26 '15 at 13:50
  • That doesn't produce a 10MB file. Also, you either need to remove base64 or add a base64 -d somewhere; what you have now only outputs characters used by the Base64 encoding. – Gilles 'SO- stop being evil' Nov 26 '15 at 23:23
0

Grep for ASCII (sub-set of UTF-8) chars, on Linux/GNU:

dd if=/dev/random bs=1 count=1G | egrep -ao "\w" | tr -d '\n'
  • 1
    base64 might be considerably more efficient. – muru Nov 26 '15 at 11:08
  • haha, doh! That's what happens if you over think the question :) You should offer that as an answer :) – Alastair McCormack Nov 26 '15 at 11:10
  • I'm looking for some way to as many UTF8 characters as I can. If I do, I'll post that as answer. :) – muru Nov 26 '15 at 11:11
  • dd if=/dev/random bs=1 count=10M | egrep -ao "\w" | tr -d '\n' > file10MB.txt gets stuck. It is too slow. – Message Passing Nov 26 '15 at 13:27
  • 1
    @muru you really need to use urandom here, random will get stuck really fast because it needs entropy. This is not fast, but it will get the job done. dd if=/dev/urandom bs=1 count=1G | egrep -ao "\w" | tr -d '\n' – theduke Jan 8 '17 at 11:23
0

If you want non-ASCII characters, then you'll need a way to build valid UTF-8 sequences. The chance that two consecutive bytes yielding a valid UTF-8 is very low.

Instead, this Python script creates random 8 bit values that can be converted in Unicode chars, then written out as UTF-8:

import random
import io

char_count = 0

with io.open("random-utf8.txt", "w", encoding="utf-8") as my_file:

    while char_count <= 1000000 * 1024:
        rand_long = random.getrandbits(8)

        # Ignore control characters
        if rand_long <= 32 or (rand_long <= 0x9F and rand_long > 0x7F):
            continue

        unicode_char = unichr(rand_long)
        my_file.write(unicode_char)
        char_count += 1

You could also change it to use a random 16 bit number which would yield non-latin values.

It's not fast but fairly accurate.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.