The following command does not use UTF-8: head -c 1M </dev/urandom >myfile.txt

  • 1
    As long as each char is <=7F, then you have a UTF-8 character Nov 26, 2015 at 10:16
  • After executing head -c 1M </dev/urandom >myfile.txt, if I open mytext.txt with gedit it says that there are problems with UTF-8 character encoding Nov 26, 2015 at 10:29
  • Sorry, I meant that to get valid UTF-8, you must only have single bytes of <= \x7F or build valid multi-byte UTF-8 sequences. The former is obviously easier from a random perspective. Nov 26, 2015 at 10:39
  • 1
    Do you want it covering \u0 to \U7fffffff or just \u0..\ud7ff+ \ue000..\u10ffff or only those that are currently specified in the latest Unicode spec? Nov 26, 2015 at 11:44

4 Answers 4


If you want UTF-8 encodings of code points 0 to 0x7FFFFFFF (which the UTF-8 encoding algorithm was originally designed to work on):

< /dev/urandom perl -CO -ne '
    no warnings "utf8";
    print chr(unpack("L>",$_) & 0x7fffffff)'

Nowadays, Unicode is restricted to 0..D7FF, E000..10FFFF (though some of those characters are not assigned, some of which will never be (are defined as non-characters)).

< /dev/urandom perl -CO -ne '
    no warnings "utf8";
    $c = unpack("L>","\0$_") * 0x10f800 >> 24;
    $c += 0x800 if $c >= 0xd800;
    print chr($c)'

If you only want assigned characters, you can pipe that to:

uconv -x '[:unassigned:]>;'

Or change that to:

< /dev/urandom perl -CO -ne '
    no warnings "utf8";
    $c = unpack("L>","\0$_") * 0x10f800 >> 24;
    $c += 0x800 if $c >= 0xd800;
    $c = chr $c;
    print $c if $c =~ /\P{unassigned}/'

You may prefer:

             if $c =~ /[\p{Space}\p{Graph}]/ && $c !~ /\p{Co}/

To only get graphical and spacing ones (exclude those from the private-use sections).

Now, to get 1GiB of that, you can pipe it to head -c1G (assuming GNU head), but beware the last character may be cut in the middle.


The most efficient way to create a text file with size 10 MB and UTF-8 character encoding is base64 /dev/urandom | head -c 10000000 | egrep -ao "\w" | tr -d '\n' > file10MB.txt

  • Is the egrep and tr necessary? Nov 26, 2015 at 13:50
  • That doesn't produce a 10MB file. Also, you either need to remove base64 or add a base64 -d somewhere; what you have now only outputs characters used by the Base64 encoding. Nov 26, 2015 at 23:23

Grep for ASCII (sub-set of UTF-8) chars, on Linux/GNU:

dd if=/dev/random bs=1 count=1G | egrep -ao "\w" | tr -d '\n'
  • 1
    base64 might be considerably more efficient.
    – muru
    Nov 26, 2015 at 11:08
  • haha, doh! That's what happens if you over think the question :) You should offer that as an answer :) Nov 26, 2015 at 11:10
  • I'm looking for some way to as many UTF8 characters as I can. If I do, I'll post that as answer. :)
    – muru
    Nov 26, 2015 at 11:11
  • dd if=/dev/random bs=1 count=10M | egrep -ao "\w" | tr -d '\n' > file10MB.txt gets stuck. It is too slow. Nov 26, 2015 at 13:27
  • 1
    @muru you really need to use urandom here, random will get stuck really fast because it needs entropy. This is not fast, but it will get the job done. dd if=/dev/urandom bs=1 count=1G | egrep -ao "\w" | tr -d '\n'
    – theduke
    Jan 8, 2017 at 11:23

If you want non-ASCII characters, then you'll need a way to build valid UTF-8 sequences. The chance that two consecutive bytes yielding a valid UTF-8 is very low.

Instead, this Python script creates random 8 bit values that can be converted in Unicode chars, then written out as UTF-8:

import random
import io

char_count = 0

with io.open("random-utf8.txt", "w", encoding="utf-8") as my_file:

    while char_count <= 1000000 * 1024:
        rand_long = random.getrandbits(8)

        # Ignore control characters
        if rand_long <= 32 or (rand_long <= 0x9F and rand_long > 0x7F):

        unicode_char = unichr(rand_long)
        char_count += 1

You could also change it to use a random 16 bit number which would yield non-latin values.

It's not fast but fairly accurate.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.