How to truncate file to maximum number of characters (not bytes)

Question

How can I truncate a (UTF-8 encoded) text file to given number of characters? I don't care about line lengths and the cut can be in the middle of word.

• cut seems to operate on lines, but I want a whole file.
• head -c uses bytes, not characters.

Answer 1

Some systems have a truncate command that truncates files to a number of bytes (not characters).

I don't know of any that truncate to a number of characters, though you could resort to perl which is installed by default on most systems:

perl

perl -Mopen=locale -ne '
  BEGIN{$/ = \1234} truncate STDIN, tell STDIN; last' <> "$file"

• With -Mopen=locale, we use the locale's notion of what characters are (so in locales using the UTF-8 charset, that's UTF-8 encoded characters). Replace with -CS if you want I/O to be decoded/encoded in UTF-8 regardless of the locale's charset.

• $/ = \1234: we set the record separator to a reference to an integer which is a way to specify records of fixed length (in number of characters).

• then upon reading the first record, we truncate stdin in place (so at the end of the first record) and exit.

GNU sed

With GNU sed, you could do (assuming the file doesn't contain NUL characters or sequences of bytes which don't form valid characters -- both of which should be true of text files):

sed -Ez -i -- 's/^(.{1234}).*/\1/' "$file"

But that's far less efficient, as it reads the file in full and stores it whole in memory, and writes a new copy.

GNU awk

Same with GNU awk:

awk -i /usr/share/awk/inplace.awk -v RS='^$' -e '
  {printf "%s", substr($0, 1, 1234)}' -E /dev/null "$file"

• -e code -E /dev/null "$file" being one way to pass arbitrary file names to gawk
• RS='^$': slurp mode.

Do not use -i inplace as gawk tries to load the inplace extension (as inplace or inplace.awk) from the current working directory first, where someone could have planted malware. The path of the inplace extension supplied with gawk may vary with the system, see the output of gawk 'BEGIN{print ENVIRON["AWKPATH"]}'

Shell builtins

With ksh93, bash or zsh (with shells other than zsh, assuming the content doesn't contain NUL bytes):

content=$(cat < "$file" && echo .) &&
  content=${content%.} &&
  printf %s "${content:0:1234}" > "$file"

With zsh:

read -k1234 -u0 s < $file &&
  printf %s $s > $file

Or:

zmodload zsh/mapfile
mapfile[$file]=${mapfile[$file][1,1234]}

With ksh93 or bash (beware it's bogus for multi-byte characters in several versions of bash):

IFS= read -rN1234 s < "$file" &&
  printf %s "$s" > "$file"

ksh93 can also truncate the file in place instead of rewriting it with its <>; redirection operator:

IFS= read -rN1234 0<>; "$file"

iconv + head

To print the first 1234 characters, another option could be to convert to an encoding with a fixed number of bytes per character like UTF32BE/UCS-4:

iconv -t UCS-4 < "$file" | head -c "$((1234 * 4))" | iconv -f UCS-4

head -c is not standard, but fairly common. A standard equivalent would be dd bs=1 count="$((1234 * 4))" but would be less efficient, as it would read the input and write the output one byte at a time¹. iconv is a standard command but the encoding names are not standardized, so you might find systems without UCS-4

Notes

In any case, though the output would have at most 1234 characters, it may end up not being valid text, as it would possibly end in a non-delimited line.

Also note that while those solutions wouldn't cut text in the middle of a character, they could break it in the middle of a grapheme , like a é expressed as U+0065 U+0301 (a e followed by a combining acute accent), or Hangul syllable graphemes in their decomposed forms.

---

^{¹ and on pipe input you can't use bs values other than 1 reliably unless you use the iflag=fullblock GNU extension, as dd could do short reads if it reads the pipe quicker than iconv fills it}

Answer 2

If you know that the text file contains Unicode encoded as UTF-8 you have to first decode the UTF-8 to get a sequence of Unicode character entities and split those.

I'd choose Python 3.x for the job.

With Python 3.x the function open() has an extra key-word argument encoding= for reading text-files. The description of method io.TextIOBase.read() looks promising.

So using Python 3 it would look like this:

truncated = open('/path/to/file.txt', 'rt', encoding='utf-8').read(1000)

Obviously a real tool would add command-line arguments, error handling etc.

With Python 2.x you could implement your own file-like object and decode the input file line-by-line.

Answer 3

Using Raku (formerly known as Perl6)

Raku provides high-level support for Unicode, built-in. Except for filehandle names, graphemes are converted to the Unicode Consortium's "Normalization Form C" (NFC) by default. Below is an example with Emoticons.

Sample Input:

~$ raku -e 'for (0..8) -> $i { $_.[0..$i].join.put given "\x1F600".."\x1F64F"};' > emoticons_0-to-8.txt
~$ cat emoticons_0-to-8.txt
😀
😀😁
😀😁😂
😀😁😂😃
😀😁😂😃😄
😀😁😂😃😄😅
😀😁😂😃😄😅😆
😀😁😂😃😄😅😆😇
😀😁😂😃😄😅😆😇😈

Read 54 characters¹ from the filehandle using readchars:²

~$ raku -e 'my $file = "emoticons_0-to-8.txt".IO; given $file.open { put .readchars: 54; .close; };'
😀
😀😁
😀😁😂
😀😁😂😃
😀😁😂😃😄
😀😁😂😃😄😅
😀😁😂😃😄😅😆
😀😁😂😃😄😅😆😇
😀😁😂😃😄😅😆😇😈

Reading 54 characters from the filehandle prints out the file including the \n newline on line 9. Reading 55 characters is the same as 54; reading 53 characters from the filehandle prints out the file omitting the \n newline on line 9 (data not shown). Hence we can have some confidence that multibyte characters are being handled correctly.

Presently, Raku cannot truncate files like Perl, nor do -i.bak in-place file editing/backup. Maybe the most robust way to truncate a file in Raku is to write out (i.e. spurt) to a separate _bak backup file, unlink the original if the $bak backup :e exists and is a :f file, and finally copy the backup over the original.

Below we truncate our emoticon file to 52 characters, deleting the final (purple) emoticon. Generalized file names:

~$ raku -e 'given "/path/to/original.txt".IO.open {   \
            my $bak = /path/to/original_bak.txt";     \
            spurt($bak, .readchars(52), createonly => True);  \
            unlink($_) if $bak.IO:e & $bak.IO:f;              \
            copy($bak, $_, createonly => True); .close; };'

Sample Output (switching to zsh which shows the EOF with %):

~$ zsh
~% cat emoticons_0-to-8.txt
😀
😀😁
😀😁😂
😀😁😂😃
😀😁😂😃😄
😀😁😂😃😄😅
😀😁😂😃😄😅😆
😀😁😂😃😄😅😆😇
😀😁😂😃😄😅😆😇%

---

^{¹ Use Gauss' calculation (add \n per line): raku -e 'say (10 * 4.5) + 9;' gives 54.
² Rakudo compiler defaults to a max of 65536 characters for readchars, if no character value is supplied.}

https://docs.raku.org/language/unicode
https://www.unicode.org/reports/tr15/
https://docs.raku.org/type/IO/Path#File_test_operators
https://docs.raku.org/routine/readchars
https://raku.org

Answer 4

I'd like to add another approach. Probably not the best performance wise, and much longer, but easy to understand:

#!/bin/bash

chars="$1"
ifile="$2"
result=$(cat "$ifile")
rcount=$(echo -n "$result" | wc -m)

while [ $rcount -ne $chars ]; do
        result=${result::-1}
        rcount=$(echo -n "$result" | wc -m)
done

echo "$result"

Invoke it with $ ./scriptname <desired chars> <input file>.

This removes the last char one by one until the goal is met, which seems really bad performance wise especially for bigger files. I just wanted to present this as an idea to show more possibilities.