16

How can I truncate a (UTF-8 encoded) text file to given number of characters? I don't care about line lengths and the cut can be in the middle of word.

  • cut seems to operate on lines, but I want a whole file.
  • head -c uses bytes, not characters.
3
  • Note that the GNU implementation of cut still doesn't support multi-byte characters. If it did, you could do cut -zc-1234 | tr -d '\0'. Jul 31, 2018 at 9:43
  • 1
    How do you want to handle emojis? Some are more that one character... stackoverflow.com/questions/51502486/…
    – phuzi
    Jul 31, 2018 at 9:57
  • 3
    What's a character? some symbols use several code points,
    – Jasen
    Jul 31, 2018 at 11:16

4 Answers 4

16

Some systems have a truncate command that truncates files to a number of bytes (not characters).

I don't know of any that truncate to a number of characters, though you could resort to perl which is installed by default on most systems:

perl

perl -Mopen=locale -ne '
  BEGIN{$/ = \1234} truncate STDIN, tell STDIN; last' <> "$file"
  • With -Mopen=locale, we use the locale's notion of what characters are (so in locales using the UTF-8 charset, that's UTF-8 encoded characters). Replace with -CS if you want I/O to be decoded/encoded in UTF-8 regardless of the locale's charset.

  • $/ = \1234: we set the record separator to a reference to an integer which is a way to specify records of fixed length (in number of characters).

  • then upon reading the first record, we truncate stdin in place (so at the end of the first record) and exit.

GNU sed

With GNU sed, you could do (assuming the file doesn't contain NUL characters or sequences of bytes which don't form valid characters -- both of which should be true of text files):

sed -Ez -i -- 's/^(.{1234}).*/\1/' "$file"

But that's far less efficient, as it reads the file in full and stores it whole in memory, and writes a new copy.

GNU awk

Same with GNU awk:

awk -i /usr/share/awk/inplace.awk -v RS='^$' -e '
  {printf "%s", substr($0, 1, 1234)}' -E /dev/null "$file"
  • -e code -E /dev/null "$file" being one way to pass arbitrary file names to gawk
  • RS='^$': slurp mode.

Do not use -i inplace as gawk tries to load the inplace extension (as inplace or inplace.awk) from the current working directory first, where someone could have planted malware. The path of the inplace extension supplied with gawk may vary with the system, see the output of gawk 'BEGIN{print ENVIRON["AWKPATH"]}'

Shell builtins

With ksh93, bash or zsh (with shells other than zsh, assuming the content doesn't contain NUL bytes):

content=$(cat < "$file" && echo .) &&
  content=${content%.} &&
  printf %s "${content:0:1234}" > "$file"

With zsh:

read -k1234 -u0 s < $file &&
  printf %s $s > $file

Or:

zmodload zsh/mapfile
mapfile[$file]=${mapfile[$file][1,1234]}

With ksh93 or bash (beware it's bogus for multi-byte characters in several versions of bash):

IFS= read -rN1234 s < "$file" &&
  printf %s "$s" > "$file"

ksh93 can also truncate the file in place instead of rewriting it with its <>; redirection operator:

IFS= read -rN1234 0<>; "$file"

iconv + head

To print the first 1234 characters, another option could be to convert to an encoding with a fixed number of bytes per character like UTF32BE/UCS-4:

iconv -t UCS-4 < "$file" | head -c "$((1234 * 4))" | iconv -f UCS-4

head -c is not standard, but fairly common. A standard equivalent would be dd bs=1 count="$((1234 * 4))" but would be less efficient, as it would read the input and write the output one byte at a timeΒΉ. iconv is a standard command but the encoding names are not standardized, so you might find systems without UCS-4

Notes

In any case, though the output would have at most 1234 characters, it may end up not being valid text, as it would possibly end in a non-delimited line.

Also note that while those solutions wouldn't cut text in the middle of a character, they could break it in the middle of a grapheme , like a Γ© expressed as U+0065 U+0301 (a e followed by a combining acute accent), or Hangul syllable graphemes in their decomposed forms.


ΒΉ and on pipe input you can't use bs values other than 1 reliably unless you use the iflag=fullblock GNU extension, as dd could do short reads if it reads the pipe quicker than iconv fills it

3
  • could do dd bs=1234 count=4
    – Jasen
    Jul 31, 2018 at 11:12
  • 2
    @Jasen, that would not be reliable. See edit. Jul 31, 2018 at 11:17
  • Wow! you would be handy to have nearby! I thought I knew a lot of handy Unix commands but this is an incredible list of great options. Jul 31, 2018 at 18:12
5

If you know that the text file contains Unicode encoded as UTF-8 you have to first decode the UTF-8 to get a sequence of Unicode character entities and split those.

I'd choose Python 3.x for the job.

With Python 3.x the function open() has an extra key-word argument encoding= for reading text-files. The description of method io.TextIOBase.read() looks promising.

So using Python 3 it would look like this:

truncated = open('/path/to/file.txt', 'rt', encoding='utf-8').read(1000)

Obviously a real tool would add command-line arguments, error handling etc.

With Python 2.x you could implement your own file-like object and decode the input file line-by-line.

4
  • Yeah, i could do that. But it's for CI build machines, so I'd like to rarther use some standard Linux command.
    – Pitel
    Jul 31, 2018 at 7:07
  • 5
    Whatever "standard Linux" means on your Linux flavour... Jul 31, 2018 at 7:21
  • 1
    Indeed, Python, some version of it anyway, is pretty standard these days.
    – muru
    Jul 31, 2018 at 8:02
  • I've already edited my answer with snippet for Python 3 which can explicitly process text-files. Jul 31, 2018 at 13:04
0

Using Raku (formerly known as Perl6)

Raku provides high-level support for Unicode, built-in. Except for filehandle names, graphemes are converted to the Unicode Consortium's "Normalization Form C" (NFC) by default. Below is an example with Emoticons.

Sample Input:

~$ raku -e 'for (0..8) -> $i { $_.[0..$i].join.put given "\x1F600".."\x1F64F"};' > emoticons_0-to-8.txt
~$ cat emoticons_0-to-8.txt
πŸ˜€
πŸ˜€πŸ˜
πŸ˜€πŸ˜πŸ˜‚
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒ
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…πŸ˜†
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…πŸ˜†πŸ˜‡
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…πŸ˜†πŸ˜‡πŸ˜ˆ

Read 54 charactersΒΉ from the filehandle using readchars:Β²

~$ raku -e 'my $file = "emoticons_0-to-8.txt".IO; given $file.open { put .readchars: 54; .close; };'
πŸ˜€
πŸ˜€πŸ˜
πŸ˜€πŸ˜πŸ˜‚
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒ
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…πŸ˜†
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…πŸ˜†πŸ˜‡
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…πŸ˜†πŸ˜‡πŸ˜ˆ

Reading 54 characters from the filehandle prints out the file including the \n newline on line 9. Reading 55 characters is the same as 54; reading 53 characters from the filehandle prints out the file omitting the \n newline on line 9 (data not shown). Hence we can have some confidence that multibyte characters are being handled correctly.

Presently, Raku cannot truncate files like Perl, nor do -i.bak in-place file editing/backup. Maybe the most robust way to truncate a file in Raku is to write out (i.e. spurt) to a separate _bak backup file, unlink the original if the $bak backup :e exists and is a :f file, and finally copy the backup over the original.

Below we truncate our emoticon file to 52 characters, deleting the final (purple) emoticon. Generalized file names:

~$ raku -e 'given "/path/to/original.txt".IO.open {   \
            my $bak = /path/to/original_bak.txt";     \
            spurt($bak, .readchars(52), createonly => True);  \
            unlink($_) if $bak.IO:e & $bak.IO:f;              \
            copy($bak, $_, createonly => True); .close; };'

Sample Output (switching to zsh which shows the EOF with %):

~$ zsh
~% cat emoticons_0-to-8.txt
πŸ˜€
πŸ˜€πŸ˜
πŸ˜€πŸ˜πŸ˜‚
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒ
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…πŸ˜†
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…πŸ˜†πŸ˜‡
πŸ˜€πŸ˜πŸ˜‚πŸ˜ƒπŸ˜„πŸ˜…πŸ˜†πŸ˜‡%

ΒΉ Use Gauss' calculation (add \n per line): raku -e 'say (10 * 4.5) + 9;' gives 54.
Β² Rakudo compiler defaults to a max of 65536 characters for readchars, if no character value is supplied.

https://docs.raku.org/language/unicode
https://www.unicode.org/reports/tr15/
https://docs.raku.org/type/IO/Path#File_test_operators
https://docs.raku.org/routine/readchars
https://raku.org

-1

I'd like to add another approach. Probably not the best performance wise, and much longer, but easy to understand:

#!/bin/bash

chars="$1"
ifile="$2"
result=$(cat "$ifile")
rcount=$(echo -n "$result" | wc -m)

while [ $rcount -ne $chars ]; do
        result=${result::-1}
        rcount=$(echo -n "$result" | wc -m)
done

echo "$result"

Invoke it with $ ./scriptname <desired chars> <input file>.

This removes the last char one by one until the goal is met, which seems really bad performance wise especially for bigger files. I just wanted to present this as an idea to show more possibilities.

3
  • Yeah, this is definitely horrible for performance. For a file of length n, wc counts on the order of O(n^2) total bytes for a target point half way into the file. It should be possible to binary-search instead of linear search by using a variable that you increase or decrease, like echo -n "${result::-$chop}" | wc -m or something. (And while you're at it, make it safe even if the file contents start with -e or something, maybe using printf). But you still won't beat methods that only look at each input character once, so probably not worth it. Aug 1, 2018 at 1:19
  • You're definitely right, more of a technical answer rather than a practical answer. You could also reverse it to add char by char to the $result until it matches the desired length, but if desired length is a high number it's just as inefficient.
    – confetti
    Aug 1, 2018 at 1:24
  • 1
    You could start close to the right place by starting with $desired_chars bytes at the low end, or maybe 4*$desired_chars at the high end. But still I think it's best to use something else entirely. Aug 1, 2018 at 1:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .