13

How can I truncate a (UTF-8 encoded) text file to given number of characters? I don't care about line lengths and the cut can be in the middle of word.

  • cut seems to operate on lines, but I want a whole file.
  • head -c uses bytes, not characters.
  • Note that the GNU implementation of cut still doesn't support multi-byte characters. If it did, you could do cut -zc-1234 | tr -d '\0'. – Stéphane Chazelas Jul 31 '18 at 9:43
  • How do you want to handle emojis? Some are more that one character... stackoverflow.com/questions/51502486/… – phuzi Jul 31 '18 at 9:57
  • 2
    What's a character? some symbols use several code points, – Jasen Jul 31 '18 at 11:16
14

Some systems have a truncate command that truncates files to a number of bytes (not characters).

I don't know of any that truncate to a number of characters, though you could resort to perl which is installed by default on most systems:

perl

perl -Mopen=locale -ne '
  BEGIN{$/ = \1234} truncate STDIN, tell STDIN; last' <> "$file"
  • With -Mopen=locale, we use the locale's notion of what characters are (so in locales using the UTF-8 charset, that's UTF-8 encoded characters). Replace with -CS if you want I/O to be decoded/encoded in UTF-8 regardless of the locale's charset.

  • $/ = \1234: we set the record separator to a reference to an integer which is a way to specify records of fixed length (in number of characters).

  • then upon reading the first record, we truncate stdin in place (so at the end of the first record) and exit.

GNU sed

With GNU sed, you could do (assuming the file doesn't contain NUL characters or sequences of bytes which don't form valid characters -- both of which should be true of text files):

sed -Ez -i -- 's/^(.{1234}).*/\1/' "$file"

But that's far less efficient, as it reads the file in full and stores it whole in memory, and writes a new copy.

GNU awk

Same with GNU awk:

awk -i inplace -v RS='^$' -e '{printf "%s", substr($0, 1, 1234)}' -E /dev/null "$file"
  • -e code -E /dev/null "$file" being one way to pass arbitrary file names to gawk
  • RS='^$': slurp mode.

Shell builtins

With ksh93, bash or zsh (with shells other than zsh, assuming the content doesn't contain NUL bytes):

content=$(cat < "$file" && echo .) &&
  content=${content%.} &&
  printf %s "${content:0:1234}" > "$file"

With zsh:

read -k1234 -u0 s < $file &&
  printf %s $s > $file

Or:

zmodload zsh/mapfile
mapfile[$file]=${mapfile[$file][1,1234]}

With ksh93 or bash (beware it's bogus for multi-byte characters in several versions of bash):

IFS= read -rN1234 s < "$file" &&
  printf %s "$s" > "$file"

ksh93 can also truncate the file in place instead of rewriting it with its <>; redirection operator:

IFS= read -rN1234 0<>; "$file"

iconv + head

To print the first 1234 characters, another option could be to convert to an encoding with a fixed number of bytes per character like UTF32BE/UCS-4:

iconv -t UCS-4 < "$file" | head -c "$((1234 * 4))" | iconv -f UCS-4

head -c is not standard, but fairly common. A standard equivalent would be dd bs=1 count="$((1234 * 4))" but would be less efficient, as it would read the input and write the output one byte at a time¹. iconv is a standard command but the encoding names are not standardized, so you might find systems without UCS-4

Notes

In any case, though the output would have at most 1234 characters, it may end up not being valid text, as it would possibly end in a non-delimited line.

Also note that while while those solutions wouldn't cut text in the middle of a character, they could break it in the middle of a grapheme , like a é expressed as U+0065 U+0301 (a e followed by a combining acute accent), or Hangul syllable graphemes in their decomposed forms.


¹ and on pipe input you can't use bs values other than 1 reliably unless you use the iflag=fullblock GNU extension, as dd could do short reads if it reads the pipe quicker than iconv fills it

  • could do dd bs=1234 count=4 – Jasen Jul 31 '18 at 11:12
  • 2
    @Jasen, that would not be reliable. See edit. – Stéphane Chazelas Jul 31 '18 at 11:17
  • Wow! you would be handy to have nearby! I thought I knew a lot of handy Unix commands but this is an incredible list of great options. – Mark Stewart Jul 31 '18 at 18:12
5

If you know that the text file contains Unicode encoded as UTF-8 you have to first decode the UTF-8 to get a sequence of Unicode character entities and split those.

I'd choose Python 3.x for the job.

With Python 3.x the function open() has an extra key-word argument encoding= for reading text-files. The description of method io.TextIOBase.read() looks promising.

So using Python 3 it would look like this:

truncated = open('/path/to/file.txt', 'rt', encoding='utf-8').read(1000)

Obviously a real tool would add command-line arguments, error handling etc.

With Python 2.x you could implement your own file-like object and decode the input file line-by-line.

  • Yeah, i could do that. But it's for CI build machines, so I'd like to rarther use some standard Linux command. – Pitel Jul 31 '18 at 7:07
  • 5
    Whatever "standard Linux" means on your Linux flavour... – Michael Ströder Jul 31 '18 at 7:21
  • 1
    Indeed, Python, some version of it anyway, is pretty standard these days. – muru Jul 31 '18 at 8:02
  • I've already edited my answer with snippet for Python 3 which can explicitly process text-files. – Michael Ströder Jul 31 '18 at 13:04
0

I'd like to add another approach. Probably not the best performance wise, and much longer, but easy to understand:

#!/bin/bash

chars="$1"
ifile="$2"
result=$(cat "$ifile")
rcount=$(echo -n "$result" | wc -m)

while [ $rcount -ne $chars ]; do
        result=${result::-1}
        rcount=$(echo -n "$result" | wc -m)
done

echo "$result"

Invoke it with $ ./scriptname <desired chars> <input file>.

This removes the last char one by one until the goal is met, which seems really bad performance wise especially for bigger files. I just wanted to present this as an idea to show more possibilities.

  • Yeah, this is definitely horrible for performance. For a file of length n, wc counts on the order of O(n^2) total bytes for a target point half way into the file. It should be possible to binary-search instead of linear search by using a variable that you increase or decrease, like echo -n "${result::-$chop}" | wc -m or something. (And while you're at it, make it safe even if the file contents start with -e or something, maybe using printf). But you still won't beat methods that only look at each input character once, so probably not worth it. – Peter Cordes Aug 1 '18 at 1:19
  • You're definitely right, more of a technical answer rather than a practical answer. You could also reverse it to add char by char to the $result until it matches the desired length, but if desired length is a high number it's just as inefficient. – confetti Aug 1 '18 at 1:24
  • 1
    You could start close to the right place by starting with $desired_chars bytes at the low end, or maybe 4*$desired_chars at the high end. But still I think it's best to use something else entirely. – Peter Cordes Aug 1 '18 at 1:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.