3

I had 4000 text files with unique filenames in a directory.

Is there any Linux command to concatenate only the 1-100 files.

cat 1.txt ... 100.txt > 1.100.txt
cat 101.txt ... 200.txt > 2.200.txt
.......
.......
cat 3901.txt ... 4000.txt > 40.4000.txt

Suggestions please.

2
  • 1
    Please properly define "first" and whether the names are truly simply 1.txt 2.txt 3.txt ...
    – FelixJN
    Jul 1 at 10:09
  • 1
    or... cat {1..100}.txt > 1.100.txt
    – JJoao
    Jul 1 at 17:51

4 Answers 4

5

With zsh:

files=( *(N.n) ) i=0
while (( $#files )) {
  () {cat -- $@ > $((++i)).$@[-1]} $files[1,100]
  files[1,100]=()
}

The n glob qualifier sorts the file names numerically, you can change it to Om to Order by modification time from oldest to newest.

4

You can use head to only pick 100 files.

For example:

cat $(ls -1 --sort=time | head -n 100) > 1.100.txt

(you can change the --sort to something else or remove it to sort by name)

If spaces or newlines might occur in filenames, you can't use ls but find -print0 is safe (uses null as the separator):

find . -type f -print0 | head -z -n 100 | xargs -0 cat > 1.100.txt
2
  • Is it possible to do concatenation by providing the filenames in a text file ?
    – sunnykevin
    Jul 1 at 5:00
  • 1
    Sure, replace the ls -1 | part with cat list.txt |
    – remram
    Jul 1 at 19:10
2

I'm going to caution you to be careful of using the output of ls in any Bash loop as a matter of habit because it can lull you into a false sense of security that it will always work. (Consider what happens with spaces in the filename...) It does work in this case - but there's a better (if somewhat more complicated) way to do it in general if the unique part of the filename is numeric and sequential.

Bash has built-in integer math functionality, and you can use the seq command with it to construct a loop to do what you want:

i=0
while :; do
    (( i += 1 ))
    (( i >= 4000 )) && break

    files_to_catenate=
    for i in $(seq $i $((i + 99))); do
        files_to_catenate+=( "${i}.txt")
    done
    cat "${files_to_catenate[@]}" > "$((i / 100)).${i}.txt"
done
4
  • All the text files has a same prefix OG*********.txt (Ex. OG0017769.txt)
    – sunnykevin
    Jul 1 at 5:14
  • 1
    Provided the unique portion is numeric and sequential, the above will work. If not, the accepted answer is the only realistic way to do it. Whichever you choose is a matter of preference.
    – deriamis
    Jul 1 at 5:21
  • Bash also has range expansion, but it happens after arithmetic expansion or variable expansion. So you need eval echo {$((1 + 2))..9} to print 3 4 5 ... 9, or eval cat foo{$i..$((i+99))}.txt >> "foo$i.concat.txt", with extra quoting for parts that might contain spaces. This would be an alternative to your loop. Perhaps using range-expansion to make an array would be sane. BTW, with files=( $(seq --format=%g.txt $i $((i+99)) ) ) you also wouldn't need a loop. Jul 2 at 2:39
  • Be very careful of eval. It’s not a bad idea in your particular application, but I wouldn’t make it my first idea when solving a problem. In any case, I try not to build very large arrays (in more than the hundreds of elements) in Bash. It’s not great at large arrays. The cost of a loop is much less than the cost of the disk access here anyway.
    – deriamis
    Jul 3 at 4:05
2

This is a bit creepy but: You could run something along the lines of:

for a in {1..40}
do 
  echo cat "{$((a*100-99))..$(($a*100))}.txt > $a.$(($a*100)).txt" 
done

echoing

cat {1..100}.txt > 1.100.txt
cat {101..200}.txt > 2.200.txt
...
cat {3901..4000}.txt > 40.4000.txt

And if you like it: for a in.....done | bash

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.