4

For example, we have N files (file1, file2, file3 ...)

We need first 20% of them, the result directory should be like (file1_20, file2_20, file3_20 ...).

I was thinking use wc to get the lines of the file, then times 0.2

Then use head to get 20% and then redirect to a new file, but i don't know how to automate it.

  • Do you need the first 20% of bytes, or the 20% of lines? – dotancohen Dec 6 '14 at 12:55
  • @dotancohen 20% of lines – Wilbeibi Dec 6 '14 at 14:06
6

So creating a single example to work from:

root@crunchbang-ibm3:~# echo {0..100} > file1        
root@crunchbang-ibm3:~# cat file1
    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

We can grab the size of the file in bytes with stat:

root@crunchbang-ibm3:~# stat --printf %s "file1"
294

And then using bc we can multipy the size by .2

root@crunchbang-ibm3:~# echo "294*.2" | bc
58.8

However we get a float so lets convert it to an integer for head ( dd might work here too ):

root@crunchbang-ibm3:~# printf %.0f "58.8" 
59

And finally the first twenty percent (give or take a byte) of file1:

root@crunchbang-ibm3:~# head -c "59" "file1" 
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Putting it together we could then do something like this

mkdir -p a_new_directory
for f in file*; do
    file_size=$(stat --printf %s "$f")
    percent_size_as_float=$(echo "$file_size*.2" | bc)
    float_to_int=$(printf %.0f "$percent_size_as_float")
    grab_twenty=$(head -c "$float_to_int" "$f")
    new_fn=$(printf "%s_20" "$f") # new name file1_20
    printf "$grab_twenty" > a_new_directory/$new_fn
done

where f is a place holder for any items found in the directory in which the for loop is run that matches file*

which when done:

root@crunchbang-ibm3:~# cat a_new_directory/file1_20
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 

update (to grab 20% of lines):

To grab the first approximate 20% of lines we could replace stat --printf %s "$f" with:

wc -l < "$f"

Since we are using printf and bc we can effectively round up from .5, however if a file is only 1 or 2 lines long it will miss them. So we would want to not only round up, but default to at least grabbing 1 line.

  • Note that the OP stated in a comment to the question (after this terrific answer was posted) that he need 20% of the lines, not 20% of the bytes. – dotancohen Dec 6 '14 at 14:15
  • @dotancohen Updated the answer. Hope it helps. – jmunsch Dec 6 '14 at 16:45
5

Dang. I wrote this whole big answer with an elaborate method of parsing a tar archive - it was cool. But I got to the end and I realized none of it was necessary at all. All you need is sed and a little shell math:

set ./file[1-5];i=1 n=;eval "${n:=
}       sed -n  \"$(grep -c '.\|' "$@"|
        sed 's|\(.*\):\(.*\)|\
        $i,$(((\2/5)+(i+=\2)-\2))w \1|
        ')\" <<!$n"'$(cat "$@")'"$n!$n"

There grep -c counts lines in whatever files you've globbed - I globbed file[1-5] - and hands the count to sed which then - with a little help from the shell - writes its own script. cat provides the input via here-document. This is is because I'm iffy on what might happen if sed does an open on and starts writing to one of the files cat is trying to read out to it - also I suspect it would be a little better at handling buffers than a pipe would be depending on size - but I'm not too clear on that part.

So that reads all the files in a single stream and writes the output accordingly. A little setup is neccessary to increment the file numbers correctly - hence grep and eval - nothing terrible. Here's some set -x output to show what it is doing:

+ set ./file1 ./file2 ./file3 ./file4 ./file5
+ i=1 n=
+ + grep -c .\| ./file1 ./file2 ./file3 ./file4 ./file5
        sed s|\(.*\):\(.*\)|\
        $i,$(((\2/5)+(i+=\2)-\2))w \1|

+ eval 
       sed -n  "
        $i,$(((18400/5)+(i+=18400)-18400))w ./file1

        $i,$(((18411/5)+(i+=18411)-18411))w ./file2

        $i,$(((18415/5)+(i+=18415)-18415))w ./file3

        $i,$(((18418/5)+(i+=18418)-18418))w ./file4

        $i,$(((18421/5)+(i+=18421)-18421))w ./file5" <<!
$(cat "$@")
!

+ cat ./file1 ./file2 ./file3 ./file4 ./file5
+ sed -n 
        1,3681w ./file1

        18401,22083w ./file2

        36812,40495w ./file3

        55227,58910w ./file4

        73645,77329w ./file5

As you can see, the lines are addressed based on each file's position in the stream, and are written as they are read to their respective filenames. Importantly though, this makes no attempt to handle any non-portable characters in a pathname - in particular, newlines in path names are a non-starter in this case as the sed write command delimits filename arguments on newlines. The situation is easily worked around if necessary with ln if you require it though.

I should also mention that there is a limit to the number of write file descriptors sed can support in a single script. The spec says:

[sed is required] to support at least ten distinct wfiles, matching historical practice on many implementations. Implementations are encouraged to support more, but conforming applications should not exceed this limit.

So the command as written above should be portable to any POSIX system for up to 10 concurrent read/write files. If this sort of thing were incorporated into a published script or app in which more might be wanted it could be worth running a few checks before processing real data in /tmp. Like:

: & set '"" "" "" "" "" "" "" "" "" "" ';n='
' f=/tmp/$$$!'_$((i+=1))' MAXw=[num]
while eval "set '$1$1' $1;exec <<!$n\$(((i=0)+\$#))$n!$n 
      i=\$(sed \"$(IFS=\ ;printf "\nw $f%.0s" $1)\")"
      [ "$(($#==i?(_i=i-1):(MAXw=_i)))" -lt "$MAXw" ]
do :;done; rm "/tmp/$$$!"*; unset _i i f n

...which should fairly portably guage sed's capability in that area. GNU sed stalled out at 4093 concurrently opened wfiles for me in about a second, but that is likely my system's max, and can be affected w/ ulimit as well. When it was through - because the check doubles $i's value for each try - $_i was left at 2560 and $i at 5120. I default to setting $MAXw to the safer $_i above at loop close - mostly because I'm unsure if all seds will properly set their return if they cannot open a wfile - but the reader can do with it what they will.

Note that the initial [num] value for $MAXw should be an actual number - whatever your maximum wanted wfiles might be - and not literally [num].

About the here-document again - I consider it - or something like it - a good idea in this case. sed must maintain its write descriptors while it reads and so what it might do with identical in/out names I do not know - but I don't guess it is a chance worth taking when alternatives are so readily available to us.

My test files were generated like:

for n in 1 2 3 4 5
do : & seq -s "$(printf "%015s--$n--%015s\n\t")" "$!" >"file$n"
done

...which gets fairly sequential pseudo-random numbers from the kernel in abandoned process PIDs. The file contents were purposely designed to indicate a mismatch in the split. Here's what a sample set looks like before and after:

Before:

for f in file[1-5]; do
nl -ba "$f" | sed -n '$p;$=;1,3p
'; done

     1  1               --1--             
     2          2               --1--     
     3          3               --1--     
  3681          3681               --1--  
3681
     1  1               --2--             
     2          2               --2--     
     3          3               --2--     
  3683          3683               --2--  
3683
     1  1               --3--             
     2          2               --3--     
     3          3               --3--     
  3684          3684               --3--  
3684
     1  1               --4--             
     2          2               --4--     
     3          3               --4--     
  3684          3684               --4--  
3684
     1  1               --5--             
     2          2               --5--     
     3          3               --5--     
  3685          3685               --5--  
3685

If the formatting looks a little funky this is probably because seq doesn't insert the -separator string before the first output line. The important thing is that sed, seq and nl all appear to agree on the line numbers. Anyway...

After: ...

  sed -n 
  1,737w ./file1

  3682,4418w ./file2

  7365,8101w ./file3

  11049,11785w ./file4

  14733,15470w ./file5
  ...
     1  1               --1--           
     2          2               --1--   
     3          3               --1--   
   737          737               --1-- 
737
     1  1               --2--           
     2          2               --2--   
     3          3               --2--   
   737          737               --2-- 
737
     1  1               --3--           
     2          2               --3--   
     3          3               --3--   
   737          737               --3-- 
737
     1  1               --4--           
     2          2               --4--   
     3          3               --4--   
   737          737               --4-- 
737
     1  1               --5--           
     2          2               --5--   
     3          3               --5--   
   738          738               --5-- 
738

And there it is - simple, efficient, and streamed.

4

Using the tools you mentioned + find:
get percentage of lines or bytes1 with head -n perc file or head -c perc file,
where perc is given by (( count / 5 )),
where count is given by wc -l < file or wc -c < file,
finally, write the output to corresponding file_20.

Note: the / operator rounds down to the nearest integer so any file* with lines/bytes count < 5 (hence perc = 0) will produce an empty file*_20 file.

get first 20% - lines:

mkdir some_dir_name
find . -maxdepth 1 -iname 'file*' -exec sh -c 'head -n $(( $(wc -l < "$0") / 5 )) "$0" > some_dir_name/"$0"_20' {} \;

get first 20% - bytes:

mkdir some_dir_name
find . -maxdepth 1 -iname 'file*' -exec sh -c 'head -c $(( $(wc -c < "$0") / 5 )) "$0" > some_dir_name/"$0"_20' {} \;

1
Note that, depending on text layout, the two methods may yield significantly different results, e.g. for a 10-line text sample:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua.


Abstract

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi
ut aliquip ex ea commodo consequat. 

Duis aute irure dolor in reprehenderit in voluptate velit esse cillum...

first 20% of the total number of lines = first 2 lines:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua.

first 20% of the total number of bytes = first line (truncated):

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.