For example, we have N files (file1, file2, file3 ...)
We need first 20% of them, the result directory should be like (file1_20, file2_20, file3_20 ...).
I was thinking use wc to get the lines of the file, then times 0.2
Then use head to get 20% and then redirect to a new file, but i don't know how to automate it.
So creating a single example to work from:
root@crunchbang-ibm3:~# echo {0..100} > file1
root@crunchbang-ibm3:~# cat file1
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100
We can grab the size of the file in bytes with stat:
root@crunchbang-ibm3:~# stat --printf %s "file1"
294
And then using bc we can multipy the size by .2
root@crunchbang-ibm3:~# echo "294*.2" | bc
58.8
However we get a float so lets convert it to an integer for head ( dd might work here too ):
root@crunchbang-ibm3:~# printf %.0f "58.8"
59
And finally the first twenty percent (give or take a byte) of file1:
root@crunchbang-ibm3:~# head -c "59" "file1"
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Putting it together we could then do something like this
mkdir -p a_new_directory
for f in file*; do
file_size=$(stat --printf %s "$f")
percent_size_as_float=$(echo "$file_size*.2" | bc)
float_to_int=$(printf %.0f "$percent_size_as_float")
grab_twenty=$(head -c "$float_to_int" "$f")
new_fn=$(printf "%s_20" "$f") # new name file1_20
printf "$grab_twenty" > a_new_directory/$new_fn
done
where f is a place holder for any items found in the directory in which the for loop is run that matches file*
which when done:
root@crunchbang-ibm3:~# cat a_new_directory/file1_20
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
To grab the first approximate 20% of lines we could replace stat --printf %s "$f" with:
wc -l < "$f"
Since we are using printf and bc we can effectively round up from .5, however if a file is only 1 or 2 lines long it will miss them. So we would want to not only round up, but default to at least grabbing 1 line.
Dang. I wrote this whole big answer with an elaborate method of parsing a tar archive - it was cool. But I got to the end and I realized none of it was necessary at all. All you need is sed and a little shell math:
set ./file[1-5];i=1 n=;eval "${n:=
} sed -n \"$(grep -c '.\|' "$@"|
sed 's|\(.*\):\(.*\)|\
$i,$(((\2/5)+(i+=\2)-\2))w \1|
')\" <<!$n"'$(cat "$@")'"$n!$n"
There grep -c counts lines in whatever files you've globbed - I globbed file[1-5] - and hands the count to sed which then - with a little help from the shell - writes its own script. cat provides the input via here-document. This is is because I'm iffy on what might happen if sed does an open on and starts writing to one of the files cat is trying to read out to it - also I suspect it would be a little better at handling buffers than a pipe would be depending on size - but I'm not too clear on that part.
So that reads all the files in a single stream and writes the output accordingly. A little setup is neccessary to increment the file numbers correctly - hence grep and eval - nothing terrible. Here's some set -x output to show what it is doing:
+ set ./file1 ./file2 ./file3 ./file4 ./file5
+ i=1 n=
+ + grep -c .\| ./file1 ./file2 ./file3 ./file4 ./file5
sed s|\(.*\):\(.*\)|\
$i,$(((\2/5)+(i+=\2)-\2))w \1|
+ eval
sed -n "
$i,$(((18400/5)+(i+=18400)-18400))w ./file1
$i,$(((18411/5)+(i+=18411)-18411))w ./file2
$i,$(((18415/5)+(i+=18415)-18415))w ./file3
$i,$(((18418/5)+(i+=18418)-18418))w ./file4
$i,$(((18421/5)+(i+=18421)-18421))w ./file5" <<!
$(cat "$@")
!
+ cat ./file1 ./file2 ./file3 ./file4 ./file5
+ sed -n
1,3681w ./file1
18401,22083w ./file2
36812,40495w ./file3
55227,58910w ./file4
73645,77329w ./file5
As you can see, the lines are addressed based on each file's position in the stream, and are written as they are read to their respective filenames. Importantly though, this makes no attempt to handle any non-portable characters in a pathname - in particular, newlines in path names are a non-starter in this case as the sed write command delimits filename arguments on newlines. The situation is easily worked around if necessary with ln if you require it though.
I should also mention that there is a limit to the number of write file descriptors sed can support in a single script. The spec says:
[
sedis required] to support at least ten distinctwfiles, matching historical practice on many implementations. Implementations are encouraged to support more, but conforming applications should not exceed this limit.
So the command as written above should be portable to any POSIX system for up to 10 concurrent read/write files. If this sort of thing were incorporated into a published script or app in which more might be wanted it could be worth running a few checks before processing real data in /tmp. Like:
: & set '"" "" "" "" "" "" "" "" "" "" ';n='
' f=/tmp/$$$!'_$((i+=1))' MAXw=[num]
while eval "set '$1$1' $1;exec <<!$n\$(((i=0)+\$#))$n!$n
i=\$(sed \"$(IFS=\ ;printf "\nw $f%.0s" $1)\")"
[ "$(($#==i?(_i=i-1):(MAXw=_i)))" -lt "$MAXw" ]
do :;done; rm "/tmp/$$$!"*; unset _i i f n
...which should fairly portably guage sed's capability in that area. GNU sed stalled out at 4093 concurrently opened wfiles for me in about a second, but that is likely my system's max, and can be affected w/ ulimit as well. When it was through - because the check doubles $i's value for each try - $_i was left at 2560 and $i at 5120. I default to setting $MAXw to the safer $_i above at loop close - mostly because I'm unsure if all seds will properly set their return if they cannot open a wfile - but the reader can do with it what they will.
Note that the initial [num] value for $MAXw should be an actual number - whatever your maximum wanted wfiles might be - and not literally [num].
About the here-document again - I consider it - or something like it - a good idea in this case. sed must maintain its write descriptors while it reads and so what it might do with identical in/out names I do not know - but I don't guess it is a chance worth taking when alternatives are so readily available to us.
My test files were generated like:
for n in 1 2 3 4 5
do : & seq -s "$(printf "%015s--$n--%015s\n\t")" "$!" >"file$n"
done
...which gets fairly sequential pseudo-random numbers from the kernel in abandoned process PIDs. The file contents were purposely designed to indicate a mismatch in the split. Here's what a sample set looks like before and after:
Before:
for f in file[1-5]; do
nl -ba "$f" | sed -n '$p;$=;1,3p
'; done
1 1 --1--
2 2 --1--
3 3 --1--
3681 3681 --1--
3681
1 1 --2--
2 2 --2--
3 3 --2--
3683 3683 --2--
3683
1 1 --3--
2 2 --3--
3 3 --3--
3684 3684 --3--
3684
1 1 --4--
2 2 --4--
3 3 --4--
3684 3684 --4--
3684
1 1 --5--
2 2 --5--
3 3 --5--
3685 3685 --5--
3685
If the formatting looks a little funky this is probably because seq doesn't insert the -separator string before the first output line. The important thing is that sed, seq and nl all appear to agree on the line numbers. Anyway...
After: ...
sed -n
1,737w ./file1
3682,4418w ./file2
7365,8101w ./file3
11049,11785w ./file4
14733,15470w ./file5
...
1 1 --1--
2 2 --1--
3 3 --1--
737 737 --1--
737
1 1 --2--
2 2 --2--
3 3 --2--
737 737 --2--
737
1 1 --3--
2 2 --3--
3 3 --3--
737 737 --3--
737
1 1 --4--
2 2 --4--
3 3 --4--
737 737 --4--
737
1 1 --5--
2 2 --5--
3 3 --5--
738 738 --5--
738
And there it is - simple, efficient, and streamed.
Using the tools you mentioned + find:
get percentage of lines or bytes1 with head -n perc file or head -c perc file,
where perc is given by (( count / 5 )),
where count is given by wc -l < file or wc -c < file,
finally, write the output to corresponding file_20.
Note: the / operator rounds down to the nearest integer so any file* with lines/bytes count < 5 (hence perc = 0) will produce an empty file*_20 file.
get first 20% - lines:
mkdir some_dir_name
find . -maxdepth 1 -iname 'file*' -exec sh -c 'head -n $(( $(wc -l < "$0") / 5 )) "$0" > some_dir_name/"$0"_20' {} \;
get first 20% - bytes:
mkdir some_dir_name
find . -maxdepth 1 -iname 'file*' -exec sh -c 'head -c $(( $(wc -c < "$0") / 5 )) "$0" > some_dir_name/"$0"_20' {} \;
1
Note that, depending on text layout, the two methods may yield significantly different results, e.g. for a 10-line text sample:
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua.
Abstract
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi
ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum...
first 20% of the total number of lines = first 2 lines:
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua.
first 20% of the total number of bytes = first line (truncated):
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
