Tag Info

New answers tagged

1

In Perl: $ perl -F',' -lane 'push @{$k{$F[0]}},@F[1..$#F]; END{$,=",";print $_,@{$k{$_}} for keys(%k)}' file 2,apple 1,cat,dog 3,human Or, for sorted output: $ perl -F',' -lane 'push @{$k{$F[0]}},@F[1..$#F]; END{$,=",";print $_,@{$k{$_}} for sort keys(%k)}' file 1,cat,dog 2,apple 3,human This has the ...


5

Assuming that the first column is strictly ordered: $ awk -F, '$1==last {printf ",%s",$2;next} NR>1{print""} {last=$1;printf "%s",$0} END{print""}' file 1,cat,dog 2,apple 3,human Alternatively, allowing the input lines in any order (and output lines in no guaranteed order): $ awk -F, '{a[$1]=a[$1]","$2} END{for (i in a)print i a[i]}' file 1,cat,dog ...


4

I prefer the variant of not changing the existing data , but adding the sort criteria as new column, and removing that auxiliary sorting field at the end of the pipe: awk -F, 'BEGIN {u["kg"]=1000; u["g"]=1}; {print $1*u[$2], $0}' file | sort -n | cut -d" " -f2-


2

You might be better off converting the units in the file, sorting them and using the resulting stored file. sed -r 's/^([0-9]+),kg/\1000,g/' $file | sort -n sed doesn't understand math, so if you have non-intigers you'll have to use something else. The following does the fast parse with sed, but uses bc to do actual math if needed. sed -r ...


4

If your file is too large to hold in memory, you could do: $ awk -F, -v OFS="," '$2=="kg"{$1=1000*$1}1;' file | sort -n | awk -F, -v OFS="," '$2=="kg"{$1=$1/1000}1;' 1000,g,dog 1,kg,cat 20,g,apple


1

It is not possible by only using grep. You have to use another tool e.g. sort: $ grep -e apple -e mango *.txt | sort -t: -k2,2 1.txt:apple 3.txt:apple 2.txt:mango 4.txt:mango


6

You need to specify where sort keys end. Otherwise they end at the end of the line. And to apply numeric sort to one key only, that's with n appended to the key spec. -n alone would turn numeric sort on globally. sort -t'|' -k1,1 -k2,2n


0

$ ls log101.gz log102.gz log103.gz log104.gz log105.gz log106.gz log10.gz log1.gz $ ls | sort -t . -n -k1.4 log1.gz log10.gz log101.gz log102.gz log103.gz log104.gz log105.gz log106.gz


4

I would do this in Perl: $ perl -lane '$k{"$F[0] $F[1]"}+=$F[2]+$F[3]; END{print "$_ $k{$_}" for keys(%k) }' file 2 1019 15 2 1021 4 2 1030 6 2 1031 8 2 1022 9 Or awk: awk '{a[$1" "$2]+=$3+$4}END{for (i in a){print i,a[i]}}' file If you want the output sorted according to the second column you could just pipe to sort: awk '{a[$1" ...


1

You could pre-sort the data and let awk handle the details: sort -n infile | awk 'NR>1 && p!=$2 {print p,s} {s+=$3+$4} {p=$2}' Output: 1019 15 1021 19 1022 28 1030 34 If you really want the keep the first column, do something like this: sort -n infile | awk 'NR>1 && p!=$1FS$2 {print p,s} {s+=$3+$4} {p=$1FS$2}' Output: 2 1019 ...


4

Maybe this could help, but is column 1 always 2 and does results depend on it? awk '{ map[$2] += $3 + $4; } END { for (i in map) { print "2", i, map[i] | "sort -t't'" } }' file or as mentioned by glenn jackman in comments about sorting: gawk '{ map[$2] += $3 + $4; } END { PROCINFO["sorted_in"] = "@ind_str_asc"; for (i in map) { print 2, i, map[i] } }' ...


3

As you have specified (but consider wurtel's comment on your question): generate_keys A > B # adjust this call however that program is defined to work awk ' NR==FNR { a[NR]=$1 ; next } !b[$1]++ { print a[FNR] } ' A B


2

$ getkeys A > B $ sort B | uniq -c | awk '{if($1 == 1) print $2}' > C $ paste B A | fgrep -f C | cut -f2- Explanation: First generate the keys for each line. Then count the number of times each key occurs, and save those keys that occur once in file C. Then join the keys with the lines using paste, match those lines with the list of unique keys ...


2

can't test without some sample data, but something like this: paste <(generate-keys "$filename") "$filename" | awk '! seen[$1]++ {print $2}'


0

I found a way on OpenBSD: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/sort.1?query=sort&sec=1 -H Use a merge sort instead of a radix sort. This option should be used for files larger than 60MB. But this isn't a fully OK solution, since it takes too much space... x>100 GByte isn't enough for it..


0

(Sorry, cannot post comments yet.) Are you sorting a single multi gig file? Is it a text file? How wide is the average line? The problem might be line width, or it may be how many lines, or a combination of the two. Can you post a test file I can use? Or how can I accurately make a test file that resembles yours? Segfault might be due to the fact that ...


2

Combining the different answers: When you want to sort the file in pieces, try using split: LARGETMP=/var/tmp mkdir ${LARGETMP} N_LINES=100000 # Adjust when to still too large or too small split --lines=${N_LINES} bigfile splitted_ for small in splitted*; do sort -u -T ${LARGETMP} ${small} > sorted_${small} rm ${small} done echo "Done with sorting ...


2

Another possibility would be to sort each of the files separately and then merge them: for f in *txt; do sort -u "$f" -T tmp/ > "$f".sorted done sort -mu *sorted The -m option causes sort to merge already sorted files instead of attempting to sort them. That should result in a far lower memory usage and should avoid the segfault.


1

The problem is that the individual files are unsorted, i.e. if you used something like sort -u file* > sortedFile, sort would have to load the contents of all files and then sort them. I assume this is inefficient given that you probably do not have more than 120 gigs of ram. I would suggest that you first sort all files individually, and the merge them ...


2

In the past, I've had to sort files that are too big for sort. I assume this is also your problem, though if you provide more information, perhaps we can better diagnose your issue. The solution to my issue was to break up the file with grep as a pre-processor. Take a look at your data to see where the clumps will be. I'm assuming it is decently spread ...


1

Sort will need a harddisk location to store temporary data. /tmp is not the best place. The sort command stores working data in temporary disk files. You can use the flag -T to specify a large temp dir (on a partition with a lot of free space).


5

Similar to Archemar's suggestion, you could script this with ed: printf %s\\n ${linenr}m${addr} w q | ed -s infile i.e. linenr # is the line number m # command that moves the line addr=$(( linenr + 1 )) # if you move the line down addr=$(( linenr - 2 )) # if you move the line up w ...


1

there is a vi command called move m you can use vi in text mode : ex $line_number=7 $line_up=$(($line_number + 1 )) (echo ${line_number}m${line_up} ; echo wq ) | ex foo where foo is your file


1

Assuming your keys don't contain colons or newlines and your values don't contain newlines: for key in "${!MYARRAY[@]}"; do printf '%s:%s\n' "$key" "${MYARRAY[$key]}" done | sort -t : -k 2n If you need to arrange the key in the order given by the values, read back the output: IFS=$'\n'; set -f sorted_keys=($( for key in "${!MYARRAY[@]}"; do ...



Top 50 recent answers are included