New answers tagged

1

I cannot speak for vendor specific implementations, but the UNIX sort implementation splits large files into smaller files, sorts these files and then combines the sorted smaller files into an aggregated sorted output. The only limitation is the disk space for the smaller files created intermediately by sort, but the files can be redirected to an arbitrary ...


33

The sort that you find on Linux comes from the coreutils package and implements an External R-Way merge. It splits up the data into chunks that it can handle in memory, stores them on disc and then merges them. The chunks are done in parallel, if the machine has the processors for that. So if there was to be a limit, it is the free disc space that sort can ...


0

Based on https://blog.mafr.de/2010/05/23/sorting-large-files/ and http://unix.stackexchange.com/a/88704/9689 : split -n l/20 input input- for inpf in input-* ; do sort --parallel="$(nproc --all)" "${inpf}" > sorted-"{$inpf}" done sort -m sorted-input-* > sorted-input Update: From answers above we see that sort already does what mentioned ...


0

Looks like your work can be reduced to: perl -e ' use POSIX qw(strftime); my ($start, $num) = (30_000_000, 10_000); my @all = ( $start .. $start + $num -1 ); # 10,000 numbers my @odds = grep { $_ % 2 == 1 } @all; my @evens = grep { $_ % 2 == 0 } @all; my $filename = strftime "%Y%m%d", localtime; my $fh; open $fh, ...


1

Following up on @cas's comment to another answer: six_months=$(date -d "6 months ago" "+%Y%m%d") for f in *.Txt; do file_date=${f%.Txt} [[ $file_date > $six_months ]] && echo "$f" done | xargs awk 'FNR > 1 {print $3}' | sort -u > unique_ids_in_last_6_months The for loop prints out the "eligible" filenames. xargs passes the ...


1

Use awk to do the work, and SQLite for the dates. sqlite3 <<< "select date('2016-04-20', '-6 month');" 2015-10-20 Dandy, ain't it? awk has string functions to insert/delete the hyphens SQLite needs and, yea, splits on tab delimiters. awk 'NR == 1 {next}; { IDS[$3]++ }; END {for (K in IDS) {print K}}' ids 123456 345678 234567 Guaranteed ...


2

It is not trivial to find the exact date 6 months ago, especially if the current date would be the 31st of some month. But if you know how to do this with find and -mtime, I would just touch the files depending on the date in their name: for x in *.Txt; do dd=${x%.Txt} touch -t "$dd"0000 "$x" done and then use the mtime


4

Not reproduced here? $ shuf -i1-10000000 > t.in $ sort -S50M -T. t.in --compress-program=lzop # ^z $ file sort* | tee >(wc -l) > >(grep -v lzop) 7 $ fg # ^c $ sort --version | head -n1 sort (GNU coreutils) 8.25 What I'm guessing the issue is, is due to failure to fork() the compression process due to the large mem size, and then falling ...


0

Here is an example: split("delta alfa charlie bravo", ech) for (fox in ech) { gol = ech[fox] hot = fox - 1 while (hot && ech[hot] > gol) { ech[hot+1] = ech[hot] hot-- } ech[hot+1] = gol } http://rosettacode.org/wiki/Sorting_algorithms/Insertion_sort#AWK


6

You want -o auto: join -t, -j 1 -a 1 -a 2 -o auto john jane From man join: -o FORMAT obey FORMAT while constructing output line If FORMAT is the keyword 'auto', then the first line of each file determines the number of fields output for each line. Or better explained from GNU Coreutils: join invocation: ‘-o auto’ If the keyword ...


4

You could explicitly specify the output format LC_ALL=C join -o0,1.2,2.2 -j1 -a1 -a2 -t',' john jane which produces apple,green,red banana,,yellow cherry,red,yellow kiwi,,green orange,orange, The key thing here is that the join field can also be referenced within the output format using 0, which is quite useful in the context of unpairable lines


1

This command almost does it; it omits a trailing comma if the key only appears in file1. Don't have time to fully debug now: awk -F, 'BEGIN{OFS=","} FNR==NR{val[$1]=$2;next} {val[$1]=val[$1] "," $2}END{for (key in val) {print key, val[key]}}' john jane Output: apple,green,red banana,,yellow cherry,red,yellow kiwi,,green orange,orange


0

If you have GNU or FreeBSD sort, you can use the -V or --version-sort option, after first using sed to swap the date format (and then sed again to change the date format back): ls -1 | sed -E -e 's/^(.*\.)(..)(..)(..)(.*)$/\1\4\3\2\5/' | sort -V | sed -E -e 's/^(.*\.)(..)(..)(..)(.*)$/\1\4\3\2\5/' Ideally, you should just rename the files ...


0

The following piped sequence uses sed to first change file names that are in the format *.DDMMYY.* to the format *|DD|MM|YY|*. The reformatted output is piped to sort where the '|' is used as the field separator and sorted first by YY (-k4n), then by MM (-k3n), and finally by DD (-k2n). Then, the sorted output is piped back into sed where the filename is ...


7

With tail -f, the sort will keep waiting, because it wants to sort the whole file, which will never happen. If you want to remove all duplicate lines ever (so there's just one occurance), you'll need something more sophisticated. If you just want to remove duplicates that are one after another, leave off the sort and just do tail -f | uniq.


1

Here's an abuse of bash arrays; it splits the timestamp apart and creates array entries based on the YYMMDD order, then prints the array back out in order. declare -a array for file in foo.*.bar do [[ $file =~ foo.([[:digit:]]{2})([[:digit:]]{2})([[:digit:]]{2}).bar ]] && \ { index="${BASH_REMATCH[3]}${BASH_REMATCH[2]}${BASH_REMATCH[1]}" ...



Top 50 recent answers are included