Tag Info

Hot answers tagged

12

Here is the processing: a[$0]: look at the value of key $0, in associative array a. If it does not exist, create it. a[$0]++: increment the value of a[$0], return the old value as value of expression. If a[$0] does not exist, return 0 and increment a[$0] to 1 (++ operator returns numeric value). !a[$0]++: negate the value of expression. If a[$0]++ return ...


12

Let's see, !a[$0]++ first a[$0] we look at the value of a[$0] (array a with whole input line ($0) as key). If it does not exist ( ! is negation in test will eval to true) !a[$0] we print the input line $0 (default action). Also, we add one ( ++ ) to a[$0], so next time !a[$0] will evaluate to false. Nice, find!! You should have a look at code ...


8

A few ways: awk awk '!a[$1]++' file This is a very condensed way of writing this: awk '{if(! a[$1]){print; a[$1]++}}' file So, if the current first field ($1) is not in the a array, print the line and add the 1st field to a. Next time we see that field, it will be in the array and so will not be printed. Perl perl -ane '$k{$F[0]}++ or print' file ...


8

awk '{ split($2,a,","); for (i in a) print $1, a[i]; }' input.txt Example: ~) $ echo -e "Main1 one,two,th\nMain2 five,six,seven,eight " | awk '{ split($2,a,","); for (i in a) print $1, a[i]; }' Main1 one Main1 two Main1 th Main2 five Main2 six Main2 seven Main2 eight


8

You can count the unique columns with following pipe: $ awk '{for (i=1; i<=NF; ++i) a[i]=a[i]$i; } END { for (i in a) print a[i] }' foo \ | sort -u | wc -l The awk command transposes your input, the resulting lines are sorted, only unique lines are kept (-u) and at the end all (unique) lines (i.e. the transposed columns) are counted (wc -l). Note ...


6

find / -type f -size +20000k -exec ls -lh {} \; 2> /dev/null | awk '{ print $NF ": " $5 }' | sort -hrk 2,2 /: find starts in main directory -type f: find only regular files -size +20000k: with size 20000k or more -exec ls -lh {} \;: for each file which has been found execute ls -lh, i.e. list them in long format and human readable size (with suffix ...


6

Here are some choices: Shell while read main rest; do for i in ${rest//,/$'\n'}; do echo $main $i done done < file Perl perl -lane 'print "$F[0] $_" for split(/,/,$F[1])' file awk awk -F'[, ]' '{for(i=2;i<=NF;i++){print $1,$i}}' file


6

How about: while read code device state junk; do if [[ $code == $message ]]; then echo "Translated: $device-$state" fi done <CODE-LIST.txt Using extra processes (i.e. forking awk everytime) will slow it a lot. read will read multiple fields, separated by $IFS (default value is all white space). The last variable listed will receive the ...


6

join combines the files (needing sorted inputs): $ join <(sort text.txt) <(sort vendor.xml) 00:10:f6 vendor="micro" 00:10:f6 vendor="micro" 03:48:03 vendor="apple" 8f:91:34 vendor="dell" 93:ab:c6 vendor="sun" So all what's left is to add uniq -c to do the counting: $ join <(sort text.txt) <(sort vendor.xml) | uniq -c 2 00:10:f6 ...


5

Using sed and column: $ sed -E 's/ intron_([^:]*):[^[:space:]]*/ \1/' file | column -t id target_id length eff_length 1 FBgn0000721 1136 243.944268 1 FBgn0000721 1122 240.237419 2 FBgn0264373 56 0 The key part of this is the substitute command: s/ intron_([^:]*):\S*/ \1/ It looks for intron_ and saves everything after intron_ ...


5

$ awk '/SETTINGS START/,/SETTINGS END/ {if (FNR==NR) {a=a"\n"$0}} FNR==NR{next} /SETTINGS START/{print substr(a,2)} /SETTINGS START/,/CUSTOMIZATION SETTINGS END/{next} 1' file2 file1 ANJALI NISHA // +++ CUSTOMIZATION SETTINGS START +++ WELCOME ALL // +++ CUSTOMIZATION SETTINGS END +++ PREETI MONA Explanation awk implicitly loops through files line ...


4

If the first column is always 5 characters long, you can simply use uniq: uniq -w 5 file If not, use awk: awk '$1!=a{print $0; a=$1}' file The first one would definitely be faster with a huge file.


4

$ awk 'FNR==NR{a[$1]=$2;next} ($1 in a) {print $1,a[$1],$2}' file2 file1 aa 45 32 bb 31 15 cc 50 78 Explanation: awk implicitly loops through each file, one line at a time. Since we gave it file2 as the first argument, it is read first. file1 is read second. FNR==NR{a[$1]=$2;next} NR is the number of lines that awk has read so far and FNR is the ...


4

I don't know if this will solve your actual issue. Since you haven't shown us the actual input file, I have no idea if what you are attempting will even work. In any case, to fix the error you're getting, use this: awk '/CREATE DATABASE/{x="F"++i;}(x){print > x;}' file2 That modifies the script to only attempt to print to file x if the variable x is ...


4

Pure awk: $ awk -F'[, ]' 'NR==FNR{n[$2]=$1;next}{m[$3]+=n[$1]} END{for(i in m){print i " " m[i]}}' \ file1 file2 degree1 2 degree2 5 Or you can put it into a script like this: #!/usr/bin/awk -f BEGIN { FS="[, ]" } { if (NR == FNR) { n[$2] = $1; next; } else { m[$3] += n[$1]; } } END { for (i in m) { ...


4

@artm showed a technique where you double-quote the awk script and escape various characters. Here are 3 other techniques Break out of the single quote to let the shell expand the variable usrpid=$(awk '$1 == "'"$USR"'" {print $2}' file) Pass the shell variable into an awk variable usrpid=$(awk -v usr="$USR" '$1 == usr {print $2}' file) If the ...


4

The following example counts the times I am mentioned without needing the lastloggedin file: $ last | awk '$1=="yeti" { ++count } END { print count }' 106 If you insist in using or are forced to use the lastloggedin file, you can do it this way: $ last > lastloggedin $ awk '$1=="yeti" { ++count } END { print count }' lastloggedin 106 Use ...


4

Perl: perl -ne ' BEGIN {$width = 10} while (length) { ($word = substr($_,0,$width)) =~ s/^\s+|\s+$//g; # trim whitespace $word ||= "-"; print $word, " "; substr($_,0,$width) = ""; } print "\n"; ' file


4

Could use awk for the whole thing. Awk is also far quicker than while read loops. awk -vM="$Message" '$1==M{print "Translated:",$2"-"$3}' CODE-LIST.txt Explanation -vM="$Message" - Create a variable named M with the value from $Message $1==M - If the first field is equal to variable M($Message) {print "Translated:",$2"-"$3} - ...


3

You could do this simply through Perl+regex combination. perl -pe 's/(\{(?:[^{}]|(?1))*\})(*SKIP)(*F)|,/|/g' file Example: $ perl -pe 's/(\{(?:[^{}]|(?1))*\})(*SKIP)(*F)|,/|/g' file (999969|2500|"777777888"|0|"45265"|"65522"|NULL|10001|2014-09-15 10:27:07.287|2014-09-15 10:28:49.085|2014-09-15 ...


3

Let's start with a sample input file: $ cat dump.txt some: other email: proxy2@domain.com fullName: LDAP Proxy2 gui: 987 other: thing To extract the desired lines and reformat them as CSV: $ awk '/email:|fullName:|gui:/{s=s", "$0} END{print substr(s,3)}' dump.txt email: proxy2@domain.com, fullName: LDAP Proxy2, gui: 987 How it works: ...


3

$ top -bn1 | grep free KiB Mem: 8117084 total, 6578888 used, 1538196 free, 302216 buffers KiB Swap: 8060924 total, 26004 used, 8034920 free, 1564856 cached $ top -bn1 | grep -oP '\S+(?=\s+free)' 1544132 8034920 requires GNU grep, but you've tagged "linux", so you're OK For just the "Mem": top -bn1 | grep -oP 'Mem.*\s\K\S+(?=\s+free)'


3

The -exec ls -lh {} \; makes find execute the command ls -lh on each file found {}, and '\;' indicates that each file is handed to the command seperately. From the man page: -exec command ; Execute command; true if 0 status is returned. All following arguments to find are taken to be arguments to the command ...


3

Why don't you directly give / as parameter to df -k? df -k / | tail -n +2 This will give the list without the header.


3

(((...))), but how to find identical columns? $ printf '%s\n' '1 0 0 0 0 0' '0 1 1 1 0 0' | awk -vSUBSEP='=' ' { for (i=1; i<NF; i++) for (j=i+1; j<=NF; j++) if ($i==$j) M[i,j]++ } END{ for (m in M) if (M[m]==NR) print m }' 5=6 2=3 2=4 3=4 For all columns i<j of each row, increment M[i,j] whenever the ...


3

awk has tow modes of invocation one is with a program text on the command line and the other with a program from a file. This is stated in the synopsis of the awk man page (this one from mawk on Ubunut 12.04): mawk [-W option] [-F value] [-v var=value] [--] 'program text' [file ...] mawk [-W option] [-F value] [-v var=value] [-f program-file] ...


3

Strictly speaking, when you call awk, you refer to the interpreter, not the language. The language is called AWK. awk (or mawk, nawk) is only utility, which will execute programs written in AWK programming language. As POSIX defined, awk program is: program If no -f option is specified, the first operand to awk shall be the text of the awk ...


3

The details of this will depend on how variable your input is. If we can assume that JOURNEY is unchanging and that the numbers you want added to it will never be more or less than two characters (01-99), this will work: perl -pe 's/^.(\d+) ## ignore the first character and capture ## as many digits as possible after it. ...


3

In python, start with the input file as an argument: import sys res = [] # list of lists for line in open(sys.argv[1]): try: x, y = line.split() # split on space except ValueError: line = line.rstrip() x, y = line.split(',') # retry with comma for l in res: if x in l: if y not in l: ...


3

For general purpose tabular alignment, you want the column utility. For example: ( printf 'PID\tUSER\tAPPNAME\n' printf '%s\t%s\t%s\n' "1" "john" "foo bar" printf '%s\t%s\t%s\n' "12345678" "someone_with_a_long_name" "pop tart" ) | column -t -s $'\t' Results in: PID USER APPNAME 1 john foo bar ...



Only top voted, non community-wiki answers of a minimum length are eligible