Tag Info

Hot answers tagged

8

The availability of parsers in nearly every programming language is one of the advantages of JSON as a data-interchange format. Rather than trying to implement a JSON parser, you are likely better off using either a tool built for JSON parsing such as jq or a general purpose script language that has a JSON library. For example, using jq, you could pull out ...


8

You can use awk for this: $ awk -F'[]]|[[]' \ '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 } $0 ~ /^\[/ && $2 >= "2014-04-08 02:00" { p=0 } p { print $0 }' log Where: -F specifies the characters [ and ] as field separators using a regular expression $0 references a complete line $2 ...


7

You don't need a temporary file, indeed. awk -F'|' '{gsub("[ \t\r]", "", $0)}{print $1,tolower($2$3$4),$5}' OFS='|' inputfile The first part of the expression does what you do using tr. The second part conbines fields 2-4 and lowercases those and prints it with the other two fields. For your input, it'd produce: 45008657|a.long|0 49210987|a.steven|3262 ...


7

It sounds like you're asking for a solution to the bin-packing problem. So far as we know, there isn't a way to do it both optimally and quickly (but we don't know there isn't one, either; its an open question). But you can get close. One utility to do it is called datapacker. Haven't used that one; found it by searching. I'm sure there are more. ...


5

awk ' NR==FNR {vals[$1] = $2 " " $3 " " $4; next} !($1 in vals) {vals[$1] = "0 0 0"} {$(NF+1) = vals[$1]; print} ' file2 file1 TABLES REF-IO HEAD-IO DIFF-IO REF-SELECT HEAD-SELECT DIFF-SELECT test 200 500 -300 5 7 -2 exam 2 3 -1 0 7 -7 final 2 1 1 12 6 6 mail 4 2 2 0 0 0 TOTAL 208 506 -298 20 23 -3


5

Using awk: awk '{print $NF}' inputfile For your sample input, it'd produce: ant tiger rabbit lion monkey donkey Using GNU grep: grep -oP '\S+$' inputfile Using sed: sed 's/.* //' inputfile Using perl: perl -pe 's/.* //' inputfile


5

All you need is a simple grep: grep -Fwf gene_list.txt gene_info.txt The options used are: -w : Search for whole words, this ensures that the gene name ERK1 will not match the gene ERK12 (-w is not a standard option but is fairly common) -f : Read the patterns to be searched for from a file. In this case gene_list.txt. -F : Treat the patterns as ...


5

You could use perl. Straight from the faq -- quoting from perldoc perlfaq6: How do I substitute case-insensitively on the LHS while preserving case on the RHS? Here's a lovely Perlish solution by Larry Rosler. It exploits properties of bitwise xor on ASCII strings. $_= "this is a TEsT case"; $old = 'test'; $new = ...


4

From an example on the awk wiki: awk 'FNR==NR {arr[$0];next} $6 in arr' gene_list info_list Explanation copied from the wiki entry: FNR == NR: This test is true when the number of records is equal to the number of records in the file. This is only true for the first file, for the second file NR will be equal to the number of lines of file1 + FNR. ...


4

With your original attempt, you can put all the commands in the same pipeline if you use the sponge utility: cat Results.txt | tr -d " \t\r" | awk 'BEGIN { FS = "|" } ; { print $1"|"tolower($2) tolower($3) tolower($4)"|"$5 }' | sponge Results.txt sponge will store its stdin in memory until it reaches the end of input, then write to file. This ...


4

The simplistic approach would be to just use sed and replace the string with the correct one. So, if the current time is 2014-04-02 21:34:13 you could simply run sed 's/2014-03-31 13:56:01/2014-04-02 21:34:13/' file Since, presumably, you want to do this dynamically, you can pass sed the result of the relevant date call instead: sed "s/2014-03-31 ...


4

Yes. Using bash script : #!/bin/bash groupsize=0 groupcnt=0 cat unix_StackExchange_question.txt | while read fsize fname do [ "$groupsize" == "0" ] && echo "Group : GROUP_$groupcnt" echo -en "\t $fname\n" ((groupsize+=$fsize)) #cp $fname GROUP_$groupcnt if [ "$groupsize" -gt "5000000" ] then ...


4

Check out dategrep at https://github.com/mdom/dategrep Description: dategrep searches the named input files for lines matching a date range and prints them to stdout. If dategrep works on a seekable file, it can do a binary search to find the first and last line to print pretty efficiently. dategrep can also read from stdin if one the filename ...


4

If you don't mind the order of output: $ awk -F',' 'NF>1{a[$1] = a[$1]","$2}END{for(i in a){print i""a[i]}}' file jkl, words and numbers, words and numbers, words and numbers, words and numbers abc, 12345, 56345, 15475, 123345 ghi, something else, something else, something else, something else def, text and nos, text and nos, text and nos, text and nos ...


4

You can use awk for this: awk '/^2009/{a=$0;next}{print a" "$0} ' file.txt This will only prepend a space to any lines before the first match of 2009, you can set a default string to prepend like this: awk 'BEGIN{a="My default prepend string";}/^2009/{a=$0;next}{print a" "$0} ' file.txt


4

If you trim the replace to pqrstu, try this: Input: abcdef Abcdef AbCdEf ABcDeF Ouput: $ perl -lpe 's/$_/$_^lc($_)^"pqrstu"/ei' file pqrstu Pqrstu PqRsTu PQrStU If you want replace with prstuvxyz, may be this: $ perl -lne '@c=unpack("(A4)*",$_); $_ =~ s/$_/$_^lc($_)^"pqrstu"/ei; $c[0] =~ s/$c[0]/$c[0]^lc($c[0])^"vxyz"/ei; print $_,$c[0]' ...


4

Portable solution using awk: awk -v find=abcdef -v rep=pqrstu '{ lwr=tolower($0) offset=index(lwr, tolower(find)) if( offset > 0 ) { printf "%s", substr($0, 0, offset) len=length(find) for( i=0; i<len; i++ ) { out=substr(rep, i+1, 1) if( substr($0, offset+i, 1) == substr(lwr, offset+i, 1) ) printf "%s", ...


4

Portable solution using sed: sed ' :1 /[aA][bB][cC][dD][eE][fF]/!b s//\ &\ pqrstu\ PQRSTU\ /;:2 s/\n[[:lower:]]\(.*\n\)\(.\)\(.*\n\).\(.*\n\)/\2\ \1\3\4/;s/\n[^[:lower:]]\(.*\n\).\(.*\n\)\(.\)\(.*\n\)/\3\ \1\2\4/;t2 s/\n.*\n//;b1' It's a bit easier with GNU sed: search=abcdef replace=pqrstuvwx sed -r ":1;/$search/I!b;s//\n&\n$replace\n/;:2 ...


4

Here is an awk solution: $ awk 'FNR==NR{a[$1];next}($1 in a){++a[$1]} END{for(i in a){print i" "a[i]}}' file1.txt 1.txt 2.txt put 1 get 1 move 2 Explanation FNR==NR{a[$1];next}: when processing file1.txt, we mark the occurence of $1 in associative array a. ($1 in a){++a[$1]}: when processing 1.txt and 2.txt, we check if $1 is existed in associative ...


3

awk -v sizelimit=5000000 -v outputfilename=shorter_list \ 'BEGIN {target=outputfilename ".0000"}; '\ '{sum+=$1; '\ 'if(sum>sizelimit) { file_index++; target=outputfilename "." sprintf("%04d",file_index); sum=$1;}; '\ 'print $0 >target}' file should do what you want. You have to adapt the size limit, though. I used a smaller value for testing (which ...


3

Here is awk code to skip only the initial comments and then print $1 on the remaining lines: gawk -F: -v c=1 '/^[^#]/ {c=0} c==0 { print $1 }' ourlog Before the program starts, the variable c is set to 1. As soon as a non-comment line is found, c is set to zero and it stays that way for the rest of the execution. When c==0, the print statement is ...


3

This should work with bash: #!/bin/bash accumulated_size=0 file_counter=0 out_file=file_0000 while read size name; do if [ "$accumulated_size" -gt 5368709120 ]; then (( file_counter += 1 )) out_file=$(printf 'file_%04d' "$file_counter") echo -n >"out_file" accumulated_size=0 fi echo "$size $name" >>"$out_file" (( ...


3

You can maybe do: awk '$10 ~ /^[0-9a-zA-Z]+$/ {count++} END{print count}' file This also works to me: to check from 0 to z, being all numbers and letters in the range. awk '$10 ~ /^[0-z]+$/ {count++} END{print count}' file The idea is to check if the given field ($10 in your case) is just based on a set of alphanumeric characters. This is accomplished ...


3

The answer is to use the --use-lc-numeric gawk option. --use-lc-numeric This forces gawk to use the locale's decimal point character when parsing input data. Although the POSIX standard requires this behavior, and gawk does so when --posix is in effect, the default is to follow traditional behavior and use a period as the decimal point, ...


3

To introduce a blank line in a file at the same position as changes in actors in file1.txt, try: $ awk -F '|' '{save=$0; getline<"file1.txt"} NR>1 && $2!=prev {print ""} {prev=$2; print save}' file2.txt 1234|oneflewovercuckoosnest^asgoodasitgets 5678|theshining 4321|batmanbegins^darkknight The above works quite a bit like your code except ...


3

awk ' NR==FNR { a[$1,$2] = $3 val[$1,$2] = $4 next } ($1,$2) in a { for (n=$3-2; n<=$3+2; n++) if (a[$1,$2] == n) { $4+=val[$1,$2] break } } {print} ' file2 file1 AT1 AT22 24 2 ATAGATA ATTATAT AT2 AT24 22 0 ATAGATA ATTATAT AT3 AT23 263 4 ...


3

A little shorter Perl solution: perl -pe 'print "\n" if ($l =~ /name\d+/ && $_ !~ /$&/);$l=$_;' input if last line ($l) was name\d+ and current line not last match, then print new line assign current line to $l A more general solution perl -pe 'print "\n" if ($l =~ /\|([^\|]+)/ && $_ !~ /$1/);$l=$_;' input


3

Here's a case insensitive Perl solution: perl -F'\|' -lape 'unless($F[1]=~/^$pre$/i || $.==1){print ""}; $pre=$F[1]' file Explanation: The -a splits the input lines into the @F array, making perl act like awk. The -F is the field separator -p means print each input line -l adds a \n to each print call, so print "" prints a newline. ...


3

POSIXly: sed ':1 s/^\(.*|\)\{0,1\}\([0-9.]*[0-9][0-9.]*\)-\(|.*\)\{0,1\}$/\1-\2\3/ t1' Or with awk in a more intuitive way: awk -F '|' -v 'OFS=|' '{ for (i = 1; i <= NF; i++) if ($i ~ /^[0-9]*(\.[0-9]+)?-$/) $i = "-" substr($i, 1, length($i)-1) print}' (that version is stricter in what it considers to be a number, the sed ...


3

This is not a problem which should be solved with regex (although technically you could find a way to do it). Since you are going to be using this more than once, you might want to use some real datetime functions rather than just parsing strings. The following Awk script may work for you. # Expects time strings in the form of `[2014-04-25 15:38:23]` with ...



Only top voted, non community-wiki answers of a minimum length are eligible