Tag Info

Hot answers tagged

12

You can just use uniq with its -f option: uniq -f 4 input.txt From man uniq: -f, --skip-fields=N avoid comparing the first N fields Actually this will display the first line: [Fri Oct 31 20:27:05 2014] The Brown Cow Jumped Over The Moon If that is a problem you can do: tac input.txt | uniq -f 4 or if you don't have tac but your tail ...


11

Pure shell implementation: while read -r filename content ; do printf '%s\n' "$content" >> "${filename}.seq" done < /source/file


10

With input: <abc>def< <firstword>anotherword</firstword> <ghi>klm< Use: sed 's/<\([^>]*\)>\(.*\)<$/<\1>\2<\/\1>/' input Output: <abc>def</abc> <firstword>anotherword</firstword> <ghi>klm</ghi> The sed line only affects lines ending in < (because of the ...


10

A version in perl, using negative lookaheads: $ perl -0pe 's/\n(?!([0-9]{8}|$))//g' test.txt 20141101 server contain dump 20141101 server contain nothing {uekdmsam ikdas jwdjamc ksadkek} ssfjddkc * kdlsdlsddsfd jfkdfk 20141101 server contain dump -0 allows the regex to be matched across the entire file, and \n(?!([0-9]{8}|$)) is a negative lookahead, ...


8

Using awk: awk '{printf "%s\n", $2>$1".seq"}' file From the nominated file, print the second field in each record ($2) to a file named after the first field ($1) with .seq appended to the name. As Thor points out in the comments, for a large dataset, you may exhaust the file descriptors, so it would be wise to close each file after writing: awk '{printf ...


8

To select only lines modulo N with awk try awk '!(NR%2)' file or awk 'NR%3==0' file Here NR denotes number of rows processed so far. In your specific case (remove all lines with Y): $ echo 'YYYYYY XXXXXX XXXXXX YYYYYY XXXXXX XXXXXX' | awk '!(NR%3==1)' XXXXXX XXXXXX XXXXXX XXXXXX


8

If you have GNU sed, you could use the n~m (n skip m) address notation sed '1~3d' file which deletes every third line, starting at the first.


7

I've looked at a few approaches. Basically when it comes to serious IO (like n writes to n files), I try and get out of the command line and into a real language. I've benchmarked each with a 10,000 line by 64-char DNA-like numbered grid. The command for generating test data is at the end but here are the benchmarks. me (C++) 0m1.991s me ...


7

x is your awk variable, $x is something different.... $0 corresponds to the input string contents of the line just read, while $1...$n correspond to the breakdown fields of $0 contents using the default separator. awk lets you to programmatically access the fields using a variable, i.e., if the variable MyValue contains 5, $MyValue is the same as $5. What ...


6

$ sed ':again;$!N;$!b again; s/{[^}]*}//g' file This is that wants anyway. Explanation: :again;$!N;$!b again; This reads the whole file into the pattern space. :again is a label. N reads in the next line. $!b again branches back to the again label on the condition that this is not the last line. s/{[^}]*}//g This removes all expressions in braces. ...


5

join fileA fileB That assumes files are sorted on the first column (which they are in your example).


5

awk 'FNR==2 {if (/some string/) print FILENAME; nextfile}' ./* Some awks don't have "nextfile".


5

One way would be: $ perl -lne 's/^/\n/ if $.>1 && /^\d+/; printf "%s",$_' file 20141101 server contain dump 20141101 server contain nothing {uekdmsam ikdas jwdjamc ksadkek} ssfjddkc * kdlsdlsddsfd jfkdfk 20141101 server contain dump However, .that also removes the final newline. To add it again, use: $ { perl -lne 's/^/\n/ if $.>1 ...


5

With awk: awk -F'.' '{print $1}' file -F option change default field separator(space) to dot(.). $1 is index of field position(with . field separator). {ILMN_1343291 TGTGTTGAGAGCTTCTCAGACTATCCACCTTTGGGTCGCTTTGCTGTTCG NM_001402}.{5} ^^ field index is $1 ^^$2 With rev and awk: rev file | awk ...


5

POSIX defined length in awk is a string function, argument taken as a string. Using length with an array as argument is unspecified behavior. In some implementations of awk like gawk (version >= 3.1.6), OS X version of AWK, you can use length with an array as argument, it will return number of elements in array. Array in awk is associative array, looping ...


5

I'm not sure this is better than doing it in memory, but with a sed that reads out its infile for every line in its infile and another on the other side of a pipe alternating Hold space with input lines... cat <<\IN >/tmp/tmp Row1,10 Row2,20 Row3,30 Row4,40 IN </tmp/tmp sed -e 'i\ ' -e 'r /tmp/tmp' | sed -n '/./!n;h;N;/\n$/D;G;s/\n/ /;P;D' ...


4

Use awk to split the fields on the , (-F,) and substitute out the " (gsub): awk -F, '{gsub(/"/,""); print $1,$2}' file Firstname Lastname Alex Johnson Baran Atasoy


4


4

awk '!seen[substr($0,27)]++' file


4

If I correctly understand the question you want to get only values after =, and store the these values in separate files, based on second field(?). If I'm right try something like this: $ awk -F'[ =]' '{print $6>"file_"$2".txt"}' file The result: $ ls -1 file_leftWallPhi.txt file_leftWallUSf.txt file_leftWallrhoPhi.txt file_leftWallrhoUSf.txt ...


4

Assuming the extensions are all-digit: perl -pi -e 's/\.\d+$//' /path/to/file -i does in-place editing (like in sed). \d means digits, and $ denotes the end of the line. With awk: awk 'gsub(/\.[0-9]+$/,"")' /path/to/file gawk has an in-place editing option in newer versions, but I am not sure how portable that is. gsub supports an optional parameter, ...


4

Using awk it is straightforward, just set your field separator as .: awk -F. '{print $1}' file Another approach, using the shell (in this case bash): while IFS=.; read -r lines _; do line+=("$lines"); done <file printf "%s\n" "${line[@]}" ILMN_1343291 ...


3

Try this one: awk -F ']' '{a[$2]=$1}END{for(i in a){print a[i]"]"i}}'


3

Other possibilities: sed sed -n 's/^n\*:\(.*\)/\1/p' grep grep -Po '^n\*:\K.*'


3

You have most of it already. You just need to change your -exec to a -printf find /var/warehouse/* -type f \( -name "*.avi" -o -name "*.mkv" -o -name "*.flv" -o -name "*.mp4" \) -printf "%f\n" %f will print the name of the file that was found, ignoring the path to it. You may also want to consider changing your -name to -iname (case insensitive match) so ...


3

This is really easy to do with sed, actually. You just get as many as you can in one go, then try, try again: sed -e :t -e 's/\(<[^<]*\)[0-9]\{1,\}\([^>]*>\)/\1\2/g;tt' I tried it with the following random bits of input: <Number1> 234234 </Nu994845mb6er8>' 234234 <000000000000000000000000000000000000>> <a1> 2 ...


3

You either need a loop around a substitution command (possible in both sed and perl), or a nested substitution command (perl only). I prefer the latter approach; it's a bit more general: perl -pe 's/\<([^>]*)\>/do{$a = $1; $a =~ s,\d,,g; "\<" . $a . "\>"}/ge;' Example input: <a1> 2 <34b5c> 6 7 def Output: <a> 2 ...


3

Here's a perl approach, essentially the same as Anthon's sed solution: $ perl -pe 's/<(.+)>(.+)</<$1>$2<\/$1>/' file <sss>ss</sss>


3

You could assign multiplay field separator by -F awk -F[\",] '{print $2,$5} Same with sed sed 's/"\|,[^,]\+$//g;s/,/ /' or sed -E 's/"(\w+)","(\w+).*/\1 \2/' Same with tr and cut tr -s ',"' ' ' | cut -d" " -f2-3 At last but not a last: just bash builtin's while IFS=, read a b c do echo "${a//\"/} ${b//\"/}" done Or may be more complicable ...


3

awk 'FNR == NR { onecol[$0]=1;next; }; $1 in onecol' fileB fileA



Only top voted, non community-wiki answers of a minimum length are eligible