Tag Info

New answers tagged

0

Everyone has posted awesome awk answers, which I like very much. A variation to @cuonglm replacing grep with sed: sed 's/[^0-9]//g' example.log | paste -sd'+' - | bc The sed strips everything except for the numbers. The paste -sd+ - command joins all the lines together as a single line The bc evaluates the expression


2

Another GNU awk one: awk -vRS='[0-9]+' '{n+=RT};END{print n}' A perl one: perl -lne'$n+=$_ for/\d+/g}{print$n' A POSIX one: tr -cs 0-9 '[\n*]' | grep . | paste -sd+ - | bc


1

sed 's/=/ /' file | awk '{ sum+=$2 } END { print sum}'


4

POSIXly: $ grep -o '[[:digit:]]*' file | paste -sd'+' | bc 784


3

You can try this: awk -F"[^0-9]+" '{ sum += $2 } END { print sum+0; }' file


6

With a newer version (4.x) of GNU awk: awk 'BEGIN {FPAT="[0-9]+"}{s+=$1}END{print s}' With other awks try: awk -F '[a-z=]*' '{s+=$2}END{print s}'


1

Based on my proposal from the other thread: awk ' BEGIN { srand(); do a[int(100*rand()+1)]; while (length(a)<10) } NR in a ' ~/orig.txt > ~/short.txt this could be changed to create both files: awk ' BEGIN { srand(); do a[int(100*rand()+1)]; while (length(a)<10) } NR in a { print > "short.txt" } !(NR in a) { print > "rest.txt" ...


0

I believe you can do everything on the command line, but some problems are better solved with an actual programming language. As an example, a python based solution to your problem would be: import random import pprint with open("file.txt", "w") as f: # create a file filled with numbers from 00 to 99 f.writelines(map(lambda x: "%02d\n" % x, ...


1

There are many ways of doing this. The simplest are probably awk and perl: GNU gawk. If you have a relatively recent version of gawk installed, you can do inline editing like this: gawk -i inplace -F"|" -vOFS="|" '$2=="Orange"{$2="Orange_Edit"}1;' file The -i inplace tells gawk to edit the file inplace, the -F"|" sets the field separator to | and the ...


1

If that string is in a shell variable, and that shell is zsh, you can do: string='5 S comment=whatever Unspecific text fruit=apple animal=mouse sky=blue' new_string=${string/(#b)(comment=*)(fruit*)/$match[2] $match[1]} If that shell is ksh93: new_strong=${string//@(comment=*)@(fruit*)/\2 \1} If that shell is a POSIX shell (like those two, bash or the ...


2

You can use perl: $ perl -i.bak -F'\|' -aple 's/Orange/Orange_Edit/ if $F[1] eq "Orange"' file or safer version: $ perl -i.bak -F'\|' -aple '$F[1]=~s/^Orange$/Orange_Edit/; $_=join "|",@F' file


2

You could do it in 2 steps with awk like awk '{$2="VAL";print}' input_file > temp_file mv temp_file input_file That will split the input file on white space, substitute the value in the second column (use any other number than 2 for different column) so it will now be "VAL". Unless you change it with OFS the output field delimiter will be a space. If ...


1

i=0 { paste res.? res.?? res.??? while paste ./res."$((i+=1))"[0-9][0-9][0-9] do :; done; } >outfile I don't think this is as complicated as all that - you've already done the hard work by ordering the filenames. Just don't open all of them at the same time, is all. Another way: pst() if shift "$1" then paste "$@" fi set ...


3

awk '{$1=$1;print}' or shorter: awk '{$1=$1};1' Would trim leading and trailing space or tab characters and also squeeze sequences of tabs and spaces into a single space. That works because when you assign something to one of the fields, awk rebuilds the whole record (as printed by print) by joining all fields ($1, ..., $NF) with OFS (space by ...


2

sed -i 's/\r/\n/g' thefile.txt "classic" macos used \r as the end-of-line character. *nix uses \n


1

Given the amount of files, line sizes, etc. involved, I think it will surpasses the default sizes of the tools (awk, sed, paste, *, etc) I would create a small program for this, it would not have 10,000 files open, nor a line of hundred of thousands in length (10,000 files of 10 (max size of line in the example)). It only requires an ~10,000 array of ...


1

You didn't ask for ruby, but this is pretty fun: ruby -rcsv -e ' columns = CSV.read(ARGV.shift, :col_sep => "\t").transpose trimmed = columns.select {|col| col if col[1..-1].find {|elem| not elem.nil?}} trimmed.transpose.each {|row| puts row.join("\t")} ' file produces col1 col2 col4 col6 1 2 3 43 1 3 ...


2

Perl solution, suitable for smaller files that can fit into memory. For larger files, you'd need to process the file twice. I assumed the input files are tab separated. The %empty hash keeps the list of indices of the empty columns. If a column is populated, it's removed from the hash. So, once the whole file is saved into the @array, the %empty contains ...


0

In practice I'd probably use Aet3miirah's answer most of the time and alexey's answer is wonderful when wanting to navigate through the lines (also, it also works with less). OTOH, I really like another approach (which is kind of the reversed Gilles' answer: sed -n '/dog 123 4335/,$p' When called with the -n flag, sed does not print by default the lines ...


8

If chaos' answer isn't applicable (because you don't have the required permissions), you can batch up the paste calls as follows: ls -1 res.* | split -l 1000 -d - lists for list in lists*; do paste $(cat $list) > merge${list##lists}; done paste merge* > final.res This lists the files 1000 at a time in files named lists00, lists01 etc., then pastes ...


10

If you have root permissions on that machine you can temporarily increase the "maximum number of open file descriptors" limit: ulimit -Hn 10240 # The hard limit ulimit -Sn 10240 # The soft limit And then paste res.* >final.res After that you can set it back to the original values. A second solution, if you cannot change the limit: for f in ...


2

Try to execute it on this way: ls res.*|xargs paste >final.res You can also split the batch in parts and try something like: paste `echo res.{1..100}` >final.100 paste `echo res.{101..200}` >final.200 ... and at the end combine final files paste final.* >final.res


0

The option -i in sed isn't quite compatible between Linux and Unix and it may not be available on other operating systems. Secondly you can't use -i and IO-redirection (>) together, because it's excludes it-self. Using in-place editing already modifying the file. So if you'd like to edit file in-place (into the same file), it's better to use ex: ex ...


1

There's nothing wrong with using a temporary variable: variables are made to store data to be used several times. In this case, you can combine the two grep/sed calls into a single call to GNU grep, sed or awk. It's easy with sed: pass the -n option to output only explicitly printed lines, and print the lines on which a substitution is made. echo "$( ...


1

Here is a pure awk solution: awk ' NR<=5 { next } { for(i=1; i<=NF; i++) print $i > "File"++c} NF==2 { c=0 } ' It skippes five lines of header, prints the fields on the remaining data in files with names defined by a counter, which is reset on lines with only two fields.


0

Here's another sed plus tr plus printf: tr -s '[:space:]' \\n <infile | sed "/^[0-9]/,${$( printf '\nw file%b\nn' \ 1 2 3 4 5 6 7 8 '9\n}\c' )" I ran it on my clipboard as piped out by xsel in /tmp after copying your example to the buffer and afterwards did... head -n3 /tmp/file? ...which printed... ==> /tmp/file1 <== 1 2 3 ...


0

Can be done in bare awk, but first what comes to my mind is sed+awk combo: sed '1,5d;s/^\([[:digit:]]\)/\n\1/' file | awk 'BEGIN{RS="\n\n"}{for(i=1;i<=NF;i++){print $i > "file"i".txt"}}' The point is to add (with sed) new line to all lines which start from digit, and after that redefine record separator in awk.


1

Here's a one liner of the iw command that have the same output as you. iw dev wlp1s0 link | grep 'SSID:\|signal' | awk '{printf "%s ", $2$3}' My output: ZyXEL-AP-2,4GHz -46dBm


0

Something like that ? iw dev wlp1s0 link | grep -E '^\s*(SSID|signal):\s' | sed -r 's/^\s*(SSID|signal):\s//' | awk '{printf $0}' grep will accept (SSID|signal): so it will match both SSID: and signal:. -E is optional, but if you don't use it, think about escaping special meaning character. Here, it would be \(SSID\|signal\): The same regex can be used ...


0

You could use awk: echo $( iw dev wlp1s0 | awk '/(SSID|signal): /{$1 = ""; print;}' )


-1

echo -e 'one two three\nfour five six\nseven eight nine' |awk -F " " '{print $NF }'


0

Probably the simplest command is sed '1s/^[^|]*|[^|]*|//g' input > output This sed command will only on line 1 do the substitution. For this input: foo|bar|baz|doo|bee|doo 123|456 789|101|112 it will produce that output: baz|doo|bee|doo 123|456 789|101|112


0

This will cut out the first 2 pipe-delimited fields of line 1, and copy the rest of the file (line 2 onward) without changes. (head -n 1 input | cut -d '|' -f 3- ; tail -n +2 input) > output A slightly more efficient but syntactically trickier way to do it is: { head -n 1 input | cut -d '|' -f 3- ; tail -n +2 input; } > output


0

The cut command should do the job if I guess at what you are referring to. echo "foo|bar|baz" | cut -d"|" -f3 So if you are only wanting to edit the first row then use the head command as well. head -n1 filename | cut -d"|" -f3 You could also use sed to do the job. head -n1 filename | sed 's/^.*|.*|//g'


0

try this command head -2 inputFile.txt | cut -d "|" -f 3- > outputFile.txt


1

If there is no other limitation here your script sed '/^HEADER/{:1;N;/TAIL/!b1;/DOG/d}' text Just for fun other variants of awk: one: awk ' BEGIN{prn=1} /^HEADER/{block=1} block{ if(/DOG/) prn=0 if(!/^TAIL/){ line=line $0 "\n" next } } prn{print line $0} /^TAIL/{ ...


0

The new edited script it's ok but stops on a new line here is the error root@localhost:~/python# ./compare.py input1.txt input2.txt output.txt Traceback (most recent call last): File "./compare.py", line 46, in <module> for row in csvReader: File "/usr/lib/python3.2/codecs.py", line 300, in decode (result, consumed) = ...


2

With perl: perl -0777 -lne 'print for grep !/DOG/, /^HEADER.*?TAIL.*?\n/mgs' your-file With awk: awk '! inside {if (/^HEADER/) {inside = 1; skip = 0; content = ""} else next} /DOG/{skip = 1; next} ! skip {content=content $0 "\n"} /^TAIL/ {if (!skip) printf "%s", content; inside = 0}' your-file


0

You could use sqlite for joining your data: $ sqlite3 <<EOT .mode csv CREATE TABLE file1 (A11,A12,A13,A14,A15,A16,A17,A18,A19,A110,A111); CREATE TABLE file2 (B11,B12,B13,B14); .import file1 file1 .import file2 file2 SELECT DISTINCT file1.*, ifnull(file2.B14,"NOACT") FROM file1 LEFT JOIN file2 ON abs(julianday(file1.A11) - julianday(file2.B11))*86400.0 ...


1

I was looking for something similar to grep, where it will exit 1 if a match is not found. Here is the equivalent with awk: awk ' BEGIN {z=1} /bravo/ {z=0; print} END {exit z} ' alpha.txt Example


0

with awk awk ' NR == 1 {print;next} { n[$3]++ val[$3] += $1 adj[$3] += $2 path[$3] = $4 } END { for (ko in n) printf "%.2f %.2f %s %s\n", val[ko]/n[ko], adj[ko]/n[ko], ko, path[ko] } ' file | column -t


0

Maybe GNU's datamash is something for you: $ datamash -H -g3,4 mean 1 mean 2 < file GroupBy(ko) GroupBy(pathway) mean(pvalue) mean(padj) ko00620 Pyruvatemetabolism 15,373333333333 0,0066666666666667 ko00051 Fructoseandmannosemetabolism 12,215 0,005


1

Probably not the answer you're looking for, but what the heck, it's fun: #!/usr/bin/env perl use strict; use warnings; use DBI; my $dbh = DBI->connect( 'dbi:CSV:', undef, undef, { f_ext => '.csv/r', csv_sep_char => "\t", csv_quote_char => undef, csv_escape_char => undef, } ); my ...


0

Use the Perl's i flag for in-place edits: $ perl -pi -e '$_ = "" if m{^\\\\}' file.txt And if you want to pipe the output of find to it, use xargs: $ find . -type f | xargs perl -pi -e '$_ = "" if m{^\\\\}' When using find's -exec instruction: $ find . -type f -exec perl -pi -e '$_ = "" if m{^\\\\}' {} \;


2

These are effectively multi-line records separated by a blank line. Awk is great for handling this kind of data: pactl list sink-inputs | awk -v RS="" '/VLC/' If you want to be really nit-picky about not including the bottom part of the record after the first occurrence of "VLC", then: pactl list sink-inputs | awk -v RS="" -v FS="\n" '/VLC/{ for(i=1; ...


0

First answer : For all that is text manipulation, Perl is always a good possibility : perl -e 'while(<>){print if(!/(\\\\)+/)}' file > file.new But if you only want to use the shell, this could work too : grep -v '\\\\' file > file.new It works well :) Edit : If you can't create a new file, Perl can still do that : perl -e 'open FILE_R, ...


3

Perl one-liner: perl -pe's/\[gene=([^\]]*)\K\]/$h{$1}++?"$h{$1}]":"]"/e' yourfile Explained: -p: execute the code for each line of the file (stored in $_) and print $_ at the end. -e: code. s/regex/replacement/e: Match regex and replace it with replacement on $_. The regex: /\[gene= # match [gene= ([^\]]*) # match anything but "]" and put it a ...


0

Use cut to extract the columns you need (providing the column numbers for us would have been a kindness), and sed to alter the text: grep ... | cut -d, -f2,9,10,11,26 | sed -e 's/CURRENCY/META/' -e 's/"//g' outputs 20150518 11:36:09,7A104802E728,4529800000,123456789,META Another approach, since your input data looks like well-formed CSV (ignoring the ...


5

With ed: ed -s <<'IN' r !pactl list sink-inputs /VLC/+,$d ?Sink Input?,.p q IN It reads the command output into the text buffer, deletes everything after the first line matching VLC and then prints from the previous line matching Sink Input up to current line. With sed: pactl list sink-inputs | sed -n 'H;/Sink Input/h;/VLC/{x;p;q}' It appends ...


5

I'd use Perl's paragraph mode: pactl list sink-inputs | perl -00ne 'print if s/(.*?VLC.*?\n).*/$1/ms' The -00 sets the input record separator to \n\n so a "line" is a paragraph. Then, the substitution will match everything until the first VLC and then anything until the 1st newline and save them as $1. Everything after that is removed (since we're ...



Top 50 recent answers are included