awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

5
votes
3answers
540 views

Extracting fixed width records with no delimiter from a single line

I need to extract strings of text from a single file containing one very long line of text with no delimiters. Using the sample line below, these are the following known facts: ...
5
votes
3answers
307 views

find length of longest line in all text files in a directory

I know how to get length of the longest line in a text file with awk awk ' { if ( length > L ) { L=length} }END{ print L}' file.txt but how can I get the length of the longest line of all files ...
5
votes
2answers
222 views

Sum in block with AWK (restart the sum when the pattern changes)

I have a file like this: A 100 A 200 A 300 #sum=600 B 400 B 500 #sum=900 A 600 A 700 A 800 #sum=2100 I would like the output to be: A 600 B 900 A 2100 C sum_of_C D sum_of_D I can do that with ...
5
votes
3answers
238 views

Replace multiple strings in a single pass

I'm looking for a way to replace placeholder strings in a template file with concrete values, with common Unix tools (bash, sed, awk, maybe perl). It is important that the replacement is done in a ...
5
votes
2answers
174 views

How to use shell to derive an IPv6 address from a MAC address?

We know that we can use the MAC address to create an interface identifier, e.g. for a link-local IPv6 address which should be unique in the Network. The image shows the way to do this: My ...
5
votes
4answers
460 views

how can we use awk to get multiple words after a significant word?

I want to connect to my DB only once to do multiple SELECT queries as follows: #!/bin/bash #Begin Code query=$(echo "SELECT COUNT(*) from USER_IPTable; SELECT User from USER_IPTable; ...
5
votes
2answers
110 views

Adding corresponding Numbers in a Range

I have a dataset with 30000 lines. I want to add all the corresponding values (i.e. Column 2) between the range 0-50 and then 50.1-100 and then 100.1-150 etc. (presented in column 1), and this ...
5
votes
3answers
4k views

Filter a .CSV file based on the 5th column values of a file and print those records into a new file

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , ...
5
votes
2answers
225 views

awk + paste for cleaning up PATH?

I have seen this code in .cshrc init files on a few machines. I went through a few awk tutorials in trying to understand how it works, but I am still unable to decrypt it. setenv PATH `echo $PATH | ...
5
votes
1answer
349 views

Delete an array in awk

In awk, I can clear an array with a loop, making it an empty array, which is equivalent to deleting it. for (key in array) delete array[key]; Is there a simpler way? Can I completely delete an ...
5
votes
2answers
114 views

How would I add a prefix to input I receive from a pipe via awk and color the prefix conditionally?

Is there a way to add a prefix to lines received (and filtered) via awk - yes I know that's possible so far - and then conditionally (e.g. based on the existence of the $PS1 variable of the ...
5
votes
5answers
669 views

Pattern matching across multiple lines

What is a quick and easy way to validate if the pattern "301 domaname.com 200" is present within BASH when spread across multiple lines of output? I was using something like the following: awk ...
5
votes
1answer
2k views

Sorting a file based on one column using Unix and Awk

I need to sort the input file according the 6th column, which is the score. Input File: Sc2/80 20 . A T 86 Pass N=2 F=5;U=4 Sc2/80 20 . A C 80 Pass N=2 F=5;U=4 Sc2/60 55 . G T 90 Pass N=2 F=5;U=4 ...
5
votes
3answers
1k views

killing processes automatically

I need to kill all processs in a certain shell excluding certain processes. Like sh which is my shell. And the comand. This is what currently in my shell right now. rcihp146 :/home/msingh2> ps ...
5
votes
1answer
128 views

Inderect references aka value of a value in awk

I'm searching for a way to solve this problem in awk. Input: X 1 Y 2 Z 3 X 4 Y 5 The output should look like this: X 5 Y 7 Z 3 I see two problems here: the first one is the indirect reference in ...
5
votes
1answer
472 views

Is getline in awk command safe to use?

I got weird comments on posting an awk answer on SO which uses the getline function. Here is the link to that answer. After posting my answer, a user came up with the below comment,(I don't criticize ...
5
votes
2answers
259 views

how to add command line calculator to bash that uses comma as decimal mark?

How can I add a command line calculator to my bash? I have found some, but all of them use the full stop as decimal mark, but I want to have it to use the comma as decimal mark as most of the world ...
5
votes
1answer
616 views

grep: display filename once, then display context with line numbers

Our source code has error codes scattered throughout. Finding them is easy with grep, but I'd like a bash function find_code that I can execute (eg. find_code ####) which will provide output along ...
5
votes
2answers
568 views

Use a script parameter in awk

Here is my script (to find the files that contain a specified pattern) : find . -type f -exec awk -v vawk="$1" '/'"$vawk"'/ {c++} c>0 {print ARGV[1] ; exit 0 } END { if (! c) {exit 1}}' \{\} \; ...
5
votes
3answers
201 views

The ERE regex to split() string between a delimiter and end-of-word

I'm using a long gawk 3.1.6 script to do a complex conversion of Zim markdown text into GtkDialog code and am stuck on the following problem... Sample ASCII input... [[link|label label]] [[link]] ...
5
votes
2answers
910 views

Inotifywait for large number of files in a directory

What I want to do is, to monitor a directory (not recursive, just one) for new files created and append those files to one single big file as they are being written. The number of files that are ...
5
votes
2answers
549 views

Gawk: Passing arrays to functions

Stuck with GNU awk 3.1.6 and think I've worked around its array bugs but still have what looks like a scope problem in a 600-line awk program. Need to verify understanding of array scope in awk to ...
5
votes
5answers
155 views

Replace data at specific positions in txt file using data from another file

I have a text file in the below format: $data This is the experimental data good data This is good file datafile 1 4324 3673 6.2e+11 7687 67576 2 3565 8768 8760 5780 8778 "This is line ...
5
votes
2answers
151 views

Grepping for a block of text with parts that can be optional

I have multiple entries that describes an event in a very large log file, say A.log. I would like to do two things with the event entries in the log file: Count the number of occurrences of each ...
5
votes
2answers
601 views

Use awk to insert a line after N output

I have a text file filled with a bunch of data. I need to pull out lines that contain specific strings. I accomplished this with awk using the following: awk '/pattern1|pattern2|pattern3/ {print;}' ...
5
votes
1answer
2k views

Fast elimination of duplicate lines across multiple files

I have a huge amount of data in which each (data-)line should be unique. There are a lot of files in one folder in which this is already true. It is about 15GB splitted into roughly 170 files with ...
5
votes
3answers
128 views

How to use awk through multiple files?

I want a solution for the general case (N folder) I'm using awk to process a file and extract its content and put it in a variable then echo it: This is the file: H1 H2 H3 H4 H5 H6 H7 H8 H9 not ...
5
votes
3answers
828 views

extracting “tag” information from git with a shell script

The vc bundle is a neat little package that extracts information about a git repo for easy insertion into a LaTeX document. It doesn't currently extract information about whether the current commit is ...
5
votes
3answers
179 views

Increment numbers inside strings by an offset

I want to search a text file for strings like these. [task0] [task1] ... [task1613] and increment each by a given offset, in this case by 3640. So, with the example I gave above, I'd get the ...
5
votes
1answer
47 views

Count duplicate rows with constraint

I have the following file: A B 1990 A C 2001 A C 2013 A B 2001 B C 2013 A D 2001 A D 2014 I need to count duplicate pairs (in the 1st and 2nd column) and to each such pair assign the lowest value ...
4
votes
7answers
334 views

How can I groups numbers in a file

I have a file with numbers in float format. I can review them via sort -rn numbers.txt | less I would like though to be able to "group" them. I.e. easily see how many are in the same range. To give an ...
4
votes
6answers
305 views

want the last line when duplicate awk?

I have the file: key value blah blah blah blah blahblah man1 boy1 blah blah man1 boy2 man1 boy1 I do this to remove duplicate lines: awk '/man1/ { print $1,$2} ' file | awk '!x[$0]++' and the ...
4
votes
4answers
1k views

How to remove duplicate lines with awk whilst keeping empty lines?

Below awk command removes all duplicate lines as explained here: awk '!seen[$0]++' If the text contains empty lines, all but one empty line will be deleted. How can I keep all empty lines whilst ...
4
votes
7answers
439 views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
4
votes
5answers
7k views

extract the 4th column from a csv file using unix command

I have a csv file formatted as below. "col1","col2","col3","col4" "1","text1","<p>big html text</p> ","4th column" "2","text2","<p>big2 html2 text2</p> ...
4
votes
3answers
409 views

pipe line with non-deterministic output

I have happened upon a command that sometimes works and sometimes does not, even when executed multiple times in rapid succession in a bash shell (I have not tested the behavior in other shells). The ...
4
votes
3answers
1k views

Summing up an array inside of awk?

I have the following piece of code: sum1= sum2= declare -a a echo $temp | awk '{split($0,a,","); name=a[1] ; for(i=2;i<=4;i++) sum1+=a[i] ; for(i=5;i<=7;i++) sum2+=a[i] }' This code is ...
4
votes
5answers
3k views

Convert formatted dates to seconds since the epoch

I have a file: pablo tty8 Thu Nov 1 12:51:21 2012 still logged in (unknown tty8 Thu Nov 1 12:50:57 2012 - Thu Nov 1 12:51:21 2012 (00:00) pablo tty2 Thu Nov 1 12:50:39 2012 still logged in pablo ...
4
votes
3answers
602 views

Replace “o” with “0” in large wordlist and save the original word?

I have a large text file with words. Every word is on a single line (typical wordlist). I want to replace all characters "o" with the number "0" in every word, and the new formed word to be pasted on ...
4
votes
4answers
1k views

remove blank lines in awk display

I was trying to figure out a solution for this question. I wanted to use awk for the solution. My input file is something like below. -bash-3.2$ cat file ramesh ramesh_venkat ramesh3_venkat3 ...
4
votes
7answers
577 views

How to split a filename list in 5GB sets?

I have a pretty big list of files which I like to group in smaller list of 5GB each for further processing. The file list is in the following form: "size" (from du) "tab" "absolute path", like this ...
4
votes
4answers
8k views

removing the first and the last character of every line from command line

I am trying to remove the first and the last characters of everyline in a text file and save the resulting truncated version in a new file. Does anyone have an idea about how to do that efficiently ...
4
votes
4answers
172 views

How to remove all comments from a file?

I have a file with a few lines commented out: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I just want to print all the uncommented code: foo bar stuff morestuff ...
4
votes
4answers
1k views

Redirecting Output of print to multiples files in awk

I want to print same lines to 2 different files. What I am doing right now is: print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file1.txt"; print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file2.txt"; ...
4
votes
3answers
1k views

awk command for parsing a file

I have a following text file. I am showing you the first 3 lines. chrom st end gene strand c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 c12 c13 c14 chr6 3345 3543 geneA + 36 -23 ...
4
votes
6answers
2k views

Remove lines that do not start with a pattern from a given set of patterns

I have a file that contains data like this: report aaaaaaaa - .. -th bbbbbbbbb -to ccccccccc .. --. Question: I want to remove any line that does not start with following strings : report ...
4
votes
6answers
103 views

Only show the lines that are in all textfiles at least once

a.txt cat a.txt a b x c b.txt cat b.txt d e a f the q: SOMEMAGICK *.txt a Q: how can I show only the lines that are in all *.txt files?
4
votes
5answers
1k views

How to remove character and space from a string

Hello I have one file with output Name : ABC Name : CDE Name : ZYS I want to get the result ABC CDE ZYS How can i get it through SED command. I have tried. sed s/'Name' but ...
4
votes
3answers
1k views

Show lines matching a pattern and the 4 lines before each

For example, from this file: CREATE SYNONYM I801XS07 FOR I8010.I801XT07 * ERROR at line 1: ORA-00955: name is already used by an existing object CREATE SYNONYM I801XS07 FOR ...
4
votes
2answers
159 views

Comma separated words to next line in new file

I have a input file like this (input.txt): Main1 one,two,three,four Main2 five,six,seven,eight I want the output like this in output file (output.txt): Main1 one Main1 two Main1 three Main1 four ...