awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
4answers
2k views

Using sed/awk to remove anything after first space

aaaaaaaa 09 bbbbbbbb 90 ccccccccccccccc 89 ddddd 09 Using sed/awk/replace, in the above text I want to remove anything that comes after the first space in each line. For example the output ...
3
votes
1answer
786 views

grep on line before the match plus the match

zzzzzzzzz aaaaaaaaa bbbbbbbbb & ccccccccc & ddddddddd hhhhhhhhh eeeeeeeee fffffffff & ggggggggg & in the above line, what i want is to grep/sed/awk (any method is fine) line that ...
1
vote
1answer
235 views

Insert a space before the first comma in a csv file

I have a huge CSV file which contains 100000 rows. However, I need to do some processing and so for that I need to have a single space before the first comma. My current data is something like below. ...
0
votes
1answer
513 views

Awk with multiple delimiters

I am trying to execute awk on something like this : <tag x="whatever" y="http://mysite.com/somewords/word1/word2/word3/word4"> I would like to extract word2 which comes forever after word1, ...
3
votes
2answers
5k views

Insert text in specific lines of a file?

How can insert some text in specific lines of a file? What is most simplest method that I can use? (bash, sed, awk?) What I want to do, might be simple, but I don't know where I should start. It's ...
4
votes
3answers
635 views

using awk to split a line on single spaces not multiples

I'm trying to split a line that I have no control over the format of. If parameter 7 and 8 are missing which is possible they will be replaced by a space so I would end up with, field1 field2 field3 ...
4
votes
1answer
174 views

Matching Five Columns in two Files using Awk

I have two Input files. File1: s2/80 20 . A T 86 F=5;U=4 s2/20 10 . G T 90 F=5;U=4 s2/90 60 . C G 30 ...
3
votes
4answers
438 views

How to Prefix a column values with an apostrophe ( ' )?

I have a CSV file with multiple columns and 1000's of records, I need to prefix all the values of one of the columns (lets say 2nd column) with an apostrophe ' except in the first line or header line, ...
7
votes
3answers
339 views

Distributing a script: Should I use /bin/gawk or /usr/bin/gawk for shebang?

Is gawk in /bin or /usr/bin usually? I would go with #!/usr/bin/env gawk but then I can't use arguments. Right now I'm using #!/bin/gawk -f. The script is very long and contains a lot of single ...
5
votes
3answers
4k views

Filter a .CSV file based on the 5th column values of a file and print those records into a new file

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , ...
0
votes
2answers
113 views

why does `awk` fail when I put it in a bash script

on the prompt: ls | awk '{printf("%s\t\t%s\n", $9, $3);}' succeeds in printing what it is supposed to be printing. If I put it in a file, I get four or five empty lines (about the number of lines ...
-1
votes
1answer
47 views

A triangle of forward differences in gawk

I need a gawk script (not a one-liner, I need a function). Given a row of integers i.e. 0 2 4 1 9 5 1 1 the function should generate successive forward absolute differences, so: 0 2 4 1 9 5 1 1 2 ...
4
votes
5answers
490 views

Multi-line grep beween two tags, only show the last match

I'm having a program that intermingles data segments into log output: log message log message ----BEGIN INLINE DATA---- data data data -----END INLINE DATA----- log messge I'm looking for something ...
3
votes
3answers
289 views

awk read error bad address

I have a file that is roughly 30GB. I am using, awk ' $1 <= 2000 ' myfile.txt >> myfile_new.txt But, I get this error after a while - awk: read error (Bad address) I guess this is ...
1
vote
1answer
43 views

Unable to unregister vms

I have the following command to extract the name of virtual machines currently available in the host system. vmslist=$(vboxmanage list vms | awk '{print $1}') I pass this list to a for loop in ...
2
votes
3answers
5k views

Multiple pattern match and print in single line

I need to match two patterns in a log file and need to get the next line of the one of the pattern (out of two patterns) that is matched, finally need to print these three values in a single line. ...
4
votes
2answers
2k views

recursive search for a pattern, then for each match print out the specific SEQUENCE: line number, file name, and no file contents

What I am after is almost exactly the same as can be found here, but I want the format "line number, separator, filename, newline" in the results, thus displaying the line number at the beginning of ...
2
votes
4answers
656 views

Format a Date field of a .CSV file with multiple commas in a string field

I have a .CSV file (file.csv) whose data are all enclosed in double quotes. Sample format of the file is as below: column1,column2,column3,column4,column5,column6, column7, Column8, Column9, Column10 ...
2
votes
2answers
272 views

Shell script and adding large values in a delimited file problem

I have a script that reads through a delimited file and adds up the 3rd element in the file for each record. For a majority of the data files, this works fine except for one. I have one data file were ...
2
votes
2answers
112 views

Awk not rendering any result

My goal is to get the file created in the current month in a directory. It seems that the command is correct but not rendering any result: Date=`date '+%b'` echo $Date Oct ls -l | awk -v d="$Date" ...
3
votes
1answer
441 views

Walking Multidimensional arrays in Mawk

I'm able to do this properly in gawk, but when I tried to post my code to the machine where it will run, I realized it was using mawk... $ cat multidim.gawk # test of multidimensional arrays // { ...
3
votes
2answers
961 views

Replace a pattern in a file with a long string

I'm writing a little shell script to generate a directory listing. To make the output easily customizable, the script just builds a HTML-table and than should replace a specific token in a template ...
3
votes
4answers
1k views

Align text to center with padding on both sides

Is there an easy way to align text to the center with padding on both sides, with the column width being the longest line from the input? For example, this: aaa bbbb c ddddd ee will turn into this ...
5
votes
3answers
398 views

Deleting Lines if there is a dot in the 5th Column

I have a file and I want to remove all the line if it has a dot in the 5th Column. Input File: sc2/80 20 . A C 80 PASS N=2 F=5;U=4 sc2/60 ...
2
votes
1answer
166 views

How to extract single value from single line tuples?

I have a dump file of a database in the following form ('value1','value2',value3','value4'),('value1','value2',value3','value4'),... I want first to have each tuple in a single line and next i ...
1
vote
2answers
1k views

Printing fields using awk

< pool.sam awk ' /./ {printf $1} {printf $7+1,"\t"} {printf $3,"\t"} { if($2 !=16) {print "\t", "+";} else {print "\t","-";} {printf $4,"\t" ,length($10)+$4, "\t", "1"} }' I am ...
3
votes
1answer
419 views

Efficient way to print lines from a massive file using awk, sed, or something else? [duplicate]

If I had a plain-text file containing 8 million lines and I want to print out lines 4,000,010 to 4,000,000 to the screen, which would be more efficient: awk or sed? There is no pattern to the text, ...
1
vote
2answers
96 views

AWK special character

If i want to awk the below pattern in a file Mobilenumber '91_987986787688899' was successful I used the below code awk '/Mobilenumber '91_987986787688899' was successful/ {print $0}' Log Filename ...
0
votes
1answer
1k views

shell script to print rows if there is a value in column 2

1 **2** 3 4 3 1 4 2 1 4 3 2 4 **2** 1 3 How can I print all rows, on which there is a 2 in the second column.
1
vote
2answers
278 views

how to calculate the average value of the column(big number of columns) that have the row ID?

I want to calculate the average value of my data when they have the same ID in the first column. I want to calculate the average values for the remaining columns that have the same value in the first ...
3
votes
3answers
938 views

Count the number of rows in each column of a text file

I have a text file with the following format: 5 3 1 2 3 4 ..... ..... i.e. space separated 3 columns of numbers/ However, some of the rows may be like: 2 3 1 So, I want to detect if the text file has ...
1
vote
1answer
230 views

How to use awk or sed to convert csv diffs into more readable format

Can anyone give me the example of how to use awk or sed (not sure which one, as I haven't use either of these much, as I have mostly been using grep and cut to work with csv data) to convert ...
2
votes
1answer
73 views

How to only output a column that has constant neighbors?

INPUT AA XXX Y1Y ZZZ GG dhz rr (AAAa) XXX Y2Y ZZZ TT GGGG UU OUTPUT Y1Y Y2Y The input lines could vary.. only the XXX before the Y1Y and the ZZZ after Y1Y is constant (they are the neighbors of ...
1
vote
2answers
573 views

Way to write new filename on wildcard input?

I have some experience with Inix terminals from science internships I've been part of, mainly using a few utilities like grep, awk, and sed but there is one thing I've been trying to figure out for a ...
12
votes
9answers
2k views

How to perform a multi line grep

How would you perform a grep for text that appears on two lines? For example: pbsnodes is a command I use that returns the utilization of a linux cluster root$ pbsnodes node1 state = free ...
5
votes
4answers
185 views

How to add each output of awk

How would you perform addition on the output of every line from awk? For example cat foo | awk '{print $1} 42 42 42 I'm looking for the answer to 42+42+42 (126). I'm trying to write a for loop ...
1
vote
2answers
627 views

awk,how to change the value of the file using parameter sets?

I am trying to reset my data using awk. I have a data of 54 columns where I want to change six of the column values using the following script. #!/bin/bash -x # optimal parameters set ...
1
vote
1answer
145 views

Retrieving two matches from repeated pattern

How can I retrieve matches from two repeated patterns: mappedname & its value in the text below?. MapName:abc someDiffName=someDiffValue x=3 y=6 NeedyMappedName=NeedyMappedValue . . Flag=true ...
1
vote
3answers
578 views

How to add Multiple Searches in AWK command

Im trying to search for the files Created in June,July and August months. I used this method ls -lrth|awk '/[Jun][Jul][Aug]/ {print}' but it ain't working. Please help me correct the syntax.
3
votes
2answers
210 views

Join: Two files - but only append last two columns

Given the files: 1.txt 1, abc, 123, 456, 789 2, lmn, 123, 456, 789 3, pqr, 123, 456, 789 2.txt 1, abc, 123, 000, 000 3, lmn, 123, 000, 000 9, opq, 123, 000, 000 OUTPUT.txt ID, NAME, X, ...
3
votes
3answers
286 views

Join every other column with sed or awk

I have a large text file (666000 colums) in the format A B C D E F Desired output AB CD EF How can we do it in sed or awk. I have tried a couple of things but nothing seems to be working. Please ...
6
votes
5answers
241 views

Get filesystem name and usage with `df -h`

df -h | grep test | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: /test /test/drv0 /test/drv1 df -h | grep test | cut -d '%' -f1 | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: 10 20 15 I want to ...
2
votes
1answer
96 views

Regular Expressions within a string in AWK using if/then control structure?

PROBLEM I want to specifically use AWK (GAWK, MAWK, whatever) in an AWK script to sum up the values in field 9 for every record that contains "runner__.jpx" (the underscores are placeholders for two ...
2
votes
3answers
2k views

Comparing files line by line in awk with delimiter

file1: abc|123|check def|456|map ijk|789|globe lmn|101112|equator file2: abc|123|check def|456|map ijk|789|equator lmn|101112|globe EXPECTED OUTPUT: ijk|789|equator lmn|101112|globe Current ...
2
votes
4answers
183 views

AWK Search massive file and write to variable name

I want to pull data that I will need to reformat from a simulation file I am working with to prepare it as input for running other simulations. The target data is an optimized chemical z-matrix. The ...
1
vote
2answers
97 views

How can I reformat tabular data using AWK?

I have a file contain following output. ora.abc.db 1 ONLINE ONLINE serverA Open 2 ONLINE ONLINE serverB Open ora.xyz.db 1 ...
1
vote
1answer
1k views

How to find out the content of a XML file using Unix Sed/Awk? [duplicate]

I have a XML file(MyXML.xml) like this : <?xml version="1.0" encoding="UTF-8"?> <S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"> <S:Body> ...
2
votes
4answers
97 views

Alternative regex for {}

I am trying to replace all of this line, but the numbers: looktype="123" so only the numbers show up. Is this possible in any easy way? {sub ("look type=\"[0-9]{0,3}", "TEST")} I am trying this ...
0
votes
1answer
259 views

how to assign values of the column for given conditional statements

I need help with a script to assign values of the column with a given conditional argument. To be specific, here is my data and my condition input data.file: 0.4 3.6 -2.4 0.1 -2.0 1.6 2.2 3.6 ...
6
votes
4answers
855 views

uniq a csv file ignoring a column, awk maybe?

Given this file (annotations are not part of file, but form part of explanation)... x,a,001,b,c,d,y x,a,002,b,c,e,yy x,bb,003,b,d,e,y x,c,004,b,d,e,y x,c,005,b,d,e,y # nb - dupe of row 4 ...