awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

5
votes
2answers
129 views

Distributing a script: Should I use /bin/gawk or /usr/bin/gawk for shebang?

Is gawk in /bin or /usr/bin usually? I would go with #!/usr/bin/env gawk but then I can't use arguments. Right now I'm using #!/bin/gawk -f. The script is very long and contains a lot of single ...
0
votes
2answers
90 views

why does `awk` fail when I put it in a bash script

on the prompt: ls | awk '{printf("%s\t\t%s\n", $9, $3);}' succeeds in printing what it is supposed to be printing. If I put it in a file, I get four or five empty lines (about the number of lines ...
2
votes
3answers
1k views

Multiple pattern match and print in single line

I need to match two patterns in a log file and need to get the next line of the one of the pattern (out of two patterns) that is matched, finally need to print these three values in a single line. ...
4
votes
5answers
182 views

Multi-line grep beween two tags, only show the last match

I'm having a program that intermingles data segments into log output: log message log message ----BEGIN INLINE DATA---- data data data -----END INLINE DATA----- log messge I'm looking for something ...
-1
votes
1answer
45 views

A triangle of forward differences in gawk

I need a gawk script (not a one-liner, I need a function). Given a row of integers i.e. 0 2 4 1 9 5 1 1 the function should generate successive forward absolute differences, so: 0 2 4 1 9 5 1 1 2 ...
4
votes
2answers
659 views

recursive search for a pattern, then for each match print out the specific SEQUENCE: line number, file name, and no file contents

What I am after is almost exactly the same as can be found here, but I want the format "line number, separator, filename, newline" in the results, thus displaying the line number at the beginning of ...
3
votes
4answers
511 views

Align text to center with padding on both sides

Is there an easy way to align text to the center with padding on both sides, with the column width being the longest line from the input? For example, this: aaa bbbb c ddddd ee will turn into this ...
2
votes
4answers
309 views

Format a Date field of a .CSV file with multiple commas in a string field

I have a .CSV file (file.csv) whose data are all enclosed in double quotes. Sample format of the file is as below: column1,column2,column3,column4,column5,column6, column7, Column8, Column9, Column10 ...
2
votes
2answers
94 views

Shell script and adding large values in a delimited file problem

I have a script that reads through a delimited file and adds up the 3rd element in the file for each record. For a majority of the data files, this works fine except for one. I have one data file were ...
2
votes
2answers
106 views

Awk not rendering any result

My goal is to get the file created in the current month in a directory. It seems that the command is correct but not rendering any result: Date=`date '+%b'` echo $Date Oct ls -l | awk -v d="$Date" ...
3
votes
1answer
171 views

Walking Multidimensional arrays in Mawk

I'm able to do this properly in gawk, but when I tried to post my code to the machine where it will run, I realized it was using mawk... $ cat multidim.gawk # test of multidimensional arrays // { ...
5
votes
3answers
179 views

Deleting Lines if there is a dot in the 5th Column

I have a file and I want to remove all the line if it has a dot in the 5th Column. Input File: sc2/80 20 . A C 80 PASS N=2 F=5;U=4 sc2/60 ...
3
votes
2answers
311 views

Replace a pattern in a file with a long string

I'm writing a little shell script to generate a directory listing. To make the output easily customizable, the script just builds a HTML-table and than should replace a specific token in a template ...
2
votes
1answer
98 views

How to extract single value from single line tuples?

I have a dump file of a database in the following form ('value1','value2',value3','value4'),('value1','value2',value3','value4'),... I want first to have each tuple in a single line and next i ...
1
vote
2answers
296 views

Printing fields using awk

< pool.sam awk ' /./ {printf $1} {printf $7+1,"\t"} {printf $3,"\t"} { if($2 !=16) {print "\t", "+";} else {print "\t","-";} {printf $4,"\t" ,length($10)+$4, "\t", "1"} }' I am ...
3
votes
1answer
165 views

Efficient way to print lines from a massive file using awk, sed, or something else?

If I had a plain-text file containing 8 million lines and I want to print out lines 4,000,010 to 4,000,000 to the screen, which would be more efficient: awk or sed? There is no pattern to the text, ...
0
votes
1answer
482 views

shell script to print rows if there is a value in column 2

1 **2** 3 4 3 1 4 2 1 4 3 2 4 **2** 1 3 How can I print all rows, on which there is a 2 in the second column.
1
vote
2answers
73 views

AWK special character

If i want to awk the below pattern in a file Mobilenumber '91_987986787688899' was successful I used the below code awk '/Mobilenumber '91_987986787688899' was successful/ {print $0}' Log Filename ...
1
vote
2answers
121 views

how to calculate the average value of the column(big number of columns) that have the row ID?

I want to calculate the average value of my data when they have the same ID in the first column. I want to calculate the average values for the remaining columns that have the same value in the first ...
3
votes
3answers
236 views

Count the number of rows in each column of a text file

I have a text file with the following format: 5 3 1 2 3 4 ..... ..... i.e. space separated 3 columns of numbers/ However, some of the rows may be like: 2 3 1 So, I want to detect if the text file has ...
1
vote
2answers
216 views

Way to write new filename on wildcard input?

I have some experience with Inix terminals from science internships I've been part of, mainly using a few utilities like grep, awk, and sed but there is one thing I've been trying to figure out for a ...
1
vote
1answer
164 views

How to use awk or sed to convert csv diffs into more readable format

Can anyone give me the example of how to use awk or sed (not sure which one, as I haven't use either of these much, as I have mostly been using grep and cut to work with csv data) to convert ...
5
votes
5answers
15k views

Multiline pattern match using sed, awk or grep

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. ...
2
votes
1answer
55 views

How to only output a column that has constant neighbors?

INPUT AA XXX Y1Y ZZZ GG dhz rr (AAAa) XXX Y2Y ZZZ TT GGGG UU OUTPUT Y1Y Y2Y The input lines could vary.. only the XXX before the Y1Y and the ZZZ after Y1Y is constant (they are the neighbors of ...
12
votes
8answers
802 views

How to perform a multi line grep

How would you perform a grep for text that appears on two lines? For example: pbsnodes is a command I use that returns the utilization of a linux cluster root$ pbsnodes node1 state = free ...
1
vote
1answer
100 views

Retrieving two matches from repeated pattern

How can I retrieve matches from two repeated patterns: mappedname & its value in the text below?. MapName:abc someDiffName=someDiffValue x=3 y=6 NeedyMappedName=NeedyMappedValue . . Flag=true ...
5
votes
4answers
155 views

How to add each output of awk

How would you perform addition on the output of every line from awk? For example cat foo | awk '{print $1} 42 42 42 I'm looking for the answer to 42+42+42 (126). I'm trying to write a for loop ...
2
votes
4answers
135 views

AWK Search massive file and write to variable name

I want to pull data that I will need to reformat from a simulation file I am working with to prepare it as input for running other simulations. The target data is an optimized chemical z-matrix. The ...
1
vote
2answers
373 views

awk,how to change the value of the file using parameter sets?

I am trying to reset my data using awk. I have a data of 54 columns where I want to change six of the column values using the following script. #!/bin/bash -x # optimal parameters set ...
2
votes
3answers
138 views

how to create a data file with a column of date from day1 to dayN?

I was wondering if I could generate a hourly or daily time step date in a column starting from some year in the past to some day in the near past or today. to be more clear, I want to create one ...
1
vote
3answers
99 views

How to add Multiple Searches in AWK command

Im trying to search for the files Created in June,July and August months. I used this method ls -lrth|awk '/[Jun][Jul][Aug]/ {print}' but it ain't working. Please help me correct the syntax.
3
votes
3answers
125 views

Join every other column with sed or awk

I have a large text file (666000 colums) in the format A B C D E F Desired output AB CD EF How can we do it in sed or awk. I have tried a couple of things but nothing seems to be working. Please ...
3
votes
2answers
133 views

Join: Two files - but only append last two columns

Given the files: 1.txt 1, abc, 123, 456, 789 2, lmn, 123, 456, 789 3, pqr, 123, 456, 789 2.txt 1, abc, 123, 000, 000 3, lmn, 123, 000, 000 9, opq, 123, 000, 000 OUTPUT.txt ID, NAME, X, ...
6
votes
5answers
221 views

Get filesystem name and usage with `df -h`

df -h | grep test | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: /test /test/drv0 /test/drv1 df -h | grep test | cut -d '%' -f1 | sed -e 's/*%.^ //g;s/.*[ ^I]//' Output: 10 20 15 I want to ...
10
votes
7answers
4k views

How to print the longest line in a file?

I'm looking for the simplest method to print the longest line in a file. I did some googling and surprisingly couldn't seem to find an answer. I frequently print the length of the longest line in a ...
2
votes
1answer
75 views

Regular Expressions within a string in AWK using if/then control structure?

PROBLEM I want to specifically use AWK (GAWK, MAWK, whatever) in an AWK script to sum up the values in field 9 for every record that contains "runner__.jpx" (the underscores are placeholders for two ...
2
votes
3answers
559 views

Comparing files line by line in awk with delimiter

file1: abc|123|check def|456|map ijk|789|globe lmn|101112|equator file2: abc|123|check def|456|map ijk|789|equator lmn|101112|globe EXPECTED OUTPUT: ijk|789|equator lmn|101112|globe Current ...
1
vote
1answer
50 views

how to clean free row/column spaces after the row of your data

I have a problem in cleaning my file where there are unnecessary free spaces below my last data row. This is created when I am copying and pasting my data from excel to text. I want my text file to ...
2
votes
4answers
81 views

Alternative regex for {}

I am trying to replace all of this line, but the numbers: looktype="123" so only the numbers show up. Is this possible in any easy way? {sub ("look type=\"[0-9]{0,3}", "TEST")} I am trying this ...
1
vote
1answer
446 views

How to find out the content of a XML file using Unix Sed/Awk? [duplicate]

I have a XML file(MyXML.xml) like this : <?xml version="1.0" encoding="UTF-8"?> <S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"> <S:Body> ...
20
votes
10answers
4k views

How can I write to the second line of a file from the command line?

I have an external program that produces an output file (largish, 20K lines possible). I need to insert a new line between the existing line 1 and line 2. I've been looking at awk and sed - I use ...
1
vote
3answers
148 views

how to calculate the average value of the column that have the row ID?

I want to calculate the average value of my data when they have the same ID (in my specific case, the same day but different time in a day). This is meant to calculate the daily average value of my ...
0
votes
1answer
82 views

how to assign values of the column for given conditional statements

I need help with a script to assign values of the column with a given conditional argument. To be specific, here is my data and my condition input data.file: 0.4 3.6 -2.4 0.1 -2.0 1.6 2.2 3.6 ...
1
vote
1answer
83 views

doubt regarding use of split in awk

arr=($(echo "great#man" | awk '{split($0,a,"#"); } END {print a[0]; print a[1];}')) echo " ${arr[0]} " echo " ${arr[1]}" My expected output is: great man But I am getting great ...
6
votes
4answers
497 views

uniq a csv file ignoring a column, awk maybe?

Given this file (annotations are not part of file, but form part of explanation)... x,a,001,b,c,d,y x,a,002,b,c,e,yy x,bb,003,b,d,e,y x,c,004,b,d,e,y x,c,005,b,d,e,y # nb - dupe of row 4 ...
9
votes
2answers
609 views

Script to quickly test all keyboard keys

I need to check some notebooks for bad keyboard keys, and so I'd like to speed that up as much as possible. I found nothing for this specific task, so my idea is a script that reads the pressed keys ...
4
votes
3answers
750 views

Summing up an array inside of awk?

I have the following piece of code: sum1= sum2= declare -a a echo $temp | awk '{split($0,a,","); name=a[1] ; for(i=2;i<=4;i++) sum1+=a[i] ; for(i=5;i<=7;i++) sum2+=a[i] }' This code is ...
2
votes
3answers
173 views

How does this awk align the spacing?

Can someone please tell me what does this do? cat infile| awk ' { for(i=3;i<=NF;i++){ $2=$2" "$i } printf "%-4s %s\n", $1,$2 ; }' > outfile.txt infile contains: 1 John Smith 2 ...
4
votes
4answers
230 views

Joining two files with unique identifier

I have two files with approximately 12900 and 4400 entries respectively, that I want to join. The files contain location information for all landbased weather observing stations around the globe. The ...
2
votes
2answers
394 views

How to remove an apostrophe ( ' ) from couple of columns of a .CSV file?

I have a .CSV file with 7 fields, and the 3rd and 4th columns of the file has a number starting with an apostrophe ( ' ). Please see the example below. col0,col1,col2,col3,col4,col5,col6, ...