awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

0
votes
2answers
229 views

bash script, printing multiple lines that matching to a specific patern

I have a log file and I'm making a script to do some actions. An action is to print a specific area of the log. Every block at the log starts with a specific time stamp and inside the block may have ...
2
votes
2answers
111 views

Convert numeric string from file to exit code in shell

I'd like to grep /etc/password for a username and return the last two digits of a user-id number as the return code. This is because I have an application whose main way of interacting with other ...
0
votes
0answers
690 views

gsub command in awk [duplicate]

I am trying to replace the field separator from "," to "|" except for the strings which are under double quotes " " . The following command works fine : awk -F\" '{for(i=1;i<=NF;i+=2) ...
2
votes
1answer
125 views

gawk with gawkextlib

In my system (Slackware64-14.1) gawk is installed in version 4.1.0. I'd like to install gawkextlib to have access to some of its features, namely xml. From the README file of gawkextlib-0.4.9: ...
3
votes
1answer
72 views

Automatically load a user function at startup with awk?

I guess this answer from awk separator preferences already gives a hint, but I thought I'd ask explicitly: Is there an equivalent to a .bashrc file for awk, where I could define my own library ...
-2
votes
1answer
47 views

delete first two line in awk script [closed]

How can I delete the first two line in awk script? From blabla blala 1 2 3 4 To 1 2 3 4
4
votes
2answers
65 views

Concatenating fields from lines with different numbers of fields

Given input like this: x y a b c t p q w w t a b c d p q r I'd like to concatenate fields from field 3 up to but not including the last one. If field 3 is the last field, I'd like to insert a ...
1
vote
2answers
184 views

using awk to create a LaTex table from my data in a text file

So I have used awk a few times to add or remove text from each line of a text file. What I would like to learn how to do now is to add some characters to each line in multiple places. What this is ...
5
votes
3answers
266 views

removing redundancy from output columns

What is a *NIX way of removing redundancy in a case where I have pairwise comparisons like these in two columns A B B A A C A D C A D A B C C B A B and B A ...
3
votes
1answer
31 views

How to calculate (weighted) majority over columns?

I have two files: the first one contains lines where the first column is an ID, and the remaining columns are votes, e.g. 3242 -1 1 -1 1 1 1 In a first step I want to calculate the majority for ...
5
votes
5answers
508 views

Pattern matching across multiple lines

What is a quick and easy way to validate if the pattern "301 domaname.com 200" is present within BASH when spread across multiple lines of output? I was using something like the following: awk ...
1
vote
2answers
54 views

Replace “-from” with “this” in the begining of line

I want to replace "-from" with "this" in the begining of line. This should happen when that line has 'R' at the end and line above it has "D" at the end. For example for the block shown below: ...
5
votes
2answers
98 views

Adding corresponding Numbers in a Range

I have a dataset with 30000 lines. I want to add all the corresponding values (i.e. Column 2) between the range 0-50 and then 50.1-100 and then 100.1-150 etc. (presented in column 1), and this ...
4
votes
3answers
99 views

awk separator preferences

I write ad hoc awk commands in cygwin to process tab delimited files. Pretty much every command starts with: BEGIN {FS="\t";OFS="\t";ORS="\r\n";} How can I make these separators the default to ...
4
votes
2answers
303 views

How can I round all numbers in a file?

How can I round all numbers in a file that countains several columns of numbers up to a certain precision? Can this be done with awk? A single line looks like this: text - 0.1655456615 - 0.158645 - ...
3
votes
6answers
969 views

Remove lines that do not start with a pattern from a given set of patterns

I have a file that contains data like this: report aaaaaaaa - .. -th bbbbbbbbb -to ccccccccc .. --. Question: I want to remove any line that does not start with following strings : report ...
2
votes
1answer
92 views

replace text after match

aaaaaaaaa bbbbbbbbb -th ccccccccc_RD \ DDDDDDDD FFFFFFFF -th GGGGGGG_RD \ using sed/grep/awk etc, In the above text, if a line ends with "RD \" , replace "-th" with "-to" at the ...
1
vote
2answers
673 views

How to sum a bash array of numbers (some in scientific notation)?

Since the following command using bc does not work for numbers in scientific notation, I was wondering about an alternative, e.g. using awk? sum=$( IFS="+"; bc <<< "${arrValues[*]}" )
1
vote
4answers
1k views

Using sed/awk to remove anything after first space

aaaaaaaa 09 bbbbbbbb 90 ccccccccccccccc 89 ddddd 09 Using sed/awk/replace, in the above text I want to remove anything that comes after the first space in each line. For example the output ...
3
votes
1answer
542 views

grep on line before the match plus the match

zzzzzzzzz aaaaaaaaa bbbbbbbbb & ccccccccc & ddddddddd hhhhhhhhh eeeeeeeee fffffffff & ggggggggg & in the above line, what i want is to grep/sed/awk (any method is fine) line that ...
1
vote
1answer
183 views

Insert a space before the first comma in a csv file

I have a huge CSV file which contains 100000 rows. However, I need to do some processing and so for that I need to have a single space before the first comma. My current data is something like below. ...
0
votes
1answer
408 views

Awk with multiple delimiters

I am trying to execute awk on something like this : <tag x="whatever" y="http://mysite.com/somewords/word1/word2/word3/word4"> I would like to extract word2 which comes forever after word1, ...
2
votes
2answers
4k views

Insert text in specific lines of a file?

How can insert some text in specific lines of a file? What is most simplest method that I can use? (bash, sed, awk?) What I want to do, might be simple, but I don't know where I should start. It's ...
4
votes
3answers
506 views

using awk to split a line on single spaces not multiples

I'm trying to split a line that I have no control over the format of. If parameter 7 and 8 are missing which is possible they will be replaced by a space so I would end up with, field1 field2 field3 ...
4
votes
1answer
152 views

Matching Five Columns in two Files using Awk

I have two Input files. File1: s2/80 20 . A T 86 F=5;U=4 s2/20 10 . G T 90 F=5;U=4 s2/90 60 . C G 30 ...
3
votes
4answers
364 views

How to Prefix a column values with an apostrophe ( ' )?

I have a CSV file with multiple columns and 1000's of records, I need to prefix all the values of one of the columns (lets say 2nd column) with an apostrophe ' except in the first line or header line, ...
7
votes
3answers
245 views

Distributing a script: Should I use /bin/gawk or /usr/bin/gawk for shebang?

Is gawk in /bin or /usr/bin usually? I would go with #!/usr/bin/env gawk but then I can't use arguments. Right now I'm using #!/bin/gawk -f. The script is very long and contains a lot of single ...
4
votes
3answers
3k views

Filter a .CSV file based on the 5th column values of a file and print those records into a new file

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , ...
0
votes
2answers
107 views

why does `awk` fail when I put it in a bash script

on the prompt: ls | awk '{printf("%s\t\t%s\n", $9, $3);}' succeeds in printing what it is supposed to be printing. If I put it in a file, I get four or five empty lines (about the number of lines ...
-1
votes
1answer
47 views

A triangle of forward differences in gawk

I need a gawk script (not a one-liner, I need a function). Given a row of integers i.e. 0 2 4 1 9 5 1 1 the function should generate successive forward absolute differences, so: 0 2 4 1 9 5 1 1 2 ...
4
votes
5answers
383 views

Multi-line grep beween two tags, only show the last match

I'm having a program that intermingles data segments into log output: log message log message ----BEGIN INLINE DATA---- data data data -----END INLINE DATA----- log messge I'm looking for something ...
3
votes
3answers
258 views

awk read error bad address

I have a file that is roughly 30GB. I am using, awk ' $1 <= 2000 ' myfile.txt >> myfile_new.txt But, I get this error after a while - awk: read error (Bad address) I guess this is ...
1
vote
1answer
42 views

Unable to unregister vms

I have the following command to extract the name of virtual machines currently available in the host system. vmslist=$(vboxmanage list vms | awk '{print $1}') I pass this list to a for loop in ...
2
votes
3answers
4k views

Multiple pattern match and print in single line

I need to match two patterns in a log file and need to get the next line of the one of the pattern (out of two patterns) that is matched, finally need to print these three values in a single line. ...
4
votes
2answers
2k views

recursive search for a pattern, then for each match print out the specific SEQUENCE: line number, file name, and no file contents

What I am after is almost exactly the same as can be found here, but I want the format "line number, separator, filename, newline" in the results, thus displaying the line number at the beginning of ...
2
votes
4answers
526 views

Format a Date field of a .CSV file with multiple commas in a string field

I have a .CSV file (file.csv) whose data are all enclosed in double quotes. Sample format of the file is as below: column1,column2,column3,column4,column5,column6, column7, Column8, Column9, Column10 ...
2
votes
2answers
192 views

Shell script and adding large values in a delimited file problem

I have a script that reads through a delimited file and adds up the 3rd element in the file for each record. For a majority of the data files, this works fine except for one. I have one data file were ...
2
votes
2answers
112 views

Awk not rendering any result

My goal is to get the file created in the current month in a directory. It seems that the command is correct but not rendering any result: Date=`date '+%b'` echo $Date Oct ls -l | awk -v d="$Date" ...
3
votes
1answer
357 views

Walking Multidimensional arrays in Mawk

I'm able to do this properly in gawk, but when I tried to post my code to the machine where it will run, I realized it was using mawk... $ cat multidim.gawk # test of multidimensional arrays // { ...
3
votes
2answers
755 views

Replace a pattern in a file with a long string

I'm writing a little shell script to generate a directory listing. To make the output easily customizable, the script just builds a HTML-table and than should replace a specific token in a template ...
3
votes
4answers
866 views

Align text to center with padding on both sides

Is there an easy way to align text to the center with padding on both sides, with the column width being the longest line from the input? For example, this: aaa bbbb c ddddd ee will turn into this ...
5
votes
3answers
306 views

Deleting Lines if there is a dot in the 5th Column

I have a file and I want to remove all the line if it has a dot in the 5th Column. Input File: sc2/80 20 . A C 80 PASS N=2 F=5;U=4 sc2/60 ...
2
votes
1answer
142 views

How to extract single value from single line tuples?

I have a dump file of a database in the following form ('value1','value2',value3','value4'),('value1','value2',value3','value4'),... I want first to have each tuple in a single line and next i ...
1
vote
2answers
943 views

Printing fields using awk

< pool.sam awk ' /./ {printf $1} {printf $7+1,"\t"} {printf $3,"\t"} { if($2 !=16) {print "\t", "+";} else {print "\t","-";} {printf $4,"\t" ,length($10)+$4, "\t", "1"} }' I am ...
3
votes
1answer
337 views

Efficient way to print lines from a massive file using awk, sed, or something else? [duplicate]

If I had a plain-text file containing 8 million lines and I want to print out lines 4,000,010 to 4,000,000 to the screen, which would be more efficient: awk or sed? There is no pattern to the text, ...
1
vote
2answers
88 views

AWK special character

If i want to awk the below pattern in a file Mobilenumber '91_987986787688899' was successful I used the below code awk '/Mobilenumber '91_987986787688899' was successful/ {print $0}' Log Filename ...
0
votes
1answer
1k views

shell script to print rows if there is a value in column 2

1 **2** 3 4 3 1 4 2 1 4 3 2 4 **2** 1 3 How can I print all rows, on which there is a 2 in the second column.
1
vote
2answers
209 views

how to calculate the average value of the column(big number of columns) that have the row ID?

I want to calculate the average value of my data when they have the same ID in the first column. I want to calculate the average values for the remaining columns that have the same value in the first ...
3
votes
3answers
696 views

Count the number of rows in each column of a text file

I have a text file with the following format: 5 3 1 2 3 4 ..... ..... i.e. space separated 3 columns of numbers/ However, some of the rows may be like: 2 3 1 So, I want to detect if the text file has ...
1
vote
1answer
210 views

How to use awk or sed to convert csv diffs into more readable format

Can anyone give me the example of how to use awk or sed (not sure which one, as I haven't use either of these much, as I have mostly been using grep and cut to work with csv data) to convert ...