Questions tagged [awk]

A pattern-directed scanning and processing language.

493 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
0answers
436 views

How to use awk-it in Emacs?

I'm a pretty new Emacs user and I want to call awk from Emacs. After googling around I found awk-it at http://www.emacswiki.org/emacs/AWK_It. However I still can't understand how to use it after I ...
3
votes
3answers
117 views

How to grep data of a new line when capturing a keyword?

I have a file which contains below sample line having a common keyword P00001121 in each line. [H00,P00001121] CHANNEL__OPEN:TCP_ADDRESS='10.32.130.202' TCP_PORT='80'" [H00,P00001121] COMMS_QUEUING:...
3
votes
1answer
922 views

I want to parse a Terraform vars file and store variables from the TF vars file as vars in a Makefile

I have two files: a Terraform variables file (vars.tf), and a Makefile. What I want to be able to do, is to have the Makefile "read" some of the variables in the Terraform vars.tf file. The vars.tf ...
3
votes
3answers
500 views

search column 2 in csv file for value, if value, then insert “invalid” and shift cells right

I have csv file that is auto generated by a script but for some of the records (line items) received I need to search column 2 and if the values contains "*.app" I need to print "INVALID" into column ...
3
votes
1answer
182 views

Merge 2 huge files matching multiple columns and preserving the order (print matching and non-matching values) - scale up from awk

I'm having trouble with merging data from 2 files. It is genetic data with chromosome, position, reference and alternate alleles and I need to merge the files by matching all 4 of those columns - with ...
3
votes
3answers
2k views

Sum multiple columns

I need to do multiple sums; my input file is: DATE|NATION|CITY|FILES|REVENUE|FREQUENCY|INVESTMENT 20170807|USA|VIRGINIA|TIMES|1919150|1779|282075 20170807|USA|NYC|ROADS|92877|41|1599 20170808|USA|...
3
votes
3answers
99 views

How to use an input file of non-exact filenames for an operation?

I have a file of filenames for files in a certain directory. However, some of the file names may: Have spaces in the original filename replaced with underscores (so directory/file with spaces ...
3
votes
3answers
1k views

awk with regex for delimiter

I want to get the first word in every line from a file. Unfortunately a lot of lines begin with space(s). So I try to get the firs word with the following: awk -F'[ \t]+' '{print $1}' < MyFile.txt,...
3
votes
3answers
215 views

How to extract logs between two time stamps, line may or may not start with datetime

How to extract logs between two time stamps, line may or may not start with datetime. I am trying below and it is only extracting the lines that starts with datetime only. Datetime format is 2014-04-...
2
votes
1answer
17 views

How to introduce a new header to a CSV file based on condition using gawk

I have a csv file from I wish to extract only column number 7 and 11. Depending on the value of column 7 which has either value OK or KO I wish to insert a column next to it called value where the ...
2
votes
0answers
200 views

compare two files on the basis of first columns of two file and get matching and non-matching rows

I have to compare two tab-delimited files on the basis of 1st column and I have to print the entire row of the first file if the first columns match with the second file. Also, I need to extract those ...
2
votes
1answer
54 views

AWK: Extract lines with values that match and less than values in another file?

I have a file called file1 on my Linux machine with these contents: ID Time Energy 43 4.38665978376386365240533e-05 3.215e-02 43 4.38666558838421885677290e-05 4.788e-01 48 4....
2
votes
1answer
168 views

Passing PID from ps command to netstat for port #

I have a process that can have multiple instances. ps shows the instance name, whereas netstat of course gives the name of the binary instead of the instance name. My goal is to use something like ...
2
votes
1answer
95 views

awking for command options from man page

I am trying to auto-generate tab-completions for different commands. I am piping man pages for different commands into awk, then searching for command line options (eg. -shortopt --long-option) ...
2
votes
1answer
81 views

compare columns from two different files and PRINT RECORDS FROM FIRST FILE those that DO NOT match from second file

I would like to compare columns from file one to two. Where column 2 of file1 should un-match from column 1 or 2 from file2 and print output from file 1. file1. cat test.head20.R2.fastq.tab @...
2
votes
1answer
69 views

how to awk if with multiple line condition

i have some problem with our log , need some alert if have some match condition. some log below condition : 2019-07-20|20:25|sorry system error|183 2019-07-20|20:25|Internal Error|8 2019-07-20|20:25|...
2
votes
2answers
93 views

How to select first occurrence from between two patterns including them

What I have: interface GigabitEthernet0/2/1/6.2061 description --> Some description here service-policy input BESTEFFORT-IN service-policy output Egress_30M vrf VPN-s2s ipv4 mtu 1500 ipv4 ...
2
votes
1answer
86 views

Why awk script is shuffling the last two fields?

My working file looks like this Time[s] A_in B_in C_in D_in E_in 0.000000E+00 1.120200E-02 1.432000E-01 2.331500E-04 2.349800E-03 2.800000E-03 2....
2
votes
1answer
84 views

Remove deduplicates in a very large file()

In this given input file, there are 4 columns. We have to remove the duplicates but there is a catch. There is a preference order C2>C3>C4. so in the output there is only one row with a, one row with ...
2
votes
1answer
90 views

uniq -c Equivalent for Groups of Lines of Arbitrary Count

I've got a file of ~1-2 million lines that I'm trying to reduce down by counting duplicate groups of lines, preserving order. uniq -c works okay : $ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz"...
2
votes
2answers
87 views

merging two files according to the specified row

I have file1.txt looks as follows: search http://google.com mail https://gmail.com file2.txt looks like this: This is a search tool This is your mail account I need to merge them in the way ...
2
votes
2answers
757 views

Parsing FIX messages for further analytics

I'm currently trying to parse my FIX messages to get 2 columns showing currency (tag55) and price (tag133) but having difficulty using 'awk' as it appears the desired part of the message isn't split ...
2
votes
1answer
204 views

Formatting and fill missed data based on column

I have the data in this format and I want to sort it based on the label in each column. Would want to replace the missing field by a comma (,) like shown below 2018-03-13 02:09:12,AB=123,BC=6,CAB=Jax,...
2
votes
2answers
2k views

how to run awk from specific line number stored in a variable

I want to get the linenumbers of the first appearance of "bar" after the apperance of "foo". This should run in a while loop, for the whole file. Like this: test_file: foo xxx xxx xxx bar bar xxx ...
2
votes
1answer
131 views

replacing values in one with the values in another file

I have a csv file named eche in following format: INCON,--,INITIAL,CONDITIONS,FOR*****,ELEMENTS,AT,TIME ,0.315570E+13 VC76,0.10000000E+00,0.2837726135782E+08,0.6756896308414E+02 1K02,0.10000000E+00,0....
2
votes
1answer
268 views

Gnu AWK split by pattern and insert columns , but get more delimiter

I have used same awk file to process two different string. So wired. cat test.awk BEGIN{FPAT="([^,])*|(\"[^\"]+\")";OFS=","}{$4="TDP,-1,-1,0,0"OFS$4;print $0} echo "a,b,b,b,b,b,b,b,b,b,b,b,\"a,b\"" ...
2
votes
2answers
664 views

compare files basis two columns and add field

I have files with column-wise dates and time in "YYYY MM DD HHMM" format plus a variable (temperature) and want to convert them into YYYY DDD format (and keep hour and temperature as is). They look ...
2
votes
0answers
273 views

use gff2fasta instead of a bash script to get parts of DNA sequences out of a full genome

EDIT and a solution Because my original question was badly phrased and I was trying to re-invent the wheel I am answering my own question now (maybe it helps someone else): gff2fasta is a tool which ...
2
votes
4answers
125 views

summing two matrixes each one in a different file

i have 2 files each contains like the following: file 1: data : [ 56, 34, 23, 54, 90, 234, 53, 12 ] file 2: data : [ 42, 56, 23, 98, 90, 23, 53, 32 ] i want to sum 1st value from ...
2
votes
1answer
271 views

awk in solaris 5.8 / get value from two fields/lines

I'm new to UNIX-Solaris, and I got to make a script and I'm having an issue with a value I require. Basically I need to detect if two process are running. I find out I can get that info using the ...
2
votes
0answers
1k views

awk to process very large file

I have a file here where I want to tag each line with FILEB and FILEC as references. If column 1 of fileb or filec is in column 2 of filea, get the column 2 of fileb or filec (if column 1 file c ...
2
votes
3answers
951 views

Finding a specific character at one to many positions in a file

I am looking for a way to iterate through fixed format text files and looking for a specific character in one to many specific character positions, including the first character position, within any ...
2
votes
2answers
213 views

sed match word1 and word2 to get word in between

Im having issues with trying to pull an ip address our of a log file call full. This is what the line in the file looks like [2016-11-10 11:22:42] NOTICE[24518] chan_sip.c: Registration from '"...
2
votes
2answers
2k views

Read from and append to file at the same time while preserving end-of-line

Conceptually I have an easy task... I have [loosely] structured data in a file: Testing: debug, default CXXFLAGS <100's of additional output lines> Testing: release, default CXXFLAGS <100's ...
2
votes
0answers
170 views

Paging with follow less +F and colorize with awk

I would like to paginate a file with follow, i.e. $less +F and also colorize the output on the screen. I want to use less because I can leave the follow mode, browse the file and then re-enter the ...
2
votes
2answers
4k views

how to set a variable to multiple values with awk?

I have a file similar to the following : x 0 1 x 1 1 and I am essentially trying to see if both fields are equal to one another or not for every row. But the problem is that this file contains x ...
2
votes
2answers
204 views

Exchanging One Column From File1 with files from File2

I have two files. The first file has 11 columns, for example: 1 2 3 4 5 6 7 8 9 10 11 The second has 10 columns and might look like this: 11 22 33 44 55 66 77 88 99 100 What I want to do is look ...
2
votes
3answers
826 views

Using awk to merge 2 files

I would like to merge 2 files based on column 1, 2, 3. I have tried the awk command below but is not working. awk 'NR==FNR {h[$1FS$2FS$3]=$4; next}{k=$1FS$2FS$3; if (k in h) print $1,$2,$3,$4,h[k] ;...
2
votes
2answers
547 views

Compare sets of values from two files and print the output in a third file using awk

I have a problem comparing two files. I'll try my best to explain. I have two files f1 and f2 which I have to compare on basis of two sets Parameter(p) and Value(v). Basically f1 contains a list of ...
1
vote
1answer
47 views

I want to merge the output of top with the start time from ps

I have several services running and a bash script that displays their current state using top e.g. top -b -n1 -p `pgrep -f "proc1|proc2|proc3|proc4|proc5" | tr '\n' ',' | # ...
1
vote
1answer
88 views

How to compare two column from two files with specific condition in awk

I have a data file A.tsv (field separator = \t) : id clade mutation 243 40A titi,xixi,lolo 254 20B titi,toto,jiji,lala 261 267 20B lala,jiji,jojo and a template file B.tsv (field separator = \t) : ...
1
vote
0answers
63 views

How to awk only output-lines containing a keyword?

I am in the process of writing a small script, which checks browser-tabs on certain keywords. If the keyword is present, then the script closes the tab. The script is actually a simple "block-...
1
vote
2answers
66 views

How to divide multiple values within a column by a constant?

I am trying to divide the values in 2 columns by a certain constant. However, there are multiple values in each column that are separated by a colon. The file is very large (24 files, 2-3 GB each) For ...
1
vote
0answers
46 views

What is good practice: write awk inside bash script, or call bash command inside awk script?

I was asigned to write a shell script to do some check on server, such as backup files, firewall status, etc. Personally, I don't like bash script's syntax, so I consider writing most of the logic in ...
1
vote
1answer
33 views

Problem with awk getline and reading file in nest if statements

Before I added the else if/else, when i only just had the: if (/Easy: /) .. part The program would launch when I put awk -f oscp-notes.awk and then put input, &c. It would then print out ...
1
vote
1answer
53 views

Awk's behaviour changes when inside a script

I have a text file, "invoice.txt" containing a line beginning "Vendor Invoice Number" which contains an order number field of the form RCTI- followed by alphanumeric characters ...
1
vote
3answers
52 views

Find matching text in consecutive lines and increment numeric value in the following line

I'm trying to find a string(Name2) in a file and match the version(*abc-xyz-0-197) in the next line and update the ID line with an incremented value. How can this be done? The file looks like this: ...
1
vote
1answer
51 views

Replace string with line range from another file, including modification of line range on replace

I have two files of code, the first (File1) including a few lines of text to be included and the second (File2) including strings that will be replaced with a range of lines from the first file. The ...
1
vote
0answers
115 views

Using awk to join on multiple fields

I have file1 as follows: 2010,92614,0,1 2010,92614,4,1 and file2 as follows: 2010,0,907894 2010,1,17788 2010,2,2827 2010,3,1212 2010,4,669914 I would like to join the two files on fields 1 and 3 of ...

1
2 3 4 5
10