Questions tagged [awk]

A pattern-directed scanning and processing language.

516 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4 votes
0 answers
479 views

How to use awk-it in Emacs?

I'm a pretty new Emacs user and I want to call awk from Emacs. After googling around I found awk-it at http://www.emacswiki.org/emacs/AWK_It. However I still can't understand how to use it after I ...
user avatar
3 votes
3 answers
189 views

How to grep data of a new line when capturing a keyword?

I have a file which contains below sample line having a common keyword P00001121 in each line. [H00,P00001121] CHANNEL__OPEN:TCP_ADDRESS='10.32.130.202' TCP_PORT='80'" [H00,P00001121] COMMS_QUEUING:...
user avatar
  • 31
3 votes
2 answers
2k views

I want to parse a Terraform vars file and store variables from the TF vars file as vars in a Makefile

I have two files: a Terraform variables file (vars.tf), and a Makefile. What I want to be able to do, is to have the Makefile "read" some of the variables in the Terraform vars.tf file. The vars.tf ...
user avatar
3 votes
3 answers
887 views

search column 2 in csv file for value, if value, then insert "invalid" and shift cells right

I have csv file that is auto generated by a script but for some of the records (line items) received I need to search column 2 and if the values contains "*.app" I need to print "INVALID" into column ...
user avatar
  • 171
3 votes
2 answers
206 views

Merge 2 huge files matching multiple columns and preserving the order (print matching and non-matching values) - scale up from awk

I'm having trouble with merging data from 2 files. It is genetic data with chromosome, position, reference and alternate alleles and I need to merge the files by matching all 4 of those columns - with ...
user avatar
3 votes
3 answers
3k views

Sum multiple columns

I need to do multiple sums; my input file is: DATE|NATION|CITY|FILES|REVENUE|FREQUENCY|INVESTMENT 20170807|USA|VIRGINIA|TIMES|1919150|1779|282075 20170807|USA|NYC|ROADS|92877|41|1599 20170808|USA|...
user avatar
  • 31
3 votes
3 answers
124 views

How to use an input file of non-exact filenames for an operation?

I have a file of filenames for files in a certain directory. However, some of the file names may: Have spaces in the original filename replaced with underscores (so directory/file with spaces ...
user avatar
  • 223
3 votes
3 answers
2k views

awk with regex for delimiter

I want to get the first word in every line from a file. Unfortunately a lot of lines begin with space(s). So I try to get the firs word with the following: awk -F'[ \t]+' '{print $1}' < MyFile.txt,...
user avatar
  • 131
3 votes
3 answers
227 views

How to extract logs between two time stamps, line may or may not start with datetime

How to extract logs between two time stamps, line may or may not start with datetime. I am trying below and it is only extracting the lines that starts with datetime only. Datetime format is 2014-04-...
user avatar
  • 31
3 votes
4 answers
2k views

how to run awk from specific line number stored in a variable

I want to get the line numbers of the first appearance of bar after the appearance of foo. This should run in a while loop, for the whole file. Like this: test_file: bar foo xxx xxx xxx bar bar xxx ...
user avatar
  • 31
2 votes
0 answers
262 views

How to awk only output-lines containing a keyword?

I am in the process of writing a small script, which checks browser-tabs on certain keywords. If the keyword is present, then the script closes the tab. The script is actually a simple "block-...
user avatar
2 votes
1 answer
25 views

How to introduce a new header to a CSV file based on condition using gawk

I have a csv file from I wish to extract only column number 7 and 11. Depending on the value of column 7 which has either value OK or KO I wish to insert a column next to it called value where the ...
user avatar
2 votes
1 answer
112 views

AWK: Extract lines with values that match and less than values in another file?

I have a file called file1 on my Linux machine with these contents: ID Time Energy 43 4.38665978376386365240533e-05 3.215e-02 43 4.38666558838421885677290e-05 4.788e-01 48 4....
user avatar
2 votes
1 answer
377 views

Passing PID from ps command to netstat for port #

I have a process that can have multiple instances. ps shows the instance name, whereas netstat of course gives the name of the binary instead of the instance name. My goal is to use something like ...
user avatar
2 votes
1 answer
112 views

awking for command options from man page

I am trying to auto-generate tab-completions for different commands. I am piping man pages for different commands into awk, then searching for command line options (eg. -shortopt --long-option) ...
user avatar
2 votes
1 answer
263 views

how to awk if with multiple line condition

i have some problem with our log , need some alert if have some match condition. some log below condition : 2019-07-20|20:25|sorry system error|183 2019-07-20|20:25|Internal Error|8 2019-07-20|20:25|...
user avatar
2 votes
2 answers
239 views

How to select first occurrence from between two patterns including them

What I have: interface GigabitEthernet0/2/1/6.2061 description --> Some description here service-policy input BESTEFFORT-IN service-policy output Egress_30M vrf VPN-s2s ipv4 mtu 1500 ipv4 ...
user avatar
2 votes
1 answer
90 views

Why awk script is shuffling the last two fields?

My working file looks like this Time[s] A_in B_in C_in D_in E_in 0.000000E+00 1.120200E-02 1.432000E-01 2.331500E-04 2.349800E-03 2.800000E-03 2....
user avatar
  • 33
2 votes
2 answers
116 views

Remove duplicates in a very large file()

In this given input file, there are 4 columns. We have to remove the duplicates but there is a catch. There is a preference order C2>C3>C4. so in the output there is only one row with a, one row with ...
user avatar
2 votes
1 answer
130 views

uniq -c Equivalent for Groups of Lines of Arbitrary Count

I've got a file of ~1-2 million lines that I'm trying to reduce down by counting duplicate groups of lines, preserving order. uniq -c works okay : $ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz"...
user avatar
  • 413
2 votes
2 answers
926 views

Parsing FIX messages for further analytics

I'm currently trying to parse my FIX messages to get 2 columns showing currency (tag55) and price (tag133) but having difficulty using 'awk' as it appears the desired part of the message isn't split ...
user avatar
2 votes
1 answer
241 views

Formatting and fill missed data based on column

I have the data in this format and I want to sort it based on the label in each column. Would want to replace the missing field by a comma (,) like shown below 2018-03-13 02:09:12,AB=123,BC=6,CAB=Jax,...
user avatar
  • 37
2 votes
1 answer
185 views

replacing values in one with the values in another file

I have a csv file named eche in following format: INCON,--,INITIAL,CONDITIONS,FOR*****,ELEMENTS,AT,TIME ,0.315570E+13 VC76,0.10000000E+00,0.2837726135782E+08,0.6756896308414E+02 1K02,0.10000000E+00,0....
user avatar
  • 21
2 votes
1 answer
320 views

Gnu AWK split by pattern and insert columns , but get more delimiter

I have used same awk file to process two different string. So wired. cat test.awk BEGIN{FPAT="([^,])*|(\"[^\"]+\")";OFS=","}{$4="TDP,-1,-1,0,0"OFS$4;print $0} echo "a,b,b,b,b,b,b,b,b,b,b,b,\"a,b\"" ...
user avatar
  • 21
2 votes
2 answers
717 views

compare files basis two columns and add field

I have files with column-wise dates and time in "YYYY MM DD HHMM" format plus a variable (temperature) and want to convert them into YYYY DDD format (and keep hour and temperature as is). They look ...
user avatar
  • 21
2 votes
0 answers
305 views

use gff2fasta instead of a bash script to get parts of DNA sequences out of a full genome

EDIT and a solution Because my original question was badly phrased and I was trying to re-invent the wheel I am answering my own question now (maybe it helps someone else): gff2fasta is a tool which ...
user avatar
  • 143
2 votes
4 answers
126 views

summing two matrixes each one in a different file

i have 2 files each contains like the following: file 1: data : [ 56, 34, 23, 54, 90, 234, 53, 12 ] file 2: data : [ 42, 56, 23, 98, 90, 23, 53, 32 ] i want to sum 1st value from ...
user avatar
2 votes
1 answer
310 views

awk in solaris 5.8 / get value from two fields/lines

I'm new to UNIX-Solaris, and I got to make a script and I'm having an issue with a value I require. Basically I need to detect if two process are running. I find out I can get that info using the ...
user avatar
  • 73
2 votes
0 answers
1k views

awk to process very large file

I have a file here where I want to tag each line with FILEB and FILEC as references. If column 1 of fileb or filec is in column 2 of filea, get the column 2 of fileb or filec (if column 1 file c ...
user avatar
  • 53
2 votes
3 answers
1k views

Finding a specific character at one to many positions in a file

I am looking for a way to iterate through fixed format text files and looking for a specific character in one to many specific character positions, including the first character position, within any ...
user avatar
  • 21
2 votes
2 answers
269 views

sed match word1 and word2 to get word in between

Im having issues with trying to pull an ip address our of a log file call full. This is what the line in the file looks like [2016-11-10 11:22:42] NOTICE[24518] chan_sip.c: Registration from '"...
user avatar
  • 21
2 votes
2 answers
2k views

Read from and append to file at the same time while preserving end-of-line

Conceptually I have an easy task... I have [loosely] structured data in a file: Testing: debug, default CXXFLAGS <100's of additional output lines> Testing: release, default CXXFLAGS <100's ...
user avatar
2 votes
0 answers
208 views

Paging with follow less +F and colorize with awk

I would like to paginate a file with follow, i.e. $less +F and also colorize the output on the screen. I want to use less because I can leave the follow mode, browse the file and then re-enter the ...
user avatar
  • 121
2 votes
2 answers
226 views

Exchanging One Column From File1 with files from File2

I have two files. The first file has 11 columns, for example: 1 2 3 4 5 6 7 8 9 10 11 The second has 10 columns and might look like this: 11 22 33 44 55 66 77 88 99 100 What I want to do is look ...
user avatar
  • 33
2 votes
3 answers
952 views

Using awk to merge 2 files

I would like to merge 2 files based on column 1, 2, 3. I have tried the awk command below but is not working. awk 'NR==FNR {h[$1FS$2FS$3]=$4; next}{k=$1FS$2FS$3; if (k in h) print $1,$2,$3,$4,h[k] ;...
user avatar
  • 21
2 votes
2 answers
555 views

Compare sets of values from two files and print the output in a third file using awk

I have a problem comparing two files. I'll try my best to explain. I have two files f1 and f2 which I have to compare on basis of two sets Parameter(p) and Value(v). Basically f1 contains a list of ...
user avatar
1 vote
1 answer
78 views

Substitution in html file using awk

My awk script reads records from file1, finds the same records in file2 and substitutes alternate positions (of the record) with a defined symbol in that. But few of the values are not getting ...
user avatar
1 vote
2 answers
142 views

How to programmatically maintain umask in /etc/profile and /etc/bashrc

I am looking for a way to programmatically maintain a consistent umask in the files /etc/profile and /etc/bashrc. They have an entry such as follows: if [ $UID -gt 199 ] && [ "`/usr/bin/...
user avatar
  • 11
1 vote
1 answer
48 views

Unix Script - On specific pattern, need to exit tail -f, and use awk to run separate script

So, I need to make a modification to a start script to clean up the /tmp once the application is fully loaded. After some research, I found a command for my script that does this: tail -f /home/user1/...
user avatar
  • 133
1 vote
1 answer
151 views

How to find the standard deviation of values in a file using awk to parse 500 files at once?

I have 500 files each with a column of values I need to find the standard deviations for. I've figured out this much: awk '{sum+=$5; array[NR]=$5} END {for(x=1;x<=NR;x++){sumsq+=((array[x]-(sum/NR))...
user avatar
  • 11
1 vote
0 answers
75 views

Output of 2 commands get the difference (ls vs du)

So analyzing an issue at work I found that, when I created some datafiles on Oracle, the datafiles didn't reserve the space in the Filesystem, so I went to the sapdata filesystem where I put the ...
user avatar
1 vote
0 answers
56 views

awk printf -x producing different output on different mawk versions

Here is the problem. Running: echo '<position x="25" y="-9"/>' | awk -F '[="]' '{ printf ("%.2x", $6) }' produces different output on bash GNU bash, version 5....
user avatar
  • 141
1 vote
1 answer
60 views

I want to merge the output of top with the start time from ps

I have several services running and a bash script that displays their current state using top e.g. top -b -n1 -p `pgrep -f "proc1|proc2|proc3|proc4|proc5" | tr '\n' ',' | # ...
user avatar
1 vote
1 answer
104 views

How to compare two column from two files with specific condition in awk

I have a data file A.tsv (field separator = \t) : id clade mutation 243 40A titi,xixi,lolo 254 20B titi,toto,jiji,lala 261 267 20B lala,jiji,jojo and a template file B.tsv (field separator = \t) : ...
user avatar
  • 499
1 vote
2 answers
95 views

How to divide multiple values within a column by a constant?

I am trying to divide the values in 2 columns by a certain constant. However, there are multiple values in each column that are separated by a colon. The file is very large (24 files, 2-3 GB each) For ...
user avatar
1 vote
0 answers
92 views

What is good practice: write awk inside bash script, or call bash command inside awk script?

I was asigned to write a shell script to do some check on server, such as backup files, firewall status, etc. Personally, I don't like bash script's syntax, so I consider writing most of the logic in ...
user avatar
  • 1,047
1 vote
1 answer
72 views

Problem with awk getline and reading file in nest if statements

Before I added the else if/else, when i only just had the: if (/Easy: /) .. part The program would launch when I put awk -f oscp-notes.awk and then put input, &c. It would then print out ...
user avatar
1 vote
1 answer
72 views

Awk's behaviour changes when inside a script

I have a text file, "invoice.txt" containing a line beginning "Vendor Invoice Number" which contains an order number field of the form RCTI- followed by alphanumeric characters ...
user avatar
  • 11
1 vote
3 answers
56 views

Find matching text in consecutive lines and increment numeric value in the following line

I'm trying to find a string(Name2) in a file and match the version(*abc-xyz-0-197) in the next line and update the ID line with an incremented value. How can this be done? The file looks like this: ...
user avatar
1 vote
1 answer
302 views

Replace string with line range from another file, including modification of line range on replace

I have two files of code, the first (File1) including a few lines of text to be included and the second (File2) including strings that will be replaced with a range of lines from the first file. The ...
user avatar

1
2 3 4 5
11