Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gawk]

gawk is the GNU implementation of awk. The awk utility interprets a special-purpose programming language that makes it possible to handle simple data-reformatting jobs with just a few lines of code.

1
vote
1answer
22 views

AWK if loop - If any value is less than one

I produce a huge multi-column output file running some analytical software on angle populations. Sadly, the angles are defines as -180° to 180° and I need 0°-360°. What I want do to is take this ...
-1
votes
1answer
24 views

awk compare array to array on previous line

I would appreciate your help. I am extracting information from data files that are in some cases more than a terabyte in size. Variables on each line are are separated by whitespace. Quantity of ...
1
vote
1answer
16 views

awk printing out an associative array with OFS

I have the following awk code. It first determines the field numbers of interest in the header line and then it prints the output of those fields. The issue is that I am using an associative array to ...
1
vote
3answers
61 views

I want to know which word is duplicated on a line and how many times it appears in a file

example: I have file a.txt that contains the following: aunt bear bear true I also have a file b.txt that contains the following: tru tru high the end result I want is : for a.txt : bear x 2 for ...
1
vote
1answer
36 views

How to replace substrings with values taken from a dictionary indexed by the substrings

I need to parse a big file replacing possibly occurring substrings, that are part of possibly occurring strings that match a regex, with a value taken from an array whose indexes are the substrings in ...
0
votes
2answers
41 views

AWK - Need to display total value of field and field number

I have data below that needed to count or total the value start with specific field. Sample below the needed field to compute is from Field5. For sample Field5 total is 42, field6 total is 107, etc.. ...
1
vote
3answers
70 views

AWK - How to count stored or index on array

I have data that need to be count how many occurrences. 1 2 1 2 3 1 And used the awk command below and work properly. awk -F'","' '{ a[$1]++ } END { for (n in a) print n "=" a[n] } ' text.txt And ...
1
vote
1answer
70 views

Improve performance when using “system” call (shell escape) processing large files in awk

I have an awk script that processes very large files that look something like this: K1353 SF3987.7PD833391.4 KARE K1353 SF3987.2KD832231.4 MEAKE K1332 IF4987.7RP832231.2 LEAOS K1329 SF2787.7KD362619....
0
votes
1answer
30 views

Gawk cross referencing

I'm having a issue with cross referencing data using gawk, when I use: gawk -F: 'FNR==NR{a[$1]=$2;next} $2 in a{print $1 FS a[$2]}' email.phone.txt name.email.txt > name.phone.txt Example of ...
1
vote
3answers
75 views

If line contains pattern, print variable to column

I have the file DEMLIR-GEO_OPT-1-distance-1.coordLog: REQUESTED STRUCTURE DATA Distance vector r(i,j) between the atom i and j in ANGSTROM r(1,5) = 0.944776 0.190651 1....
0
votes
0answers
20 views

Modifying text file using (g)awk [duplicate]

Is it possible to make a modification to a text file for example and save those modification to the same file using (g)awk script/command? All I see is only a modification to the printed output and ...
-1
votes
2answers
26 views

extract specific fields from a file using a character delimiter

I have a 1gb file and would like to get the third and fifth fields. The file is delimited by (,): input file: luis, jose, jesus atoche, mark, john 1,2,3,4,5 aa 1, bb 2, cc3, dd 4, ee5 output file: ...
0
votes
1answer
26 views

modify between (only) 1st occurrence of patterns (inclusive of patterns) and keep processing

There are several questions about matching between only the first occurrence of two patterns (e.g., this) — but they all seem to rely on exit, which is no good, as I want to keep processing the file. ...
9
votes
4answers
958 views

awk assign to multiple variables at once

I'm trying to pull two numerical values out of a string and assign them to variables using awk (gawk is what I'm using specifically). I want to pull the major and minor version numbers out of a tmux ...
4
votes
2answers
265 views

GNU awk ERRNO not set on command failure

I was trying run a few commands using getline() function of GNU awk and print the error number (errno) value returned. But for simple failure cases of non existent directory/file the variable doesn't ...
1
vote
1answer
65 views

Using NF with FPAT – what am I doing wrong?

When NF is used with FPAT regex, the comma is considered a field. I prefer using NF and FPAT: 1) NF – to limit the output to the actual number of fields for the record 2) FPAT – to handle an ...
5
votes
1answer
97 views

Why does gawk treat `0123` as a decimal number when coming from the input data?

According to $ man gawk, the strtonum() function can convert a string into a number: strtonum(str) Examine str, and return its numeric value. If str begins ...
6
votes
4answers
451 views

Why does regex `“\.pdf”` match `/…/pdf…/…` in gawk, but not in mawk?

From How can I extract only the pid column and only the pathname column in the lsof output? awk '{ for (i=9; i<=NF; i++) { if ($i ~ "string" && $1 != "wineserv" && $5 == "REG" &...
0
votes
2answers
29 views

Using awk how to compare 2 fields and print something out

Let say I have file1 with the following content: 1,2,3,4,12,6 1,2,3,5,333,6 1,2,3,12,5,6 1,2,3,44,101,6 While file2 has the following content: 22,4,111,1,6,7,12 22,4,56,55,6,7,12 22,4,12,12,6,7,12 ...
1
vote
2answers
101 views

Shell-Script Bash

I have this loop in the shell script, but it seems I don't fully understand what it does: especially gawk -v for ((k=$1; $k<3 + 3 ; k=$k + $3)) do echo $k | gawk -v s=4 '{print $1*s}' done ...
2
votes
4answers
131 views

How to break line after matching case but needs to go back a couple of characters

I have the following sample: 1. TNT 00:00 2. Swing From The Gutters 7:34 3. Ten-Day Interval 13:29 4. I Set My Face To The Hillside 18:16 5. The Equator 24:26 6. A Simple Way To Go Faster Than Light ...
1
vote
1answer
154 views

Unexpected split behavior

I read about Awk split behavior here: [...] the fs argument to the split function (see String Functions) shall be interpreted as extended regular expressions. These can be either ERE tokens or ...
-2
votes
1answer
64 views

Can I use awk to perform some number or string operations, without specifying a file as input?

Awk has builtin functions which are operating on numbers and strings. That is exactly what a programming language does. That makes me wonder if I can use awk to perform some number or string ...
-3
votes
1answer
204 views

Are `match()`, `gensub()`, `gsub()` and `sub()` only concerned about records not fields?

In gawk, is it correct that match(), gensub(), gsub() and sub() don't consider the fields, and are not affected by change to FS and OFS? In other words, are they only concerned about records not ...
1
vote
4answers
339 views

How shall I perform multiline matching and substitution using awk?

In a text file, ignoring any trailing whitespace at the end of each line, I assume that if a line is not ended with a digit, then there is a line break between the line and the next line, and I would ...
1
vote
3answers
64 views

Organizing data by a header

I have two csv files. File 1 contains header. File 2 contains data file 1 format : file1.csv id,abc,xyz,aaa,bbb,ccc file 2 format : file2.csv id,source,value 1,abc,100 1,xyz,...
1
vote
1answer
20 views

(g)awk: Substituting expression from another script

I am building a generic script which will be re-used from several other scripts. For each case, I need to extract Nth field in the generic script; in some cases the double quotes should be removed ...
0
votes
1answer
26 views

How to join two files using two columns in unix

I have file a and file lookup and the structure is below. vi a.txt empid|ename|sal_grade|MANAGER_ID 1|raj|A|202 2|MAN|B|203 3|JOS|C|204 vi lookup.txt Grade|sal|manager_id A|$100000|202 A|$1000|099 ...
-1
votes
3answers
67 views

Remove a whole line beginning with a pattern occurring anywhere after another pattern

How can I remove any line beginning with --- occurring in any line number after a line containing # Match (the first occurrence in any line number)? EDIT: The relationship between #Match with the ...
2
votes
3answers
1k views

awk: extract string from a field [closed]

in the input fields are separated by pipe sign: CCCC|Sess C1|s1 DA=yy07:@##;/u/t/we DDDDD|Sess C2|s4 DB=yy8:@##;/u/ba I want to get output where last field is changed (extracted only what is between ...
0
votes
1answer
82 views

gawk process multiple files inside directory

I have multiple text files in a folder that I want to process through gawk, rather than process them each individually is it possible I can process them all at once without merging them into 1 .txt ...
0
votes
1answer
140 views

awk print from passwd but only with userid > 1000

Ok I've done: cat /etc/passwd | awk -F: '{ print $1 }' -F ==> Specify a file seperator for delineating data fields in a line $1 ==> Represents the first field in the input but how to only print ...
0
votes
1answer
62 views

gawk job getting killed, too large of input?

I can't seem to understand why a script I've been running fine up until now fails on a slightly larger input file. If I run the script on a file that is 820000x75 (lines by columns), the script works ...
4
votes
4answers
424 views

How to create a new file with required columns from different multiple files in linux?

I have a directory ballgown, in which there are around 1000 subdirectories as sample names. Each subdirectory has a file t_data.ctab. The filename is the same in all subdirectories. ballgown |...
0
votes
3answers
96 views

resume reading a log file from the point I left it last time [duplicate]

I have a log file which is continuously updating(new line added) after few time period. I am fetching only error messages from the file in every 10 minutes. Initially, at 1st time I fetched all line ...
1
vote
1answer
94 views

Gawk partial match

My gawk script isn't matching all potential results, it's only matching one of them.. Example - script: gawk -F: "FNR==NR{a[$2]=$1;next} $1 in a{print a[$1] FS $2}" username.email.txt email.phone....
0
votes
1answer
31 views

trying to execute base62 operation in unix with more than 25 digits in a row, but awk treating a default value

Below is my code: #!/usr/bin/ksh awk 'BEGIN {FS="|"; flag=0;} { x=$2 rem=int(x)%62 quo=int(x/62) flag=0 while(quo>62) { sub_rem=int(quo)%62 quo=int(quo/62) if(flag==0) { grp_rem=sub_rem } else { ...
2
votes
4answers
2k views

Passing arguments to awk script

I have a awk script where i want to be able to pass N arguments into it and also read from stdin. I would like to be able to do something like tail -f logfile | my_cool_awk_scipt var1 var2 var3 ... ...
0
votes
2answers
160 views

Awk command failing during the redirection with null sting value

I want to split the large file into pieces. Filename and the number records per file after split will be provided as arguments/variables to awk. EG: FILENAME=A_20 (say it has 100 records), ...
3
votes
1answer
295 views

Clone complex array in awk

I have an array like this ... a["a"] = "aa" a["b"] = "ab" a["c"] = "ac" a["d"]["a"] = "ada" a["d"]["b"]["a"] = "adba" a["e"]["a"]["a"]["a"] ...
1
vote
1answer
186 views

Gawk: backspace preceding a newline doesn't work

I'm using gawk (4.1.3) and it seems I have struck upon a trivial problem. The backspace character doesn't seem to work if it is immediately followed by a newline. awk 'BEGIN{for(i=1;i<=3;i++) ...
4
votes
1answer
961 views

Speed up text processing

I have the following script to process a file with some data: First, the header is printed to the output file. Then 60000 lines from the input are randomly picked and printed to the output (the ...
2
votes
1answer
107 views

Print random lines of file but with maximum count of a certain value

I have a input file from which I want to randomly choose a line. In a column in every line is a value, and I want only a number of lines up to a maximum count of this value to be choosen randomly, not ...
0
votes
1answer
88 views

processing multiple files with awk problem

I've the input files with fields separated with colon: main: one:111:222:333 fiv:333:222:333 two:123:234:500 file1: one:111:222:333 two:123:234:501 file2: one:111:222:333 thr:-:234:232 fiv:999:...
1
vote
1answer
110 views

socat and gawk piped

I'm trying to manipulate a TCP stream using gawk. The TCP server streams out a text string separated by whitespaces. I want to add an extra string at the beginning and then separate the remaining ...
1
vote
1answer
31 views

Using GAWK to sum certain numbers line by line

My lines will look as such X 2.63766751981802 Y 1.68326762144808 Z 0 XX 2.63577810877244 YY 1.67475013544567 ZZ 0 Using GNU awk, how do I print (X-XX) + (Y-YY) on the end of every line? Just to ...
1
vote
1answer
97 views

Concerning AWK programming, what do the -f switch and ~ operator do? [closed]

I can't seem to find anything on these switches. I know the "-F" option can be used to change the delimiter but that's the extent of what I've found other than it looks like maybe a compiling option; ...
1
vote
2answers
103 views

awk - processing two files

I have two files which I need to compare by some fields: reference file: 42:B:0 43:A:1 44:A:1 45:A:1 target file: 42:!:1 43:B:0 44:A:1 45:B:2 What I need I can already achieve with combined '...
1
vote
1answer
65 views

searching multiple files for a line with bigger number in column 3 of matched lines

I have multiple files with contents similar to: main file1: test01:6733:4370:5342 test02:7776:2018:1001 test03:9865:5632:1429 test04:8477:4757:1890 test05:8019:8860:5298 test06:5602:3100:6995 test07:...
0
votes
1answer
32 views

setting more variables insite awk if ? :

I get error in such simple test with two variables: $ echo test|awk '{$2 != ""? o="ABC" r="123" : o="123 r="ABC"}END{print "o:"o" r:"r}' awk: cmd. line:1: {$2 != ""? o="ABC" r="123" : o="123 r="ABC"}...