Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gawk]

gawk is the GNU implementation of awk. The awk utility interprets a special-purpose programming language that makes it possible to handle simple data-reformatting jobs with just a few lines of code.

1
vote
1answer
66 views

Improve performance when using “system” call (shell escape) processing large files in awk

I have an awk script that processes very large files that look something like this: K1353 SF3987.7PD833391.4 KARE K1353 SF3987.2KD832231.4 MEAKE K1332 IF4987.7RP832231.2 LEAOS K1329 SF2787.7KD362619....
0
votes
1answer
27 views

Gawk cross referencing

I'm having a issue with cross referencing data using gawk, when I use: gawk -F: 'FNR==NR{a[$1]=$2;next} $2 in a{print $1 FS a[$2]}' email.phone.txt name.email.txt > name.phone.txt Example of ...
1
vote
3answers
74 views

If line contains pattern, print variable to column

I have the file DEMLIR-GEO_OPT-1-distance-1.coordLog: REQUESTED STRUCTURE DATA Distance vector r(i,j) between the atom i and j in ANGSTROM r(1,5) = 0.944776 0.190651 1....
0
votes
0answers
20 views

Modifying text file using (g)awk [duplicate]

Is it possible to make a modification to a text file for example and save those modification to the same file using (g)awk script/command? All I see is only a modification to the printed output and ...
-1
votes
2answers
25 views

extract specific fields from a file using a character delimiter

I have a 1gb file and would like to get the third and fifth fields. The file is delimited by (,): input file: luis, jose, jesus atoche, mark, john 1,2,3,4,5 aa 1, bb 2, cc3, dd 4, ee5 output file: ...
0
votes
1answer
26 views

modify between (only) 1st occurrence of patterns (inclusive of patterns) and keep processing

There are several questions about matching between only the first occurrence of two patterns (e.g., this) — but they all seem to rely on exit, which is no good, as I want to keep processing the file. ...
9
votes
4answers
843 views

awk assign to multiple variables at once

I'm trying to pull two numerical values out of a string and assign them to variables using awk (gawk is what I'm using specifically). I want to pull the major and minor version numbers out of a tmux ...
4
votes
2answers
257 views

GNU awk ERRNO not set on command failure

I was trying run a few commands using getline() function of GNU awk and print the error number (errno) value returned. But for simple failure cases of non existent directory/file the variable doesn't ...
1
vote
1answer
61 views

Using NF with FPAT – what am I doing wrong?

When NF is used with FPAT regex, the comma is considered a field. I prefer using NF and FPAT: 1) NF – to limit the output to the actual number of fields for the record 2) FPAT – to handle an ...
5
votes
1answer
91 views

Why does gawk treat `0123` as a decimal number when coming from the input data?

According to $ man gawk, the strtonum() function can convert a string into a number: strtonum(str) Examine str, and return its numeric value. If str begins ...
7
votes
4answers
436 views

Why does regex `“\.pdf”` match `/…/pdf…/…` in gawk, but not in mawk?

From How can I extract only the pid column and only the pathname column in the lsof output? awk '{ for (i=9; i<=NF; i++) { if ($i ~ "string" && $1 != "wineserv" && $5 == "REG" &...
0
votes
2answers
28 views

Using awk how to compare 2 fields and print something out

Let say I have file1 with the following content: 1,2,3,4,12,6 1,2,3,5,333,6 1,2,3,12,5,6 1,2,3,44,101,6 While file2 has the following content: 22,4,111,1,6,7,12 22,4,56,55,6,7,12 22,4,12,12,6,7,12 ...
1
vote
2answers
99 views

Shell-Script Bash

I have this loop in the shell script, but it seems I don't fully understand what it does: especially gawk -v for ((k=$1; $k<3 + 3 ; k=$k + $3)) do echo $k | gawk -v s=4 '{print $1*s}' done ...
2
votes
4answers
131 views

How to break line after matching case but needs to go back a couple of characters

I have the following sample: 1. TNT 00:00 2. Swing From The Gutters 7:34 3. Ten-Day Interval 13:29 4. I Set My Face To The Hillside 18:16 5. The Equator 24:26 6. A Simple Way To Go Faster Than Light ...
1
vote
1answer
137 views

Unexpected split behavior

I read about Awk split behavior here: [...] the fs argument to the split function (see String Functions) shall be interpreted as extended regular expressions. These can be either ERE tokens or ...
-2
votes
1answer
47 views

Can I use awk to perform some number or string operations, without specifying a file as input?

Awk has builtin functions which are operating on numbers and strings. That is exactly what a programming language does. That makes me wonder if I can use awk to perform some number or string ...
-3
votes
1answer
184 views

Are `match()`, `gensub()`, `gsub()` and `sub()` only concerned about records not fields?

In gawk, is it correct that match(), gensub(), gsub() and sub() don't consider the fields, and are not affected by change to FS and OFS? In other words, are they only concerned about records not ...
1
vote
4answers
253 views

How shall I perform multiline matching and substitution using awk?

In a text file, ignoring any trailing whitespace at the end of each line, I assume that if a line is not ended with a digit, then there is a line break between the line and the next line, and I would ...
0
votes
3answers
61 views

Organizing data by a header

I have two csv files. File 1 contains header. File 2 contains data file 1 format : file1.csv id,abc,xyz,aaa,bbb,ccc file 2 format : file2.csv id,source,value 1,abc,100 1,xyz,...
1
vote
1answer
20 views

(g)awk: Substituting expression from another script

I am building a generic script which will be re-used from several other scripts. For each case, I need to extract Nth field in the generic script; in some cases the double quotes should be removed ...
0
votes
1answer
25 views

How to join two files using two columns in unix

I have file a and file lookup and the structure is below. vi a.txt empid|ename|sal_grade|MANAGER_ID 1|raj|A|202 2|MAN|B|203 3|JOS|C|204 vi lookup.txt Grade|sal|manager_id A|$100000|202 A|$1000|099 ...
-1
votes
3answers
67 views

Remove a whole line beginning with a pattern occurring anywhere after another pattern

How can I remove any line beginning with --- occurring in any line number after a line containing # Match (the first occurrence in any line number)? EDIT: The relationship between #Match with the ...
2
votes
3answers
901 views

awk: extract string from a field [closed]

in the input fields are separated by pipe sign: CCCC|Sess C1|s1 DA=yy07:@##;/u/t/we DDDDD|Sess C2|s4 DB=yy8:@##;/u/ba I want to get output where last field is changed (extracted only what is between ...
0
votes
1answer
65 views

gawk process multiple files inside directory

I have multiple text files in a folder that I want to process through gawk, rather than process them each individually is it possible I can process them all at once without merging them into 1 .txt ...
0
votes
1answer
134 views

awk print from passwd but only with userid > 1000

Ok I've done: cat /etc/passwd | awk -F: '{ print $1 }' -F ==> Specify a file seperator for delineating data fields in a line $1 ==> Represents the first field in the input but how to only print ...
0
votes
1answer
55 views

gawk job getting killed, too large of input?

I can't seem to understand why a script I've been running fine up until now fails on a slightly larger input file. If I run the script on a file that is 820000x75 (lines by columns), the script works ...
4
votes
4answers
381 views

How to create a new file with required columns from different multiple files in linux?

I have a directory ballgown, in which there are around 1000 subdirectories as sample names. Each subdirectory has a file t_data.ctab. The filename is the same in all subdirectories. ballgown |...
0
votes
3answers
93 views

resume reading a log file from the point I left it last time [duplicate]

I have a log file which is continuously updating(new line added) after few time period. I am fetching only error messages from the file in every 10 minutes. Initially, at 1st time I fetched all line ...
1
vote
1answer
86 views

Gawk partial match

My gawk script isn't matching all potential results, it's only matching one of them.. Example - script: gawk -F: "FNR==NR{a[$2]=$1;next} $1 in a{print a[$1] FS $2}" username.email.txt email.phone....
0
votes
1answer
30 views

trying to execute base62 operation in unix with more than 25 digits in a row, but awk treating a default value

Below is my code: #!/usr/bin/ksh awk 'BEGIN {FS="|"; flag=0;} { x=$2 rem=int(x)%62 quo=int(x/62) flag=0 while(quo>62) { sub_rem=int(quo)%62 quo=int(quo/62) if(flag==0) { grp_rem=sub_rem } else { ...
2
votes
4answers
1k views

Passing arguments to awk script

I have a awk script where i want to be able to pass N arguments into it and also read from stdin. I would like to be able to do something like tail -f logfile | my_cool_awk_scipt var1 var2 var3 ... ...
0
votes
2answers
147 views

Awk command failing during the redirection with null sting value

I want to split the large file into pieces. Filename and the number records per file after split will be provided as arguments/variables to awk. EG: FILENAME=A_20 (say it has 100 records), ...
3
votes
1answer
267 views

Clone complex array in awk

I have an array like this ... a["a"] = "aa" a["b"] = "ab" a["c"] = "ac" a["d"]["a"] = "ada" a["d"]["b"]["a"] = "adba" a["e"]["a"]["a"]["a"] ...
1
vote
1answer
172 views

Gawk: backspace preceding a newline doesn't work

I'm using gawk (4.1.3) and it seems I have struck upon a trivial problem. The backspace character doesn't seem to work if it is immediately followed by a newline. awk 'BEGIN{for(i=1;i<=3;i++) ...
4
votes
1answer
951 views

Speed up text processing

I have the following script to process a file with some data: First, the header is printed to the output file. Then 60000 lines from the input are randomly picked and printed to the output (the ...
2
votes
1answer
102 views

Print random lines of file but with maximum count of a certain value

I have a input file from which I want to randomly choose a line. In a column in every line is a value, and I want only a number of lines up to a maximum count of this value to be choosen randomly, not ...
0
votes
1answer
87 views

processing multiple files with awk problem

I've the input files with fields separated with colon: main: one:111:222:333 fiv:333:222:333 two:123:234:500 file1: one:111:222:333 two:123:234:501 file2: one:111:222:333 thr:-:234:232 fiv:999:...
1
vote
1answer
104 views

socat and gawk piped

I'm trying to manipulate a TCP stream using gawk. The TCP server streams out a text string separated by whitespaces. I want to add an extra string at the beginning and then separate the remaining ...
1
vote
1answer
31 views

Using GAWK to sum certain numbers line by line

My lines will look as such X 2.63766751981802 Y 1.68326762144808 Z 0 XX 2.63577810877244 YY 1.67475013544567 ZZ 0 Using GNU awk, how do I print (X-XX) + (Y-YY) on the end of every line? Just to ...
1
vote
1answer
92 views

Concerning AWK programming, what do the -f switch and ~ operator do? [closed]

I can't seem to find anything on these switches. I know the "-F" option can be used to change the delimiter but that's the extent of what I've found other than it looks like maybe a compiling option; ...
1
vote
2answers
101 views

awk - processing two files

I have two files which I need to compare by some fields: reference file: 42:B:0 43:A:1 44:A:1 45:A:1 target file: 42:!:1 43:B:0 44:A:1 45:B:2 What I need I can already achieve with combined '...
1
vote
1answer
65 views

searching multiple files for a line with bigger number in column 3 of matched lines

I have multiple files with contents similar to: main file1: test01:6733:4370:5342 test02:7776:2018:1001 test03:9865:5632:1429 test04:8477:4757:1890 test05:8019:8860:5298 test06:5602:3100:6995 test07:...
0
votes
1answer
32 views

setting more variables insite awk if ? :

I get error in such simple test with two variables: $ echo test|awk '{$2 != ""? o="ABC" r="123" : o="123 r="ABC"}END{print "o:"o" r:"r}' awk: cmd. line:1: {$2 != ""? o="ABC" r="123" : o="123 r="ABC"}...
1
vote
0answers
350 views

strings concatenation on variable

how to collect add do existing "b" values next matching? input: 1:2:3:4:5 6:7:8:9:0 1:3:2:4:5 $ echo -e "1:2:3:4:5\n6:7:8:9:0\n1:3:2:4:5"|awk -F: '$1=="1"{b="line "NR" "$2;print b}' line 1 2 line 3 ...
0
votes
2answers
1k views

awk: How to use different field separators, and operate on different columns [closed]

For multiple files, I'm trying to extract columns containing a specific string, but the field separators in the files are different, and the string will appear in a different column depending on the ...
0
votes
3answers
1k views

Rounding a number in awk

I’m trying to find a way to round up a few numbers. The topic I found: https://stackoverflow.com/questions/2395284/round-a-divided-number-in-bash I am using the following command in my bash script.. ...
1
vote
1answer
347 views

awk print with a condition expression with a ternary operator

Sample input: 1:2:3:0:5 1:2:3::5 Awk processing: $ echo -e "1:2:3:0:5\n1:2:3::5"|awk -F: '{if($4!="") print NR,$4; else print NR,"-"}' 1 0 2 - Question: can above be run in a short if notation? I ...
-3
votes
1answer
99 views

How Do I compare text in 2 files and export to file 3? [duplicate]

I have 2 files 1.txt aab aac aad 2.txt aaa aab aac File 3 should contain aab aac
0
votes
2answers
86 views

check if no line contains string in given field [closed]

I need to search for thousands of strings one by one if they exists on a second column and if not do action against each such string which was not found in second column of any line file: line a ...
0
votes
1answer
38 views

two input files data processing

I have two huge input files: file1 contains 3 tabulator separated fields: field1: UID field2: username field3: Real Name Example of file1: 644 u11496 Real name1 640 u1309 Real name2 ...