Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gawk]

gawk is the GNU implementation of awk. The awk utility interprets a special-purpose programming language that makes it possible to handle simple data-reformatting jobs with just a few lines of code.

1
vote
1answer
66 views

Improve performance when using “system” call (shell escape) processing large files in awk

I have an awk script that processes very large files that look something like this: K1353 SF3987.7PD833391.4 KARE K1353 SF3987.2KD832231.4 MEAKE K1332 IF4987.7RP832231.2 LEAOS K1329 SF2787.7KD362619....
-1
votes
2answers
192 views

awk - replace pattern

how do I search the following pattern by awk in a file. '\b' The pattern is one of the lines in a file. I have to add text next to the pattern. Like this: '\b' TEXT Any of these commands are not ...
0
votes
1answer
27 views

Gawk cross referencing

I'm having a issue with cross referencing data using gawk, when I use: gawk -F: 'FNR==NR{a[$1]=$2;next} $2 in a{print $1 FS a[$2]}' email.phone.txt name.email.txt > name.phone.txt Example of ...
1
vote
3answers
74 views

If line contains pattern, print variable to column

I have the file DEMLIR-GEO_OPT-1-distance-1.coordLog: REQUESTED STRUCTURE DATA Distance vector r(i,j) between the atom i and j in ANGSTROM r(1,5) = 0.944776 0.190651 1....
0
votes
1answer
917 views

how to get multiple command output as variable in awk

I am trying to execute 3 command and get the output as a variable which needs some percentage calculation and display the same as the example. can this be executed as script. First command output $ ...
0
votes
0answers
20 views

Modifying text file using (g)awk [duplicate]

Is it possible to make a modification to a text file for example and save those modification to the same file using (g)awk script/command? All I see is only a modification to the printed output and ...
-1
votes
2answers
25 views

extract specific fields from a file using a character delimiter

I have a 1gb file and would like to get the third and fifth fields. The file is delimited by (,): input file: luis, jose, jesus atoche, mark, john 1,2,3,4,5 aa 1, bb 2, cc3, dd 4, ee5 output file: ...
0
votes
1answer
26 views

modify between (only) 1st occurrence of patterns (inclusive of patterns) and keep processing

There are several questions about matching between only the first occurrence of two patterns (e.g., this) — but they all seem to rely on exit, which is no good, as I want to keep processing the file. ...
1
vote
1answer
92 views

Concerning AWK programming, what do the -f switch and ~ operator do? [closed]

I can't seem to find anything on these switches. I know the "-F" option can be used to change the delimiter but that's the extent of what I've found other than it looks like maybe a compiling option; ...
0
votes
2answers
72 views

gawk - Dynamic creation of text file using input file fields

I have a text file that looks like this: UICEX_0001 UICEX_0001_T1.bam UICEX_0001_C2.bam chr1:16946335 chr19:9064309 chr8:10480278 ...
9
votes
4answers
843 views

awk assign to multiple variables at once

I'm trying to pull two numerical values out of a string and assign them to variables using awk (gawk is what I'm using specifically). I want to pull the major and minor version numbers out of a tmux ...
4
votes
2answers
257 views

GNU awk ERRNO not set on command failure

I was trying run a few commands using getline() function of GNU awk and print the error number (errno) value returned. But for simple failure cases of non existent directory/file the variable doesn't ...
1
vote
1answer
173 views

Extracting trace information using gawk

I am using gawk to extract trace information from a mrt file in order to use it further for analysis. I have successfully been able to extract trace information from a pcap file format but cannot ...
1
vote
1answer
61 views

Using NF with FPAT – what am I doing wrong?

When NF is used with FPAT regex, the comma is considered a field. I prefer using NF and FPAT: 1) NF – to limit the output to the actual number of fields for the record 2) FPAT – to handle an ...
5
votes
1answer
91 views

Why does gawk treat `0123` as a decimal number when coming from the input data?

According to $ man gawk, the strtonum() function can convert a string into a number: strtonum(str) Examine str, and return its numeric value. If str begins ...
7
votes
4answers
436 views

Why does regex `“\.pdf”` match `/…/pdf…/…` in gawk, but not in mawk?

From How can I extract only the pid column and only the pathname column in the lsof output? awk '{ for (i=9; i<=NF; i++) { if ($i ~ "string" && $1 != "wineserv" && $5 == "REG" &...
2
votes
4answers
131 views

How to break line after matching case but needs to go back a couple of characters

I have the following sample: 1. TNT 00:00 2. Swing From The Gutters 7:34 3. Ten-Day Interval 13:29 4. I Set My Face To The Hillside 18:16 5. The Equator 24:26 6. A Simple Way To Go Faster Than Light ...
1
vote
1answer
31 views

Using GAWK to sum certain numbers line by line

My lines will look as such X 2.63766751981802 Y 1.68326762144808 Z 0 XX 2.63577810877244 YY 1.67475013544567 ZZ 0 Using GNU awk, how do I print (X-XX) + (Y-YY) on the end of every line? Just to ...
3
votes
2answers
817 views

Combine two operations in one command - awk

I need to change - FROM Car Bus - TO Helicopter Airplane This two commands are sufficient. awk -i inplace '{sub(/Car/,"Helicopter")}1' file awk -i inplace '{sub(/Bus/,"Airplane")}1' file And ...
1
vote
1answer
67 views

Word frequency gawk memory leak

I have the following bash script: #!/usr/bin/env bash grep -e '^[a-zA-Z]\{4,8\}$' data/words3.txt | tr '[:upper:]' '[:lower:]' | sort -u > data/passphrase-words.txt function wordfrequency() { ...
2
votes
5answers
4k views

How do I append/prepend a timestamp to grep output?

I am running a small linux server at home, and I am writing a script to log the temperature of the CPU cores every 5 seconds, but I need timestamps for it to be useful. So far I have something that ...
0
votes
3answers
61 views

Organizing data by a header

I have two csv files. File 1 contains header. File 2 contains data file 1 format : file1.csv id,abc,xyz,aaa,bbb,ccc file 2 format : file2.csv id,source,value 1,abc,100 1,xyz,...
2
votes
3answers
2k views

How to get the last occurrence of lines between two patterns from a file?

I have a log file which reports on the output of a process, I'd like to extract all lines from between the last occurrence of two patterns. The patterns will be along the lines of; Summary process ...
0
votes
2answers
28 views

Using awk how to compare 2 fields and print something out

Let say I have file1 with the following content: 1,2,3,4,12,6 1,2,3,5,333,6 1,2,3,12,5,6 1,2,3,44,101,6 While file2 has the following content: 22,4,111,1,6,7,12 22,4,56,55,6,7,12 22,4,12,12,6,7,12 ...
4
votes
4answers
3k views

display the missing icmp_seq count | ping command

I am looking for something which shows me the missing ICMP package when running ping command or which can display failed in the output. Example below Current output where we are missing the icmp_seq ...
1
vote
2answers
99 views

Shell-Script Bash

I have this loop in the shell script, but it seems I don't fully understand what it does: especially gawk -v for ((k=$1; $k<3 + 3 ; k=$k + $3)) do echo $k | gawk -v s=4 '{print $1*s}' done ...
1
vote
1answer
137 views

Unexpected split behavior

I read about Awk split behavior here: [...] the fs argument to the split function (see String Functions) shall be interpreted as extended regular expressions. These can be either ERE tokens or ...
0
votes
2answers
63 views

Centos gawk command line usage

I am trying to run the following command on Centos however I get an error? Unexpected token near '(' gawk { if($0=="id323"){print $0} } doc.txt
0
votes
1answer
84 views

AWK ending standalone script

I started to do this.. to swap fields in a csv file but I don't get to finish the script. What I want to do is to concatenate fields from 2 to 7 and group them by the first field. Fields 1, 8 and 9 ...
0
votes
2answers
104 views

Grep output concerns

Okay so I'm using grep on a external HDD example, M:/ grep -rhI "bananas" . > out.txt which would output any lines within " M:/ " containing " bananas " However I would like to output the ...
-2
votes
2answers
53 views

Data Manipulation

I have a data as mentioned below. host_name Server1.domain.com contacts DL - Desktop contact_groups ravi, raj, rahim host_name Server2.domain.com contact_groups DL-Server host_name Server3....
0
votes
1answer
66 views

Unix file edit for pattern substitution

I have a file with repetitive patterns like below: set m [ meas new $proj rajesh ] meas $m set_type lvf_timing abcde erft 1213 234 456 897 00000000000000 once upon a time set t [ table new $m ...
-2
votes
1answer
47 views

Can I use awk to perform some number or string operations, without specifying a file as input?

Awk has builtin functions which are operating on numbers and strings. That is exactly what a programming language does. That makes me wonder if I can use awk to perform some number or string ...
1
vote
4answers
253 views

How shall I perform multiline matching and substitution using awk?

In a text file, ignoring any trailing whitespace at the end of each line, I assume that if a line is not ended with a digit, then there is a line break between the line and the next line, and I would ...
-3
votes
1answer
184 views

Are `match()`, `gensub()`, `gsub()` and `sub()` only concerned about records not fields?

In gawk, is it correct that match(), gensub(), gsub() and sub() don't consider the fields, and are not affected by change to FS and OFS? In other words, are they only concerned about records not ...
0
votes
3answers
1k views

Rounding a number in awk

I’m trying to find a way to round up a few numbers. The topic I found: https://stackoverflow.com/questions/2395284/round-a-divided-number-in-bash I am using the following command in my bash script.. ...
0
votes
2answers
51 views

Print N number of observations that are non-missing from specific column in delimited file

Say I have file.txt that is pipe-delimited, and I want to take a look at 10 non-missing observations from the 20th column to ensure they appear in the correct format. Would I use the awk command and ...
1
vote
1answer
20 views

(g)awk: Substituting expression from another script

I am building a generic script which will be re-used from several other scripts. For each case, I need to extract Nth field in the generic script; in some cases the double quotes should be removed ...
0
votes
1answer
25 views

How to join two files using two columns in unix

I have file a and file lookup and the structure is below. vi a.txt empid|ename|sal_grade|MANAGER_ID 1|raj|A|202 2|MAN|B|203 3|JOS|C|204 vi lookup.txt Grade|sal|manager_id A|$100000|202 A|$1000|099 ...
2
votes
1answer
148 views

Keep lines containing only 3 fields

I have a .txt file in this format: USER:EMAIL:PHONE but some lines only have USER:EMAIL I want to extract only the lines containing 3 fields, example USER:EMAIL:PHONE
-1
votes
3answers
67 views

Remove a whole line beginning with a pattern occurring anywhere after another pattern

How can I remove any line beginning with --- occurring in any line number after a line containing # Match (the first occurrence in any line number)? EDIT: The relationship between #Match with the ...
2
votes
3answers
901 views

awk: extract string from a field [closed]

in the input fields are separated by pipe sign: CCCC|Sess C1|s1 DA=yy07:@##;/u/t/we DDDDD|Sess C2|s4 DB=yy8:@##;/u/ba I want to get output where last field is changed (extracted only what is between ...
0
votes
1answer
65 views

gawk process multiple files inside directory

I have multiple text files in a folder that I want to process through gawk, rather than process them each individually is it possible I can process them all at once without merging them into 1 .txt ...
0
votes
1answer
134 views

awk print from passwd but only with userid > 1000

Ok I've done: cat /etc/passwd | awk -F: '{ print $1 }' -F ==> Specify a file seperator for delineating data fields in a line $1 ==> Represents the first field in the input but how to only print ...
0
votes
1answer
55 views

gawk job getting killed, too large of input?

I can't seem to understand why a script I've been running fine up until now fails on a slightly larger input file. If I run the script on a file that is 820000x75 (lines by columns), the script works ...
4
votes
4answers
381 views

How to create a new file with required columns from different multiple files in linux?

I have a directory ballgown, in which there are around 1000 subdirectories as sample names. Each subdirectory has a file t_data.ctab. The filename is the same in all subdirectories. ballgown |...
0
votes
3answers
93 views

resume reading a log file from the point I left it last time [duplicate]

I have a log file which is continuously updating(new line added) after few time period. I am fetching only error messages from the file in every 10 minutes. Initially, at 1st time I fetched all line ...
1
vote
1answer
86 views

Gawk partial match

My gawk script isn't matching all potential results, it's only matching one of them.. Example - script: gawk -F: "FNR==NR{a[$2]=$1;next} $1 in a{print a[$1] FS $2}" username.email.txt email.phone....
3
votes
1answer
1k views

Using awk to process a mapping file with dynamic number of columns

I am wondering how to process information as following: mapping.txt: 80 001 002 81 011 012 013 014 82 021 022 ... input.txt: 81 103823044 80 103823054 81 103823064 ... Desired output.txt: ...
7
votes
3answers
2k views

Surprising result using awk floating point arithmetic

I've been trying to get awk to do some trivial arithmetic, which involves carrying some values from one line to the next. Here is a minimal example pair, for comparison. The first example is expected ...