Questions tagged [awk]

A pattern-directed scanning and processing language.

Filter by
Sorted by
Tagged with
0 votes
6 answers
296 views

Rewriting log parsing script in Perl or Python (get rid of awk)

I need accomplish a task of filtering activity of bots in the log file. Solution should only show records meeting following criteria user logged in, user changed password, user logged off within ...
7 votes
1 answer
487 views

Why does awk -F work for most letters, but not for the letter "t"?

July 2022 mac os Monterey V12.1 awk --version 20200816 GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin21) Why does awk -F work for most letters, but NOT for the letter t? I have ...
-1 votes
2 answers
27 views

Remove lines if the second field is blank

sed -i here is my file 1 a 2 b 3 4 d 5 6 e i want to get the intermediate file 1 a 2 b 4 d 6 e and the final file >1 a >2 b >4 d >6 e
-4 votes
0 answers
53 views

How to calculate nodes, edges and degrees of TXT file using LINUX | UNIX (GITBASH)? [closed]

I am building a network by Web-Crawling of txt file extracted from WIKIPEDIA, Using UNIX commands (the tool is GitBash). Already found the first lines of the file but need to count the number of ...
-1 votes
2 answers
30 views

how to add a number as well as format change to column value and print it in 3rd column

I have these two columns with header: CHR BP 1 78901 i need to add a character to the first column numbers (chr). I also need to make a third column in which one score being added to the second ...
2 votes
9 answers
117 views

How to split a single line of fields delimited by commas into multiple lines with a specific number of fields in each line?

I have a file like so Hello,Hi,Hullo,Hammers,Based,Random For n=2, output must be like so Hello,Hi Hullo,Hammers Based,Random For n=3, output must be like so Hello,Hi,Hullo Hammers,Based,Random How ...
11 votes
11 answers
31k views

can we print the last word of each line in linux using sed command?

suppose , if there is a file consisting of following lines , if they are 12345 567 7878 66 er3 t45t y6y46y 4y6 y656y y5y 46y6 65y7 y66uyuy yy46y6y The output has to look like: 66 ...
-1 votes
1 answer
35 views

How to extract specific word from excel cell using awk command?

I need help on how to extract specific word from excel cell using awk command? Filename: Auditlog.csv Column: D {"CreationTime":"2022-05-24T19:40:38","Id":"034ad6cv-...
8 votes
1 answer
6k views

Stop processing single line in awk after successful match?

Is there a way to stop processing a single line in awk? Is there something like break or continue that works on pattern-action pairs rather than control structures within an action? Suppose I have ...
4 votes
5 answers
387 views

Finding the line number of first occurrence of a text in bash script

I need to find out what is the line number of first occurrence of a given search string that should be in the start of a line in a text file and store it in a variable in my bash script. For example I ...
0 votes
2 answers
511 views

Printing the difference between adjacent values in a column to a new column

I have a file A 1 A 2 A 4 A 6 I want to print the difference between adjacent values (below-above) in column 2 to a new column 3, to get this A 1 A 2 1 A 4 2 A 6 2 I have discovered something like ...
6 votes
5 answers
13k views

Use awk results as parameters to another command

I'm extracting rows from a set of text files with awk. The files look like this: 1000 1 75 1000 2 76 1001 1 76 1001 2 80 I'm searching several directories of these with this ...
0 votes
4 answers
48 views

Grep line containing a word OR has length in a specific range?

1598427@931 PDD 220624P00051000 ohlc=0,0,0,0 vol=0 oi=424 nbbo=69@2316/113@532 nbbo2=69@145/113@95 PDD 220617C00051000 ohlc=0,0,0,0 vol=0 oi=434 nbbo=530@1921/710@1496 nbbo2=530@31/710@115 PDD ...
0 votes
1 answer
43 views

how to convert list of txt files in one directory to make another list of csv files of same name, not one csv file?

i need to convert list of txt files in one directory to list of csv files of same file names .I don,t need to convert one csv file
3 votes
7 answers
376 views

awk/sed find indexes of the first and the last capital letter in a string

I have several kinds of strings like this: the example string1 --AbbbAnde--- the example string2 abksjiRNNBBKUGFLYFYLF the example string3 -ankNUGUYUBUIGCafrg-- the example string4 BNKJUGFVULNK-Kew-...
0 votes
2 answers
6k views

How to assign variable in awk

I have a code like below: $ awk -F'[]]|[[]' \ '$0 ~ /^\[/ && $2 >= "2014-04-07 23:00" { p=1 } $0 ~ /^\[/ && $2 >= "2014-04-08 02:00:01" { p=0 } ...
0 votes
2 answers
64 views

Awk substr index 0

I just discovered that substr() in awk accepts either 0 or 1 as the initial index in a string. I tested this in Gawk 5.1.0 and MacOS awk 20070501. awk 'BEGIN {print substr("abcd", 0, 1)}' ...
1 vote
5 answers
73 views

How to print several match in the same line with awk

Hi have a tabular file (separator =\t) A.txt : 205 toto patho 207 tata benign 208 titi likely_patho and B.txt: 210 lolo patho 211 lala benign 212 lili benign And I would like to have a file that ...
0 votes
1 answer
2k views

Converting csv file (generated by db query) to html and color code the table based on Column Value

(Using Shell Script) Need to generate html report from a CSV file which is an output of a DB query and then color code it based on column values. For example, from the given CSV ile, if any cell under ...
3 votes
3 answers
3k views

Sum multiple columns

I need to do multiple sums; my input file is: DATE|NATION|CITY|FILES|REVENUE|FREQUENCY|INVESTMENT 20170807|USA|VIRGINIA|TIMES|1919150|1779|282075 20170807|USA|NYC|ROADS|92877|41|1599 20170808|USA|...
915 votes
10 answers
1.8m views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
110 votes
7 answers
69k views

Is there a basic tutorial for grep, awk and sed? [closed]

I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...
10 votes
2 answers
509 views

awk "date" | getline var caches its value - but only sometimes

As part of a larger awk script I needed to convert an arbitrary date string into seconds since the epoch. This isn't available as an awk function so I thought I could restort to calling date on each ...
1 vote
1 answer
2k views

How to automatically detect and write to usb with variable spaces in its name

I am doing the second BASH exercise from TLDP Bash-Scripting Guide, and I have most of it figured out up until the part when it comes time to copy the compressed files to an inserted USB. Home ...
3 votes
2 answers
2k views

I want to parse a Terraform vars file and store variables from the TF vars file as vars in a Makefile

I have two files: a Terraform variables file (vars.tf), and a Makefile. What I want to be able to do, is to have the Makefile "read" some of the variables in the Terraform vars.tf file. The vars.tf ...
-1 votes
1 answer
32 views

Capturing section of first line and output all lines matching pattern

I have output that looks like, from a directory listing: -rw-r--r-- user users 5432 1 Jun 2020 foo.txt -rw-r--r-- user users 1928 1 Jun 2020 bar.txt -rw-r--r-- user users 3333 21 Jul 2020 baz0.txt -...
1 vote
2 answers
77 views

How to create a combination of all the numbers after the first digit, in separate lines [duplicate]

I have a big data with rows like this: This data is in txt format: 0 1,5,6,4 2,23,43,5,6 1 4 5 2 3 5 4 5 6 7 56,65 Each line considers being from the first digit and the other series of digits ...
0 votes
1 answer
41 views

Include spaces and tabs in "awk" search and replace

Another user helped me earlier to fix something I'm doing with awk, where I search for a string at any point in all files and replace two numbers in the same line when I find it. awk -i inplace '/^...
0 votes
1 answer
74 views

How to get lines which are unique for 1 column but are same for another column?

I am looking for an awk solution which would fetch lines which are unique based on 2nd column when the 1st & 3rd and 4th columns are the same. constraints : column 3 and column 4 has to be same ...
0 votes
0 answers
39 views

AWK command for summary statistics of array with several columns and a grouping variable

I got a data file with time-series data. Each line of the 5+ GB csv file freqs.csv has the appearance: start_freq stop_freq measurement1 measurement2 ... measurementk Frequencies are represented on ...
1 vote
2 answers
45 views

awk get column value from file 2 for file 1 if value matches in 2 columns [duplicate]

I have 2 files, trying to set column value that is taken from file2 into file1 when their columns match file1: signup||20200620|A3332|en|LA||ACTIVE signup||20200620|B4443|en|CA|66001|ACTIVE signup||...
0 votes
2 answers
56 views

replace text in specific field with variables from loop

I have a huge file to parse & need to search and replace text but in specific fields, sharing a small sample for reference named dest. first row is header for reference. cat dest ID|NAME|COMPANY|...
0 votes
2 answers
50 views

Combining files column-wise by first column as key (using grep or awk, etc.)

awk -F '\t' 'NR==FNR{F1[$0];next}$1 in F1{print}' 1.txt 2.txt 1.txt 1 2 3 4 5 2.txt 2 a 5 3 than i get the file is 2 a 5 3 but i want to get file 1 2 a 3 4 5 3 or 1 ...
0 votes
5 answers
72 views

Creating a sequence with a specific string and varying numbers and letters

I need to create a single row with columns that have a shared string, vary by number, and share repeating letters. My desired output looks like: SNP1a SNP1b SNP2a SNP2b ... SNP3502a SNP3502b I am ...
0 votes
1 answer
39 views

Using AWK to sum up two column for each block

I would like to calculate for each block/("team") a summary of $4 & $5 in awk Sample input: ABC Team 1109;BB; EE;B;M 1108;GG; KK;B;M 1104;KK; CC;F;S 1103;LL; JJ;B;XL CBS Team 600;AA; AA;...
-1 votes
1 answer
47 views

How to extract the characters after a forward slash using awk

How can I extract the substring after a / and compare the extracted value against another file ? cat file /abc|98357| abc/aabb|023947| xyz/0102|293847| pqr/pqrst|29384| 9864/truse|253123| expected ...
2 votes
4 answers
3k views

How to extract unique values from column #x with its corresponding values of column #y?

I have a comma',' FS filename as csv with n number of columns. I need to extract the unique value from colm.#1 with only corresponding values in colm.#10. So basically the column 10 is the date which ...
2 votes
5 answers
254 views

Add a column from one file as last column to another file

I would like to make a loop in which a certain column (in my case column 4) from a text file is added as last column to a new text file. I have in total around 500 text files (V1-V500) from which I ...
-1 votes
2 answers
52 views

get my file by awk - split by last occurrence

awk -F here is my file a_b_1 b_c_3 c_1 final file a_b 1 b_c 3 c 1 how to get my final file by awk
2 votes
7 answers
265 views

Replace values in fifth column

I have multiple text Name 1:10:34 date short_id 10 Name 1:10:45 date short_id 10 Name 1:20:54 date short_id 20 Name 1:30:43 date short_id 30 Name 1:40:43 date short_id 40 Name 1:40:13 date short_id ...
0 votes
1 answer
47 views

How do I extract the filename from a path in Awk if I have different length filenames with a path? [duplicate]

I have an Awk script that counts the average of rows from every file from a given directory and its subdirectories and I have to print the filename and the average from a file like this example.c : 5 ...
0 votes
1 answer
40 views

Add information to HTML table with awk

I am trying to get the numerical and percentage outputs of free together in a table, which I send to myself per mail. I have tried to combine these into one table for quite a while, but nothing seems ...
0 votes
7 answers
70 views

How can aggregate lines in CSV based on value of one column

My input is the following sorted CSV file (using : as the field delimiter instead of the usual comma): version:device 1.0.0:dev1 1.0.0:dev2 1.2.3:dev3 1.3.4:dev4 1.3.4:dev5 I want to aggregate it so ...
-2 votes
1 answer
52 views

How to prefix a value of variable before the each line coming from the tail command using the awk command

I am coming from this question and I need to make the filename value to be stored in a variable and to use this variable in the print command instead of a static value These were my trials but I am ...
3 votes
5 answers
200 views

How do we find the maximum value of Column 3 from a range of values of Column 1?

I have this dataset on an Ubuntu machine: 37.500 0.0000 0.005605 37.750 0.0000 -0.027858 38.000 0.0000 -0.060678 38.250 0.0000 -0.088557 38.500 0.0000 -0.109210 38.750 0.0000 -0.122482 39....
1 vote
1 answer
174 views

Why is my command giving wrong output?

I m trying to output lines if col 1 has digit value , but its not working. My sample file doesn't have digit in col 1 so shouldnt awk return nothing in this case? my code awk -F 'BEGIN {OFS=FS}{ if ( (...
2 votes
3 answers
84 views

replace a string by using part of the string in the same line

I have a file like the following on a Linux machine, and the gene_id is wrong. So I would like to replace the string "PB" in the gene_id with the transcript_id from the same line. $ cat try....
0 votes
1 answer
43 views

When does OFS change?

Except when set by the user or an user script, the value of OFS seems to always be an space. Even when the input use tabs: $ printf 'one\ntwo\tthree\nfour' | awk '{NF=NF}1' | sed -n l one$ two three$ ...
0 votes
1 answer
66 views

How does awk handle collisions in a hashmap?

Does awk use separate chaining, open addressing or does it have its own way of handling collisions in a hashmap? Do gawk and nawk implement the same algorithm? Thank you.
0 votes
2 answers
82 views

Parsing CSV with AWK, returning fields to bash var with line breaks

I have to migrate a password database (Keepass) by using a csv file of it, to a new application by using its API. The API is updated with post requests, those needs JSON data format. What I need to do ...

1
2 3 4 5
156