Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
The results are in! See what nearly 90,000 developers picked as their most loved, dreaded, and desired coding languages and more in the 2019 Developer Survey.

Questions tagged [awk]

A pattern-directed scanning and processing language.

0
votes
1answer
10 views

How to check if all first column fields are unique

I have a file where strings are separated by a special string (not a comma or delimiter) e.g. <vvv>. I want to check if all strings in the first field are unique. If duplicate rows for the same ...
0
votes
1answer
23 views

compare a certain number of lines between columns of two files

I have a thousand files and I need to verify if they all have exactly the same information in the second column up to a certain number of lines. Below an example. I would like to print the names of ...
0
votes
2answers
43 views

Find and replace Pipe using awk command

I am trying to grep column 7th with pipe delimiter (|) data in that I have to removed pipe (|) from middle of text. Sample file: col1|col2|col3|col4|col5|col6|col7|col8|col9|col10 value1|value2|...
0
votes
0answers
19 views

Modifying text file using (g)awk [duplicate]

Is it possible to make a modification to a text file for example and save those modification to the same file using (g)awk script/command? All I see is only a modification to the printed output and ...
0
votes
3answers
62 views

How to extract first line that matches pattern from a file and if pattern doesn't exist print fixed output?

I have a text file shown below: Doc_A 123 abc Doc_A 456 def Doc_A 789 ghi Doc_B 123 abc Doc_B 456 def Doc_C 123 abc Doc_C 456 def Doc_C 789 ghi Doc_C 101 jkl And a reference file Doc_A Doc_B Doc_C ...
1
vote
2answers
65 views

find a pair of words that appear the most of the times together

I have 10 text files, in each file i have a chapter from a book, i want to find the pair of words that appear the most of the time together in a line i.e: chapter1: hello world good boy green sun ...
-1
votes
1answer
31 views

find the word that appears the most at the beginning of a line from entire paragraph

I have a paragraph and I want to know which word appears the most at the beginning of a line from all paragraph for example: paragraph: Hello my name is X Nice to meet you Hello my name is Y so ...
-4
votes
0answers
24 views

Linux script to move item from one folder to another

I want to write a script to read a file which contains the below information.The script needs to process each line. Folder A goes to Folder AX condition=name starts with RED Folder B goes to Folder ...
1
vote
1answer
34 views

awk syntax error inside script while assigning it to a variable

Basically memory is a variable. In a script, I am trying to remote login to one server and execute free command to check memory usage and using awk I'm getting the usage in percentage. But I am ...
7
votes
6answers
2k views

How to compare two different files line by line in unix?

File1: 123 234 345 456 File2: 123 234 343 758 Expected output: File3: TRUE TRUE FALSE FALSE so the code should compare two files and print 'TRUE' if it matches otherwise it should print 'FALSE' ...
5
votes
8answers
680 views

Delete nth line (when counted from the bottom)

There are unknown number of lines in a file. How to delete nth line (when counted from the bottom) with one-liner command (you may use more than one if it is necessary) on Unix platform.
0
votes
3answers
33 views

Extracting file name with spaces using awk

I have this line of output: M file.txt Getting fed into this awk program: awk '{printf("\t%s: %s %s\n", FNR, $1, $2)}' which returns 1: M file.txt This works and is the usual case. However, if ...
1
vote
1answer
26 views

How to split file based on number of columns?

I have a file containing numeric data. Each line has a varying number of columns. I want to Split the file in to multiple files based on number of columns in a Line. Each Line may have columns varying ...
-1
votes
2answers
24 views

extract specific fields from a file using a character delimiter

I have a 1gb file and would like to get the third and fifth fields. The file is delimited by (,): input file: luis, jose, jesus atoche, mark, john 1,2,3,4,5 aa 1, bb 2, cc3, dd 4, ee5 output file: ...
0
votes
1answer
27 views

Get Lines between pattern 1 and pattern2 which may accure multiple times

I try to print out the lines between two different patterns(including the patterns) which may accure multiple times in a file. The second pattern may accure besides the startpattern-endpattern-rhytm. ...
0
votes
1answer
21 views

modify between (only) 1st occurrence of patterns (inclusive of patterns) and keep processing

There are several questions about matching between only the first occurrence of two patterns (e.g., this) — but they all seem to rely on exit, which is no good, as I want to keep processing the file. ...
0
votes
1answer
17 views

Ignoring first record in awk

BEGIN{FS=","; t1 = 0; n = 0; avg = 0; sum=0; printf("%-10s %9s\n%-10s %9s","Name", "Average", "----", "-------") } { if ($3 > 0){ t1 += $3; n++; } sum = 0; if($3 &...
2
votes
2answers
42 views

goto a specific line in awk to start processing [on hold]

I want the awk to go to the beginning of the block of lines to start processing it again once it hit the bottom of that block and processed its last line. Basically, if block of text is <START ...
1
vote
1answer
30 views

Element comparison between subsequent lines, assigning a value for each null output

I have an output which looks like this: ASCII file I want to pick up the last element from the rows that contain the string "Fe\s2b". But the thing is each occurrence of "H\s\s1\s\s\s\s\s\s4861.33A" ...
2
votes
1answer
40 views

search for single or multiple occurrences of the samestring and edit next line

I have a file that sometimes has one occurrence or multiple occurrences of the same string. I am needing to search for one or more occurrences in the file and go to the next line and keep the values ...
-1
votes
1answer
25 views

Sort based on second column and list from highest to lowest

TGAAACATGCCTTTCACCTCCTGCCATGATTGTGAGGCCT 2 GTGTTGCCCAGGTTGGTTTCAAACTCAGCTCAAGTGATCT 1 GAGACTCCTTCAGGAGACTGGTCCCCTGTCCTCGCCTCTC 3 CATGTTCTCCTGGAAGCCATGTGAGGAAGGCACATCATGG 8 ...
2
votes
1answer
171 views

Regex in IF condition in awk

I have awk script file as below. I need to add another condition in the if statement to check if the string contains atleast one alphabet. How can I add the extra condition to the present if statement?...
0
votes
1answer
21 views

Allign file based on field values

I have a file like below with two fields ending with *.alarms.gz and *.values.gz, want to allign the files like all the alarms.gz in first field and all the values.gz in second field. How can i ...
1
vote
1answer
48 views

Bash - How to update what's inside the quotes, but only if it's a number?

For this project, we have hundreds of applications using composer.json. I'm using a Bash script to go through and update the version numbers of any file that contain an application that is getting a ...
2
votes
3answers
178 views

How to get single character after space?

How can I obtain below output? I want the first field as it is and a single character after space. echo "Hello world" Hellow If it also has a 3rd field than the beginning character of the 3rd field ...
0
votes
1answer
30 views

Renaming multiple paired files, deleting varying barcode character string in middle?

I have a bunch of paired files with unneeded barcode tags within the middle of the file, for example: LIB008983_TRA00020080_TAAGGCGA-TATCCTCT_L001_R1.fastq.gz LIB008983_TRA00020080_TAAGGCGA-...
0
votes
3answers
53 views

awk or grep? - print keyword and following columns

I did some benchmarks and the output is a single file containing multiple results with the following format [blaaa] 1st run T/V N NB P Q Time ========================== 2 3 4 1 2 1....
-1
votes
3answers
79 views

Extract files which have more than 5 lines, and keep same filename [on hold]

I have multipe files containing errors in annual reports. Each file name have .mvt extension. It is not necessary that every file has error. I have found that when errors appear OR not there is always ...
0
votes
4answers
36 views

Match the column heading and print the values of the column using awk

How to print column free using awk or sed command? Depends on vmstat version column, free may change from column number 5 to something else. bash-3.00$ vmstat 1 1|tail -2 r b w swap free re mf pi po ...
1
vote
2answers
54 views

Combining two outputs using AWK

I have a bash command (on macOS High Sierra) as: sudo cat /private/var/log/system.log*| awk 'BEGIN {print "\n"} {print "Month\tDay\tTime\t\tSystemMessage"} /SHUTDOWN_TIME/{print $1,$2,$3,$8}'|sort -...
0
votes
0answers
36 views

How to rearrange a data?

I have Following type of data: each block starting with "Atom An X Y Z", I have 4 rows and 5 columns for each number point. I can arrange somehow for 3 columns and 2 rows with following code. awk '!(...
0
votes
2answers
42 views

Pick up successive lines containing keywords in order

I have a tab-separated file that looks as follows: $ cat file GCF_000015405.1_ASM1540v1.dist_nbr_anntn WP_011558474.1 1159543 1160595 -4330977 polyketide synthase [Mycobacterium] ...
0
votes
1answer
14 views

Invalid char error when running .awk file

I am appending a value 100 to every line in a file with the code below. This code works well when I run it in terminal but when I save the code in .awk file and run it I am getting an error. How to ...
0
votes
2answers
46 views

Extracting lines to new files

Say I have a large CSV file with a header and several columns. For the purpose of this question I will consider a small file with just two columns. We can call it use_rep. user_id,rep 885,500K+ 22565,...
1
vote
5answers
59 views

Question about text formatting using sed of awk

I want to format text using awk or sed. I have text looking like, H1 (dxfault) H337 (Unxr) H1106 (hast) H1129 (h1129) H1370 (hlan1370) I want to remove the letter H and everything after the first ...
1
vote
2answers
52 views

Remove duplicates from particular columns

I have a file in below format, where columns are separated with comma. [1], Value1, UAC, AB [2.2], Check1, BOH D2A D2A BOH, SD [63], name2, MFB MFB, k ... I want ...
1
vote
3answers
66 views

How to compare two CSV files and display unique records?

I have two csv file as below, i want to compare file1 with file2 and if records present in file2 that complete row should remove from file1. field to compare here is ID and in original file its at ...
0
votes
0answers
18 views

Delete the second line which matches a pattern keeping the first line if it starts with DA [on hold]

I have a file which has following lines Input DA0183930283940484088X802004830202**0102937**60 DU0235363283940484088X802004830202**0102937**20 ------------------------------------------- ------------...
0
votes
0answers
24 views

Inserting column headings in awk command

I have a raw file (gx.txt) having below sample contents shown below. TIME;PCS/Main:GxCCAInitialSuccES;PCS/Main:GxCCAInitialSuccSILP;PCS/Main:GxCCATermSuccES;PCS/Main:GxCCATermSuccSILP;PCS/Main:...
0
votes
1answer
18 views

Replacing tabs with spaces from nth column to last column

I have a tab-separated file which looks as follows: GCF_000014165.1_ASM1416v1.dist_nbr_anntn WP_011558474.1 1155234 1156286 44173 polyketide synthase [Mycobacterium] GCF_000014165....
1
vote
2answers
71 views

Get logs for last 30 minutes

I want to extract the logs between the current timestamp and 30 minutes before. I developed the below script but it's not working properly it shows only timestamp lines whereas I have to get ...
1
vote
2answers
43 views

awk for filtering the contents on two files

$ file1.txt 12345,865221,APPLE,ZZ,QQ,DD,GG,APPLE-FRUIT 12346,865222,MANGO,ZZ,QQ,DD,GG,MANGO-FRUIT 12347,865222,GRAPE,ZZ,QQ,DD,GG,GRAPE-FRUIT $file2.txt APPLE-FRUIT,10KG MANGO-FRUIT,12KG I have ...
0
votes
1answer
41 views

Using IDs to extract ID headers and add to second column in file B

I have a file A that contains accession number family name and name and file B that contains ID and sequences. I want to use the accession number in B to retrieve the family name and virus name in A ...
4
votes
4answers
494 views

extract characters between two commas?

I have a file with ~ 3 million rows, here is the first few lines of my file: head out.txt NA NA NA NA NA gene85752,gene85753 gene85752,gene85753 gene85752,gene85753 ...
0
votes
2answers
62 views

text processing between 2 patterns in awk to give selective unique output

I have below input file, Policy Name: KE15-LOCALHOST-APP-RADIX-DAILY Policy Type: Standard Active: yes Include: /appussd /home/ussd2ke /var/...
0
votes
1answer
38 views

Replace group of similar text in string from nth position with one string else increment the value by +1

I have a text file with following lines for example. Input DD0TRANSID000019021210504250003379433005533665506656000008587201902070168304000.0AK 0000L00000.00 N ...
0
votes
2answers
43 views

Add two doubles on bash

I'm trying to add two doubles y1=0.17580197E-01 y2=0.11979236E-02 sum=`echo $y1+$y2 | bc -l` the above script gives me sum = -2.704405652. How do i resolve this issue?
1
vote
1answer
56 views

Awk syntax, strange variable?

I have this awk command: awk -F, 'NR==FNR{_1[$1]++;next}_1[$1]' file1 file2 I actually almost know what the program is doing, but I don't know what _1 means. The intention is to compare two files, ...
0
votes
3answers
81 views

Replace text in lines in a file with increments

I have a file with multiple lines (no. of lines unknown) DD0TRANSID000019021210504250003379433005533665506656000008587201902070168304000.0AK 0000L00000.00 N 01683016832019021220190212N0000....
0
votes
1answer
32 views

awk filtering one column log file

I'm trying to use the following command to extract a time period forthe current day date and grep for the specific message as you can see below: awk -v date="$(date +%Y%m%d')" '$1="date" && $...