Questions tagged [awk]
A pattern-directed scanning and processing language.
7,032
questions
0
votes
3answers
45 views
Combine two files using common column
I have 2 files.
File 1 is:
burnj01,Joe Burns
steves02,Santosh Steve
File 2 is:
burnj01,0001
burnj01,0002
burnj01,0010
burnj,0100
burnj01,2000
steves02,2048
steves02,2049
steves02,2091
and I would ...
-1
votes
1answer
49 views
sed command to replace multiple lines after interval [closed]
I would like to replace the first character C with H for 23 lines (say, lines 83 to 106) after every interval of 82 lines. How can I do this using sed or awk?
1
vote
1answer
88 views
How to compare two column from two files with specific condition in awk
I have a data file A.tsv (field separator = \t) :
id clade mutation
243 40A titi,xixi,lolo
254 20B titi,toto,jiji,lala
261
267 20B lala,jiji,jojo
and a template file B.tsv (field separator = \t) :
...
0
votes
2answers
59 views
How to move multiple files to multiple folders in linux?
I have a folder containing *.fastq.gz files (R1 and R2) for many samples. For example, in the folder "Raw_WGS", I have several files like
Sample_1_R1.fastq.gz, Sample_1_R2.fastq.gz, ...
1
vote
7answers
265 views
awk print columns separated by space [closed]
I have *.dat files like below in a directory:
$ cat holiday_us.dat
20210101 1 New Year's Day
20210102 2 Labor Day
20210103 1 Independence Day
Output I want:
20210101_New Year'sDay
...
3
votes
4answers
103 views
Find IP addresses in file excluding local IPs
The task: find and print lines from a file that contain any (IPv4) IP address except those on the local LAN (local LAN uses class C addresses of the 192.168.1.x variety). I (having cribbed bits from ...
0
votes
5answers
77 views
Remove block of lines from file with awk
I have a file looking like:
---
some metadata
...
---
Some real content.
...
Important content.
Expected output:
Some real content.
...
Important content.
To keep things simple, let's consider that ...
1
vote
3answers
81 views
How to extract certain columns with AWK from one CSV file to another?
I have a very big CSV log file that contains fields like this:
aaa=somedata1,bbb=somedata2,ccc=somedata3,eee=somedata5,hhh=somedata8
aaa=somedata1,ddd=somedata4,fff=somedata6,hhh=somedata8
aaa=...
10
votes
2answers
809 views
Why is there a syntax error if I don't write 'if' in an END block of AWK?
I'm learning AWK these days for text processing. But I'm very much confused about AWK syntax. I read on Wikipedia that the syntax follows this format:
(conditions) {actions}
I assumed I can follow ...
-4
votes
5answers
97 views
Duplicate lines containing a string with edits [closed]
I need to find a way to duplicate lines in a file containing a string with small edits. The dupe would have to show up above the original line. Would sed or awk be my best bet?
Ideally replacing ...
1
vote
1answer
52 views
How to create folders based on part of a filename and also extract & rename specific columns using awk
the following code extracts data from csv files contained in a folder, creates new folders (based on filename and date), creates and stores multiple csv files in the said folders. more info on this is ...
2
votes
2answers
76 views
How to compare two column of two file and print not matching pattern with awk
I have a data file A.tsv (field separator = \t) :
id clade mutation
243 40A siti,toto,mumu
254
267 40B lala,sisi,sojo
and a template file B.tsv (field separator = \t) :
40A toto,xixi,saxa
40B ...
1
vote
4answers
113 views
How to compare two column of two file and print the number of match with awk
I have a data file A.tsv (field separator = \t) :
id clade mutation
243 40A titi,toto,lala
254
267 40B lala,jiji,jojo
and a template file B.tsv (field separator = \t) :
40A lala,toto,xixi,...
4
votes
2answers
182 views
How to split CSV file and create multiple CSV files based on a column
I have a csv file with the format:
Input.csv:
TIMESTAMP,Data1,Data2,Data3,Data4
"2021-01-03 00:00:00",80953,3.243183,2.943338,358.0123
"2021-01-03 00:01:00",80954,2.173187,1.990327,...
0
votes
2answers
44 views
awk paragraph with string and the following paragraph in pgn chess games
How can I get a paragraph with the string and the following paragraph with awk?
The file looks like this:
[Event "Chess game"]
[Date "2021.01.08"]
[White "Player A"]
[...
-1
votes
5answers
88 views
Replace characters on specific lines and specific columns
Consider this sample.txt:
ATO N X B
AT H1 X BT
ATOM H25 X BAA
ATOM H3 X BUTZ
ATOM CA X BAT
I want to replace X's from lines 2-4, ...
1
vote
2answers
61 views
ksh88 AIX merge two files based on partial string match in one column
Using ksh88 on AIX machine, I've tried a whole slew of things that are ultimately not working.
File_A has 2 columns without headers: the user group that created the directory in column 1, and the full ...
0
votes
1answer
48 views
awk: how to extract strings directly into bash array?
let's say I have this:
ARRAY=()
NUM_EXTRACTED=$(cat somefile.txt | awk '/magic/{i++; print > i "_extracted.txt"} END {print i}')
for (( i=1; i<=$NUM_EXTRACTED; i++)); do
LINE=$(cat &...
-1
votes
2answers
56 views
How to merge two files based on the matching of one column?
I have two files, B.csv:
1,AD
2,AB
3,AC
5,AF
7,AE
and C.csv:
1,x
3,z
5,y
How do I get this output:
1,AD,x
2,AB,
3,AC,z
5,AF,y
7,AE,
by matching the common column 1 in both of the files?
0
votes
2answers
47 views
awk to vlookup and print
I got a comma-separated file
# cat data
smartplayer,2222,off
smartplayer,1111,on
scorer,0000,
I want to search for smartplayer in $1, get its state from $3
and print like this
off-smart-player 2222
...
0
votes
2answers
63 views
How to combine data from two file with the header in awk
I have two files, A.tsv and B.tsv:
A.tsv (field separator = \t):
Sample ID Internal Control Result Consensus
4686427 Pass Not Detected Not Available
4666275 Pass Detected Not ...
0
votes
1answer
48 views
How to use BASH variable for AWK filtering results
I have this script, I'm getting first letter of my directory name, and I wan't to get directory names that start with the same letter and write to textfile. The for loop is looping list of directory ...
-1
votes
4answers
106 views
How to split columns of a CSV file into separate files?
I have csv file namely list.csv as shown below,
Pcissicola19,cissicola39,12xbauhiniae
BGDHLHFA_02833,DGDFDEGP_00879,POPGJMOL_04119
BGDHLHFA_01427,DGDFDEGP_03106,POPGJMOL_01558
BGDHLHFA_01618,...
-1
votes
1answer
89 views
grep logs between two timestamps
2021-02-10 08:17:38 dell AptDaemon.Worker: INFO: Processing transaction /org/debian/apt/transaction/08a244f7b8ce4fad9f6b304aca9eae7a
2021-02-10 08:17:50 dell AptDaemon.Worker: INFO: Finished ...
0
votes
2answers
28 views
Removing strings after first space only in first column
The following file is tab-separated. I tried to remove from the first column NbLab330C00 64506568 the number after space in order to get NbLab330C00.
$ head LAB330_TE_annotation.gff3
##gff-version 3 ...
0
votes
0answers
48 views
how to replace a substring of a line after multiple keywords/patterns have matched
I need to go through thousands of files and find lines that start with
# disabledRule:
On these lines I need to look for a series of keywords, namely fileinto or fileinto_copy followed by create and ...
-1
votes
1answer
54 views
extracting multiple columns from multiple files and merge them to a single table [closed]
I have nearly 500 files each with more that 120 columns and some of the columns are blank.
hghflf.maf
lnuegflkn.maf
jdfrlbvfl11.maf
jkfbrhw4rkb4.maf
....
here is an example of the head of data for ...
1
vote
1answer
30 views
Repeat the value first column based on key column - Linux
I have a pipe delimited file like this where the second column is the key that groups the ids in the first column.
456566|23233|
456545|23233|
456745|23233|
456456|23234|
456356|23234|
I am trying to ...
0
votes
2answers
82 views
Pass awk variable to command and read output
I would like to sort file in parts delimeted by string, but it is a larger part of processing in the awk.
I have read that it could be possible to store lines between delimeter and then use asort for ...
0
votes
3answers
25 views
Unable to get the nth column data due to changing output of uptime command
Below is the output of uptime from Solaris where I am extracting the third last column:
uptime
8:30pm up 23 day(s), 17:46, 5 users, load average: **2.79**, 1.79, 1.53
I always need to get the ...
2
votes
2answers
80 views
extracting substring
I have the following file
Nt01 maker mRNA 143295 155540 . + . ID=Nitab4.5_0006317g0010.1;Parent=Nitab4.5_0006317g0010;Name=Nitab4.5_0006317g0010.1;_AED=0.08;_eAED=0.08;_QI=0|0.45|0.25|...
-1
votes
4answers
58 views
print all line that match my pattern until last end pattern match
I have a file like below, I need to print all line that match my pattern.
used the below code but end pattern stops after the first pattern match, I need to print until last end pattern match.
Input
...
4
votes
4answers
265 views
Reformat timestamp in a pipe delimited file
I have a pipe delimited file with date time in this format yyyymmddhhmmss as below.
John|Doe|TEST|20210728120821|john@test.com
John|Davis|TEST|20210828120821|john@test.com
John|Smith|TEST|...
0
votes
1answer
65 views
How to concatenate and sum multiple line in a file using awk
I have a file A.csv (field separator = ,):
Lane,SampleID,Index,# Reads,# Perfect Index Reads,# One Mismatch Index Reads,# of >= Q30 Bases (PF),Mean Quality Score (PF)
1,4331099,AACCATAGAA-...
0
votes
0answers
29 views
passing shell variable to awk and get awk output in shell variable [duplicate]
I have passed argument to myscript.sh that returns a value in variable m. Problem is that I am not able to pass shell variable (runvalues) to awk script that search for a pattern. Therefore value of ...
0
votes
1answer
41 views
Arithmetic between 2 files generating a series of new files (Pt 2)
This is a more specific follow-up to a previous question (Arithmetic between 2 files generating a series of new files).
I have a tab separated model input file I would like to vary for an ensemble ...
0
votes
3answers
73 views
Split a row by a delimiter
I have a space-separated text file, e.g.
text1a text2a id1 text4a text5a
text1b text2b id2 text4b text5b
text1c text2c id3,id4 text4c text5c
text1d text2d id5,id6,id7 text4d,text4di text5d
The file ...
4
votes
6answers
940 views
how to perform mathematical operations on numbers in a file using perl or awk?
I have a fileA.txt:
RS0255_RS0083:115,124,129,141,143,168,170,180
RS0343_RS0083:112,113,163,175,181
RS0343_RS0255:94,101,107,164,179,183
I would like to perform mathematical operations on the numbers ...
6
votes
5answers
541 views
grep output from long to wide
I have a file of patterns and I want to return all the line numbers where the pattern was found, but in a wide format and not long/spread.
Example:
fileA.txt
Germany
USA
UK
fileB.txt
USA
USA
Italy
...
1
vote
6answers
102 views
Arithmetic between 2 files generating a series of new files
I have a tab separated model input file I would like to vary for an ensemble analysis formatted similar to this
input.txt
/* Preciptation in mm */
10 30 40 50 23
### Species description
*** sp_name ...
1
vote
1answer
182 views
How to change date format in Linux Redhat
Is it possible to change date format from example Fri 12 Feb 18:27:34 +08 2021 to just simply DD-MM-YYYY?
Specifically, to change date format in last command
-3
votes
5answers
97 views
How to read list in JSON file using awk and sed
I have a list in JSON file apps.json:
{
"properites":{
"apps":["apple","orange","mango"]
}
}
I need to read this file and get of objects.
I used:
...
0
votes
1answer
56 views
How to change 1 line of text with existing entries split to multiple lines
RHEL 7 system.
I have an output file that looks like
eDTG=20210210-1540
Sent 74004956
Completed 72185529 (97.54%)
Lost 18119427 (2.46%)
QPS, SERVFAIL QPS 60122 2905
eDTG=20210210-1601
...
-1
votes
4answers
118 views
Count lines where the second field starts with a specific character
I have a file that looks like this
header start
stuff
header end
pos LV file LVG size
1 a1 AAA BBB 100
2 b1 AAC BBB 1000
3 a3 AAB BBB 47
4 b6 AAC BBB 1000
I need to count the lines that ...
-1
votes
1answer
37 views
Extract single field from string in android emulator environment [closed]
I am running android emulator with the adb shell
I try to extract the field _id=number from a query output.
The "raw" result looks like:
Row: 9991 last_time_contacted=0, phonetic_name=NULL, ...
0
votes
4answers
83 views
How to show only line after the matched one?
I have a file A.tsv (field separator = \t):
for Research Use Only
[Header]
Test Name Il
Run ID 2102
...
0
votes
2answers
50 views
How to compare data from 2 files then create an output file that has the missing data in the order of file 2
I'm new to shell scripting and would like to create the below. I believe I could use something like grep -xvFf file1 file2 but I'm not sure how to create output file with the correct order.
I have ...
1
vote
2answers
55 views
AWK pattern-code-block syntax: how to run code blocks by pattern in AWK?
Suppose I have a couple of line formats:
this:foo ...
that: foo ...
And I want to run two different routines for one form or the other:
cat text_file \
| awk -F ':' '/:foo/ { # do this } # else print ...
0
votes
3answers
104 views
Parse yaml file without using outside tool [duplicate]
I am new to bash, dragged into a problem where I need to update some dynamic parameters in a yaml file
Based on query (search4.subsearch1.name), i need to update
name: old_name
into
name: new_name
...
-1
votes
2answers
71 views
How to print data in a template from multiple file in awk
I have a main file A.txt (field separator = \t) :
Sample ID Internal Control Result Consensus Sequence Lane Index Set Index ID
2154686427 Pass Detected Not Available 1,2,3,4 1 ...
