Questions tagged [awk]
A pattern-directed scanning and processing language.
7,058
questions
-1
votes
4answers
109 views
How to split columns of a CSV file into separate files?
I have csv file namely list.csv as shown below,
Pcissicola19,cissicola39,12xbauhiniae
BGDHLHFA_02833,DGDFDEGP_00879,POPGJMOL_04119
BGDHLHFA_01427,DGDFDEGP_03106,POPGJMOL_01558
BGDHLHFA_01618,...
-1
votes
1answer
115 views
grep logs between two timestamps
2021-02-10 08:17:38 dell AptDaemon.Worker: INFO: Processing transaction /org/debian/apt/transaction/08a244f7b8ce4fad9f6b304aca9eae7a
2021-02-10 08:17:50 dell AptDaemon.Worker: INFO: Finished ...
0
votes
2answers
28 views
Removing strings after first space only in first column
The following file is tab-separated. I tried to remove from the first column NbLab330C00 64506568 the number after space in order to get NbLab330C00.
$ head LAB330_TE_annotation.gff3
##gff-version 3 ...
0
votes
0answers
49 views
how to replace a substring of a line after multiple keywords/patterns have matched
I need to go through thousands of files and find lines that start with
# disabledRule:
On these lines I need to look for a series of keywords, namely fileinto or fileinto_copy followed by create and ...
-1
votes
1answer
54 views
extracting multiple columns from multiple files and merge them to a single table [closed]
I have nearly 500 files each with more that 120 columns and some of the columns are blank.
hghflf.maf
lnuegflkn.maf
jdfrlbvfl11.maf
jkfbrhw4rkb4.maf
....
here is an example of the head of data for ...
1
vote
1answer
30 views
Repeat the value first column based on key column - Linux
I have a pipe delimited file like this where the second column is the key that groups the ids in the first column.
456566|23233|
456545|23233|
456745|23233|
456456|23234|
456356|23234|
I am trying to ...
0
votes
2answers
83 views
Pass awk variable to command and read output
I would like to sort file in parts delimeted by string, but it is a larger part of processing in the awk.
I have read that it could be possible to store lines between delimeter and then use asort for ...
0
votes
3answers
25 views
Unable to get the nth column data due to changing output of uptime command
Below is the output of uptime from Solaris where I am extracting the third last column:
uptime
8:30pm up 23 day(s), 17:46, 5 users, load average: **2.79**, 1.79, 1.53
I always need to get the ...
2
votes
2answers
80 views
extracting substring
I have the following file
Nt01 maker mRNA 143295 155540 . + . ID=Nitab4.5_0006317g0010.1;Parent=Nitab4.5_0006317g0010;Name=Nitab4.5_0006317g0010.1;_AED=0.08;_eAED=0.08;_QI=0|0.45|0.25|...
-1
votes
4answers
60 views
print all line that match my pattern until last end pattern match
I have a file like below, I need to print all line that match my pattern.
used the below code but end pattern stops after the first pattern match, I need to print until last end pattern match.
Input
...
4
votes
4answers
265 views
Reformat timestamp in a pipe delimited file
I have a pipe delimited file with date time in this format yyyymmddhhmmss as below.
John|Doe|TEST|20210728120821|john@test.com
John|Davis|TEST|20210828120821|john@test.com
John|Smith|TEST|...
0
votes
1answer
66 views
How to concatenate and sum multiple line in a file using awk
I have a file A.csv (field separator = ,):
Lane,SampleID,Index,# Reads,# Perfect Index Reads,# One Mismatch Index Reads,# of >= Q30 Bases (PF),Mean Quality Score (PF)
1,4331099,AACCATAGAA-...
0
votes
0answers
29 views
passing shell variable to awk and get awk output in shell variable [duplicate]
I have passed argument to myscript.sh that returns a value in variable m. Problem is that I am not able to pass shell variable (runvalues) to awk script that search for a pattern. Therefore value of ...
0
votes
1answer
41 views
Arithmetic between 2 files generating a series of new files (Pt 2)
This is a more specific follow-up to a previous question (Arithmetic between 2 files generating a series of new files).
I have a tab separated model input file I would like to vary for an ensemble ...
0
votes
3answers
73 views
Split a row by a delimiter
I have a space-separated text file, e.g.
text1a text2a id1 text4a text5a
text1b text2b id2 text4b text5b
text1c text2c id3,id4 text4c text5c
text1d text2d id5,id6,id7 text4d,text4di text5d
The file ...
4
votes
6answers
943 views
how to perform mathematical operations on numbers in a file using perl or awk?
I have a fileA.txt:
RS0255_RS0083:115,124,129,141,143,168,170,180
RS0343_RS0083:112,113,163,175,181
RS0343_RS0255:94,101,107,164,179,183
I would like to perform mathematical operations on the numbers ...
6
votes
5answers
542 views
grep output from long to wide
I have a file of patterns and I want to return all the line numbers where the pattern was found, but in a wide format and not long/spread.
Example:
fileA.txt
Germany
USA
UK
fileB.txt
USA
USA
Italy
...
1
vote
6answers
102 views
Arithmetic between 2 files generating a series of new files
I have a tab separated model input file I would like to vary for an ensemble analysis formatted similar to this
input.txt
/* Preciptation in mm */
10 30 40 50 23
### Species description
*** sp_name ...
1
vote
1answer
183 views
How to change date format in Linux Redhat
Is it possible to change date format from example Fri 12 Feb 18:27:34 +08 2021 to just simply DD-MM-YYYY?
Specifically, to change date format in last command
-3
votes
5answers
97 views
How to read list in JSON file using awk and sed
I have a list in JSON file apps.json:
{
"properites":{
"apps":["apple","orange","mango"]
}
}
I need to read this file and get of objects.
I used:
...
0
votes
1answer
56 views
How to change 1 line of text with existing entries split to multiple lines
RHEL 7 system.
I have an output file that looks like
eDTG=20210210-1540
Sent 74004956
Completed 72185529 (97.54%)
Lost 18119427 (2.46%)
QPS, SERVFAIL QPS 60122 2905
eDTG=20210210-1601
...
-1
votes
4answers
119 views
Count lines where the second field starts with a specific character
I have a file that looks like this
header start
stuff
header end
pos LV file LVG size
1 a1 AAA BBB 100
2 b1 AAC BBB 1000
3 a3 AAB BBB 47
4 b6 AAC BBB 1000
I need to count the lines that ...
-1
votes
1answer
37 views
Extract single field from string in android emulator environment [closed]
I am running android emulator with the adb shell
I try to extract the field _id=number from a query output.
The "raw" result looks like:
Row: 9991 last_time_contacted=0, phonetic_name=NULL, ...
0
votes
4answers
83 views
How to show only line after the matched one?
I have a file A.tsv (field separator = \t):
for Research Use Only
[Header]
Test Name Il
Run ID 2102
...
0
votes
2answers
50 views
How to compare data from 2 files then create an output file that has the missing data in the order of file 2
I'm new to shell scripting and would like to create the below. I believe I could use something like grep -xvFf file1 file2 but I'm not sure how to create output file with the correct order.
I have ...
1
vote
2answers
55 views
AWK pattern-code-block syntax: how to run code blocks by pattern in AWK?
Suppose I have a couple of line formats:
this:foo ...
that: foo ...
And I want to run two different routines for one form or the other:
cat text_file \
| awk -F ':' '/:foo/ { # do this } # else print ...
0
votes
3answers
107 views
Parse yaml file without using outside tool [duplicate]
I am new to bash, dragged into a problem where I need to update some dynamic parameters in a yaml file
Based on query (search4.subsearch1.name), i need to update
name: old_name
into
name: new_name
...
-1
votes
2answers
71 views
How to print data in a template from multiple file in awk
I have a main file A.txt (field separator = \t) :
Sample ID Internal Control Result Consensus Sequence Lane Index Set Index ID
2154686427 Pass Detected Not Available 1,2,3,4 1 ...
5
votes
2answers
282 views
replace lines in one file with lines in another by line number
I have a fileA.txt with many lines and I want to replace specific lines with other specific lines from a second file fileB.txt which also has many lines
For example:
fileA.txt
Italy
Korea
USA
England
...
0
votes
3answers
68 views
Relative log file matching with linux?
I already know how to use grep or awk to match something in a log file, but is there a way to do the following?
Match a specific line in a file...and include it in the result.
From the matched line ...
0
votes
3answers
76 views
find text in one file and replace with text from another file SED or AWK
Ubuntu 18 build
I am trying to find this text, [define( 'AUTH_KEY',] without [] in /var/www/file1.txt, and then copy the line and replace the same line in /var/www/file2.php. I have tried multiple ...
-1
votes
3answers
57 views
awk text processing take values of limits
i have a txt file like this
32 1
32 2
32 3
32 4
32 5
17 8
17 9
17 10
17 11
14 33
14 34
14 35
and i want to get txt file lke this one ! so we take the limits MIN and MAX of a value in field one from ...
-3
votes
3answers
43 views
awk columns based on condition with header
I have a text file like below.
variable1
10 20 30
40 50 60
variable2
2 4 40
3 2 1
variable3
2 4 2
3 2 1
If col3>20 the output should be like below with header(variable1/2) values
...
1
vote
0answers
63 views
How to awk only output-lines containing a keyword?
I am in the process of writing a small script, which checks browser-tabs on certain keywords. If the keyword is present, then the script closes the tab. The script is actually a simple "block-...
0
votes
1answer
53 views
Compare three columns of two files
I have two files that have some similar and some unique columns. I'd like to compare the first 3 columns that are similar and print if they match, along with each of their unique columns.
File1:
A cat ...
0
votes
4answers
84 views
Finding a string and remove all contents between two delimiters
I have searched and I don't know what I'm doing wrong but I can't find an answer to this question.
I have a file with all text stored as a single line. I am needing to find a pattern and remove all ...
0
votes
1answer
39 views
How to compare column in two different csv file and replace column from another file
I have a requirement to match column 4 (Grade) from File1 with column 1 (Grade) of File2. If it matches, replace column 4 (Grade) of File1 with column2 (Marks) of File2. If it doesn't match, then ...
1
vote
1answer
99 views
Check number in another file if in range using shell script and output all information for that match
I have two input files
File A
chr1 1167639 0
chrX 114868305 14
chr1 1167653 0
chr11 62389325 2
chr9 140127768 17
File B
chr10 73082492 73082831 SLC29A3
chr11 46332562 ...
0
votes
2answers
41 views
How to merge two files of different number of lines using blank line condition?
I have two files, one have diffrent section separted by blank line as follows
a11 a12 a13
b11 b12 b13
c11 c12 c13
a21 a22 a23
b21 b22 b23
c21 c22 c23
d21 d22 d23
a31 a32 a33
b31 b32 b33
the second ...
1
vote
2answers
67 views
How to divide multiple values within a column by a constant?
I am trying to divide the values in 2 columns by a certain constant. However, there are multiple values in each column that are separated by a colon. The file is very large (24 files, 2-3 GB each) For ...
1
vote
2answers
56 views
awk: change line to itself parsed with command. Multiline text
I'm trying to answer a question. It involves using the command recode. A specific line has to be parsed by the command leaving the rest untouched. The file is an xml file. Example:
<ITEM_ID>...
-2
votes
4answers
127 views
Replace 0 with 1000 in Port 0 And increment by +1 in following lines
I want to increment port numbers starting from 1000. I know how to make all port 0 when they are assigned numbers with this command
sed 's/port [0-9]\{1,5\}/port 0/g'
but do not know how to do the ...
1
vote
1answer
54 views
Merge multiple CSV files from subfolders into a unique file and add file name in a new column
I have a bunch of CSV files in many subfolders (only 1 level of subfolders from the parent directory, nothing is nested further).
Each CSV file has three columns; i, k, v with a mix of string and ...
-2
votes
1answer
56 views
sed XML parse select text between tags with special characters while deleting other blocks
I am trying to parse an XML which consists of blocks between <JOB> and </JOB> , I am parsing within a do while loop , while reading the block between <JOB> and </JOB> , I am ...
1
vote
1answer
54 views
How does this awk command work?
Consider the file test.txt:
pfg025G
pfg025T
pfg034T
pfg039G
Now consider the following awk command and its output:
awk '(NR>1) {print "s/"p"/"$1"/g"}{p=$1}' test.txt
...
0
votes
4answers
168 views
split a multifasta file into different files with new headers
Input file: 131751_pphA.fasta
>ID:NDNDCOEC_02118 |[Genus species]|strain|PANS_1_2_annot.gbk|pphA|855|NODE_3_length_422941_cov_112.146787422941(422941):170566-171420:1 ^^ Genus species strain strain....
-2
votes
2answers
26 views
Move matched line with expr1 before another matched line with expr2 above in a file containing several expr2
I have a folder containing several files. In each of these files there might be one or more occurrences of the word implicit, and in some of the files with implicit there might be the sentence use ...
1
vote
2answers
95 views
How to collapse consecutive numbers into ranges and get the average
I have many .txt files with the following information
in 4 columns and thousands of rows:
chr10 73121691 18 SLC29A3
chr10 73121692 14 SLC29A3
chr10 73121693 10 SLC29A3
chr10 73120590 15 SLC29A3
chr10 ...
0
votes
2answers
48 views
awk - Remove a line, if a specific column contains as substring an element from another textfile
I have the following tsv-file (extract):
File 1:
NC_002163.1 RefSeq source 1 1641481 . + . organism=Campylobacter jejuni subsp. jejuni NCTC 11168;mol_type=genomic DNA;strain=NCTC 11168;...
5
votes
8answers
874 views
How to count the lines containing one of two words but not both
I need to count the lines containing the words the and an in a text file (poem.txt), but not those containing both.
I've tried using
grep -c the poem.txt | grep -c an poem.txt
but this gives me the ...
