Questions tagged [awk]

A pattern-directed scanning and processing language.

Filter by
Sorted by
Tagged with
0
votes
3answers
45 views

Combine two files using common column

I have 2 files. File 1 is: burnj01,Joe Burns steves02,Santosh Steve File 2 is: burnj01,0001 burnj01,0002 burnj01,0010 burnj,0100 burnj01,2000 steves02,2048 steves02,2049 steves02,2091 and I would ...
-1
votes
1answer
49 views

sed command to replace multiple lines after interval [closed]

I would like to replace the first character C with H for 23 lines (say, lines 83 to 106) after every interval of 82 lines. How can I do this using sed or awk?
1
vote
1answer
88 views

How to compare two column from two files with specific condition in awk

I have a data file A.tsv (field separator = \t) : id clade mutation 243 40A titi,xixi,lolo 254 20B titi,toto,jiji,lala 261 267 20B lala,jiji,jojo and a template file B.tsv (field separator = \t) : ...
0
votes
2answers
59 views

How to move multiple files to multiple folders in linux?

I have a folder containing *.fastq.gz files (R1 and R2) for many samples. For example, in the folder "Raw_WGS", I have several files like Sample_1_R1.fastq.gz, Sample_1_R2.fastq.gz, ...
1
vote
7answers
265 views

awk print columns separated by space [closed]

I have *.dat files like below in a directory: $ cat holiday_us.dat 20210101 1 New Year's Day 20210102 2 Labor Day 20210103 1 Independence Day Output I want: 20210101_New Year'sDay ...
3
votes
4answers
103 views

Find IP addresses in file excluding local IPs

The task: find and print lines from a file that contain any (IPv4) IP address except those on the local LAN (local LAN uses class C addresses of the 192.168.1.x variety). I (having cribbed bits from ...
0
votes
5answers
77 views

Remove block of lines from file with awk

I have a file looking like: --- some metadata ... --- Some real content. ... Important content. Expected output: Some real content. ... Important content. To keep things simple, let's consider that ...
1
vote
3answers
81 views

How to extract certain columns with AWK from one CSV file to another?

I have a very big CSV log file that contains fields like this: aaa=somedata1,bbb=somedata2,ccc=somedata3,eee=somedata5,hhh=somedata8 aaa=somedata1,ddd=somedata4,fff=somedata6,hhh=somedata8 aaa=...
10
votes
2answers
809 views

Why is there a syntax error if I don't write 'if' in an END block of AWK?

I'm learning AWK these days for text processing. But I'm very much confused about AWK syntax. I read on Wikipedia that the syntax follows this format: (conditions) {actions} I assumed I can follow ...
-4
votes
5answers
97 views

Duplicate lines containing a string with edits [closed]

I need to find a way to duplicate lines in a file containing a string with small edits. The dupe would have to show up above the original line. Would sed or awk be my best bet? Ideally replacing ...
1
vote
1answer
52 views

How to create folders based on part of a filename and also extract & rename specific columns using awk

the following code extracts data from csv files contained in a folder, creates new folders (based on filename and date), creates and stores multiple csv files in the said folders. more info on this is ...
2
votes
2answers
76 views

How to compare two column of two file and print not matching pattern with awk

I have a data file A.tsv (field separator = \t) : id clade mutation 243 40A siti,toto,mumu 254 267 40B lala,sisi,sojo and a template file B.tsv (field separator = \t) : 40A toto,xixi,saxa 40B ...
1
vote
4answers
113 views

How to compare two column of two file and print the number of match with awk

I have a data file A.tsv (field separator = \t) : id clade mutation 243 40A titi,toto,lala 254 267 40B lala,jiji,jojo and a template file B.tsv (field separator = \t) : 40A lala,toto,xixi,...
4
votes
2answers
182 views

How to split CSV file and create multiple CSV files based on a column

I have a csv file with the format: Input.csv: TIMESTAMP,Data1,Data2,Data3,Data4 "2021-01-03 00:00:00",80953,3.243183,2.943338,358.0123 "2021-01-03 00:01:00",80954,2.173187,1.990327,...
0
votes
2answers
44 views

awk paragraph with string and the following paragraph in pgn chess games

How can I get a paragraph with the string and the following paragraph with awk? The file looks like this: [Event "Chess game"] [Date "2021.01.08"] [White "Player A"] [...
-1
votes
5answers
88 views

Replace characters on specific lines and specific columns

Consider this sample.txt: ATO N X B AT H1 X BT ATOM H25 X BAA ATOM H3 X BUTZ ATOM CA X BAT I want to replace X's from lines 2-4, ...
1
vote
2answers
61 views

ksh88 AIX merge two files based on partial string match in one column

Using ksh88 on AIX machine, I've tried a whole slew of things that are ultimately not working. File_A has 2 columns without headers: the user group that created the directory in column 1, and the full ...
0
votes
1answer
48 views

awk: how to extract strings directly into bash array?

let's say I have this: ARRAY=() NUM_EXTRACTED=$(cat somefile.txt | awk '/magic/{i++; print > i "_extracted.txt"} END {print i}') for (( i=1; i<=$NUM_EXTRACTED; i++)); do LINE=$(cat &...
-1
votes
2answers
56 views

How to merge two files based on the matching of one column?

I have two files, B.csv: 1,AD 2,AB 3,AC 5,AF 7,AE and C.csv: 1,x 3,z 5,y How do I get this output: 1,AD,x 2,AB, 3,AC,z 5,AF,y 7,AE, by matching the common column 1 in both of the files?
0
votes
2answers
47 views

awk to vlookup and print

I got a comma-separated file # cat data smartplayer,2222,off smartplayer,1111,on scorer,0000, I want to search for smartplayer in $1, get its state from $3 and print like this off-smart-player 2222 ...
0
votes
2answers
63 views

How to combine data from two file with the header in awk

I have two files, A.tsv and B.tsv: A.tsv (field separator = \t): Sample ID Internal Control Result Consensus 4686427 Pass Not Detected Not Available 4666275 Pass Detected Not ...
0
votes
1answer
48 views

How to use BASH variable for AWK filtering results

I have this script, I'm getting first letter of my directory name, and I wan't to get directory names that start with the same letter and write to textfile. The for loop is looping list of directory ...
-1
votes
4answers
106 views

How to split columns of a CSV file into separate files?

I have csv file namely list.csv as shown below, Pcissicola19,cissicola39,12xbauhiniae BGDHLHFA_02833,DGDFDEGP_00879,POPGJMOL_04119 BGDHLHFA_01427,DGDFDEGP_03106,POPGJMOL_01558 BGDHLHFA_01618,...
-1
votes
1answer
89 views

grep logs between two timestamps

2021-02-10 08:17:38 dell AptDaemon.Worker: INFO: Processing transaction /org/debian/apt/transaction/08a244f7b8ce4fad9f6b304aca9eae7a 2021-02-10 08:17:50 dell AptDaemon.Worker: INFO: Finished ...
0
votes
2answers
28 views

Removing strings after first space only in first column

The following file is tab-separated. I tried to remove from the first column NbLab330C00 64506568 the number after space in order to get NbLab330C00. $ head LAB330_TE_annotation.gff3 ##gff-version 3 ...
0
votes
0answers
48 views

how to replace a substring of a line after multiple keywords/patterns have matched

I need to go through thousands of files and find lines that start with # disabledRule: On these lines I need to look for a series of keywords, namely fileinto or fileinto_copy followed by create and ...
-1
votes
1answer
54 views

extracting multiple columns from multiple files and merge them to a single table [closed]

I have nearly 500 files each with more that 120 columns and some of the columns are blank. hghflf.maf lnuegflkn.maf jdfrlbvfl11.maf jkfbrhw4rkb4.maf .... here is an example of the head of data for ...
1
vote
1answer
30 views

Repeat the value first column based on key column - Linux

I have a pipe delimited file like this where the second column is the key that groups the ids in the first column. 456566|23233| 456545|23233| 456745|23233| 456456|23234| 456356|23234| I am trying to ...
0
votes
2answers
82 views

Pass awk variable to command and read output

I would like to sort file in parts delimeted by string, but it is a larger part of processing in the awk. I have read that it could be possible to store lines between delimeter and then use asort for ...
0
votes
3answers
25 views

Unable to get the nth column data due to changing output of uptime command

Below is the output of uptime from Solaris where I am extracting the third last column: uptime 8:30pm up 23 day(s), 17:46, 5 users, load average: **2.79**, 1.79, 1.53 I always need to get the ...
2
votes
2answers
80 views

extracting substring

I have the following file Nt01 maker mRNA 143295 155540 . + . ID=Nitab4.5_0006317g0010.1;Parent=Nitab4.5_0006317g0010;Name=Nitab4.5_0006317g0010.1;_AED=0.08;_eAED=0.08;_QI=0|0.45|0.25|...
-1
votes
4answers
58 views

print all line that match my pattern until last end pattern match

I have a file like below, I need to print all line that match my pattern. used the below code but end pattern stops after the first pattern match, I need to print until last end pattern match. Input ...
4
votes
4answers
265 views

Reformat timestamp in a pipe delimited file

I have a pipe delimited file with date time in this format yyyymmddhhmmss as below. John|Doe|TEST|20210728120821|john@test.com John|Davis|TEST|20210828120821|john@test.com John|Smith|TEST|...
0
votes
1answer
65 views

How to concatenate and sum multiple line in a file using awk

I have a file A.csv (field separator = ,): Lane,SampleID,Index,# Reads,# Perfect Index Reads,# One Mismatch Index Reads,# of >= Q30 Bases (PF),Mean Quality Score (PF) 1,4331099,AACCATAGAA-...
0
votes
0answers
29 views

passing shell variable to awk and get awk output in shell variable [duplicate]

I have passed argument to myscript.sh that returns a value in variable m. Problem is that I am not able to pass shell variable (runvalues) to awk script that search for a pattern. Therefore value of ...
0
votes
1answer
41 views

Arithmetic between 2 files generating a series of new files (Pt 2)

This is a more specific follow-up to a previous question (Arithmetic between 2 files generating a series of new files). I have a tab separated model input file I would like to vary for an ensemble ...
0
votes
3answers
73 views

Split a row by a delimiter

I have a space-separated text file, e.g. text1a text2a id1 text4a text5a text1b text2b id2 text4b text5b text1c text2c id3,id4 text4c text5c text1d text2d id5,id6,id7 text4d,text4di text5d The file ...
4
votes
6answers
940 views

how to perform mathematical operations on numbers in a file using perl or awk?

I have a fileA.txt: RS0255_RS0083:115,124,129,141,143,168,170,180 RS0343_RS0083:112,113,163,175,181 RS0343_RS0255:94,101,107,164,179,183 I would like to perform mathematical operations on the numbers ...
6
votes
5answers
541 views

grep output from long to wide

I have a file of patterns and I want to return all the line numbers where the pattern was found, but in a wide format and not long/spread. Example: fileA.txt Germany USA UK fileB.txt USA USA Italy ...
1
vote
6answers
102 views

Arithmetic between 2 files generating a series of new files

I have a tab separated model input file I would like to vary for an ensemble analysis formatted similar to this input.txt /* Preciptation in mm */ 10 30 40 50 23 ### Species description *** sp_name ...
1
vote
1answer
182 views

How to change date format in Linux Redhat

Is it possible to change date format from example Fri 12 Feb 18:27:34 +08 2021 to just simply DD-MM-YYYY? Specifically, to change date format in last command
-3
votes
5answers
97 views

How to read list in JSON file using awk and sed

I have a list in JSON file apps.json: { "properites":{ "apps":["apple","orange","mango"] } } I need to read this file and get of objects. I used: ...
0
votes
1answer
56 views

How to change 1 line of text with existing entries split to multiple lines

RHEL 7 system. I have an output file that looks like eDTG=20210210-1540 Sent 74004956 Completed 72185529 (97.54%) Lost 18119427 (2.46%) QPS, SERVFAIL QPS 60122 2905 eDTG=20210210-1601 ...
-1
votes
4answers
118 views

Count lines where the second field starts with a specific character

I have a file that looks like this header start stuff header end pos LV file LVG size 1 a1 AAA BBB 100 2 b1 AAC BBB 1000 3 a3 AAB BBB 47 4 b6 AAC BBB 1000 I need to count the lines that ...
-1
votes
1answer
37 views

Extract single field from string in android emulator environment [closed]

I am running android emulator with the adb shell I try to extract the field _id=number from a query output. The "raw" result looks like: Row: 9991 last_time_contacted=0, phonetic_name=NULL, ...
0
votes
4answers
83 views

How to show only line after the matched one?

I have a file A.tsv (field separator = \t): for Research Use Only [Header] Test Name Il Run ID 2102 ...
0
votes
2answers
50 views

How to compare data from 2 files then create an output file that has the missing data in the order of file 2

I'm new to shell scripting and would like to create the below. I believe I could use something like grep -xvFf file1 file2 but I'm not sure how to create output file with the correct order. I have ...
1
vote
2answers
55 views

AWK pattern-code-block syntax: how to run code blocks by pattern in AWK?

Suppose I have a couple of line formats: this:foo ... that: foo ... And I want to run two different routines for one form or the other: cat text_file \ | awk -F ':' '/:foo/ { # do this } # else print ...
0
votes
3answers
104 views

Parse yaml file without using outside tool [duplicate]

I am new to bash, dragged into a problem where I need to update some dynamic parameters in a yaml file Based on query (search4.subsearch1.name), i need to update name: old_name into name: new_name ...
-1
votes
2answers
71 views

How to print data in a template from multiple file in awk

I have a main file A.txt (field separator = \t) : Sample ID Internal Control Result Consensus Sequence Lane Index Set Index ID 2154686427 Pass Detected Not Available 1,2,3,4 1 ...