Questions tagged [awk]
A pattern-directed scanning and processing language.
7,688
questions
-1
votes
2
answers
49
views
Replace empty spaces with TAB except in last column
I have a file that looks like this.
How can I replace each empty space with a TAB except in the last column that starts with ID?
chr1 13513 14763 Medtr1g004950 . + JCVI gene . ID=Medtr1g004950;Note=...
0
votes
4
answers
140
views
awk loop over range of records write to separate files
I have a file with several thousand lines describing the output from an elastic wave mode
code. The size of the file is variable depending on the frequency, the thickness of the
model, and the number ...
- 1
2
votes
4
answers
161
views
Unable to exclude lines containing ^M from grep
I have the following file from which I want to extract only Removed '2022-01-30_01-00-05', at the end.
Removing '2022-01-30_01-00-05'...
0.46% complete (00:03:45 remaining)^M 5.49% complete (00:00:...
- 253
4
votes
7
answers
322
views
awk: deal with newline separation in wrong place
I have some data that looks like this:
abc
123
456
789
def
111
222
333
ghi
999
888
777
666
i.e. the records are separated by multiple newlines but in the wrong place. What I want is to get it like ...
- 145
-3
votes
3
answers
138
views
Any idea how to calculate median value of each part of column
I have a data like this:
111 5
111 6
111 1
222 8
222 9
222 1
222 3
555 9
555 7
555 6
and for each value of $1, I want to get the median value of all the values of $2 for that $1 if possible using AWK ...
- 163
0
votes
1
answer
34
views
print byte from number in awk
I can print a byte from a string literal like:
awk 'BEGIN {print "\001"}' | cat -v
But I need to print a byte of the result of a bitwise OR.
So how can I print a byte from a number?
Gawk is ...
- 5
0
votes
0
answers
32
views
How to break a long line into max length by word and use that programatically in a git commit [duplicate]
I have a scenario where I need to convert a long string of text and break it by words not exceeding a set line length (70 characters). I need to replace the space with a '\n' where the line break ...
0
votes
2
answers
179
views
Compare one column in one file, against another column and if they match, append column
I would like to compare one column from one file against another column in another file. If the columns match, I would like to append the first column from the first file, as the first column of the ...
- 3
0
votes
1
answer
79
views
Reading multiple zipped files in awk script
I have an awk script named text_processing.awk . I need to pass multiple zipped files named 01JAN21.txt.gz , 02JAN21.txt.gz etc. to this script for further processing. The output file takes the date ...
user511618
2
votes
1
answer
61
views
Merge two files using first column
I have two different files and I'd like to do a merge of their information using the first column.
File1.txt
A,info1,info2
234,info3,info4
CD,info5,info6
File2.txt
234,ccc,bb
CD,aaa,dd
Expected ...
- 161
2
votes
2
answers
122
views
Matching the first occurrence only of a string then adding N lines awk
Hi would like extract N number of lines after matching the first incidence of a string printing the N lines following using awk. The string is repeated a number of times in the file I am ...
- 25
1
vote
0
answers
15
views
How to import environment variable in awk? [duplicate]
I have the following code:
export my_var=3
cat /etc/passwd | awk {'print $1'}
Now I need to use my my_var in my awk pseudo code.
How can I apply this?
- 21
1
vote
2
answers
85
views
AWK: Insert in random selected lines target terms after source terms from a dictionary
Note: I have already asked a similar question in AWK: Quick way to insert target words after a source term and I am at the beginner level of AWK.
This question considers the insertion of multiple ...
- 161
2
votes
1
answer
190
views
After AWK the number of lines has been increased
After running my AWK script
awk -i inplace '(NR==FNR){a[$1];next}
(FNR in a) && gsub(/\<Source Term\>/,"& Target Term")
1
' <(shuf -n 198058 -i 1-$(wc -l &...
- 161
0
votes
0
answers
28
views
I need to replace placeholders in a yaml file with the lines from another file [duplicate]
I have a text file looking like this
Name: somename
username:some alphanumeric string
password:some alphanumeric string
And a yaml file looking like this
apiVersion: v1
kind: secret
metadata:
name: ...
0
votes
1
answer
69
views
AWK: Quick way to insert target words after a source term
I'm not familiar with awk. For inserting a single target term after a source term in 198058 random lines I have this code here
awk -i inplace '(NR==FNR){a[$1];next}
(FNR in a) && gsub(/\&...
- 161
0
votes
2
answers
89
views
Average of lines with different conditions at header and tail of files
Edit: edited it for clarity and made the example files minimal and more reproducible for easier help. Thanks!
I have files that gave 1000+lines. Each file is formatted with the same amount of lines. ...
- 27
0
votes
1
answer
67
views
How to group docker stats by container name prefix?
How to group docker stats --no-stream by container prefix, and show the sum of RAM and CPU and network and IO?
If we consider the first two columns of the docker stats --no-stream command to be like ...
- 199
1
vote
1
answer
46
views
AWK: Add missing lines in csv file and set specific column to "0"
(Note: this question is heavily shortened because the detailed question was blocked with "This looks like spam.")
I'm having a hard time adding missing lines to a series of data in a ...
- 13
1
vote
2
answers
274
views
Gunzip inside awk script
I am creating an output file inside an awk script which I want to compress then.
Input file - Marks.txt
Student1:AP:Maths:30:Science:43
Student2:AP:Maths:23:Science:35
Student3:Non_AP:Maths:17:Science:...
user511618
1
vote
2
answers
143
views
How to programmatically maintain umask in /etc/profile and /etc/bashrc
I am looking for a way to programmatically maintain a consistent umask in the files /etc/profile and /etc/bashrc. They have an entry such as follows:
if [ $UID -gt 199 ] && [ "`/usr/bin/...
- 11
4
votes
5
answers
278
views
Manipulating text using shell script: How can I fill in "missing" lines?
I have a list of data, like a CSV but some lines are missing a value. I'd like to generate a value for the missing line based on the lines before and after using linux shell script.
Take this table ...
- 43
0
votes
0
answers
27
views
How to join two files into one in multiple folders using specific columns and add file names where they originated from
I would like to ask, how can I join multiple files into one using a specific column and name it by the originating folder.
Let me explain: I have the following files, having multiple columns and rows:
...
- 5
7
votes
1
answer
304
views
Is "warning: range of the form `[o-b]' is locale dependent" a bug in gawk 4.1.0?
While writing some awk code I was using the --lint option for gawk 4.1.0.
To my surprise I get a warning
warning: range of the form `[o-b]' is locale dependent
but my code has only a foo-bar outside ...
- 801
0
votes
6
answers
93
views
linux change date format in file with full year
I am trying to change the date format in file with the full year. This is my data.
06/30/21 07/01/21 05436841182400056721972 random text
07/06/21 07/07/21 05436841188400057034635 random text
07/17/21 ...
- 981
0
votes
1
answer
99
views
Awk-Relational processing. Editing on one column must reflect on column on another line number specified by third column of actual line
relational column processing using awk sed
File 1 ---> Scrambled data in column (Old stale files with intermediate test data)
mL9A7hajHyuVIQr1HNP7ThYfj9yBUd
Iq4iqnH4UftLgGUSobLeti0hkmdMn7
...
user414574
0
votes
1
answer
54
views
how get ssh awk remote values
im using this script
[test@sys-master ~]$ parallel -k -j 100 sshpass -p test1213'!' ssh -oStrictHostKeyChecking=no -oCheckHostIP=no -q root@10.0.{} "cat /proc/loadavg | awk -F \" \" '{...
- 23
0
votes
1
answer
148
views
Using grep or awk, how can I count the number of characters in every line but also ignore certain lines based on their first character?
Very simply, I have this kind of text in a file:
#number
3222
#number
3311
I'm using:
awk '{print length}' file.txt
and it prints out:
7
4
7
4
But I just want it to print the length of the ...
0
votes
1
answer
47
views
Pass variable to context match awk [duplicate]
I am trying to extract context from a file with awk from a start pattern until an end pattern.
Here is simplified sample input to reproduce:
$ cat file
1
2
3
4
5
6
When I pass the start pattern ...
- 501
2
votes
6
answers
697
views
Using grep/awk to find/print certain value in a text file
Assume that i have a text file containing the following 5 lines:
Tue 18 2022 car model: Toyota , car motor: 2001 , car color: blue , year of production: 2018
Thu 19 2022 car model: Mercedes , car ...
- 89
-1
votes
3
answers
90
views
How to concatenate several file with * in the path using xargs or parallel
I have a command that fetch a path to a file for each line in a CSV file :
awk -F, 'NR>0 {print "/mnt/datagenetique/ANALYSIS/Infectiologie/COVID-WGS/Analyse/" $2 "_*/dragen-covidseq/&...
- 499
0
votes
1
answer
86
views
How to use the awk command to combine columns from one file to another matching by ID?
I have a file that looks like this:
FID IID Country Smoker Cancer_Type Age
1 RQ34365-4 1 2 1 70
2 RQ22067-0 1 3 1 58
3 RQ22101-7 1 1 1 61
4 RQ14754-1 2 3 1 70
And another file with 16 columns.
Id ...
- 1
-4
votes
3
answers
78
views
Filter file by row values
I need to select the rows containing a certain character in the 1st column and having the digits in the 1st column greater than a specific value.
Specifically, I need to select the rows containing a '...
- 239
0
votes
2
answers
96
views
Group based on multiple rows and columns
An example of the data is as follows. The first two columns are IDs and the third column is the frequency.
1 2 99
2 3 62
4 5 80
4 4 98
5 5 79
6 1 98
The first and second columns are the same person ...
- 25
0
votes
4
answers
75
views
How might I comment out with sed or awk a block of two lines (a unique line followed by an indented nonunique one)?
I wrote the following code:
#!/usr/bin/bash
wireless_state=$(ip l |grep "3: wlp" |sed -E 's/^.*state ([DOWNUP]{2,4}).*/\1/')
ethernet_state=$(ip l |grep "2: en" |sed -E 's/^.*...
- 319
0
votes
1
answer
57
views
How to merge two files based on a single key column and auto fix fixed columns and fill missing data
I have two sets of files which attempting to merge into a single file and filling in associated missing data.
Files are all , delimited
The first file contains 13 columns with the 8th column ...
- 65
-1
votes
1
answer
187
views
remove duplicate lines across multiple txt files [closed]
I have 12 text files all in one folder, each with about 5 million lines, each file has no duplicate line on its own but there are duplicated across multiple files, I want to remove the duplicate lines ...
-3
votes
2
answers
54
views
Change the number of values in column [closed]
i have a file like this exemple:
2931
7895
6688
7895
2931
2931
6688
and I want to generate a file that contain this column and another one like this:
2931 1
7895 2
6688 3
7895 2
2931 1
2931 1
6688 3
- 163
0
votes
7
answers
742
views
awk extract numbers from a string
There are multiple related Questions, seems they don't use awk to solve the problem.
Extracting positive/negative floating-point numbers from a string
How to extract the numbers from a filename
echo ...
- 411
-1
votes
4
answers
77
views
How to store and use two variable from a csv file for each line
I have a csv file A.csv (separator : ,):
sample,run
2071410189,RUN120
2071436273,RUN119
2071507844,RUN120
2071627356,RUN120
2071627370,RUN118
And I want to concatenate a file that each "sample&...
- 499
3
votes
2
answers
269
views
How do I add a line of text to the middle of a file using awk?
This is an great answer.
https://stackoverflow.com/a/6739327/15603477
But Still a little bit confused me.
without a variable, awk '/^nameserver/ { printf("nameserver 127.0.0.1\n")} {print}' ...
- 411
0
votes
2
answers
224
views
How to find/replace and increment a matched number only with awk?
https://stackoverflow.com/a/14348899/15603477
I do understand the following answer.
sed -r 's/(.*)(\?cache_version=)([0-9]+)(.*)/echo "\1\2$((\3+1))\4"/ge' file
But I wonder how to do it in ...
- 411
-1
votes
2
answers
68
views
Find unique line in two different files based on specific field
I need to compare two txt files containing several lines and create a third txt file with only lines that are unique to file 1.
An example of file 1 is the following:
../../A/folder/fname.gz | -12.36 |...
- 370
0
votes
1
answer
144
views
using awk to print line number with output lines
context: grep for multiple strings in file on different lines (ie. whole file, not line based search)?
egrep -nH 'Dansk|Norsk|Svenska' file3
or using ripgrep
rg 'Dansk|Norsk|Svenska' file3
But ...
- 411
6
votes
8
answers
497
views
Make change operations on substring only
In a file that has any garbled text before and after a section that is marked by patterns START and END (specific strings that occur only once each and in the correct order and on the same line). I ...
- 10.6k
-2
votes
1
answer
167
views
add header to a big csv file >5GB
i have a big csv file (5gb) with 300 columns, and i want to add a simple header via the commande line :
row-0;row1;row2 ... row299
i used sed (without a loop) :
sed -i 1i'row-0;row-1;row-2 .... ; row-...
- 75
0
votes
2
answers
81
views
sort by column not working as expected
cat marks.txt returns
1) Amit Physics 80
2) Rahul Maths 90
3) Shyam Biology 87
4) Kedar English 85
5) Hari History 89
awk -F" " '{ print $4"\t" $0 }' marks....
- 411
-3
votes
1
answer
129
views
tab separated value in awk
https://stackoverflow.com/questions/5374239/tab-separated-values-in-awk#comment23678233_5376527
copied the exact the code.
echo "$line"
will return
LOAD_SETTLED LOAD_INIT 2011-01-...
- 411
2
votes
2
answers
433
views
AWK printf syntax
I though that the syntax for printf statements is printf format, item1, item2, ... as described e.g. here
However, in this question printf is used like this: printf NR "%s ", $0 and it works!...
- 153
3
votes
2
answers
451
views
awk NR%2{}1 meaning
I am trying to understand this code:
awk 'NR%2{printf "%s ",$0;next;}1' yourFile
Now I try to customize it.
Given this error.txt content:
KEY 4048:1736 string
3
KEY 0:1772 string
1
KEY 4192:...
- 411