Questions tagged [awk]

A pattern-directed scanning and processing language.

Filter by
Sorted by
Tagged with
-1 votes
2 answers
49 views

Replace empty spaces with TAB except in last column

I have a file that looks like this. How can I replace each empty space with a TAB except in the last column that starts with ID? chr1 13513 14763 Medtr1g004950 . + JCVI gene . ID=Medtr1g004950;Note=...
user avatar
0 votes
4 answers
140 views

awk loop over range of records write to separate files

I have a file with several thousand lines describing the output from an elastic wave mode code. The size of the file is variable depending on the frequency, the thickness of the model, and the number ...
user avatar
  • 1
2 votes
4 answers
161 views

Unable to exclude lines containing ^M from grep

I have the following file from which I want to extract only Removed '2022-01-30_01-00-05', at the end. Removing '2022-01-30_01-00-05'... 0.46% complete (00:03:45 remaining)^M 5.49% complete (00:00:...
user avatar
  • 253
4 votes
7 answers
322 views

awk: deal with newline separation in wrong place

I have some data that looks like this: abc 123 456 789 def 111 222 333 ghi 999 888 777 666 i.e. the records are separated by multiple newlines but in the wrong place. What I want is to get it like ...
user avatar
  • 145
-3 votes
3 answers
138 views

Any idea how to calculate median value of each part of column

I have a data like this: 111 5 111 6 111 1 222 8 222 9 222 1 222 3 555 9 555 7 555 6 and for each value of $1, I want to get the median value of all the values of $2 for that $1 if possible using AWK ...
user avatar
0 votes
1 answer
34 views

print byte from number in awk

I can print a byte from a string literal like: awk 'BEGIN {print "\001"}' | cat -v But I need to print a byte of the result of a bitwise OR. So how can I print a byte from a number? Gawk is ...
user avatar
  • 5
0 votes
0 answers
32 views

How to break a long line into max length by word and use that programatically in a git commit [duplicate]

I have a scenario where I need to convert a long string of text and break it by words not exceeding a set line length (70 characters). I need to replace the space with a '\n' where the line break ...
user avatar
0 votes
2 answers
179 views

Compare one column in one file, against another column and if they match, append column

I would like to compare one column from one file against another column in another file. If the columns match, I would like to append the first column from the first file, as the first column of the ...
user avatar
0 votes
1 answer
79 views

Reading multiple zipped files in awk script

I have an awk script named text_processing.awk . I need to pass multiple zipped files named 01JAN21.txt.gz , 02JAN21.txt.gz etc. to this script for further processing. The output file takes the date ...
user avatar
2 votes
1 answer
61 views

Merge two files using first column

I have two different files and I'd like to do a merge of their information using the first column. File1.txt A,info1,info2 234,info3,info4 CD,info5,info6 File2.txt 234,ccc,bb CD,aaa,dd Expected ...
user avatar
  • 161
2 votes
2 answers
122 views

Matching the first occurrence only of a string then adding N lines awk

Hi would like extract N number of lines after matching the first incidence of a string printing the N lines following using awk. The string is repeated a number of times in the file I am ...
user avatar
  • 25
1 vote
0 answers
15 views

How to import environment variable in awk? [duplicate]

I have the following code: export my_var=3 cat /etc/passwd | awk {'print $1'} Now I need to use my my_var in my awk pseudo code. How can I apply this?
user avatar
  • 21
1 vote
2 answers
85 views

AWK: Insert in random selected lines target terms after source terms from a dictionary

Note: I have already asked a similar question in AWK: Quick way to insert target words after a source term and I am at the beginner level of AWK. This question considers the insertion of multiple ...
user avatar
2 votes
1 answer
190 views

After AWK the number of lines has been increased

After running my AWK script awk -i inplace '(NR==FNR){a[$1];next} (FNR in a) && gsub(/\<Source Term\>/,"& Target Term") 1 ' <(shuf -n 198058 -i 1-$(wc -l &...
user avatar
0 votes
0 answers
28 views

I need to replace placeholders in a yaml file with the lines from another file [duplicate]

I have a text file looking like this Name: somename username:some alphanumeric string password:some alphanumeric string And a yaml file looking like this apiVersion: v1 kind: secret metadata: name: ...
user avatar
0 votes
1 answer
69 views

AWK: Quick way to insert target words after a source term

I'm not familiar with awk. For inserting a single target term after a source term in 198058 random lines I have this code here awk -i inplace '(NR==FNR){a[$1];next} (FNR in a) && gsub(/\&...
user avatar
0 votes
2 answers
89 views

Average of lines with different conditions at header and tail of files

Edit: edited it for clarity and made the example files minimal and more reproducible for easier help. Thanks! I have files that gave 1000+lines. Each file is formatted with the same amount of lines. ...
user avatar
0 votes
1 answer
67 views

How to group docker stats by container name prefix?

How to group docker stats --no-stream by container prefix, and show the sum of RAM and CPU and network and IO? If we consider the first two columns of the docker stats --no-stream command to be like ...
user avatar
1 vote
1 answer
46 views

AWK: Add missing lines in csv file and set specific column to "0"

(Note: this question is heavily shortened because the detailed question was blocked with "This looks like spam.") I'm having a hard time adding missing lines to a series of data in a ...
user avatar
1 vote
2 answers
274 views

Gunzip inside awk script

I am creating an output file inside an awk script which I want to compress then. Input file - Marks.txt Student1:AP:Maths:30:Science:43 Student2:AP:Maths:23:Science:35 Student3:Non_AP:Maths:17:Science:...
user avatar
1 vote
2 answers
143 views

How to programmatically maintain umask in /etc/profile and /etc/bashrc

I am looking for a way to programmatically maintain a consistent umask in the files /etc/profile and /etc/bashrc. They have an entry such as follows: if [ $UID -gt 199 ] && [ "`/usr/bin/...
user avatar
  • 11
4 votes
5 answers
278 views

Manipulating text using shell script: How can I fill in "missing" lines?

I have a list of data, like a CSV but some lines are missing a value. I'd like to generate a value for the missing line based on the lines before and after using linux shell script. Take this table ...
user avatar
0 votes
0 answers
27 views

How to join two files into one in multiple folders using specific columns and add file names where they originated from

I would like to ask, how can I join multiple files into one using a specific column and name it by the originating folder. Let me explain: I have the following files, having multiple columns and rows: ...
user avatar
  • 5
7 votes
1 answer
304 views

Is "warning: range of the form `[o-b]' is locale dependent" a bug in gawk 4.1.0?

While writing some awk code I was using the --lint option for gawk 4.1.0. To my surprise I get a warning warning: range of the form `[o-b]' is locale dependent but my code has only a foo-bar outside ...
user avatar
  • 801
0 votes
6 answers
93 views

linux change date format in file with full year

I am trying to change the date format in file with the full year. This is my data. 06/30/21 07/01/21 05436841182400056721972 random text 07/06/21 07/07/21 05436841188400057034635 random text 07/17/21 ...
user avatar
  • 981
0 votes
1 answer
99 views

Awk-Relational processing. Editing on one column must reflect on column on another line number specified by third column of actual line

relational column processing using awk sed File 1 ---> Scrambled data in column (Old stale files with intermediate test data) mL9A7hajHyuVIQr1HNP7ThYfj9yBUd Iq4iqnH4UftLgGUSobLeti0hkmdMn7 ...
user avatar
0 votes
1 answer
54 views

how get ssh awk remote values

im using this script [test@sys-master ~]$ parallel -k -j 100 sshpass -p test1213'!' ssh -oStrictHostKeyChecking=no -oCheckHostIP=no -q root@10.0.{} "cat /proc/loadavg | awk -F \" \" '{...
user avatar
0 votes
1 answer
148 views

Using grep or awk, how can I count the number of characters in every line but also ignore certain lines based on their first character?

Very simply, I have this kind of text in a file: #number 3222 #number 3311 I'm using: awk '{print length}' file.txt and it prints out: 7 4 7 4 But I just want it to print the length of the ...
user avatar
0 votes
1 answer
47 views

Pass variable to context match awk [duplicate]

I am trying to extract context from a file with awk from a start pattern until an end pattern. Here is simplified sample input to reproduce: $ cat file 1 2 3 4 5 6 When I pass the start pattern ...
user avatar
2 votes
6 answers
697 views

Using grep/awk to find/print certain value in a text file

Assume that i have a text file containing the following 5 lines: Tue 18 2022 car model: Toyota , car motor: 2001 , car color: blue , year of production: 2018 Thu 19 2022 car model: Mercedes , car ...
user avatar
-1 votes
3 answers
90 views

How to concatenate several file with * in the path using xargs or parallel

I have a command that fetch a path to a file for each line in a CSV file : awk -F, 'NR>0 {print "/mnt/datagenetique/ANALYSIS/Infectiologie/COVID-WGS/Analyse/" $2 "_*/dragen-covidseq/&...
user avatar
  • 499
0 votes
1 answer
86 views

How to use the awk command to combine columns from one file to another matching by ID?

I have a file that looks like this: FID IID Country Smoker Cancer_Type Age 1 RQ34365-4 1 2 1 70 2 RQ22067-0 1 3 1 58 3 RQ22101-7 1 1 1 61 4 RQ14754-1 2 3 1 70 And another file with 16 columns. Id ...
user avatar
  • 1
-4 votes
3 answers
78 views

Filter file by row values

I need to select the rows containing a certain character in the 1st column and having the digits in the 1st column greater than a specific value. Specifically, I need to select the rows containing a '...
user avatar
  • 239
0 votes
2 answers
96 views

Group based on multiple rows and columns

An example of the data is as follows. The first two columns are IDs and the third column is the frequency. 1 2 99 2 3 62 4 5 80 4 4 98 5 5 79 6 1 98 The first and second columns are the same person ...
user avatar
  • 25
0 votes
4 answers
75 views

How might I comment out with sed or awk a block of two lines (a unique line followed by an indented nonunique one)?

I wrote the following code: #!/usr/bin/bash wireless_state=$(ip l |grep "3: wlp" |sed -E 's/^.*state ([DOWNUP]{2,4}).*/\1/') ethernet_state=$(ip l |grep "2: en" |sed -E 's/^.*...
user avatar
0 votes
1 answer
57 views

How to merge two files based on a single key column and auto fix fixed columns and fill missing data

I have two sets of files which attempting to merge into a single file and filling in associated missing data. Files are all , delimited The first file contains 13 columns with the 8th column ...
user avatar
-1 votes
1 answer
187 views

remove duplicate lines across multiple txt files [closed]

I have 12 text files all in one folder, each with about 5 million lines, each file has no duplicate line on its own but there are duplicated across multiple files, I want to remove the duplicate lines ...
user avatar
-3 votes
2 answers
54 views

Change the number of values in column [closed]

i have a file like this exemple: 2931 7895 6688 7895 2931 2931 6688 and I want to generate a file that contain this column and another one like this: 2931 1 7895 2 6688 3 7895 2 2931 1 2931 1 6688 3
user avatar
0 votes
7 answers
742 views

awk extract numbers from a string

There are multiple related Questions, seems they don't use awk to solve the problem. Extracting positive/negative floating-point numbers from a string How to extract the numbers from a filename echo ...
user avatar
  • 411
-1 votes
4 answers
77 views

How to store and use two variable from a csv file for each line

I have a csv file A.csv (separator : ,): sample,run 2071410189,RUN120 2071436273,RUN119 2071507844,RUN120 2071627356,RUN120 2071627370,RUN118 And I want to concatenate a file that each "sample&...
user avatar
  • 499
3 votes
2 answers
269 views

How do I add a line of text to the middle of a file using awk?

This is an great answer. https://stackoverflow.com/a/6739327/15603477 But Still a little bit confused me. without a variable, awk '/^nameserver/ { printf("nameserver 127.0.0.1\n")} {print}' ...
user avatar
  • 411
0 votes
2 answers
224 views

How to find/replace and increment a matched number only with awk?

https://stackoverflow.com/a/14348899/15603477 I do understand the following answer. sed -r 's/(.*)(\?cache_version=)([0-9]+)(.*)/echo "\1\2$((\3+1))\4"/ge' file But I wonder how to do it in ...
user avatar
  • 411
-1 votes
2 answers
68 views

Find unique line in two different files based on specific field

I need to compare two txt files containing several lines and create a third txt file with only lines that are unique to file 1. An example of file 1 is the following: ../../A/folder/fname.gz | -12.36 |...
user avatar
  • 370
0 votes
1 answer
144 views

using awk to print line number with output lines

context: grep for multiple strings in file on different lines (ie. whole file, not line based search)? egrep -nH 'Dansk|Norsk|Svenska' file3 or using ripgrep rg 'Dansk|Norsk|Svenska' file3 But ...
user avatar
  • 411
6 votes
8 answers
497 views

Make change operations on substring only

In a file that has any garbled text before and after a section that is marked by patterns START and END (specific strings that occur only once each and in the correct order and on the same line). I ...
user avatar
  • 10.6k
-2 votes
1 answer
167 views

add header to a big csv file >5GB

i have a big csv file (5gb) with 300 columns, and i want to add a simple header via the commande line : row-0;row1;row2 ... row299 i used sed (without a loop) : sed -i 1i'row-0;row-1;row-2 .... ; row-...
user avatar
  • 75
0 votes
2 answers
81 views

sort by column not working as expected

cat marks.txt returns 1) Amit Physics 80 2) Rahul Maths 90 3) Shyam Biology 87 4) Kedar English 85 5) Hari History 89 awk -F" " '{ print $4"\t" $0 }' marks....
user avatar
  • 411
-3 votes
1 answer
129 views

tab separated value in awk

https://stackoverflow.com/questions/5374239/tab-separated-values-in-awk#comment23678233_5376527 copied the exact the code. echo "$line" will return LOAD_SETTLED LOAD_INIT 2011-01-...
user avatar
  • 411
2 votes
2 answers
433 views

AWK printf syntax

I though that the syntax for printf statements is printf format, item1, item2, ... as described e.g. here However, in this question printf is used like this: printf NR "%s ", $0 and it works!...
user avatar
  • 153
3 votes
2 answers
451 views

awk NR%2{}1 meaning

I am trying to understand this code: awk 'NR%2{printf "%s ",$0;next;}1' yourFile Now I try to customize it. Given this error.txt content: KEY 4048:1736 string 3 KEY 0:1772 string 1 KEY 4192:...
user avatar
  • 411