Frequent 'awk' Questions

913 votes

10 answers

1.8m views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...

terdon♦

215k

asked Feb 1, 2014 at 17:05

74 votes

5 answers

110k views

Pass shell variable as a /pattern/ to awk

Having the following in one of my shell functions: function _process () { awk -v l="$line" ' BEGIN {p=0} /'"$1"'/ {p=1} END{ if(p) print l >> "outfile.txt" } ' } , so when called as ...

shell awk quoting variable

branquito

977

asked Mar 21, 2014 at 15:03

12 votes

2 answers

4k views

Use a shell variable in awk

Here is my script (to find the files that contain a specified pattern): find . -type f \ -exec awk -v vawk="$1" '/'"$vawk"'/ {c++} c>0 { print ARGV[1]; exit 0 } END { if (! c) {exit 1}}' \{\} \...

bash awk variable

Nicolas

389

asked Oct 5, 2012 at 18:26

65 votes

4 answers

31k views

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++' <f q w e r $ The original code I have found on the internets read:...

text-processing awk

Alexander Shcheblikin

1,633

asked Oct 6, 2014 at 20:56

32 votes

10 answers

29k views

How to select first occurrence between two patterns including them

How can I select first occurrence between two patterns including them. Preferably using sed or awk. I have: text something P1 something content1 content2 something P2 something text something P1 ...

sed awk regular-expression

kofucii

423

asked Jan 23, 2015 at 15:59

40 votes

2 answers

81k views

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...

text-processing awk

Zach C

409

asked Dec 2, 2014 at 16:37

15 votes

2 answers

15k views

How to change a file in-place using awk? (as with "sed -i")

I have an awk script, new.awk: BEGIN { FS = OFS = "," } NR == 1 { for (i = 1; i <= NF; i++) f[$i] = i } NR > 1 { begSecs = mktime(gensub(/[":-]/, " ", &...

linux shell awk

mittu

157

asked Jan 23, 2019 at 10:44

24 votes

4 answers

10k views

Print lines between (and including) two patterns

I want to start grepping from the lines that has CK at the end of line and stop grepping when the line has D at the end. I tried grep "$CK" "$D" file..txt, but it didn't work. Input: kkkkkkkkkkk ...

text-processing sed awk grep

Rana Khan

831

asked Nov 8, 2013 at 17:28

28 votes

5 answers

71k views

Multiline pattern match using sed, awk or grep [duplicate]

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. {........

text-processing sed grep awk

user6106

asked Mar 28, 2011 at 11:15

28 votes

5 answers

49k views

How to grep for text in a file and display the paragraph that has the text?

Below is the text in the file: Pseudo name=Apple Code=42B state=fault Pseudo name=Prance Code=43B state=good I need to grep for "42B" and get the output from the above text like: Pseudo name=...

text-processing awk sed grep perl

Jaya William

381

asked Jul 14, 2013 at 0:24

53 votes

11 answers

95k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" line4-...

text-processing awk sed perl

user1228191

771

asked Feb 28, 2012 at 5:38

37 votes

8 answers

35k views

How to grep-inverse-match and exclude "before" and "after" lines

Consider a text file with the following entries: aaa bbb ccc ddd eee fff ggg hhh iii Given a pattern (e.g. fff), I would like to grep the file above to get in the output: all_lines except (...

text-processing awk sed grep

Amelio Vazquez-Reina

36.6k

asked Jul 1, 2015 at 19:22

10 votes

3 answers

4k views

How to print a line if that line or the next line do not contain a particular string

Input.txt: 8B0C remove 8B0D remove 8B0E remove 8B0F 8B10 remove 8B14 remove 8B15 remove 8B16 remove 8B17 remove 8AC0 8AC1 ...

text-processing sed awk solaris

ayrton_senna

1,001

asked Jul 16, 2015 at 19:04

36 votes

9 answers

110k views

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...

text-processing awk

LanceBaynes

36.2k

asked Jul 20, 2011 at 17:23

29 votes

10 answers

71k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...

text-processing sed awk csv

mtk

24.2k

asked Sep 20, 2012 at 8:21

9 votes

9 answers

4k views

How do I print all lines of a file with duplicate values in a certain column

Given a file like this: 1,768,12,46576457,7898 1,123,435,134,146 2,345,6756856,12312,1311 5,234,567465,12341,1341 1,3245,4356345,2442,13 9,423,2342,121,463 9,989,342,121,1212 I would like to list ...

text-processing awk csv

Bob

191

asked Jan 15, 2016 at 16:13

4 votes

2 answers

13k views

external variable in awk [duplicate]

Possible Duplicate: Use a script parameter in awk I have a script, in which a script snippet is x=3 awk '$2=$x{print $1}' infile the external variable is x, but it prompts an error in awk so ...

awk

wenzi

393

asked Nov 19, 2012 at 17:11

22 votes

14 answers

98k views

Rows to column conversion of file

Suppose I have a file: File1: PAPER TEAM MANISH NISHA GARIMA JYOUTI ........etc File2 I want: PAPER TEAM MANISH NISHA GARIMA JYOUTI Rows to column conversion of File1.

awk sed text-formatting

yisha

839

asked Nov 26, 2014 at 5:12

17 votes

8 answers

4k views

How to process an x-column text file to get a y-column one?

I have a text file: a aa aaa b bb bbb c cc ccc d dd ddd e ee eee f ff fff g gg ggg h hh hhh i ii iii j jj jjj How can I process it and get a 2 ...

text-processing awk

Just a learner

1,626

asked Sep 8, 2016 at 11:05

42 votes

15 answers

123k views

How to print certain columns by name?

I have the following file: id name age 1 ed 50 2 joe 70 I want to print just the id and age columns. Right now I just use awk: cat file.tsv | awk '{ print $1, $3 }' However, this ...

awk

Brett Thomas

523

asked Nov 22, 2011 at 16:40

33 votes

2 answers

80k views

Common lines between two files [duplicate]

I have the following code that I run on my Terminal. LC_ALL=C && grep -F -f genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed This doesn't give me the common lines between the two files. ...

text-processing awk diff uniq comm

Marwah Soliman

523

asked Oct 14, 2017 at 18:46

33 votes

3 answers

10k views

What is the meaning of '1' at the end of an awk script

I was reading this awk script awk -F"=" '{OFS="=";gsub(",",";",$2)}1' I want to know what is the function of 1 at the end of it.

awk

Mirage

1,447

asked Feb 6, 2013 at 3:18

9 votes

1 answer

2k views

How can I extract/change lines in a text file whose data are separated into fields?

How can I manipulate field-based data from the commandline? For example How can I print only lines whose Nth field is foo? How can I print only lines whose Nth field isn't foo? How can I print only ...

text-processing sed awk perl

terdon♦

215k

asked Jun 17, 2015 at 18:03

9 votes

4 answers

27k views

Print a string including single quotes and other special characters

How can I write this with an AIX script correctly? My requirement is to write this command in test.txt: clock=$(prtconf -s | awk '{print $4,$5}') I tried this command: print 'clock=$(prtconf -s | ...

shell awk command-line scripting quoting

batil

301

asked Jun 10, 2014 at 9:12

8 votes

3 answers

4k views

String replacement using a dictionary

What is a good way to do string replacements in a file using a dictionary with a lot of substituend-substituent pairs? And by a lot, I actually mean about 20 – not much, but many enough so that I want ...

text-processing sed awk gawk

k.stm

659

asked Mar 12, 2016 at 16:15

97 votes

5 answers

237k views

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string bar in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...

text-processing sed awk replace

jakub.g

2,853

asked Nov 19, 2012 at 16:41

34 votes

15 answers

99k views

How can I remove all comments from a file?

I have a file which includes comments: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I want to print the file without including any of the comments: foo bar stuff morestuff ...

shell-script text-processing awk sed grep

Questionmark

3,475

asked Sep 24, 2014 at 20:02

24 votes

7 answers

63k views

Repeat each line multiple times

Would like to have each line in a file repeated a fixed number of times. e.g. have each line repeated four times: a b c becomes: a a a a b b b b c c c c I've done some searching, and there are a ...

text-processing sed awk perl

Yimin Rong

853

asked Jul 5, 2013 at 15:48

25 votes

11 answers

23k views

Tool in UNIX to subtract dates

Is there any tool in Solaris UNIX (so no GNU tool available) to subtract dates? I know that in Linux we have gawk that can subtract one date from another. But in Solaris the maximum we have is nawk (...

shell solaris awk utilities date

jyz

657

asked Sep 9, 2010 at 19:31

14 votes

9 answers

3k views

Command line tool to "cat" pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...

shell shell-script text-processing awk

Tom Hayden

383

asked Nov 24, 2014 at 1:57

4 votes

4 answers

723 views

text processing rows to columns for a block

I have a file containing lists on Solaris: List A hi hello hw r u List B Hi Yes List C Hello I need to transpose the lists as shown below: List A List B List C hi Hi Hello ...

text-processing awk solaris csplit

John

51

asked Sep 7, 2017 at 10:44

60 votes

18 answers

38k views

Remove duplicate $PATH entries with awk command

I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my PATH environment variable. I was told that it is possible to achieve this with a one ...

bash shell awk path

Johnny Williem

733

asked Jun 14, 2012 at 4:22

102 votes

5 answers

388k views

Using awk to sum the values of a column, based on the values of another column

I am trying to sum certain numbers in a column using awk. I would like to sum just column 3 of the "smiths" to get a total of 212. I can sum the whole column using awk but not just the "smiths". I ...

awk

jake

1,165

asked Nov 14, 2015 at 4:56

26 votes

5 answers

90k views

How to extract logs between two time stamps

I want to extract all logs between two timestamps. Some lines may not have the timestamp, but I want those lines also. In short, I want every line that falls under two time stamps. My log structure ...

text-processing sed awk grep

Amit

261

asked Apr 9, 2014 at 20:03

18 votes

1 answer

33k views

Splitting a file into multiple files based on 1st column value [duplicate]

I have seen solutions of splitting a file with respect to pattern matching and line matching but what I want is the following. The scenario is, let's say I have a file file1 - A.B|100|20 A.B|101|...

text-processing awk

Zzrot

361

asked Jul 22, 2016 at 22:10

11 votes

4 answers

13k views

sum pair of columns based on matching fields

I have a large file in the following format: 2 1019 0 12 2 1019 3 0 2 1021 0 2 2 1021 2 0 2 1022 4 5 2 1030 0 1 2 1030 5 0 2 1031 4 4 If the values in column 2 match, I want to sum the values ...

text-processing awk sort uniq

TomPio

113

asked Mar 18, 2015 at 11:42

11 votes

13 answers

2k views

How to collapse consecutive numbers into ranges?

Given a sorted input file (or command output) that contains unique numbers, one per line, I would like to collapse all runs of consecutive numbers into ranges such that n n+1 ... n+m becomes n,n+m ...

text-processing awk perl numeric-data

don_crissti

73k

asked Sep 19, 2018 at 17:16

8 votes

3 answers

9k views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...

text-processing sed awk

turtle

2,417

asked Dec 9, 2014 at 21:23

11 votes

2 answers

35k views

How to read only the second last line of a file

I have a client that updates/writes into myfile.csv arbitrarily. I've written the following code, I left the MySQL snippet out: while true do awk_variables_value_array=`tail -n2 myfile.csv | awk -...

text-processing awk

3kstc

4,106

asked Mar 14, 2015 at 1:55

8 votes

7 answers

9k views

What is the easiest way to add a string on the beginning of every line of the file from the command line?

I am looking for a way to add some string to the beginning of every line (same string for every line). Not something customizable but rather something that will be easy to remember and available on ...

text-processing awk sed perl

syntagma

11.3k

asked Oct 8, 2013 at 18:58

110 votes

7 answers

69k views

Is there a basic tutorial for grep, awk and sed? [closed]

I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...

grep sed awk

Macha

3,640

asked Sep 25, 2010 at 18:04

84 votes

7 answers

238k views

How to read first and last line from cat output?

I have text file. Task - get first and last line from file after $ cat file | grep -E "1|2|3|4" | commandtoprint $ cat file 1 2 3 4 5 Need this without cat output (only 1 and 5). ~$ cat file | ...

bash shell sed awk grep

dmgl

973

asked Jun 25, 2014 at 10:38

24 votes

1 answer

10k views

When to use grep, less, awk, sed [closed]

I'm entering the world of Linux and at work I'm using grep more and more. By doing that I'm figuring out that sometimes it's not adequate for what I want. I was struggling with grep a few days ago ...

awk sed grep less

frankguthrie

371

asked Aug 12, 2016 at 15:05

26 votes

7 answers

86k views

Text between two tags

I want to retrieve whatever is between these two tags – <tr> </tr> – from an html doc. Now I don't have any specific html requirements that would warrant for an html parser. I just plain ...

shell-script text-processing sed awk html

TechJack

363

asked Feb 13, 2013 at 11:43

29 votes

3 answers

87k views

Count distinct values of a field in a file

I have a file with one million lines. Each line has a field called transactionid, which has repetitive values. What I need to do is to count them distinctly. No matter how many times a value is ...

text-processing awk

Olgun Kaya

527

asked Jan 11, 2012 at 14:08

32 votes

8 answers

98k views

Print odd-numbered lines, print even-numbered lines

I want to print the odd-numbered and even-numbered lines from files. I found this shell script which makes use of echo. #!/bin/bash # Write a shell script that, given a file name as the argument ...

text-processing sed awk

ixtmixilix

12.8k

asked Dec 13, 2011 at 11:36

11 votes

2 answers

46k views

Join two files with matching columns

File1.txt id No gi|371443199|gb|JH556661.1| 7907290 gi|371443198|gb|JH556662.1| 7573913 gi|371443197|gb|JH556663.1| 7384412 gi|371440577|gb|JH559283.1| ...

awk join

jack

3,683

asked Jul 18, 2012 at 20:27

31 votes

2 answers

22k views

Why does awk do full buffering when reading from a pipe

I'm reading from a serial port connected to a gps device sending nmea strings. A simplified invocation to illustrate my point: $ awk '{ print $0 }' /dev/ttyPSC9 GPGGA,073651.000,6310.1043,N,...

awk tty serial-port

Daniel Näslund

1,732

asked Mar 7, 2012 at 7:46

7 votes

5 answers

3k views

Identifying genes from a list of genes

I have a gene list file. Some thing like this SWT21 SSA1 NRP1 EFB1 TFC3 MDM10 I have another file which also contains the names of these genes in my list along with other ...

text-processing awk grep

user3138373

2,317

asked Mar 24, 2014 at 20:26

7 votes

5 answers

7k views

Display only the penultimate (second last) row of a text [duplicate]

I have a poem with an unknown number of rows and I want to display only the penultimate one. What command should I use?

text-processing awk sed tail ed

Elena

71

asked May 7, 2018 at 14:06

Questions tagged [awk]

Related Tags