Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [awk]

A pattern-directed scanning and processing language.

Filter by
Sorted by
Tagged with
913 votes
10 answers
1.8m views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
user avatar
  • 215k
74 votes
5 answers
110k views

Pass shell variable as a /pattern/ to awk

Having the following in one of my shell functions: function _process () { awk -v l="$line" ' BEGIN {p=0} /'"$1"'/ {p=1} END{ if(p) print l >> "outfile.txt" } ' } , so when called as ...
user avatar
  • 977
12 votes
2 answers
4k views

Use a shell variable in awk

Here is my script (to find the files that contain a specified pattern): find . -type f \ -exec awk -v vawk="$1" '/'"$vawk"'/ {c++} c>0 { print ARGV[1]; exit 0 } END { if (! c) {exit 1}}' \{\} \...
user avatar
  • 389
65 votes
4 answers
31k views

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++' <f q w e r $ The original code I have found on the internets read:...
user avatar
32 votes
10 answers
29k views

How to select first occurrence between two patterns including them

How can I select first occurrence between two patterns including them. Preferably using sed or awk. I have: text something P1 something content1 content2 something P2 something text something P1 ...
user avatar
  • 423
40 votes
2 answers
81k views

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
user avatar
  • 409
15 votes
2 answers
15k views

How to change a file in-place using awk? (as with "sed -i")

I have an awk script, new.awk: BEGIN { FS = OFS = "," } NR == 1 { for (i = 1; i <= NF; i++) f[$i] = i } NR > 1 { begSecs = mktime(gensub(/[":-]/, " ", &...
user avatar
  • 157
24 votes
4 answers
10k views

Print lines between (and including) two patterns

I want to start grepping from the lines that has CK at the end of line and stop grepping when the line has D at the end. I tried grep "$CK" "$D" file..txt, but it didn't work. Input: kkkkkkkkkkk ...
user avatar
  • 831
28 votes
5 answers
71k views

Multiline pattern match using sed, awk or grep [duplicate]

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. {........
user avatar
28 votes
5 answers
49k views

How to grep for text in a file and display the paragraph that has the text?

Below is the text in the file: Pseudo name=Apple Code=42B state=fault Pseudo name=Prance Code=43B state=good I need to grep for "42B" and get the output from the above text like: Pseudo name=...
user avatar
53 votes
11 answers
95k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" line4-...
user avatar
37 votes
8 answers
35k views

How to grep-inverse-match and exclude "before" and "after" lines

Consider a text file with the following entries: aaa bbb ccc ddd eee fff ggg hhh iii Given a pattern (e.g. fff), I would like to grep the file above to get in the output: all_lines except (...
user avatar
10 votes
3 answers
4k views

How to print a line if that line or the next line do not contain a particular string

Input.txt: 8B0C remove 8B0D remove 8B0E remove 8B0F 8B10 remove 8B14 remove 8B15 remove 8B16 remove 8B17 remove 8AC0 8AC1 ...
user avatar
  • 1,001
36 votes
9 answers
110k views

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...
user avatar
  • 36.2k
29 votes
10 answers
71k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
user avatar
  • 24.2k
9 votes
9 answers
4k views

How do I print all lines of a file with duplicate values in a certain column

Given a file like this: 1,768,12,46576457,7898 1,123,435,134,146 2,345,6756856,12312,1311 5,234,567465,12341,1341 1,3245,4356345,2442,13 9,423,2342,121,463 9,989,342,121,1212 I would like to list ...
user avatar
  • 191
4 votes
2 answers
13k views

external variable in awk [duplicate]

Possible Duplicate: Use a script parameter in awk I have a script, in which a script snippet is x=3 awk '$2=$x{print $1}' infile the external variable is x, but it prompts an error in awk so ...
user avatar
  • 393
22 votes
14 answers
98k views

Rows to column conversion of file

Suppose I have a file: File1: PAPER TEAM MANISH NISHA GARIMA JYOUTI ........etc File2 I want: PAPER TEAM MANISH NISHA GARIMA JYOUTI Rows to column conversion of File1.
user avatar
  • 839
17 votes
8 answers
4k views

How to process an x-column text file to get a y-column one?

I have a text file: a aa aaa b bb bbb c cc ccc d dd ddd e ee eee f ff fff g gg ggg h hh hhh i ii iii j jj jjj How can I process it and get a 2 ...
user avatar
42 votes
15 answers
123k views

How to print certain columns by name?

I have the following file: id name age 1 ed 50 2 joe 70 I want to print just the id and age columns. Right now I just use awk: cat file.tsv | awk '{ print $1, $3 }' However, this ...
user avatar
33 votes
2 answers
80k views

Common lines between two files [duplicate]

I have the following code that I run on my Terminal. LC_ALL=C && grep -F -f genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed This doesn't give me the common lines between the two files. ...
user avatar
33 votes
3 answers
10k views

What is the meaning of '1' at the end of an awk script

I was reading this awk script awk -F"=" '{OFS="=";gsub(",",";",$2)}1' I want to know what is the function of 1 at the end of it.
user avatar
  • 1,447
9 votes
1 answer
2k views

How can I extract/change lines in a text file whose data are separated into fields?

How can I manipulate field-based data from the commandline? For example How can I print only lines whose Nth field is foo? How can I print only lines whose Nth field isn't foo? How can I print only ...
user avatar
  • 215k
9 votes
4 answers
27k views

Print a string including single quotes and other special characters

How can I write this with an AIX script correctly? My requirement is to write this command in test.txt: clock=$(prtconf -s | awk '{print $4,$5}') I tried this command: print 'clock=$(prtconf -s | ...
user avatar
  • 301
8 votes
3 answers
4k views

String replacement using a dictionary

What is a good way to do string replacements in a file using a dictionary with a lot of substituend-substituent pairs? And by a lot, I actually mean about 20 – not much, but many enough so that I want ...
user avatar
  • 659
97 votes
5 answers
237k views

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string bar in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
user avatar
  • 2,853
34 votes
15 answers
99k views

How can I remove all comments from a file?

I have a file which includes comments: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I want to print the file without including any of the comments: foo bar stuff morestuff ...
user avatar
  • 3,475
24 votes
7 answers
63k views

Repeat each line multiple times

Would like to have each line in a file repeated a fixed number of times. e.g. have each line repeated four times: a b c becomes: a a a a b b b b c c c c I've done some searching, and there are a ...
user avatar
25 votes
11 answers
23k views

Tool in UNIX to subtract dates

Is there any tool in Solaris UNIX (so no GNU tool available) to subtract dates? I know that in Linux we have gawk that can subtract one date from another. But in Solaris the maximum we have is nawk (...
user avatar
  • 657
14 votes
9 answers
3k views

Command line tool to "cat" pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
user avatar
4 votes
4 answers
723 views

text processing rows to columns for a block

I have a file containing lists on Solaris: List A hi hello hw r u List B Hi Yes List C Hello I need to transpose the lists as shown below: List A List B List C hi Hi Hello ...
user avatar
  • 51
60 votes
18 answers
38k views

Remove duplicate $PATH entries with awk command

I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my PATH environment variable. I was told that it is possible to achieve this with a one ...
user avatar
102 votes
5 answers
388k views

Using awk to sum the values of a column, based on the values of another column

I am trying to sum certain numbers in a column using awk. I would like to sum just column 3 of the "smiths" to get a total of 212. I can sum the whole column using awk but not just the "smiths". I ...
user avatar
  • 1,165
26 votes
5 answers
90k views

How to extract logs between two time stamps

I want to extract all logs between two timestamps. Some lines may not have the timestamp, but I want those lines also. In short, I want every line that falls under two time stamps. My log structure ...
user avatar
  • 261
18 votes
1 answer
33k views

Splitting a file into multiple files based on 1st column value [duplicate]

I have seen solutions of splitting a file with respect to pattern matching and line matching but what I want is the following. The scenario is, let's say I have a file file1 - A.B|100|20 A.B|101|...
user avatar
  • 361
11 votes
4 answers
13k views

sum pair of columns based on matching fields

I have a large file in the following format: 2 1019 0 12 2 1019 3 0 2 1021 0 2 2 1021 2 0 2 1022 4 5 2 1030 0 1 2 1030 5 0 2 1031 4 4 If the values in column 2 match, I want to sum the values ...
user avatar
  • 113
11 votes
13 answers
2k views

How to collapse consecutive numbers into ranges?

Given a sorted input file (or command output) that contains unique numbers, one per line, I would like to collapse all runs of consecutive numbers into ranges such that n n+1 ... n+m becomes n,n+m ...
user avatar
8 votes
3 answers
9k views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
user avatar
  • 2,417
11 votes
2 answers
35k views

How to read only the second last line of a file

I have a client that updates/writes into myfile.csv arbitrarily. I've written the following code, I left the MySQL snippet out: while true do awk_variables_value_array=`tail -n2 myfile.csv | awk -...
user avatar
  • 4,106
8 votes
7 answers
9k views

What is the easiest way to add a string on the beginning of every line of the file from the command line?

I am looking for a way to add some string to the beginning of every line (same string for every line). Not something customizable but rather something that will be easy to remember and available on ...
user avatar
  • 11.3k
110 votes
7 answers
69k views

Is there a basic tutorial for grep, awk and sed? [closed]

I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...
user avatar
  • 3,640
84 votes
7 answers
238k views

How to read first and last line from cat output?

I have text file. Task - get first and last line from file after $ cat file | grep -E "1|2|3|4" | commandtoprint $ cat file 1 2 3 4 5 Need this without cat output (only 1 and 5). ~$ cat file | ...
user avatar
  • 973
24 votes
1 answer
10k views

When to use grep, less, awk, sed [closed]

I'm entering the world of Linux and at work I'm using grep more and more. By doing that I'm figuring out that sometimes it's not adequate for what I want. I was struggling with grep a few days ago ...
user avatar
26 votes
7 answers
86k views

Text between two tags

I want to retrieve whatever is between these two tags – <tr> </tr> – from an html doc. Now I don't have any specific html requirements that would warrant for an html parser. I just plain ...
user avatar
  • 363
29 votes
3 answers
87k views

Count distinct values of a field in a file

I have a file with one million lines. Each line has a field called transactionid, which has repetitive values. What I need to do is to count them distinctly. No matter how many times a value is ...
user avatar
32 votes
8 answers
98k views

Print odd-numbered lines, print even-numbered lines

I want to print the odd-numbered and even-numbered lines from files. I found this shell script which makes use of echo. #!/bin/bash # Write a shell script that, given a file name as the argument ...
user avatar
  • 12.8k
11 votes
2 answers
46k views

Join two files with matching columns

File1.txt id No gi|371443199|gb|JH556661.1| 7907290 gi|371443198|gb|JH556662.1| 7573913 gi|371443197|gb|JH556663.1| 7384412 gi|371440577|gb|JH559283.1| ...
user avatar
  • 3,683
31 votes
2 answers
22k views

Why does awk do full buffering when reading from a pipe

I'm reading from a serial port connected to a gps device sending nmea strings. A simplified invocation to illustrate my point: $ awk '{ print $0 }' /dev/ttyPSC9 GPGGA,073651.000,6310.1043,N,...
user avatar
7 votes
5 answers
3k views

Identifying genes from a list of genes

I have a gene list file. Some thing like this SWT21 SSA1 NRP1 EFB1 TFC3 MDM10 I have another file which also contains the names of these genes in my list along with other ...
user avatar
  • 2,317
7 votes
5 answers
7k views

Display only the penultimate (second last) row of a text [duplicate]

I have a poem with an unknown number of rows and I want to display only the penultimate one. What command should I use?
user avatar
  • 71

1
2 3 4 5
16