Questions tagged [awk]
A pattern-directed scanning and processing language.
838
questions
967
votes
10
answers
1.8m
views
How can I replace a string in a file(s)?
Replacing strings in files based on certain search criteria is a very common task. How can I
replace string foo with bar in all files in the current directory?
do the same recursively for sub ...
- 246k
78
votes
5
answers
119k
views
Pass shell variable as a /pattern/ to awk
Having the following in one of my shell functions:
function _process () {
awk -v l="$line" '
BEGIN {p=0}
/'"$1"'/ {p=1}
END{ if(p) print l >> "outfile.txt" }
'
}
, so when called as ...
- 1,027
14
votes
2
answers
5k
views
Use a shell variable in awk
Here is my script (to find the files that contain a specified pattern):
find . -type f \
-exec awk -v vawk="$1" '/'"$vawk"'/ {c++} c>0 { print ARGV[1]; exit 0 } END { if (! c) {exit 1}}' \{\} \...
- 419
74
votes
4
answers
39k
views
How does awk '!a[$0]++' work?
This one-liner removes duplicate lines from text input without pre-sorting.
For example:
$ cat >f
q
w
e
w
r
$ awk '!a[$0]++' <f
q
w
e
r
$
The original code I have found on the internets read:...
- 1,828
31
votes
2
answers
42k
views
How to change a file in-place using awk? (as with "sed -i")
I have an awk script, new.awk:
BEGIN { FS = OFS = "," }
NR == 1 {
for (i = 1; i <= NF; i++)
f[$i] = i
}
NR > 1 {
begSecs = mktime(gensub(/[":-]/, " ", &...
- 427
33
votes
10
answers
35k
views
How to select first occurrence between two patterns including them
How can I select first occurrence between two patterns including them. Preferably using sed or awk.
I have:
text
something P1 something
content1
content2
something P2 something
text
something P1 ...
- 433
46
votes
2
answers
93k
views
Remove lines based on duplicates within one column without sort
I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
- 469
32
votes
5
answers
56k
views
How to grep for text in a file and display the paragraph that has the text?
Below is the text in the file:
Pseudo name=Apple
Code=42B
state=fault
Pseudo name=Prance
Code=43B
state=good
I need to grep for "42B" and get the output from the above text like:
Pseudo name=...
- 421
28
votes
4
answers
13k
views
Print lines between (and including) two patterns
I want to start grepping from the lines that has CK at the end of line and stop grepping when the line has D at the end. I tried grep "$CK" "$D" file..txt, but it didn't work.
Input:
kkkkkkkkkkk
...
- 931
26
votes
4
answers
2k
views
How to safely use gawk's -i option or @include directive?
Using gawk -i inplace some-awk-code some-file (or @include "inplace" within an awk script) to edit a file in-place (or any other extension) is a security vulnerability.
Why?
How do I work ...
- 559k
61
votes
11
answers
114k
views
How to insert the content of a file into another file before a pattern (marker)?
File1 contents:
line1-file1 "1"
line2-file1 "2"
line3-file1 "3"
line4-file1 "4"
File2 contents:
line1-file2 "25"
line2-file2 "24"
Pointer-file2 "23"
line4-...
- 871
33
votes
5
answers
85k
views
Multiline pattern match using sed, awk or grep [duplicate]
Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and }
So it should be able to match
1. {}
2. {.....}
3. {........
user6106
34
votes
12
answers
79k
views
Only remove commas embedded within quotes in a comma delimited file
I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row
123,"ABC, DEV 23",345,534.202,NAME
I need to ...
- 27.9k
41
votes
8
answers
45k
views
How to grep-inverse-match and exclude "before" and "after" lines
Consider a text file with the following entries:
aaa
bbb
ccc
ddd
eee
fff
ggg
hhh
iii
Given a pattern (e.g. fff), I would like to grep the file above to get in the output:
all_lines except (...
- 41.9k
10
votes
3
answers
5k
views
How to print a line if that line or the next line do not contain a particular string
Input.txt:
8B0C
remove
8B0D
remove
8B0E
remove
8B0F
8B10
remove
8B14
remove
8B15
remove
8B16
remove
8B17
remove
8AC0
8AC1
...
- 1,091
44
votes
9
answers
130k
views
How to print only last column?
echo -e 'one two three\nfour five six\nseven eight nine'
one two three
four five six
seven eight nine
how can I do some "MAGIC" do get this output?:
three
six
nine
UPDATE:
I don't need it in this ...
- 40.7k
11
votes
9
answers
5k
views
How do I print all lines of a file with duplicate values in a certain column
Given a file like this:
1,768,12,46576457,7898
1,123,435,134,146
2,345,6756856,12312,1311
5,234,567465,12341,1341
1,3245,4356345,2442,13
9,423,2342,121,463
9,989,342,121,1212
I would like to list ...
- 211
4
votes
2
answers
14k
views
external variable in awk [duplicate]
Possible Duplicate:
Use a script parameter in awk
I have a script, in which a script snippet is
x=3
awk '$2=$x{print $1}' infile
the external variable is x,
but it prompts an error in awk
so ...
- 413
51
votes
2
answers
114k
views
Common lines between two files [duplicate]
I have the following code that I run on my Terminal.
LC_ALL=C && grep -F -f genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed
This doesn't give me the common lines between the two files. ...
- 703
22
votes
14
answers
113k
views
Rows to column conversion of file
Suppose I have a file:
File1:
PAPER TEAM MANISH NISHA GARIMA JYOUTI ........etc
File2 I want:
PAPER
TEAM
MANISH
NISHA
GARIMA
JYOUTI
Rows to column conversion of File1.
- 909
47
votes
16
answers
137k
views
How to print certain columns by name?
I have the following file:
id name age
1 ed 50
2 joe 70
I want to print just the id and age columns. Right now I just use awk:
cat file.tsv | awk '{ print $1, $3 }'
However, this ...
- 573
40
votes
3
answers
16k
views
What is the meaning of '1' at the end of an awk script
I was reading this awk script
awk -F"=" '{OFS="=";gsub(",",";",$2)}1'
I want to know what is the function of 1 at the end of it.
- 1,557
14
votes
10
answers
4k
views
Command line tool to "cat" pairwise expansion of all rows in a file
Suppose I have a file (call it sample.txt) that looks like this:
Row1,10
Row2,20
Row3,30
Row4,40
I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
- 383
10
votes
4
answers
32k
views
Print a string including single quotes and other special characters
How can I write this with an AIX script correctly? My requirement is to write this command in test.txt:
clock=$(prtconf -s | awk '{print $4,$5}')
I tried this command:
print 'clock=$(prtconf -s | ...
- 311
9
votes
1
answer
2k
views
How can I extract/change lines in a text file whose data are separated into fields?
How can I manipulate field-based data from the commandline? For example
How can I print only lines whose Nth field is foo?
How can I print only lines whose Nth field isn't foo?
How can I print only ...
- 246k
8
votes
3
answers
5k
views
String replacement using a dictionary
What is a good way to do string replacements in a file using a dictionary with a lot of substituend-substituent pairs? And by a lot, I actually mean about 20 – not much, but many enough so that I want ...
- 719
102
votes
5
answers
249k
views
Remove line containing certain string and the following line
I use this
cat foo.txt | sed '/bar/d'
to remove lines containing the string bar in the file.
I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
- 3,333
43
votes
16
answers
124k
views
How can I remove all comments from a file?
I have a file which includes comments:
foo
bar
stuff
#Do not show this...
morestuff
evenmorestuff#Or this
I want to print the file without including any of the comments:
foo
bar
stuff
morestuff
...
- 3,995
32
votes
6
answers
101k
views
Extract file name from path in awk program
I have an awk script and I have passed a CSV file to it.
awk -f script.awk /home/abc/imp/asgd.csv
What am I doing is to get FILENAME within script.awk. FILENAME gives me the whole path. As I am in ...
- 781
26
votes
11
answers
26k
views
Tool in UNIX to subtract dates
Is there any tool in Solaris UNIX (so no GNU tool available) to subtract dates? I know that in Linux we have gawk that can subtract one date from another. But in Solaris the maximum we have is nawk (...
- 677
25
votes
7
answers
69k
views
Repeat each line multiple times
Would like to have each line in a file repeated a fixed number of times.
e.g. have each line repeated four times:
a
b
c
becomes:
a
a
a
a
b
b
b
b
c
c
c
c
I've done some searching, and there are a ...
- 933
4
votes
4
answers
885
views
text processing rows to columns for a block
I have a file containing lists on Solaris:
List A
hi
hello
hw r u
List B
Hi
Yes
List C
Hello
I need to transpose the lists as shown below:
List A List B List C
hi Hi Hello
...
- 51
108
votes
5
answers
418k
views
Using awk to sum the values of a column, based on the values of another column
I am trying to sum certain numbers in a column using awk. I would like to sum just column 3 of the "smiths" to get a total of 212. I can sum the whole column using awk but not just the "smiths". I ...
- 1,225
63
votes
20
answers
42k
views
Remove duplicate $PATH entries with awk command
I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my PATH environment variable.
I was told that it is possible to achieve this with a one ...
- 773
21
votes
1
answer
1k
views
Slurp-mode in awk?
Tools like sed, awk or perl -n process their input one record at a time, records being lines by default.
Some, like awk with RS, GNU sed with -z or perl with -0ooo can change the type of record by ...
- 559k
21
votes
1
answer
41k
views
Splitting a file into multiple files based on 1st column value [duplicate]
I have seen solutions of splitting a file with respect to pattern matching and line matching but what I want is the following.
The scenario is, let's say I have a file file1 -
A.B|100|20
A.B|101|...
- 391
13
votes
2
answers
42k
views
How to read only the second last line of a file
I have a client that updates/writes into myfile.csv arbitrarily.
I've written the following code, I left the MySQL snippet out:
while true
do
awk_variables_value_array=`tail -n2 myfile.csv | awk -...
- 4,826
11
votes
13
answers
3k
views
How to collapse consecutive numbers into ranges?
Given a sorted input file (or command output) that contains unique numbers, one per line, I would like to collapse all runs of consecutive numbers into ranges such that
n
n+1
...
n+m
becomes
n,n+m
...
- 83.9k
11
votes
4
answers
13k
views
sum pair of columns based on matching fields
I have a large file in the following format:
2 1019 0 12
2 1019 3 0
2 1021 0 2
2 1021 2 0
2 1022 4 5
2 1030 0 1
2 1030 5 0
2 1031 4 4
If the values in column 2 match, I want to sum the values ...
- 113
9
votes
7
answers
10k
views
What is the easiest way to add a string on the beginning of every line of the file from the command line?
I am looking for a way to add some string to the beginning of every line (same string for every line).
Not something customizable but rather something that will be easy to remember and available on ...
- 12.5k
9
votes
3
answers
10k
views
Insert text from file inline after matching pattern in another file
I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
- 2,717
8
votes
6
answers
12k
views
Replace strings in a file based on a list of strings and a list of corresponding replacements
I am trying to replace strings in a file A:
Hello Peter, how is your dad? where is mom?
where the strings to be replaced are in file B:
Peter
dad
mom
and their corresponding replacements are in ...
- 83
6
votes
5
answers
13k
views
How do I append/prepend a timestamp to grep output?
I am running a small linux server at home, and I am writing a script to log the temperature of the CPU cores every 5 seconds, but I need timestamps for it to be useful. So far I have something that ...
- 75
112
votes
7
answers
73k
views
Is there a basic tutorial for grep, awk and sed? [closed]
I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...
- 3,810
95
votes
7
answers
256k
views
How to read first and last line from cat output?
I have text file. Task - get first and last line from file after
$ cat file | grep -E "1|2|3|4" | commandtoprint
$ cat file
1
2
3
4
5
Need this without cat output (only 1 and 5).
~$ cat file | ...
- 1,083
40
votes
8
answers
121k
views
Print odd-numbered lines, print even-numbered lines
I want to print the odd-numbered and even-numbered lines from files.
I found this shell script which makes use of echo.
#!/bin/bash
# Write a shell script that, given a file name as the argument ...
- 13.4k
33
votes
3
answers
27k
views
Why does awk do full buffering when reading from a pipe [duplicate]
I'm reading from a serial port connected to a gps device sending nmea strings.
A simplified invocation to illustrate my point:
$ awk '{ print $0 }' /dev/ttyPSC9
GPGGA,073651.000,6310.1043,N,01436....
- 1,872
30
votes
3
answers
112k
views
Count distinct values of a field in a file
I have a file with one million lines. Each line has a field called transactionid, which has repetitive values. What I need to do is to count them distinctly. No matter how many times a value is ...
- 537
29
votes
7
answers
97k
views
How to extract logs between two time stamps
I want to extract all logs between two timestamps. Some lines may not have the timestamp, but I want those lines also. In short, I want every line that falls under two time stamps. My log structure ...
- 291
27
votes
3
answers
2k
views
Why does awk stop and wait if the filename contains = and how to work around that?
awk 'processing_script_here' my=file.txt
seems to stop and wait indefinitely...
What's going on here and how do I make it work ?
- 83.9k