Questions tagged [awk]
A pattern-directed scanning and processing language.
8,214
questions
947
votes
10
answers
1.8m
views
How can I replace a string in a file(s)?
Replacing strings in files based on certain search criteria is a very common task. How can I
replace string foo with bar in all files in the current directory?
do the same recursively for sub ...
- 235k
122
votes
13
answers
369k
views
Show sum of file sizes in directory listing
The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current directory,...
- 2,314
112
votes
7
answers
71k
views
Is there a basic tutorial for grep, awk and sed? [closed]
I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...
- 3,770
107
votes
5
answers
403k
views
Using awk to sum the values of a column, based on the values of another column
I am trying to sum certain numbers in a column using awk. I would like to sum just column 3 of the "smiths" to get a total of 212. I can sum the whole column using awk but not just the "smiths". I ...
- 1,215
105
votes
6
answers
449k
views
How to use a shell command to only show the first column and last column in a text file?
I need some help to figure out how to use the sed command to only show the first column and last column in a text file. Here is what I have so far for column 1:
cat logfile | sed 's/\|/ /'|awk '{...
- 1,289
101
votes
5
answers
246k
views
Remove line containing certain string and the following line
I use this
cat foo.txt | sed '/bar/d'
to remove lines containing the string bar in the file.
I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
- 3,153
99
votes
8
answers
209k
views
How do you list number of lines of every file in a directory in human readable format.
I have a list of directories and subdirectories that contain large csv files. There are about 500 million lines in these files, each is a record. I would like to know
How many lines are in each ...
- 1,235
95
votes
7
answers
251k
views
How to read first and last line from cat output?
I have text file. Task - get first and last line from file after
$ cat file | grep -E "1|2|3|4" | commandtoprint
$ cat file
1
2
3
4
5
Need this without cat output (only 1 and 5).
~$ cat file | ...
- 1,083
89
votes
13
answers
513k
views
How to parse JSON with shell scripting in Linux?
I have a JSON output from which I need to extract a few parameters in Linux.
This is the JSON output:
{
"OwnerId": "121456789127",
"ReservationId": "r-48465168",
"Groups": [],...
- 991
86
votes
2
answers
231k
views
Replacing string based on line number
I have a situation where i want to replace a particular string in many files
Replace a string AAA with another string BBB but there are lot of strings starting with AAA or ending in AAA ,and i want ...
- 961
77
votes
5
answers
117k
views
Pass shell variable as a /pattern/ to awk
Having the following in one of my shell functions:
function _process () {
awk -v l="$line" '
BEGIN {p=0}
/'"$1"'/ {p=1}
END{ if(p) print l >> "outfile.txt" }
'
}
, so when called as ...
- 1,017
74
votes
4
answers
36k
views
How does awk '!a[$0]++' work?
This one-liner removes duplicate lines from text input without pre-sorting.
For example:
$ cat >f
q
w
e
w
r
$ awk '!a[$0]++' <f
q
w
e
r
$
The original code I have found on the internets read:...
- 1,798
73
votes
1
answer
52k
views
Difference between gawk vs. awk
Trying to understand the differences between the two functions gawk vs. awk? When would one use gawk vs awk? Or are they the same in terms of usage?
Also, could one provide an example?
- 8,303
72
votes
5
answers
134k
views
Skip the first 6 lines/rows in a text file with awk
How can I skip the first 6 lines/rows in a text file (input.txt) and process the rest with awk? The format of my awk script (program.awk) is:
BEGIN {
}
{
process here
}
END {
}
My text file is ...
- 833
71
votes
4
answers
88k
views
How to add line numbers in every line using shell command?
My file,
PSS-A (Primary A)
PSS-B (Primary B)
PSS-C (Primary C)
PSS-D (Primary D)
PSS-E (Primary E)
PSS-F (Primary F)
PSS-G (Primary G)
PSS-H (Primary H)
PSS-I (Primary I)
SPARE (SPARE)
...
- 2,732
68
votes
4
answers
99k
views
How to view all the content in an awk array?
In my understanding, awk array is something like python dict.
So I write down the code bellow to explore it:
awk '{my_dict[$1] = $2} END { print my_dict}' zen
And I got: awk: can't read value of ...
- 7,297
68
votes
12
answers
74k
views
How to print the longest line in a file?
I'm looking for the simplest method to print the longest line in a file. I did some googling and surprisingly couldn't seem to find an answer. I frequently print the length of the longest line in a ...
- 1,749
66
votes
14
answers
59k
views
Add thousands separator in a number
In python
re.sub(r"(?<=.)(?=(?:...)+$)", ",", stroke )
To split a number by triplets, e.g.:
echo 123456789 | python -c 'import sys;import re; print re.sub(r"(?<=.)(?=(?:...)+$)", ",", sys....
- 1,151
65
votes
4
answers
50k
views
Removing leading zeros from date output
I made an alias of the date command to display date in the following format:
2013.06.14.12.10.02
using this command:
alias date = date +"%Y.%m.%d.%H.%M.%S"
Everything works great, except I want ...
- 1,978
62
votes
19
answers
41k
views
Remove duplicate $PATH entries with awk command
I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my PATH environment variable.
I was told that it is possible to achieve this with a one ...
- 763
59
votes
3
answers
176k
views
How to grep lines between start and end pattern? [duplicate]
I have a file which is having following content:
zdk
aaa
b12
cdn
dke
kdn
Input1: aaa and cdn
Output 1:
aaa
b12
cdn
Input 2: zdk and dke
Output 2:
zdk
aaa
b12
cdn
dke
I could use below commands ...
- 989
57
votes
11
answers
109k
views
How to insert the content of a file into another file before a pattern (marker)?
File1 contents:
line1-file1 "1"
line2-file1 "2"
line3-file1 "3"
line4-file1 "4"
File2 contents:
line1-file2 "25"
line2-file2 "24"
Pointer-file2 "23"
line4-...
- 831
55
votes
5
answers
277k
views
How to run grep on a single column?
I want to grep the output of my ls -l command:
-rw-r--r-- 1 root root 1866 Feb 14 07:47 rahmu.file
-rw-r--r-- 1 rahmu user 95653 Feb 14 07:47 foo.file
-rw-r--r-- 1 rahmu user ...
- 19.7k
51
votes
1
answer
110k
views
Remove last character from string captured with awk [duplicate]
I need to remove the last character from a string in this command:
sudo docker stats --no-stream 39858jf8 | awk '{if (NR!=1) {print $2}}'
The result is 5.20% , I need remove the % at the end, giving ...
- 557
50
votes
5
answers
180k
views
How to replace the content of a specific column with awk?
Given: there are 40 columns in a record. I want to replace the 35th column so that the 35th column will be replaced with the content of the 35th column and a "$" symbol. What came to mind is something ...
- 1,101
49
votes
9
answers
154k
views
How do I delete the first n lines and last line of a file using shell commands?
I have a file named Element_query containing the result of a query :
SQL> select count (*) from element;
[Output of the query which I want to keep in my file]
SQL> spool off;
I want to ...
- 2,732
46
votes
2
answers
106k
views
Common lines between two files [duplicate]
I have the following code that I run on my Terminal.
LC_ALL=C && grep -F -f genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed
This doesn't give me the common lines between the two files. ...
- 653
46
votes
3
answers
180k
views
How to merge two files based on the matching of two columns?
I have file1 likes:
0 AFFX-SNP-000541 NA
0 AFFX-SNP-002255 NA
1 rs12103 0.6401
1 rs12103_1247494 0.696
1 rs12142199 0.7672
And a file2:
0 AFFX-SNP-000541 1
0 AFFX-...
- 669
46
votes
2
answers
89k
views
Remove lines based on duplicates within one column without sort
I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
- 469
45
votes
4
answers
16k
views
Reverse grepping
Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
user78433
45
votes
3
answers
64k
views
What are the exact differences between awk and cut with grep? [closed]
We know that we can get the second column of the line we want from a file using these two techniques:
awk '/WORD/ { print $2 }' filename
or
grep WORD filename| cut -f 2 -d ' '
My questions are:
...
- 8,876
44
votes
3
answers
113k
views
Piping from grep to awk not working
I am trying to grep the ongoing tail of file log and get the nth word from a line. Example file:
$ cat > test.txt <<EOL
Beam goes blah
John goes hey
Beam goes what?
John goes forget it
Beam ...
- 9,367
44
votes
16
answers
133k
views
How to print certain columns by name?
I have the following file:
id name age
1 ed 50
2 joe 70
I want to print just the id and age columns. Right now I just use awk:
cat file.tsv | awk '{ print $1, $3 }'
However, this ...
- 543
43
votes
9
answers
124k
views
How to print only last column?
echo -e 'one two three\nfour five six\nseven eight nine'
one two three
four five six
seven eight nine
how can I do some "MAGIC" do get this output?:
three
six
nine
UPDATE:
I don't need it in this ...
- 39.5k
43
votes
15
answers
116k
views
How can I remove all comments from a file?
I have a file which includes comments:
foo
bar
stuff
#Do not show this...
morestuff
evenmorestuff#Or this
I want to print the file without including any of the comments:
foo
bar
stuff
morestuff
...
- 3,915
43
votes
4
answers
97k
views
Split a file into two
I have a big file and need to split into two files. Suppose in the first file the 1000 lines should be selected and put into another file and delete those lines in the first file.
I tried using split ...
- 1,559
42
votes
2
answers
143k
views
Print lines of a file between two matching patterns [duplicate]
How can I print all the lines between two lines starting with one pattern for the first line and ending with another pattern for the last line?
Update
I guess it was a mistake to mention that this ...
- 593
42
votes
8
answers
125k
views
Using bash shell function inside AWK
Is it possible to use bash function inside AWK somehow?
Example file (string, int, int, int)
Mike 247808 247809 247810
Trying to convert values from decimal to hexadecimal.
Function defined either in ...
- 813
42
votes
5
answers
39k
views
Find intersection of lines in two files [duplicate]
If I have two files (with single columns), one like so (file1)
34
67
89
92
102
180
blue2
3454
And the second file (file2)
23
56
67
69
102
200
How do I find elements that are common in both files (...
- 1,229
41
votes
5
answers
38k
views
How to get last part of http link in Bash?
I have an http link :
http://www.test.com/abc/def/efg/file.jar
and I want to save the last part file.jar to variable, so the output string is "file.jar".
Condition: link can has different length ...
- 421
41
votes
9
answers
27k
views
How can you combine all lines that end with a backslash character?
Using a common command line tool like sed or awk, is it possible to join all lines that end with a given character, like a backslash?
For example, given the file:
foo bar \
bash \
baz
dude \
happy
...
- 18.4k
41
votes
8
answers
42k
views
How to grep-inverse-match and exclude "before" and "after" lines
Consider a text file with the following entries:
aaa
bbb
ccc
ddd
eee
fff
ggg
hhh
iii
Given a pattern (e.g. fff), I would like to grep the file above to get in the output:
all_lines except (...
- 40.3k
40
votes
3
answers
14k
views
What is the meaning of '1' at the end of an awk script
I was reading this awk script
awk -F"=" '{OFS="=";gsub(",",";",$2)}1'
I want to know what is the function of 1 at the end of it.
- 1,547
39
votes
5
answers
51k
views
Delete the first n bytes of files
I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way.
I want to delete the first 31 bytes of each file ...
- 393
39
votes
2
answers
7k
views
Why does awk -F work for most letters, but not for the letter "t"?
July 2022 mac os Monterey V12.1
awk --version 20200816
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin21)
Why does awk -F work for most letters, but NOT for the letter t?
I have ...
- 503
38
votes
4
answers
28k
views
Split long output into two columns
Is there a simple utility or script to columnate the output from one of my scripts? I have data in some form:
A aldkhasdfljhaf
B klajsdfhalsdfh
C salkjsdjkladdag
D lseuiorlhisnflkc
E sdjklfhnslkdfhn
...
- 483
37
votes
3
answers
131k
views
awk + print line only if the first field start with string as Linux1
how to print the line in case the first field start with Linux1
for example:
echo Linux1_ver2 12542 kernel-update | awk '{if ($1 ~ Linux1 ) print $0;}'
the target is to print the line , while ...
- 1,295
36
votes
8
answers
115k
views
Print odd-numbered lines, print even-numbered lines
I want to print the odd-numbered and even-numbered lines from files.
I found this shell script which makes use of echo.
#!/bin/bash
# Write a shell script that, given a file name as the argument ...
- 13.1k