Questions tagged [awk]

A pattern-directed scanning and processing language.

Filter by
Sorted by
Tagged with
947 votes
10 answers

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
terdon's user avatar
  • 235k
122 votes
13 answers

Show sum of file sizes in directory listing

The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current directory,...
MattDMo's user avatar
  • 2,314
112 votes
7 answers

Is there a basic tutorial for grep, awk and sed? [closed]

I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...
Macha's user avatar
  • 3,770
107 votes
5 answers

Using awk to sum the values of a column, based on the values of another column

I am trying to sum certain numbers in a column using awk. I would like to sum just column 3 of the "smiths" to get a total of 212. I can sum the whole column using awk but not just the "smiths". I ...
jake's user avatar
  • 1,215
105 votes
6 answers

How to use a shell command to only show the first column and last column in a text file?

I need some help to figure out how to use the sed command to only show the first column and last column in a text file. Here is what I have so far for column 1: cat logfile | sed 's/\|/ /'|awk '{...
user70573's user avatar
  • 1,289
101 votes
5 answers

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string bar in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
jakub.g's user avatar
  • 3,153
99 votes
8 answers

How do you list number of lines of every file in a directory in human readable format.

I have a list of directories and subdirectories that contain large csv files. There are about 500 million lines in these files, each is a record. I would like to know How many lines are in each ...
Hexatonic's user avatar
  • 1,235
95 votes
7 answers

How to read first and last line from cat output?

I have text file. Task - get first and last line from file after $ cat file | grep -E "1|2|3|4" | commandtoprint $ cat file 1 2 3 4 5 Need this without cat output (only 1 and 5). ~$ cat file | ...
dmgl's user avatar
  • 1,083
89 votes
13 answers

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": [],...
user3086014's user avatar
86 votes
2 answers

Replacing string based on line number

I have a situation where i want to replace a particular string in many files Replace a string AAA with another string BBB but there are lot of strings starting with AAA or ending in AAA ,and i want ...
krypto's user avatar
  • 961
77 votes
5 answers

Pass shell variable as a /pattern/ to awk

Having the following in one of my shell functions: function _process () { awk -v l="$line" ' BEGIN {p=0} /'"$1"'/ {p=1} END{ if(p) print l >> "outfile.txt" } ' } , so when called as ...
branquito's user avatar
  • 1,017
74 votes
4 answers

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++' <f q w e r $ The original code I have found on the internets read:...
Alexander Shcheblikin's user avatar
73 votes
1 answer

Difference between gawk vs. awk

Trying to understand the differences between the two functions gawk vs. awk? When would one use gawk vs awk? Or are they the same in terms of usage? Also, could one provide an example?
chrisjlee's user avatar
  • 8,303
72 votes
5 answers

Skip the first 6 lines/rows in a text file with awk

How can I skip the first 6 lines/rows in a text file (input.txt) and process the rest with awk? The format of my awk script (program.awk) is: BEGIN { } { process here } END { } My text file is ...
amatek's user avatar
  • 833
71 votes
4 answers

How to add line numbers in every line using shell command?

My file, PSS-A (Primary A) PSS-B (Primary B) PSS-C (Primary C) PSS-D (Primary D) PSS-E (Primary E) PSS-F (Primary F) PSS-G (Primary G) PSS-H (Primary H) PSS-I (Primary I) SPARE (SPARE) ...
Nainita's user avatar
  • 2,732
68 votes
4 answers

How to view all the content in an awk array?

In my understanding, awk array is something like python dict. So I write down the code bellow to explore it: awk '{my_dict[$1] = $2} END { print my_dict}' zen And I got: awk: can't read value of ...
Zen's user avatar
  • 7,297
68 votes
12 answers

How to print the longest line in a file?

I'm looking for the simplest method to print the longest line in a file. I did some googling and surprisingly couldn't seem to find an answer. I frequently print the length of the longest line in a ...
dr.bunsen's user avatar
  • 1,749
66 votes
14 answers

Add thousands separator in a number

In python re.sub(r"(?<=.)(?=(?:...)+$)", ",", stroke ) To split a number by triplets, e.g.: echo 123456789 | python -c 'import sys;import re; print re.sub(r"(?<=.)(?=(?:...)+$)", ",", sys....
user2496's user avatar
  • 1,151
65 votes
4 answers

Removing leading zeros from date output

I made an alias of the date command to display date in the following format: 2013. using this command: alias date = date +"%Y.%m.%d.%H.%M.%S" Everything works great, except I want ...
easl's user avatar
  • 1,978
62 votes
19 answers

Remove duplicate $PATH entries with awk command

I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my PATH environment variable. I was told that it is possible to achieve this with a one ...
Johnny Williem's user avatar
59 votes
3 answers

How to grep lines between start and end pattern? [duplicate]

I have a file which is having following content: zdk aaa b12 cdn dke kdn Input1: aaa and cdn Output 1: aaa b12 cdn Input 2: zdk and dke Output 2: zdk aaa b12 cdn dke I could use below commands ...
Spike's user avatar
  • 989
57 votes
11 answers

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" line4-...
user1228191's user avatar
55 votes
5 answers

How to run grep on a single column?

I want to grep the output of my ls -l command: -rw-r--r-- 1 root root 1866 Feb 14 07:47 rahmu.file -rw-r--r-- 1 rahmu user 95653 Feb 14 07:47 foo.file -rw-r--r-- 1 rahmu user ...
rahmu's user avatar
  • 19.7k
51 votes
1 answer

Remove last character from string captured with awk [duplicate]

I need to remove the last character from a string in this command: sudo docker stats --no-stream 39858jf8 | awk '{if (NR!=1) {print $2}}' The result is 5.20% , I need remove the % at the end, giving ...
Steph's user avatar
  • 557
50 votes
5 answers

How to replace the content of a specific column with awk?

Given: there are 40 columns in a record. I want to replace the 35th column so that the 35th column will be replaced with the content of the 35th column and a "$" symbol. What came to mind is something ...
Marcus Thornton's user avatar
49 votes
9 answers

How do I delete the first n lines and last line of a file using shell commands?

I have a file named Element_query containing the result of a query : SQL> select count (*) from element; [Output of the query which I want to keep in my file] SQL> spool off; I want to ...
Nainita's user avatar
  • 2,732
46 votes
2 answers

Common lines between two files [duplicate]

I have the following code that I run on my Terminal. LC_ALL=C && grep -F -f genename2.txt hg38.hgnc.bed > hg38.hgnc.goi.bed This doesn't give me the common lines between the two files. ...
Marwah Soliman's user avatar
46 votes
3 answers

How to merge two files based on the matching of two columns?

I have file1 likes: 0 AFFX-SNP-000541 NA 0 AFFX-SNP-002255 NA 1 rs12103 0.6401 1 rs12103_1247494 0.696 1 rs12142199 0.7672 And a file2: 0 AFFX-SNP-000541 1 0 AFFX-...
Dadong Zhang's user avatar
46 votes
2 answers

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
Zach C's user avatar
  • 469
45 votes
4 answers

Reverse grepping

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
user avatar
45 votes
3 answers

What are the exact differences between awk and cut with grep? [closed]

We know that we can get the second column of the line we want from a file using these two techniques: awk '/WORD/ { print $2 }' filename or grep WORD filename| cut -f 2 -d ' ' My questions are: ...
Nidal's user avatar
  • 8,876
44 votes
3 answers

Piping from grep to awk not working

I am trying to grep the ongoing tail of file log and get the nth word from a line. Example file: $ cat > test.txt <<EOL Beam goes blah John goes hey Beam goes what? John goes forget it Beam ...
Belmin Fernandez's user avatar
44 votes
16 answers

How to print certain columns by name?

I have the following file: id name age 1 ed 50 2 joe 70 I want to print just the id and age columns. Right now I just use awk: cat file.tsv | awk '{ print $1, $3 }' However, this ...
Brett Thomas's user avatar
43 votes
9 answers

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...
LanceBaynes's user avatar
  • 39.5k
43 votes
15 answers

How can I remove all comments from a file?

I have a file which includes comments: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I want to print the file without including any of the comments: foo bar stuff morestuff ...
Questionmark's user avatar
  • 3,915
43 votes
4 answers

Split a file into two

I have a big file and need to split into two files. Suppose in the first file the 1000 lines should be selected and put into another file and delete those lines in the first file. I tried using split ...
Aravind's user avatar
  • 1,559
42 votes
2 answers

Print lines of a file between two matching patterns [duplicate]

How can I print all the lines between two lines starting with one pattern for the first line and ending with another pattern for the last line? Update I guess it was a mistake to mention that this ...
Vince's user avatar
  • 593
42 votes
8 answers

Using bash shell function inside AWK

Is it possible to use bash function inside AWK somehow? Example file (string, int, int, int) Mike 247808 247809 247810 Trying to convert values from decimal to hexadecimal. Function defined either in ...
tiny's user avatar
  • 813
42 votes
5 answers

Find intersection of lines in two files [duplicate]

If I have two files (with single columns), one like so (file1) 34 67 89 92 102 180 blue2 3454 And the second file (file2) 23 56 67 69 102 200 How do I find elements that are common in both files (...
mindlessgreen's user avatar
41 votes
5 answers

How to get last part of http link in Bash?

I have an http link : and I want to save the last part file.jar to variable, so the output string is "file.jar". Condition: link can has different length ...
FunTomas's user avatar
  • 421
41 votes
9 answers

How can you combine all lines that end with a backslash character?

Using a common command line tool like sed or awk, is it possible to join all lines that end with a given character, like a backslash? For example, given the file: foo bar \ bash \ baz dude \ happy ...
Cory Klein's user avatar
  • 18.4k
41 votes
8 answers

How to grep-inverse-match and exclude "before" and "after" lines

Consider a text file with the following entries: aaa bbb ccc ddd eee fff ggg hhh iii Given a pattern (e.g. fff), I would like to grep the file above to get in the output: all_lines except (...
Amelio Vazquez-Reina's user avatar
40 votes
3 answers

What is the meaning of '1' at the end of an awk script

I was reading this awk script awk -F"=" '{OFS="=";gsub(",",";",$2)}1' I want to know what is the function of 1 at the end of it.
Mirage's user avatar
  • 1,547
39 votes
5 answers

Delete the first n bytes of files

I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way. I want to delete the first 31 bytes of each file ...
von der tann's user avatar
39 votes
2 answers

Why does awk -F work for most letters, but not for the letter "t"?

July 2022 mac os Monterey V12.1 awk --version 20200816 GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin21) Why does awk -F work for most letters, but NOT for the letter t? I have ...
Ian Christy's user avatar
38 votes
4 answers

Split long output into two columns

Is there a simple utility or script to columnate the output from one of my scripts? I have data in some form: A aldkhasdfljhaf B klajsdfhalsdfh C salkjsdjkladdag D lseuiorlhisnflkc E sdjklfhnslkdfhn ...
Hemmer's user avatar
  • 483
37 votes
3 answers

awk + print line only if the first field start with string as Linux1

how to print the line in case the first field start with Linux1 for example: echo Linux1_ver2 12542 kernel-update | awk '{if ($1 ~ Linux1 ) print $0;}' the target is to print the line , while ...
yael's user avatar
  • 1,295
36 votes
8 answers

Print odd-numbered lines, print even-numbered lines

I want to print the odd-numbered and even-numbered lines from files. I found this shell script which makes use of echo. #!/bin/bash # Write a shell script that, given a file name as the argument ...
ixtmixilix's user avatar
  • 13.1k

2 3 4 5