Questions tagged [awk]

A pattern-directed scanning and processing language.

Filter by
Sorted by
Tagged with
912 votes
10 answers
1.8m views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
user avatar
  • 215k
110 votes
7 answers
69k views

Is there a basic tutorial for grep, awk and sed? [closed]

I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...
user avatar
  • 3,640
102 votes
13 answers
330k views

Show sum of file sizes in directory listing

The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current directory,...
user avatar
  • 2,044
102 votes
5 answers
387k views

Using awk to sum the values of a column, based on the values of another column

I am trying to sum certain numbers in a column using awk. I would like to sum just column 3 of the "smiths" to get a total of 212. I can sum the whole column using awk but not just the "smiths". I ...
user avatar
  • 1,165
97 votes
5 answers
237k views

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string bar in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
user avatar
  • 2,833
84 votes
7 answers
237k views

How to read first and last line from cat output?

I have text file. Task - get first and last line from file after $ cat file | grep -E "1|2|3|4" | commandtoprint $ cat file 1 2 3 4 5 Need this without cat output (only 1 and 5). ~$ cat file | ...
user avatar
  • 973
84 votes
6 answers
377k views

How to use a shell command to only show the first column and last column in a text file?

I need some help to figure out how to use the sed command to only show the first column and last column in a text file. Here is what I have so far for column 1: cat logfile | sed 's/\|/ /'|awk '{...
user avatar
  • 1,079
83 votes
13 answers
442k views

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": [],...
user avatar
79 votes
6 answers
170k views

How do you list number of lines of every file in a directory in human readable format.

I have a list of directories and subdirectories that contain large csv files. There are about 500 million lines in these files, each is a record. I would like to know How many lines are in each ...
user avatar
  • 995
77 votes
2 answers
200k views

Replacing string based on line number

I have a situation where i want to replace a particular string in many files Replace a string AAA with another string BBB but there are lot of strings starting with AAA or ending in AAA ,and i want ...
user avatar
  • 871
74 votes
5 answers
110k views

Pass shell variable as a /pattern/ to awk

Having the following in one of my shell functions: function _process () { awk -v l="$line" ' BEGIN {p=0} /'"$1"'/ {p=1} END{ if(p) print l >> "outfile.txt" } ' } , so when called as ...
user avatar
  • 977
68 votes
1 answer
45k views

Difference between gawk vs. awk

Trying to understand the differences between the two functions gawk vs. awk? When would one use gawk vs awk? Or are they the same in terms of usage? Also, could one provide an example?
user avatar
  • 7,633
66 votes
4 answers
124k views

Skip the first 6 lines/rows in a text file with awk

How can I skip the first 6 lines/rows in a text file (input.txt) and process the rest with awk? The format of my awk script (program.awk) is: BEGIN { } { process here } END { } My text file is ...
user avatar
  • 773
65 votes
4 answers
30k views

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++' <f q w e r $ The original code I have found on the internets read:...
user avatar
63 votes
4 answers
85k views

How to view all the content in an awk array?

In my understanding, awk array is something like python dict. So I write down the code bellow to explore it: awk '{my_dict[$1] = $2} END { print my_dict}' zen And I got: awk: can't read value of ...
user avatar
  • 6,787
62 votes
4 answers
45k views

Removing leading zeros from date output

I made an alias of the date command to display date in the following format: 2013.06.14.12.10.02 using this command: alias date = date +"%Y.%m.%d.%H.%M.%S" Everything works great, except I want ...
user avatar
  • 1,888
61 votes
12 answers
66k views

How to print the longest line in a file?

I'm looking for the simplest method to print the longest line in a file. I did some googling and surprisingly couldn't seem to find an answer. I frequently print the length of the longest line in a ...
user avatar
  • 1,659
60 votes
18 answers
37k views

Remove duplicate $PATH entries with awk command

I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my PATH environment variable. I was told that it is possible to achieve this with a one ...
user avatar
59 votes
14 answers
53k views

Add thousands separator in a number

In python re.sub(r"(?<=.)(?=(?:...)+$)", ",", stroke ) To split a number by triplets, e.g.: echo 123456789 | python -c 'import sys;import re; print re.sub(r"(?<=.)(?=(?:...)+$)", ",", sys....
user avatar
  • 1,071
53 votes
11 answers
95k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" line4-...
user avatar
52 votes
3 answers
140k views

How to grep lines between start and end pattern? [duplicate]

I have a file which is having following content: zdk aaa b12 cdn dke kdn Input1: aaa and cdn Output 1: aaa b12 cdn Input 2: zdk and dke Output 2: zdk aaa b12 cdn dke I could use below commands ...
user avatar
  • 909
51 votes
5 answers
255k views

How to run grep on a single column?

I want to grep the output of my ls -l command: -rw-r--r-- 1 root root 1866 Feb 14 07:47 rahmu.file -rw-r--r-- 1 rahmu user 95653 Feb 14 07:47 foo.file -rw-r--r-- 1 rahmu user ...
user avatar
  • 18.3k
50 votes
4 answers
69k views

How to add line numbers in every line using shell command?

My file, PSS-A (Primary A) PSS-B (Primary B) PSS-C (Primary C) PSS-D (Primary D) PSS-E (Primary E) PSS-F (Primary F) PSS-G (Primary G) PSS-H (Primary H) PSS-I (Primary I) SPARE (SPARE) ...
user avatar
  • 2,386
47 votes
9 answers
141k views

How do I delete the first n lines and last line of a file using shell commands?

I have a file named Element_query containing the result of a query : SQL> select count (*) from element; [Output of the query which I want to keep in my file] SQL> spool off; I want to ...
user avatar
  • 2,386
47 votes
5 answers
161k views

How to replace the content of a specific column with awk?

Given: there are 40 columns in a record. I want to replace the 35th column so that the 35th column will be replaced with the content of the 35th column and a "$" symbol. What came to mind is something ...
user avatar
45 votes
3 answers
162k views

How to merge two files based on the matching of two columns?

I have file1 likes: 0 AFFX-SNP-000541 NA 0 AFFX-SNP-002255 NA 1 rs12103 0.6401 1 rs12103_1247494 0.696 1 rs12142199 0.7672 And a file2: 0 AFFX-SNP-000541 1 0 AFFX-...
user avatar
45 votes
4 answers
15k views

Reverse grepping

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
user avatar
42 votes
3 answers
101k views

Piping from grep to awk not working

I am trying to grep the ongoing tail of file log and get the nth word from a line. Example file: $ cat > test.txt <<EOL Beam goes blah John goes hey Beam goes what? John goes forget it Beam ...
user avatar
42 votes
1 answer
84k views

Remove last character from string captured with awk [duplicate]

I need to remove the last character from a string in this command: sudo docker stats --no-stream 39858jf8 | awk '{if (NR!=1) {print $2}}' The result is 5.20% , I need remove the % at the end, giving ...
user avatar
  • 467
42 votes
15 answers
123k views

How to print certain columns by name?

I have the following file: id name age 1 ed 50 2 joe 70 I want to print just the id and age columns. Right now I just use awk: cat file.tsv | awk '{ print $1, $3 }' However, this ...
user avatar
41 votes
3 answers
58k views

What are the exact differences between awk and cut with grep? [closed]

We know that we can get the second column of the line we want from a file using these two techniques: awk '/WORD/ { print $2 }' filename or grep WORD filename| cut -f 2 -d ' ' My questions are: ...
user avatar
  • 8,456
40 votes
4 answers
81k views

Split a file into two

I have a big file and need to split into two files. Suppose in the first file the 1000 lines should be selected and put into another file and delete those lines in the first file. I tried using split ...
user avatar
  • 1,519
40 votes
8 answers
112k views

Using bash shell function inside AWK

Is it possible to use bash function inside AWK somehow? Example file (string, int, int, int) Mike 247808 247809 247810 Trying to convert values from decimal to hexadecimal. Function defined either in ...
user avatar
  • 763
40 votes
9 answers
26k views

How can you combine all lines that end with a backslash character?

Using a common command line tool like sed or awk, is it possible to join all lines that end with a given character, like a backslash? For example, given the file: foo bar \ bash \ baz dude \ happy ...
user avatar
  • 16.8k
40 votes
2 answers
81k views

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
user avatar
  • 409
37 votes
5 answers
31k views

How to get last part of http link in Bash?

I have an http link : http://www.test.com/abc/def/efg/file.jar and I want to save the last part file.jar to variable, so the output string is "file.jar". Condition: link can has different length ...
user avatar
  • 381
37 votes
2 answers
126k views

Print lines of a file between two matching patterns [duplicate]

How can I print all the lines between two lines starting with one pattern for the first line and ending with another pattern for the last line? Update I guess it was a mistake to mention that this ...
user avatar
  • 543
37 votes
8 answers
35k views

How to grep-inverse-match and exclude "before" and "after" lines

Consider a text file with the following entries: aaa bbb ccc ddd eee fff ggg hhh iii Given a pattern (e.g. fff), I would like to grep the file above to get in the output: all_lines except (...
user avatar
35 votes
4 answers
44k views

Delete the first n bytes of files

I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way. I want to delete the first 31 bytes of each file ...
user avatar
35 votes
4 answers
25k views

Split long output into two columns

Is there a simple utility or script to columnate the output from one of my scripts? I have data in some form: A aldkhasdfljhaf B klajsdfhalsdfh C salkjsdjkladdag D lseuiorlhisnflkc E sdjklfhnslkdfhn ...
user avatar
  • 453
34 votes
9 answers
110k views

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...
user avatar
  • 36.1k
34 votes
13 answers
13k views

There must be a better way to replace single newlines only?

I am in the habit of writing one line per sentence because I typically compile things to LaTeX, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
user avatar
  • 3,263
34 votes
12 answers
48k views

How can I write to the second line of a file from the command line?

I have an external program that produces an output file (largish, 20K lines possible). I need to insert a new line between the existing line 1 and line 2. I've been looking at awk and sed - I use ...
user avatar
33 votes
15 answers
99k views

How can I remove all comments from a file?

I have a file which includes comments: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I want to print the file without including any of the comments: foo bar stuff morestuff ...
user avatar
  • 3,455
33 votes
5 answers
63k views

Delete range of lines above pattern with sed (or awk)

I have the following code that will remove lines with the pattern banana and 2 lines after it: sed '/banana/I,+2 d' file So far, so good! But I need it to remove 2 lines before banana, but I can't ...
user avatar
  • 4,432
33 votes
3 answers
26k views

Diff, show modified line from right file only

I have file a and b and I would like to output lines of b that changed since it was cloned from a. Just the modified lines, no surrounding context, no diff offset marks. How can I do that using shell ...
user avatar
33 votes
3 answers
10k views

What is the meaning of '1' at the end of an awk script

I was reading this awk script awk -F"=" '{OFS="=";gsub(",",";",$2)}1' I want to know what is the function of 1 at the end of it.
user avatar
  • 1,437
32 votes
9 answers
68k views

How to delete the last column of a file in Linux

I want to delete the last column of a txt file, while I do not know what the column number is. How could I do this? Example: Input: 1223 1234 1323 ... 2222 123 1233 1234 1233 ... 3444 125 0000 5553 ...
user avatar
  • 1,263

1
2 3 4 5
154