awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

64
votes
7answers
22k views

Is there a basic tutorial for grep, awk and sed? [closed]

I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...
36
votes
13answers
53k views

Is there a way to get the min, max, median, and average of a list of numbers in a single command?

I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script. Although my immediate situation is for ...
11
votes
5answers
25k views

Multiline pattern match using sed, awk or grep

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. ...
36
votes
4answers
4k views

Reverse grepping

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
14
votes
8answers
14k views

How to print certain columns by name?

I have the following file: id name age 1 ed 50 2 joe 70 I want to print just the id and age columns. Right now I just use awk: cat file.tsv | awk '{ print $1, $3 }' However, this ...
5
votes
3answers
19k views

How to merge two files based on the matching of two columns?

I have file1 likes: 0 AFFX-SNP-000541 NA 0 AFFX-SNP-002255 NA 1 rs12103 0.6401 1 rs12103_1247494 0.696 1 rs12142199 0.7672 And a file2: 0 AFFX-SNP-000541 1 0 ...
17
votes
11answers
3k views

There must be a better way to replace single newlines only?

I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
14
votes
10answers
43k views

Show sum of file sizes in directory listing

The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current ...
8
votes
6answers
3k views

Is there any alternative to grep's -A -B -C switches (to print few lines before and after )?

grep -A 2 -B 3 prints 2 lines after the grep string and prints 3 lines before. grep -C 3 prints 3 Lines before and 3 lines after Unfortunately, the grep I'm using does not support these ...
7
votes
3answers
5k views

Diff, show modified line from right file only

I have file a and b and I would like to output lines of b that changed since it was cloned from a. Just the modified lines, no surrounding context, no diff offset marks. How can I do that using shell ...
15
votes
5answers
34k views

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": ...
13
votes
1answer
4k views

Making a progressbar with “dialog” from rsync output

I'm looking for a way to filter/redirect rsync output in a manner where it can be fed to the "dialog --gauge" command, so I can get a nice looking progressbar during file sync. Currently I have only ...
9
votes
3answers
3k views

Process last line first using awk

I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally. The ...
7
votes
2answers
45k views

How to run grep on a single column?

I want to grep the output of my ls -l command: -rw-r--r-- 1 root root 1866 Feb 14 07:47 rahmu.file -rw-r--r-- 1 rahmu user 95653 Feb 14 07:47 foo.file -rw-r--r-- 1 rahmu user ...
6
votes
1answer
17k views

Replacing string based on line number

I have a situation where i want to replace a particular string in many files Replace a string AAA with another string BBB but there are lot of strings starting with AAA or ending in AAA ,and i want ...
6
votes
2answers
4k views

Replacing missing value blank space with zero

I have input.txt tab-delimited text file around 30K lines, I would like to check each row (s1..s30K lines) for missing value (i.e blank space) and fill the missing value with zero value.See out.txt ...
3
votes
1answer
2k views

Comparing Files based on 5 fields using Awk and Bash

I want to compare File1 and File2 (Separated by spaces) using five fields (Column 1,2,4,5,6). *Logic:*If column 1 and 2 of File1 and File2 match exactly and if the File2 has the same characters as ...
2
votes
2answers
2k views

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
1
vote
1answer
2k views

two .csv files compare using awk

I need to compare two files, File1.csv and File2.csv (Separated by ',') using Awk Logic: Column PID in both files are reference. If for same 'PID' in 'File1.csv' and 'File2.csv' and in both files ...
9
votes
7answers
6k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" ...
8
votes
4answers
11k views

How to extract logs between two time stamps

I want to extract all logs between two timestamps. Some lines may not have the timestamp, but I want those lines also. In short, I want every line that falls under two time stamps. My log structure ...
6
votes
5answers
275 views

Using awk to identify the number identical columns

I am new to all this, and this is probably a rather simple question, but I am stuck: I have a large number of individual files that contain six columns each (number of rows can vary). As a simple ...
6
votes
3answers
562 views

Is there a perl or awk solution for this problem?

I have an input file (input.txt) like below. id1 id2 name weight 53723848 12651711 timburnes 1.36667 53530214 12651711 timburnes 1.51191 53723848 53530214 timburnes 1.94 764157 ...
5
votes
8answers
685 views

How do I delete the first n lines and last line of a file using shell commands?

I have a file named Element_query containing the result of a query : SQL> select count (*) from element; [Output of the query which I want to keep in my file] SQL> spool off; I want to ...
5
votes
2answers
2k views

What is the best way to find a list of several strings within a large text file

The short, general question is: In Unix/Linux, what is the best way to find a list of several (about 150) strings within a large text file? I am asking this to all Unix/Linux experts as a general ...
4
votes
2answers
478 views

Nested 'awk' in a 'while' loop, parse two files line by line and compare column values

I need some help with a combination of awk & while loop. I have two simple files with columns (normal ones are very large), one representing simple intervals for an ID=10(of coding ...
4
votes
4answers
2k views

merge 2 rows based on the same column values

I have a file like below. 47196436 47723284 name1 1.77273 42672249 52856963 name2 1.06061 52856963 430695 name2 1.16667 55094959 380983 name3 1.55613 17926380 55584836 name4 1.02461 3213456 34211 ...
4
votes
6answers
113 views

Only show the lines that are in all textfiles at least once

a.txt cat a.txt a b x c b.txt cat b.txt d e a f the q: SOMEMAGICK *.txt a Q: how can I show only the lines that are in all *.txt files?
3
votes
1answer
849 views

Efficient way to print lines from a massive file using awk, sed, or something else? [duplicate]

If I had a plain-text file containing 8 million lines and I want to print out lines 4,000,010 to 4,000,000 to the screen, which would be more efficient: awk or sed? There is no pattern to the text, ...
2
votes
4answers
106 views

How to highlight given string in given place?

How can we modify this: <a href="http://foo.bar1">asfdlksafbar1qsasadf</a><br> <a href="http://foo.bar2">svasfbar2saldkfj</a><br> <a ...
2
votes
4answers
253 views

Sed/awk/perl: reversing order of comma separated values, preserving other text

I have this text: LABEL1 .BYTE 01, 02, 03, 04, 05 .BYTE 01, 02, 03 I need to reverse only comma separated values order: LABEL1 .BYTE 05, 04, 03, 02, 01 .BYTE 03, 02, 01 I need ...
2
votes
3answers
280 views

List all word pairs that appear in a line together in a file

How can I use awk to run through a file written as: Sarah,Masha,Helen Connor,Jessica,Jennifer,Candace Betty,Sarah,Helen John,Harold,Frank,Daisy and put out all pairs that appeared in a line ...
2
votes
4answers
1k views

Format a Date field of a .CSV file with multiple commas in a string field

I have a .CSV file (file.csv) whose data are all enclosed in double quotes. Sample format of the file is as below: column1,column2,column3,column4,column5,column6, column7, Column8, Column9, Column10 ...
2
votes
1answer
2k views

Diff the output of two `awk` commands

I'm trying to compute the difference between the output of two awk commands but my simple attempts at it seem to be failing. Here is what I'm trying: diff $(awk '{print $3}' f1.txt | sort -u) $(awk ...
2
votes
4answers
1k views

gawk join two TSV by columns (a'la sql join)

How to join, to tsv files, examples: a.tsv c 7 r z d 6 s w f 1 f f b 8 p y a 9 q x b.tsv a q a c r ccc b p bb 0 0 0 d s dddd Here I'd like to ...
1
vote
2answers
96 views

Compare and print data from different text files to one in Shell

How can I compare and print data from different text files to one in Shell. I have captures NAS details of three different boxes using SSH, now I need to combine all the three text files to one file ...
10
votes
4answers
925 views

How do you check if a file exists within awk? [-d 'filename'] failing

I'm trying to generate a list of users who have a home directory set which does not exist. It seems I should be able to do this with awk, but something is wrong with my syntax. It keeps telling me ...
9
votes
1answer
910 views

grep: display filename once, then display context with line numbers

Our source code has error codes scattered throughout. Finding them is easy with grep, but I'd like a bash function find_code that I can execute (eg. find_code ####) which will provide output along ...
9
votes
3answers
5k views

Why does awk do full buffering when reading from a pipe

I'm reading from a serial port connected to a gps device sending nmea strings. A simplified invocation to illustrate my point: $ awk '{ print $0 }' /dev/ttyPSC9 ...
8
votes
3answers
8k views

What are the exact differences between awk and cut with grep? [closed]

We know that we can get the second column of the line we want from a file using these two techniques: awk '/WORD/ { print $2 }' filename or grep WORD filename| cut -f 2 -d ' ' My questions are: ...
8
votes
2answers
307 views

Are there 2 ways to set awk vars via command line?

I noticed an O'Reilly awk example (1997) which assigned an awk variable by setting it on the command line after the program-text. It does work, but I can't find this syntax in man / info awk. Have I ...
6
votes
7answers
740 views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
6
votes
2answers
1k views

Inotifywait for large number of files in a directory

What I want to do is, to monitor a directory (not recursive, just one) for new files created and append those files to one single big file as they are being written. The number of files that are ...
6
votes
2answers
913 views

How would I sort these directory names numerically?

I'm aware that I can somehow sort this output numerically (so cpu1/ follows cpu0/) ... I could probably get something to work eventually by splitting up the string various ways with awk, but is there ...
5
votes
5answers
477 views

Replace data at specific positions in txt file using data from another file

I have a text file in the below format: $data This is the experimental data good data This is good file datafile 1 4324 3673 6.2e+11 7687 67576 2 3565 8768 8760 5780 8778 "This is line ...
5
votes
5answers
1k views

Pattern matching across multiple lines

What is a quick and easy way to validate if the pattern "301 domaname.com 200" is present within BASH when spread across multiple lines of output? I was using something like the following: awk ...
5
votes
4answers
311 views

How to add each output of awk

How would you perform addition on the output of every line from awk? For example cat foo | awk '{print $1} 42 42 42 I'm looking for the answer to 42+42+42 (126). I'm trying to write a for loop ...
4
votes
3answers
10k views

How to replace the content of specific column with awk?

Given there is 40 columns in a record. I want to replace 35th column so that the 35th column will be replace with the content of 35th column and a "$" symbol. What came up my mind is like: awk ...
4
votes
3answers
3k views

GREP / SED or AWK: Print entire paragraph in a file on pattern match [duplicate]

I have a file with hundreds of paragraphs of around 15 lines each. I need to search for a pattern, say Occurance: 1. If this pattern is found in the para, I need to print the entire paragraph. Note ...
4
votes
2answers
5k views

Processing two files using awk

I read Comparing two files using Unix and Awk. It is really interesting. I read and tested it, but I can't understand it completely and use it in other cases. I have two files. file1 has one field ...