awk - pattern-directed scanning and processing language
47
votes
6answers
8k views
Is there a basic tutorial for grep, awk and sed?
I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...
18
votes
9answers
3k views
How can I write to the second line of a file from the command line?
I have an external program that produces an output file (largish, 20K lines possible).
I need to insert a new line between the existing line 1 and line 2. I've been looking at awk and sed - I use ...
16
votes
9answers
1k views
Remove duplicate $PATH entries with awk command
I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my path environment.
I was told that it is possible to achieve this with a one line ...
15
votes
3answers
966 views
Piping from grep to awk not working
I am trying to grep the ongoing tail of file log and get the nth word from a line. Example file:
$ cat > test.txt <<EOL
Beam goes blah
John goes hey
Beam goes what?
John goes forget it
Beam ...
15
votes
9answers
6k views
Tool in UNIX to subtract dates
Is there any tool in Solaris UNIX (so no GNU tool available) to subtract dates? I know that in Linux we have gawk that can subtract one date from another. But in Solaris the maximum we have is nawk ...
12
votes
9answers
2k views
There must be a better way to replace single newlines only?
I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
11
votes
11answers
6k views
Grep huge number of patterns from huge file
I have a file that's growing about 200,000 lines a day, and it is all formed with blocks of three lines as such:
1358726575123 # key
Joseph Brunner # name
carpenter # job
...
11
votes
6answers
15k views
Is there a way to get the min, max, median, and average of a list of numbers in a single command?
I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script.
Although my immediate situation is for ...
11
votes
1answer
1k views
Making a progressbar with “dialog” from rsync output
I'm looking for a way to filter/redirect rsync output in a manner where it can be fed to the "dialog --gauge" command, so I can get a nice looking progressbar during file sync.
Currently I have only ...
10
votes
4answers
10k views
How to manipulate a CSV file with sed or awk?
How can I do the following to a CSV file using sed or awk?
Delete a column
Duplicate a column
Move a column
I have a big table with over 200 rows, and I'm not that familiar with sed.
10
votes
6answers
2k views
How to print the longest line in a file?
I'm looking for the simplest method to print the longest line in a file. I did some googling and surprisingly couldn't seem to find an answer. I frequently print the length of the longest line in a ...
9
votes
3answers
417 views
Process last line first using awk
I have a data file that I want to normalize using awk, based on the last datapoint. Therefor, I would like to access the last data point first, to normalize the data, then process normally.
The ...
8
votes
6answers
222 views
Grab certain contents of a file
So I know tools exist for this problem because I've heard about them, but I don't know what they are.
I want to do something like filter out all data but the usernames in /etc/passwd.
For example, I ...
8
votes
3answers
819 views
How to sort the string which combined with string + numeric using bash script?
This is the data what I want to sort. But sort treats the numeric to string, the data it no sorted as I expected.
/home/files/profile1
/home/files/profile10
/home/files/profile11
...
8
votes
6answers
778 views
Delete lines beginning with #
How do I delete lines beginning with a #, given that there can be whitespace on the left and right of the #?
# Master socket provides access to userdb information. It's typically
8
votes
3answers
2k views
When using awk /pattern/ { print “text”} /patern/ {print “”} is there an ELSE pattern?
Let's say I have text file like:
R1 12 324 3453 36 457 4 7 8
R2 34 2342 2525 25 25 26 26 2 2
R3 23 2342 32 52 54 543 643 63
R4 25 234 2342 4 234242
I want to use awk to process these lines ...
8
votes
3answers
2k views
Delete the first n bytes of files
I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way.
I want to delete the first 31 bytes of each file ...
8
votes
2answers
143 views
Are there 2 ways to set awk vars via command line?
I noticed an O'Reilly awk example (1997) which assigned an awk variable by setting it on the command line after the program-text. It does work, but I can't find this syntax in man / info awk. Have I ...
7
votes
10answers
547 views
What's a good way to filter a text file to remove empty lines?
I have a .csv file (on a mac) that has a bunch of empty lines, e.g.:
"1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum ...
7
votes
5answers
3k views
Remove comma between the quotes only in a comma delimited file
I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row
123,"ABC, DEV 23",345,534.202,NAME
I need to ...
7
votes
6answers
1k views
Is there any alternative to grep's -A -B -C switches (to print few lines before and after )?
grep -A 2 -B 3
prints 2 lines after the grep string and prints 3 lines before.
grep -C 3
prints 3 Lines before and 3 lines after
Unfortunately, the grep I'm using does not support these ...
7
votes
2answers
1k views
A shell script for joining two files
I want to write a shell script that get two files A and B, and get a result like this:
File A:
user_a tel_a addr_a
user_b tel_b addr_b
File B:
process_1 user_a
process_2 user_a
process_3 user_b
...
6
votes
7answers
632 views
Slick one-liner to convert a list like “1: 2, 3, 4, 5” to “1.2, 1.3, 1.4, 1.5”
Let's say I have a file that looks something like this:
23: a, b, c, d
24: b, d, f
25: c, g
and I want to get output like this:
23.a
23.b
23.c
23.d
24.b
24.d
24.f
25.c
25.g
Of course it's not ...
6
votes
7answers
255 views
Swapping an unlimited number of columns
I have a file with columns. See below for an example:
a b c ... z
1 2 3 ... 26
I'd like to swap all columns where the 1st becomes the last, the second becomes the one before last...etc..
z y x ...
6
votes
4answers
2k views
Remove line containing certain string and the following line
I use this
cat foo.txt | sed '/bar/d'
to remove lines containing the string 'bar' in the file.
I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
6
votes
2answers
941 views
sed one-liner to delete any line that does not contain lowercase letters
So, basically
THIS LINE WOULD BE DELETED
and
(THIS LINE WOULD ALSO BE DELETED)
but
Indeed, THIS LINE WOULD NOT
6
votes
6answers
2k views
How can you combine all lines that end with a backslash character?
Using a common command line tool like sed or awk, is it possible to join all lines that end with a given character, like a backslash?
For example, given the file:
foo bar \
bash \
baz
dude \
happy
...
6
votes
3answers
1k views
AWK: wrap lines to 72 characters
$ awk 'length > 72' {HOW TO PRINT THE LINEs IN PCS?} msg
ie I want it to add \n after 72 chars and continue, so initially you may need to remove all single \ns and the add them. It may be easier ...
6
votes
4answers
330 views
Find images by size: find / file / awk
I've been trying to find png image files a certain height (over 500px). I know that file will return image dimensions. Example:
$ file TestImg1a.png
TestImg1a.png: PNG image data, 764 x 200, ...
5
votes
4answers
487 views
How to append end of every line with the line itself
This is in file.txt:
redcar
bluecar
greencar
Im looking for ways to make it become:
redcar redcar
bluecar bluecar
greencar greencar
I've tried many ways using sed with no luck
5
votes
7answers
4k views
How to print only last column?
echo -e 'one two three\nfour five six\nseven eight nine'
one two three
four five six
seven eight nine
how can I do some "MAGIC" do get this output?:
three
six
nine
UPDATE:
I don't need it in this ...
5
votes
3answers
568 views
Is there a more elegant way to count words and assign that count to variables?
I have a script:
#!/bin/bash
/root/xiotech status > xiostatus.tmp
SyncCount=$(grep -c Sync xiostatus.tmp)
PauseCount=$(grep -c paused xiostatus.tmp)
CopyingCount=$(grep -c Copying xiostatus.tmp)
...
5
votes
2answers
940 views
How to use regex as field separator in awk?
I'm trying to use regex as a field seperator in awk. From my reading this seems possible but I can't get the syntax right.
rpm -qa | awk '{ 'FS == [0-9]' ; print $1 }'
awk: cmd. line:1: { FS
awk: ...
5
votes
3answers
3k views
How to sed only that lines that contains given string?
INPUT:
Select ASDF 325 sdfg sdflk lk
Select TRG 46sdg rasdftz fsgs 45
Select ASDF 6ffg sdfg 4456 sdrg
OUTPUT:
Select ASDF 325 XXXX sdflk lk
Select TRG 46sdg rasdftz fsgs 45
Select ASDF 6ffg XXXX ...
5
votes
4answers
210 views
How could I simplify this command to only use awk?
awk '/user/ {print $1 }' /etc/userdomains | sed 's/://'
the format of /etc/userdomains is
domain.tld: user
otherdomain.tld: otheruser
5
votes
5answers
238 views
How to remove multiple blank lines from a file?
I have some text-files I use to take notes in - just plain text, usually just using cat >> file. Occasionally I use a blank line or two (just return - the new-line character) to specify a new ...
5
votes
2answers
177 views
what is the meaning of 1 at the end of awk script
I was reading this awk script
awk -F"=" '{OFS="=";gsub(",",";",$2)}1'
I want to know what is the function of 1 at the end of it
5
votes
3answers
13k views
How to use regrex with AWK for string replacement in this example?
Suppose there is some text from a file:
(bookmarks
("Chapter 1 Introduction 1" "#1"
("1.1 Problem Statement and Basic Definitions 23" "#2")
("Exercises 31" "#30")
("Notes and References 42" "#34"))
)
...
5
votes
4answers
2k views
How to print certain columns by name?
I have the following file:
id name age
1 ed 50
2 joe 70
I want to print just the id and age columns. Right now I just use awk:
cat file.tsv | awk '{ print $1, $3 }'
However, this ...
5
votes
2answers
365 views
Replacing missing value blank space with zero
I have input.txt tab-delimited text file around 30K lines, I would like to check each row (s1..s30K lines) for missing value (i.e blank space) and fill the missing value with zero value.See out.txt
...
5
votes
2answers
107 views
Why does awk execute both actions?
When I run the command:
awk '/from/ {print $7} /to/ { print $7}' erroMuitoDoido.txt
The file is:
May 19 04:44:43 server postfix/smtpd[32595]: CDAB515013: ...
5
votes
2answers
249 views
How would I sort these directory names numerically?
I'm aware that I can somehow sort this output numerically (so cpu1/ follows cpu0/) ... I could probably get something to work eventually by splitting up the string various ways with awk, but is there ...
5
votes
2answers
89 views
awk + paste for cleaning up PATH?
I have seen this code in .cshrc init files on a few machines. I went through a few awk tutorials in trying to understand how it works, but I am still unable to decrypt it.
setenv PATH `echo $PATH | ...
5
votes
2answers
108 views
Sort fields inline
I'm trying to sort within a line of input over an unknown number of fields:
Input:
ab bc
bc ab
cd ef bc
bc cd ef
cd bc ab
ef ab bc cd gh
Output:
ab bc
ab bc
bc cd ef
bc cd ef
ab cb cd
ab bc cd ...
5
votes
3answers
713 views
killing processes automatically
I need to kill all processs in a certain shell excluding certain processes.
Like sh which is my shell. And the comand.
This is what currently in my shell right now.
rcihp146 :/home/msingh2> ps
...
5
votes
1answer
92 views
Inderect references aka value of a value in awk
I'm searching for a way to solve this problem in awk. Input:
X 1
Y 2
Z 3
X 4
Y 5
The output should look like this:
X 5
Y 7
Z 3
I see two problems here: the first one is the indirect reference in ...
5
votes
2answers
177 views
Use a script parameter in awk
Here is my script (to find the files that contain a specified pattern) :
find . -type f -exec awk -v vawk="$1" '/'"$vawk"'/ {c++} c>0 {print ARGV[1] ; exit 0 } END { if (! c) {exit 1}}' \{\} \;
...
5
votes
1answer
351 views
Why are capital letters included in a range of lower-case letters in an awk regex?
$ echo ABC | awk '$0 ~ /^[a-b]/'
ABC
$ echo ABC | awk '$0 ~ /^[a-a]/'
$ echo ABC | awk '$0 ~ /^a/'
$
You see. /[a-b]/ captures A, but /[a-a]/ or /a/ doesn't. Why?
5
votes
4answers
184 views
Show sum of file sizes in directory listing
The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current ...
5
votes
1answer
829 views
Fast elimination of duplicate lines across multiple files
I have a huge amount of data in which each (data-)line should be unique.
There are a lot of files in one folder in which this is already true. It is about 15GB splitted into roughly 170 files with ...
