awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

6
votes
3answers
837 views

Replace multiple strings in a single pass

I'm looking for a way to replace placeholder strings in a template file with concrete values, with common Unix tools (bash, sed, awk, maybe perl). It is important that the replacement is done in a ...
6
votes
1answer
4k views

Removing leading zeros from DATE

I made an alias of the date command to display date in the following format: 2013.06.14.12.10.02 using this command: alias date = date +"%Y.%m.%d.%H.%M.%S" Everything works great, except I ...
6
votes
4answers
1k views

find length of longest line in all text files in a directory

I know how to get length of the longest line in a text file with awk awk ' { if ( length > L ) { L=length} }END{ print L}' file.txt but how can I get the length of the longest line of all files ...
6
votes
3answers
7k views

Filter a .CSV file based on the 5th column values of a file and print those records into a new file

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , ...
6
votes
1answer
105 views

How can I extract/change lines in a text file whose data are separated into fields?

How can I manipulate field-based data from the commandline? For example How can I print only lines whose Nth field is foo? How can I print only lines whose Nth field isn't foo? How can I print only ...
6
votes
2answers
1k views

Inotifywait for large number of files in a directory

What I want to do is, to monitor a directory (not recursive, just one) for new files created and append those files to one single big file as they are being written. The number of files that are ...
6
votes
2answers
267 views

Grepping for a block of text with parts that can be optional

I have multiple entries that describes an event in a very large log file, say A.log. I would like to do two things with the event entries in the log file: Count the number of occurrences of each ...
6
votes
2answers
2k views

Use awk to insert a line after N output

I have a text file filled with a bunch of data. I need to pull out lines that contain specific strings. I accomplished this with awk using the following: awk '/pattern1|pattern2|pattern3/ {print;}' ...
6
votes
6answers
257 views

How can I avoid escaped delimiter characters in a Unix command?

I need to take the sum of a field which is tilde (~) delimited. Problem I have is that my data is also having delimiter escaped. Example 1~CEO~ashok\~kumar~1000 As we see in 3rd field above we ...
6
votes
1answer
109 views

Count duplicate rows with constraint

I have the following file: A B 1990 A C 2001 A C 2013 A B 2001 B C 2013 A D 2001 A D 2014 I need to count duplicate pairs (in the 1st and 2nd column) and to each such pair assign the lowest value ...
5
votes
4answers
1k views

How to append end of every line with the line itself

This is in file.txt: redcar bluecar greencar Im looking for ways to make it become: redcar redcar bluecar bluecar greencar greencar I've tried many ways using sed with no luck
5
votes
8answers
462 views

How do I delete the first n lines and last line of a file using shell commands?

I have a file named Element_query containing the result of a query : SQL> select count (*) from element; [Output of the query which I want to keep in my file] SQL> spool off; I want to ...
5
votes
6answers
419 views

split file into two parts, at a pattern

How to split a large file into two parts, at a pattern? Given an example file.txt: ABC EFG XYZ HIJ KNL I want to split this file at XYZ such that file1 contains lines up-to XYZ and rest of the ...
5
votes
4answers
484 views

Reduce the over use of files in text processing

Currently, I have a file (Results.txt) which has 27k rows. The file content is as below. 45008657 | A. | | Long | 0 49210987 | A. | ...
5
votes
3answers
798 views

How to understand the command awk '{for(x=1;$x;++x)print $x}'?

I know this command awk '{for(x=1;$x;++x)print $x}' will print out all columns in a line. wouldn't this ++x change x to 2, and thus print $2 first? As I understood based on this: ...
5
votes
4answers
2k views

remove blank lines in awk display

I was trying to figure out a solution for this question. I wanted to use awk for the solution. My input file is something like below. -bash-3.2$ cat file ramesh ramesh_venkat ramesh3_venkat3 ...
5
votes
5answers
837 views

Why does awk not sum to 0

I have this file and I want to sum all the number in first column. Easy: awk '{s+=$1;print $1,s}' file -0.4157 -0.4157 0.0143 -0.4014 0.5973 0.1959 -0.5679 -0.372 -0.2041 -0.5761 0.7130 0.1369 ...
5
votes
4answers
3k views

number parsing in awk

I have a text file with numbers in second column and the numbers have thousand-separator and decimal-separator: 2014-01 2,277.40 2014-02 2,282.20 2014-03 3,047.90 2014-04 4,127.60 ...
5
votes
5answers
4k views

Convert formatted dates to seconds since the epoch

I have a file: pablo tty8 Thu Nov 1 12:51:21 2012 still logged in (unknown tty8 Thu Nov 1 12:50:57 2012 - Thu Nov 1 12:51:21 2012 (00:00) pablo tty2 Thu Nov 1 12:50:39 2012 still logged in pablo ...
5
votes
3answers
1k views

Is there a more elegant way to count words and assign that count to variables?

I have a script: #!/bin/bash /root/xiotech status > xiostatus.tmp SyncCount=$(grep -c Sync xiostatus.tmp) PauseCount=$(grep -c paused xiostatus.tmp) CopyingCount=$(grep -c Copying xiostatus.tmp) ...
5
votes
3answers
17k views

How to merge two files based on the matching of two columns?

I have file1 likes: 0 AFFX-SNP-000541 NA 0 AFFX-SNP-002255 NA 1 rs12103 0.6401 1 rs12103_1247494 0.696 1 rs12142199 0.7672 And a file2: 0 AFFX-SNP-000541 1 0 ...
5
votes
3answers
1k views

Deleting Lines if there is a dot in the 5th Column

I have a file and I want to remove all the line if it has a dot in the 5th Column. Input File: sc2/80 20 . A C 80 PASS N=2 F=5;U=4 sc2/60 ...
5
votes
4answers
284 views

How to add each output of awk

How would you perform addition on the output of every line from awk? For example cat foo | awk '{print $1} 42 42 42 I'm looking for the answer to 42+42+42 (126). I'm trying to write a for loop ...
5
votes
4answers
464 views

extract lines according to a particular column

I have a CSV file like CK,ck XYZ,xyz,xyzs ABC,abc,abcs PQR,pqr,pqrs LMN,lmn, IJK,ijk, I have to check if something is written in column 3 except whitespace, then display the whole line.the output ...
5
votes
6answers
402 views

Remove duplicates values within a field

How to remove duplicate (or more) values within a selected field in a file, keeping only one copy? Exemple Input file: A 1,2,3,45,1,8,2,3 B 5,6,6,6,6,6,2,3,7 Expected output: A ...
5
votes
4answers
235 views

How could I simplify this command to only use awk?

awk '/user/ {print $1 }' /etc/userdomains | sed 's/://' the format of /etc/userdomains is domain.tld: user otherdomain.tld: otheruser
5
votes
3answers
405 views

removing redundancy from output columns

What is a *NIX way of removing redundancy in a case where I have pairwise comparisons like these in two columns A B B A A C A D C A D A B C C B A B and B A ...
5
votes
3answers
308 views

Remove duplicate fields in a given column

I'd like to remove from a given column ($2 in the example) the duplicate fields (comma separated). Input file: A 1,2,3,4 B 4,5,6,3 C 2,15 Expected output: A 1,2,3,4 B 5,6 C ...
5
votes
2answers
418 views

what is the meaning of 1 at the end of awk script

I was reading this awk script awk -F"=" '{OFS="=";gsub(",",";",$2)}1' I want to know what is the function of 1 at the end of it
5
votes
2answers
9k views

Compare two file columns

I have long text files with space-delimited fields: cat file1.txt Id leng sal mon 25671 34343 56565 5565 44888 56565 45554 6868 23343 23423 26226 6224 77765 88688 87464 6848 66776 23343 63463 ...
5
votes
1answer
3k views

Replacing multiple lines in sed or awk [closed]

I'm trying to use sed or awk to replace 5 lines in a smb file but I just don't have any idea how to deal with the newlines.
5
votes
4answers
356 views

How to replace all the first occurences in matched lines with incremental number?

I have a file like this ... 1562 first part 1563 H col3 H col4 1564 H col3 H col4 ... 3241 H col3 H col4 3242 third part ... I want to replace only the first H in every line to H#, with ...
5
votes
4answers
233 views

How to replace m part of Lines with n part of Lines?

I have a file "test" like this: RewiteEngine On RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.7$ [NC] RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.8$ [NC] RewriteCond %{REMOTE_ADDR} ...
5
votes
3answers
9k views

awk with if statements

I am trying to print from a file using awk, but my output is empty. Here is my code so far accountNum=$1 while read -r LINE || [[ -n $LINE ]] ; do awk -F',' '{ if($1==accountNum) { print ...
5
votes
2answers
2k views

Replace string with sequential index

Can someone suggest an elegant way to accomplish this? Input: test instant () test instant () ... test instant () //total 1000 lines output should be: test instant1 () test ...
5
votes
1answer
15k views

Replacing string based on line number

I have a situation where i want to replace a particular string in many files Replace a string AAA with another string BBB but there are lot of strings starting with AAA or ending in AAA ,and i want ...
5
votes
2answers
124 views

Why does awk execute both actions?

When I run the command: awk '/from/ {print $7} /to/ { print $7}' erroMuitoDoido.txt The file is: May 19 04:44:43 server postfix/smtpd[32595]: CDAB515013: ...
5
votes
2answers
163 views

How can I delete all unordered lines from a text file?

Overview Consider an ordered list interspersed with unordered elements, e.g.: Alligator Ant Falcon <-- Baboon Badger Armadillo <-- Caiman Cat How can this list be processed so that all ...
5
votes
2answers
2k views

Add thousands separator in a number

In python re.sub(r"(?<=.)(?=(?:...)+$)", ",", stroke ) To split a number by triplets, e.g.: echo 123456789 | python -c 'import sys;import re; print re.sub(r"(?<=.)(?=(?:...)+$)", ",", ...
5
votes
3answers
968 views

Extracting fixed width records with no delimiter from a single line

I need to extract strings of text from a single file containing one very long line of text with no delimiters. Using the sample line below, these are the following known facts: ...
5
votes
4answers
4k views

AWK - how to reduce the greed of a regular expression?

I have an example: echo "@article{gjn2010jucs, Author = {Grzegorz J. Nalepa}, " | awk '{ sub(/@.*,/,""); print }' Is it possible to write a regular expression that selects the shorter pattern? ...
5
votes
2answers
494 views

Sum in block with AWK (restart the sum when the pattern changes)

I have a file like this: A 100 A 200 A 300 #sum=600 B 400 B 500 #sum=900 A 600 A 700 A 800 #sum=2100 I would like the output to be: A 600 B 900 A 2100 C sum_of_C D sum_of_D I can do that with ...
5
votes
4answers
611 views

how can we use awk to get multiple words after a significant word?

I want to connect to my DB only once to do multiple SELECT queries as follows: #!/bin/bash #Begin Code query=$(echo "SELECT COUNT(*) from USER_IPTable; SELECT User from USER_IPTable; ...
5
votes
2answers
132 views

Adding corresponding Numbers in a Range

I have a dataset with 30000 lines. I want to add all the corresponding values (i.e. Column 2) between the range 0-50 and then 50.1-100 and then 100.1-150 etc. (presented in column 1), and this ...
5
votes
2answers
254 views

awk + paste for cleaning up PATH?

I have seen this code in .cshrc init files on a few machines. I went through a few awk tutorials in trying to understand how it works, but I am still unable to decrypt it. setenv PATH `echo $PATH | ...
5
votes
3answers
9k views

How can I use SED or AWK to replace placeholders in a template file with variable content that contains special characters?

CentOS 6.3 I'm trying to get a small script to send an email containing a copy of email headers in the body (for the purpose of internal reporting). The template file contains the following: ...
5
votes
2answers
153 views

How would I add a prefix to input I receive from a pipe via awk and color the prefix conditionally?

Is there a way to add a prefix to lines received (and filtered) via awk - yes I know that's possible so far - and then conditionally (e.g. based on the existence of the $PS1 variable of the ...
5
votes
5answers
1k views

Pattern matching across multiple lines

What is a quick and easy way to validate if the pattern "301 domaname.com 200" is present within BASH when spread across multiple lines of output? I was using something like the following: awk ...
5
votes
1answer
3k views

Sorting a file based on one column using Unix and Awk

I need to sort the input file according the 6th column, which is the score. Input File: Sc2/80 20 . A T 86 Pass N=2 F=5;U=4 Sc2/80 20 . A C 80 Pass N=2 F=5;U=4 Sc2/60 55 . G T 90 Pass N=2 F=5;U=4 ...
5
votes
2answers
702 views

Use a script parameter in awk

Here is my script (to find the files that contain a specified pattern) : find . -type f -exec awk -v vawk="$1" '/'"$vawk"'/ {c++} c>0 {print ARGV[1] ; exit 0 } END { if (! c) {exit 1}}' \{\} \; ...