awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

6
votes
5answers
281 views

Using awk to identify the number identical columns

I am new to all this, and this is probably a rather simple question, but I am stuck: I have a large number of individual files that contain six columns each (number of rows can vary). As a simple ...
6
votes
4answers
3k views

Split long output into two columns

Is there a simple utility or script to columnate the output from one of my scripts? I have data in some form: A aldkhasdfljhaf B klajsdfhalsdfh C salkjsdjkladdag D lseuiorlhisnflkc E sdjklfhnslkdfhn ...
6
votes
2answers
9k views

Compare two file columns

I have long text files with space-delimited fields: cat file1.txt Id leng sal mon 25671 34343 56565 5565 44888 56565 45554 6868 23343 23423 26226 6224 77765 88688 87464 6848 66776 23343 63463 ...
6
votes
3answers
3k views

AWK: wrap lines to 72 characters

$ awk 'length > 72' {HOW TO PRINT THE LINEs IN PCS?} msg ie I want it to add \n after 72 chars and continue, so initially you may need to remove all single \ns and the add them. It may be easier ...
6
votes
3answers
582 views

sum columns based on matching fields

I have encountered a problem that exceeds my basic unix knowledge and would really appreciate some help. I have a large file in the following format: 2 1019 0 12 2 1019 3 0 2 1021 0 2 ...
6
votes
1answer
2k views

Delete an array in awk

In awk, I can clear an array with a loop, making it an empty array, which is equivalent to deleting it. for (key in array) delete array[key]; Is there a simpler way? Can I completely delete an ...
6
votes
3answers
1k views

Extracting fixed width records with no delimiter from a single line

I need to extract strings of text from a single file containing one very long line of text with no delimiters. Using the sample line below, these are the following known facts: ...
6
votes
5answers
2k views

Find images by size: find / file / awk

I've been trying to find png image files a certain height (over 500px). I know that file will return image dimensions. Example: $ file TestImg1a.png TestImg1a.png: PNG image data, 764 x 200, ...
6
votes
2answers
4k views

awk: forcing a return status?

This is a followup to my earlier question. I am validating the number of fields in /etc/passwd using this handy snippit. In the following example, the users 'fieldcount1' and 'fieldcount2' have the ...
6
votes
3answers
4k views

How to subtract rows (lines) with AWK

I'm trying to figure out how I can use AWK to subtract lines. For example, imagine the input file is: 30 20 The output would be: 10 Now, as a test I am trying to calculate the "Used" memory ...
6
votes
3answers
19k views

Adding a Column of values in a tab delimited file

How can I add a Column of values in a file which has a certain number of rows. I have a input file like this: Input file: SPATA17 1 217947738 LYPLAL1 1 219383905 FAM47E 4 77192838 SHROOM3 4 ...
6
votes
2answers
5k views

Replacing missing value blank space with zero

I have input.txt tab-delimited text file around 30K lines, I would like to check each row (s1..s30K lines) for missing value (i.e blank space) and fill the missing value with zero value.See out.txt ...
6
votes
2answers
969 views

How would I sort these directory names numerically?

I'm aware that I can somehow sort this output numerically (so cpu1/ follows cpu0/) ... I could probably get something to work eventually by splitting up the string various ways with awk, but is there ...
6
votes
2answers
1k views

Number of comma-separated fields in a text file

I'm trying to build an awk statement to read this file: A 1,2,3 * A 4,5,6 ** B 1 B 4,5 * and build a file like this: A 1,2,3 * 3 1 0.333 A 4,5,6 ** 3 2 0.666 B ...
6
votes
3answers
1k views

Replace multiple strings in a single pass

I'm looking for a way to replace placeholder strings in a template file with concrete values, with common Unix tools (bash, sed, awk, maybe perl). It is important that the replacement is done in a ...
6
votes
4answers
2k views

find length of longest line in all text files in a directory

I know how to get length of the longest line in a text file with awk awk ' { if ( length > L ) { L=length} }END{ print L}' file.txt but how can I get the length of the longest line of all files ...
6
votes
2answers
381 views

how to add command line calculator to bash that uses comma as decimal mark?

How can I add a command line calculator to my bash? I have found some, but all of them use the full stop as decimal mark, but I want to have it to use the comma as decimal mark as most of the world ...
6
votes
3answers
8k views

Filter a .CSV file based on the 5th column values of a file and print those records into a new file

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , ...
6
votes
1answer
115 views

How can I extract/change lines in a text file whose data are separated into fields?

How can I manipulate field-based data from the commandline? For example How can I print only lines whose Nth field is foo? How can I print only lines whose Nth field isn't foo? How can I print only ...
6
votes
2answers
1k views

Inotifywait for large number of files in a directory

What I want to do is, to monitor a directory (not recursive, just one) for new files created and append those files to one single big file as they are being written. The number of files that are ...
6
votes
2answers
295 views

Grepping for a block of text with parts that can be optional

I have multiple entries that describes an event in a very large log file, say A.log. I would like to do two things with the event entries in the log file: Count the number of occurrences of each ...
6
votes
2answers
3k views

Use awk to insert a line after N output

I have a text file filled with a bunch of data. I need to pull out lines that contain specific strings. I accomplished this with awk using the following: awk '/pattern1|pattern2|pattern3/ {print;}' ...
6
votes
6answers
300 views

How can I avoid escaped delimiter characters in a Unix command?

I need to take the sum of a field which is tilde (~) delimited. Problem I have is that my data is also having delimiter escaped. Example 1~CEO~ashok\~kumar~1000 As we see in 3rd field above we ...
6
votes
1answer
128 views

Count duplicate rows with constraint

I have the following file: A B 1990 A C 2001 A C 2013 A B 2001 B C 2013 A D 2001 A D 2014 I need to count duplicate pairs (in the 1st and 2nd column) and to each such pair assign the lowest value ...
5
votes
8answers
1k views

How do I delete the first n lines and last line of a file using shell commands?

I have a file named Element_query containing the result of a query : SQL> select count (*) from element; [Output of the query which I want to keep in my file] SQL> spool off; I want to ...
5
votes
4answers
1k views

How to append end of every line with the line itself

This is in file.txt: redcar bluecar greencar Im looking for ways to make it become: redcar redcar bluecar bluecar greencar greencar I've tried many ways using sed with no luck
5
votes
6answers
712 views

split file into two parts, at a pattern

How to split a large file into two parts, at a pattern? Given an example file.txt: ABC EFG XYZ HIJ KNL I want to split this file at XYZ such that file1 contains lines up-to XYZ and rest of the ...
5
votes
4answers
486 views

Reduce the over use of files in text processing

Currently, I have a file (Results.txt) which has 27k rows. The file content is as below. 45008657 | A. | | Long | 0 49210987 | A. | ...
5
votes
3answers
880 views

How to understand the command awk '{for(x=1;$x;++x)print $x}'?

I know this command awk '{for(x=1;$x;++x)print $x}' will print out all columns in a line. wouldn't this ++x change x to 2, and thus print $2 first? As I understood based on this: ...
5
votes
5answers
15k views

extract the 4th column from a csv file using unix command

I have a csv file formatted as below. "col1","col2","col3","col4" "1","text1","<p>big html text</p> ","4th column" "2","text2","<p>big2 html2 text2</p> ...
5
votes
5answers
888 views

Why does awk not sum to 0

I have this file and I want to sum all the number in first column. Easy: awk '{s+=$1;print $1,s}' file -0.4157 -0.4157 0.0143 -0.4014 0.5973 0.1959 -0.5679 -0.372 -0.2041 -0.5761 0.7130 0.1369 ...
5
votes
3answers
21k views

How to merge two files based on the matching of two columns?

I have file1 likes: 0 AFFX-SNP-000541 NA 0 AFFX-SNP-002255 NA 1 rs12103 0.6401 1 rs12103_1247494 0.696 1 rs12142199 0.7672 And a file2: 0 AFFX-SNP-000541 1 0 ...
5
votes
4answers
3k views

number parsing in awk

I have a text file with numbers in second column and the numbers have thousand-separator and decimal-separator: 2014-01 2,277.40 2014-02 2,282.20 2014-03 3,047.90 2014-04 4,127.60 ...
5
votes
5answers
4k views

Convert formatted dates to seconds since the epoch

I have a file: pablo tty8 Thu Nov 1 12:51:21 2012 still logged in (unknown tty8 Thu Nov 1 12:50:57 2012 - Thu Nov 1 12:51:21 2012 (00:00) pablo tty2 Thu Nov 1 12:50:39 2012 still logged in pablo ...
5
votes
3answers
1k views

Is there a more elegant way to count words and assign that count to variables?

I have a script: #!/bin/bash /root/xiotech status > xiostatus.tmp SyncCount=$(grep -c Sync xiostatus.tmp) PauseCount=$(grep -c paused xiostatus.tmp) CopyingCount=$(grep -c Copying xiostatus.tmp) ...
5
votes
7answers
2k views

can we print the last word of each line in linux using sed command?

suppose , if there is a file consisting of following lines , if they are 12345 567 7878 66 er3 t45t y6y46y 4y6 y656y y5y 46y6 65y7 y66uyuy yy46y6y The output has to look like: 66 ...
5
votes
6answers
377 views

sed command to swap characters

My input file layout is: mm/dd/yyyy,hh:mm,other fields I need to format it as: yyyy-mm-dd hh:mm:00,other fields sample input: 01/02/1998,09:30,0.4571,0.4613,0.4529,0.4592,6042175 ...
5
votes
3answers
1k views

Deleting Lines if there is a dot in the 5th Column

I have a file and I want to remove all the line if it has a dot in the 5th Column. Input File: sc2/80 20 . A C 80 PASS N=2 F=5;U=4 sc2/60 ...
5
votes
5answers
182 views

Reformat each two-line sequence into two columns

I would like to tab my printout: 1 2 3 4 5 6 7 8 to 1 2 5 6 3 4 7 8 How to adjust it using simple command line (awk or sed)??
5
votes
4answers
325 views

How to add each output of awk

How would you perform addition on the output of every line from awk? For example cat foo | awk '{print $1} 42 42 42 I'm looking for the answer to 42+42+42 (126). I'm trying to write a for loop ...
5
votes
3answers
1k views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
5
votes
4answers
531 views

extract lines according to a particular column

I have a CSV file like CK,ck XYZ,xyz,xyzs ABC,abc,abcs PQR,pqr,pqrs LMN,lmn, IJK,ijk, I have to check if something is written in column 3 except whitespace, then display the whole line.the output ...
5
votes
5answers
225 views

Merging columns in a file using Awk

Input: ABC,SSSD,12345,NSS,12345,xxx,TS11 ,,,,,,TS21 ,,,,,,TS22 ,,,,,,BS26 ,,,,,,GPRS ABC,SSSD,12356,NSS,12356,xxx,TS11 ,,,,,,TS21 ,,,,,,TS22 ,,,,,,GPRS ,,,,,,BS26 Output: ...
5
votes
6answers
441 views

Remove duplicates values within a field

How to remove duplicate (or more) values within a selected field in a file, keeping only one copy? Exemple Input file: A 1,2,3,45,1,8,2,3 B 5,6,6,6,6,6,2,3,7 Expected output: A ...
5
votes
5answers
7k views

How can I convert tab delimited data to comma delimited data?

I'm requesting a list of ec2 snapshots via amazon's ec2 command line tool: ec2-describe-snapshots -H --hide-tags > snapshots.csv The data looks something like this: SnapshotId VolumeId ...
5
votes
4answers
237 views

How could I simplify this command to only use awk?

awk '/user/ {print $1 }' /etc/userdomains | sed 's/://' the format of /etc/userdomains is domain.tld: user otherdomain.tld: otheruser
5
votes
3answers
11k views

How to replace the content of specific column with awk?

Given there is 40 columns in a record. I want to replace 35th column so that the 35th column will be replace with the content of 35th column and a "$" symbol. What came up my mind is like: awk ...
5
votes
3answers
439 views

removing redundancy from output columns

What is a *NIX way of removing redundancy in a case where I have pairwise comparisons like these in two columns A B B A A C A D C A D A B C C B A B and B A ...
5
votes
3answers
328 views

Remove duplicate fields in a given column

I'd like to remove from a given column ($2 in the example) the duplicate fields (comma separated). Input file: A 1,2,3,4 B 4,5,6,3 C 2,15 Expected output: A 1,2,3,4 B 5,6 C ...
5
votes
3answers
352 views

Join: Two files - but only append last two columns

Given the files: 1.txt 1, abc, 123, 456, 789 2, lmn, 123, 456, 789 3, pqr, 123, 456, 789 2.txt 1, abc, 123, 000, 000 3, lmn, 123, 000, 000 9, opq, 123, 000, 000 OUTPUT.txt ID, NAME, X, ...