awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

4
votes
1answer
168 views

Looking for lines which is in one file but not in other using Unix and Awk

I have 2 files with 7 fields and I want to print the lines which are present in file1 but not in file2 based on field1 and field2. Logic: I want to print all the lines, where there is a particular ...
4
votes
3answers
236 views

Check for, and add, missing timestamps to individual lines in a file

I have a file in the following format: ABCD 01206001022T01YA022T01YA022T07SO 09:20:38 ABCD 01206001022ACION 09:24:40 ABCD 04006001021S01UK 09:24:42 ABCD 7878696621321312 23213213213213 ABCD ...
4
votes
3answers
6k views

How can I use SED or AWK to replace placeholders in a template file with variable content that contains special characters?

CentOS 6.3 I'm trying to get a small script to send an email containing a copy of email headers in the body (for the purpose of internal reporting). The template file contains the following: ...
4
votes
1answer
2k views

awk high precision arithmetic

I am looking for a way to tell awk to do high-precision arithmetic in a substitution operation. This involves, reading a field from a file and substituting it with a 1% increment on that value. ...
4
votes
2answers
278 views

only output when first columns are the same

INPUT: [USER@NOTEBOOK ~/proba] find . -type f | xargs -I "{}" md5sum "{}" | sort 18b17ef2e55f6fd2deb044943a8a769d ./yes 2b00042f7481c7b056c4b410d28f33cf ./tt.txt 2b00042f7481c7b056c4b410d28f33cf ...
4
votes
1answer
3k views

escaping a single dot with double backslash - awk

"effective awk programming" book has an example on Field-Splitting. here is the example: If you want fields to be separated by a literal period followed by any single character, use ‘FS = ...
4
votes
2answers
3k views

My awk program to change spaces into tabs doesn't work

How to write a shell script which uses awk to read in the data file students.txt and output the data in the tabbed format as shown: Surname Forename MSc Stream Date of Birth Smith John IT 15.01.1986 ...
4
votes
1answer
199 views

Matching Five Columns in two Files using Awk

I have two Input files. File1: s2/80 20 . A T 86 F=5;U=4 s2/20 10 . G T 90 F=5;U=4 s2/90 60 . C G 30 ...
4
votes
2answers
3k views

recursive search for a pattern, then for each match print out the specific SEQUENCE: line number, file name, and no file contents

What I am after is almost exactly the same as can be found here, but I want the format "line number, separator, filename, newline" in the results, thus displaying the line number at the beginning of ...
4
votes
1answer
2k views

Awk to remove line if argument is encountered in a specific column

I need to cycle through an entire file of unknown size and remove any line in which a given word (passed in as argument 1) appears in a specified column. In addition, I need to keep track of how many ...
4
votes
2answers
536 views

AWK help for filling up rest of the columns with &nbsp in file

I'm trying to get AWK filling up following text file for "empty" columns". Basic idea of finding longest line with AWK NF and checking which line has no value on that column. Then adding x times N/A ...
4
votes
5answers
2k views

extract fields from “tail -f” of a syslog stream

Example line from syslog file: Aug 1 10:25:50 10.10.10.1 id=firewall sn=XXXX time="2012-08-01 14:35:18 UTC" fw=x.x.x.x pri=6 c=1024 m=537 msg="Connection Closed" f=11 n=195273698 ...
4
votes
2answers
623 views

Replace one line with STDIN in a Makefile

One Makefile target is supposed to be in charge of: grabbing the output of $ perl Markdown.pl src/index.md (markdown to html) using that string to replace one line, something like CONTENT ...
4
votes
2answers
840 views

Moving average on a log file with awk or other unix utilities?

Scenario: I have a log file that has a few number of entry "classes", like this: R0 dx=0.00500 rb=0.00000 sn=1 3145.88 2.59 0.08 se=21315 id=16190 R0 dx=0.00300 rb=-1.00000 sn=1 3150.40 2.38 0.05 ...
4
votes
3answers
105 views

awk separator preferences

I write ad hoc awk commands in cygwin to process tab delimited files. Pretty much every command starts with: BEGIN {FS="\t";OFS="\t";ORS="\r\n";} How can I make these separators the default to ...
4
votes
1answer
988 views

Pass colors from ls through pipe to awk 'print' statement

This is a follow-up to my question from yesterday, Show sum of file sizes in directory listing. Thanks to Zero Piraeus and a point in the right direction by Mauritz Hansen, I now have function ...
4
votes
3answers
4k views

How to list all unique ip address currently connected to a specific port?

Say I want to know how many unique clients are connected to port 5222 on a server. Can you find a way better/faster/stronger than this? netstat -nt | grep ':5222.*ESTABLISHED' | awk '{ print $5 }' \ ...
4
votes
2answers
779 views

How to test if a function is defined in awk?

I am writing an awk program and I want it to run it both with mawk and gawk. I need a sorting function. I have written my own, but I want to use the builtin asort() function in gawk if available. How ...
4
votes
5answers
618 views

Multi-line grep beween two tags, only show the last match

I'm having a program that intermingles data segments into log output: log message log message ----BEGIN INLINE DATA---- data data data -----END INLINE DATA----- log messge I'm looking for something ...
4
votes
1answer
278 views

Are there awk versions that provide syntax for computing aggregations?

From time to time I find myself writing awk scripts that compute some simple statistics. For example computing a histogram, the average of a value, the standard deviation or even the variance ... ...
4
votes
6answers
383 views

What is the easiest way to check if column A and B values goes both ways with AWK? [closed]

What is the easiest way to check if column A and B values goes both ways? Output to check: Mike John John Mike Pamela Barbara Barbara Pamela Mike Paul Roger Paul Desired output Mike <-> ...
4
votes
1answer
400 views

Efficient way of comparing in awk

#!/bin/awk BEGIN { while(getline var < compareTo > 0) { orderIds[var]=var; } } { if(orderIds[$0] == "") { print $0; ...
4
votes
1answer
69 views

Insert missing data within blocks

I`m a biologist and I am trying to solve this problem, please guide me on where to start. I do not know which forum to post this in, please let me know if this place is not appropriate. I have blocks ...
3
votes
6answers
1k views

Convert string into date time stamp in gawk or awk

I want to convert following string (20140805234656) into date time stamp (2014-08-05 23:46:56).I am new to gawk and I don't know the exact syntax,how can I put - at every 5,8 and : at every 14,17 and ...
3
votes
5answers
542 views

Print last element of each row

How can one print the element after the last tab in a file? Exemple: File1 A 3 8 6 7 B 4 6 2 3 6 8 c 1 9 would return: 7 8 9
3
votes
7answers
2k views

Problem in splitting a string using awk or cut command

I have a properties file which has a password field (key value pair) PASSWORD=NDhhHcsOBofXUdUzGw5B0Q== I am trying to get the value in my shell script using the awk command password=`awk -F "=" ...
3
votes
6answers
578 views

Add a symbol before any letter

I have a text file with each line having some defined number of fields but they can differ from line to line. All I want to do is to add "=" sign in front of every value in the field in that line. ...
3
votes
3answers
9k views

How to merge two files based on the matching of two columns?

I have file1 likes: 0 AFFX-SNP-000541 NA 0 AFFX-SNP-002255 NA 1 rs12103 0.6401 1 rs12103_1247494 0.696 1 rs12142199 0.7672 And a file2: 0 AFFX-SNP-000541 1 0 ...
3
votes
3answers
1k views

reading entries from multiple files

I have few files like A.txt, B.txt and C.txt, each having a only single number in it i.e A.txt has 45 B.txt has 35 and C.txt has 100. How could I read these entries from each files and report with its ...
3
votes
5answers
178 views

removing extensions in a column

I have a file like this ILMN_1343291 TGTGTTGAGAGCTTCTCAGACTATCCACCTTTGGGTCGCTTTGCTGTTCG NM_001402.5 ILMN_1343295 CTTCAACAGCGACACCCACTCCTCCACCTTTGACGCTGGGGCTGGCATTG NM_002046.3 ILMN_1651209 ...
3
votes
3answers
157 views

Parse multiple sections of data into seperate files

I have a concatenated log file with multiple logs inside that I'm trying to parse out into individual log files (I will later rename them to the date/time of each). Each log is separated by "--- LOG ...
3
votes
2answers
219 views

Delete every Nth line in shell

I'm trying to delete a line after N lines using awk and I can't seem to get it right. The file format is like this YYYYYY XXXXXX XXXXXX YYYYYY XXXXXX XXXXXX The real example would be office3 3 1 ...
3
votes
4answers
222 views

Count how many elements in the nth column in a whole file

Input file: A 1,2,3,4 #length($2)=4 B 1,2 #length($2)=2 C 9,8,7,6,5,4 #length($2)=6 Expected output: 12 #4+2+6 A method like: awk -F '[\t,]' '{print length($2)}' but ...
3
votes
2answers
528 views

Only print rows that have 2 or more fields?

I've got a text file and I'm trying to figure out how to filter out any lines with data in only 1 field, leaving lines with 2 fields. I came across the following code: awk '$NF > 1' but I can't ...
3
votes
4answers
475 views

How to Prefix a column values with an apostrophe ( ' )?

I have a CSV file with multiple columns and 1000's of records, I need to prefix all the values of one of the columns (lets say 2nd column) with an apostrophe ' except in the first line or header line, ...
3
votes
3answers
262 views

How can I delete all text between curly brackets in a multiline text file?

Example: This is { the multiline text file } that wants { to be changed } anyway. Should become: This is that wants anyway. I have found some similar threads in the forum, but they don't seem ...
3
votes
4answers
288 views

Split the input file with line breaks

I am trying to insert line breaks on a file based on the same names which happens to be the second field in my file. So, my input file is like below. 17412193|name1|organization ...
3
votes
3answers
198 views

awk manipulation of file

I have a big file in this format, whose first few lines I am showing you including the header. gene c1 c2 c3 c4 c5 c6 c7 c8 G1*1 0 0 0 0 0 0 0 0 G2*2 0 0 0 0 1 ...
3
votes
3answers
253 views

Calculate and divide by total with AWK

Given the following data file... foo 10 bar 20 oof 50 rab 20 ... how would I print column two as a percent of the total of column two? In other words, I want... foo 10 10% ...
3
votes
3answers
549 views

calculate sum of squares using shell script in perl/awk

I have 2 files as below. file1 0.34 0.27 0.32 file2 0.15 0.21 0.15 Now, I would like to calculate the sum of squares between each column. For example, [(0.34 - 0.15)^2 + (0.27 - 0.21)^2 + ...
3
votes
4answers
1k views

Replacing pattern after nth match is found on each line?

I have a file containing lines: india;austria;japan;chile china;US;nigeria;mexico;russia I want to replace all the occurences of semicolon on each line with e.g. ;NEW;, but starting from the 2nd ...
3
votes
2answers
229 views

awk: Extracting a fixed number of rows where the last row number may vary

This question may look like a duplicate, but only at first glance. Of course, I would no longer need help in how to code a one-liner that extracts a fixed number of continuous lines (e. g. 5 in this ...
3
votes
1answer
330 views

Fastest `uniq` tool in linux

I have large text file (1.5 G), I want to know what the fastest and more reliable tool in linux is. I usualy use: awk '!x[$0]++' file.txt But when I use htop command I see my memory usage is ...
3
votes
4answers
191 views

Identifying genes from a list of genes

I have a gene list file. Some thing like this SWT21 SSA1 NRP1 EFB1 TFC3 MDM10 I have another file which also contains the names of these genes in my list along with other ...
3
votes
3answers
268 views

Easily get a particular column from output without sed or awk

Is there a quicker way of getting a couple of column of values than futzing with sed and awk? For instance, if I have the output of ls -hal / and I want to just get the file and directory names and ...
3
votes
1answer
2k views

Awk - how to compare two numbers?

This example works fine: awk -v num1=5999 -v num2=5999 'BEGIN{ print (num2==num1) ? "equal" : "not equal" }' equal This example does not work well: awk -v num1=59558711052462309110012 -v ...
3
votes
2answers
3k views

How to apply the same awk action to different files?

I am new in awk and I do not know if it is possible to write an awk script that does this: I have hundred of data files that I have to sort. For each one I use the following one-liner: awk ...
3
votes
2answers
178 views

Shell script to assign values from a lookup table is too slow

I'm trying to assign some variables from a lookup file with a shell script. I have something working but it seems unnecessary slow. Script: while read line do ...
3
votes
2answers
3k views

Extract file name from path in awk program

I have an awk script and I have passed a CSV file to it. awk -f script.awk /home/abc/imp/asgd.csv What am I doing is to get FILENAME within script.awk. FILENAME gives me the whole path. As I am in ...
3
votes
5answers
207 views

Merging Two Files columns in order

I want to merge two files. Files A.txt 001;abc;def;ghi;jkl;pqr 002;abc;def;ghi;jkl;pqr 003;abc;def;ghi;jkl;pqr 004;abc;def;ghi;jkl;pqr 005;abc;def;ghi;jkl;pqr . Second File B.txt 001;mno 002;mno ...