awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

4
votes
1answer
879 views

Pass colors from ls through pipe to awk 'print' statement

This is a follow-up to my question from yesterday, Show sum of file sizes in directory listing. Thanks to Zero Piraeus and a point in the right direction by Mauritz Hansen, I now have function ...
4
votes
3answers
4k views

How to list all unique ip address currently connected to a specific port?

Say I want to know how many unique clients are connected to port 5222 on a server. Can you find a way better/faster/stronger than this? netstat -nt | grep ':5222.*ESTABLISHED' | awk '{ print $5 }' \ ...
4
votes
2answers
737 views

How to test if a function is defined in awk?

I am writing an awk program and I want it to run it both with mawk and gawk. I need a sorting function. I have written my own, but I want to use the builtin asort() function in gawk if available. How ...
4
votes
5answers
489 views

Multi-line grep beween two tags, only show the last match

I'm having a program that intermingles data segments into log output: log message log message ----BEGIN INLINE DATA---- data data data -----END INLINE DATA----- log messge I'm looking for something ...
4
votes
1answer
260 views

Are there awk versions that provide syntax for computing aggregations?

From time to time I find myself writing awk scripts that compute some simple statistics. For example computing a histogram, the average of a value, the standard deviation or even the variance ... ...
4
votes
6answers
370 views

What is the easiest way to check if column A and B values goes both ways with AWK? [closed]

What is the easiest way to check if column A and B values goes both ways? Output to check: Mike John John Mike Pamela Barbara Barbara Pamela Mike Paul Roger Paul Desired output Mike <-> ...
4
votes
1answer
375 views

Efficient way of comparing in awk

#!/bin/awk BEGIN { while(getline var < compareTo > 0) { orderIds[var]=var; } } { if(orderIds[$0] == "") { print $0; ...
3
votes
6answers
687 views

Convert string into date time stamp in gawk or awk

I want to convert following string (20140805234656) into date time stamp (2014-08-05 23:46:56).I am new to gawk and I don't know the exact syntax,how can I put - at every 5,8 and : at every 14,17 and ...
3
votes
5answers
390 views

Print last element of each row

How can one print the element after the last tab in a file? Exemple: File1 A 3 8 6 7 B 4 6 2 3 6 8 c 1 9 would return: 7 8 9
3
votes
7answers
2k views

Problem in splitting a string using awk or cut command

I have a properties file which has a password field (key value pair) PASSWORD=NDhhHcsOBofXUdUzGw5B0Q== I am trying to get the value in my shell script using the awk command password=`awk -F "=" ...
3
votes
6answers
477 views

Add a symbol before any letter

I have a text file with each line having some defined number of fields but they can differ from line to line. All I want to do is to add "=" sign in front of every value in the field in that line. ...
3
votes
3answers
1k views

reading entries from multiple files

I have few files like A.txt, B.txt and C.txt, each having a only single number in it i.e A.txt has 45 B.txt has 35 and C.txt has 100. How could I read these entries from each files and report with its ...
3
votes
3answers
129 views

Parse multiple sections of data into seperate files

I have a concatenated log file with multiple logs inside that I'm trying to parse out into individual log files (I will later rename them to the date/time of each). Each log is separated by "--- LOG ...
3
votes
4answers
188 views

Count how many elements in the nth column in a whole file

Input file: A 1,2,3,4 #length($2)=4 B 1,2 #length($2)=2 C 9,8,7,6,5,4 #length($2)=6 Expected output: 12 #4+2+6 A method like: awk -F '[\t,]' '{print length($2)}' but ...
3
votes
2answers
378 views

Only print rows that have 2 or more fields?

I've got a text file and I'm trying to figure out how to filter out any lines with data in only 1 field, leaving lines with 2 fields. I came across the following code: awk '$NF > 1' but I can't ...
3
votes
4answers
438 views

How to Prefix a column values with an apostrophe ( ' )?

I have a CSV file with multiple columns and 1000's of records, I need to prefix all the values of one of the columns (lets say 2nd column) with an apostrophe ' except in the first line or header line, ...
3
votes
6answers
1k views

Remove lines that do not start with a pattern from a given set of patterns

I have a file that contains data like this: report aaaaaaaa - .. -th bbbbbbbbb -to ccccccccc .. --. Question: I want to remove any line that does not start with following strings : report ...
3
votes
4answers
280 views

Split the input file with line breaks

I am trying to insert line breaks on a file based on the same names which happens to be the second field in my file. So, my input file is like below. 17412193|name1|organization ...
3
votes
3answers
389 views

calculate sum of squares using shell script in perl/awk

I have 2 files as below. file1 0.34 0.27 0.32 file2 0.15 0.21 0.15 Now, I would like to calculate the sum of squares between each column. For example, [(0.34 - 0.15)^2 + (0.27 - 0.21)^2 + ...
3
votes
4answers
1k views

Replacing pattern after nth match is found on each line?

I have a file containing lines: india;austria;japan;chile china;US;nigeria;mexico;russia I want to replace all the occurences of semicolon on each line with e.g. ;NEW;, but starting from the 2nd ...
3
votes
2answers
195 views

awk: Extracting a fixed number of rows where the last row number may vary

This question may look like a duplicate, but only at first glance. Of course, I would no longer need help in how to code a one-liner that extracts a fixed number of continuous lines (e. g. 5 in this ...
3
votes
1answer
256 views

Fastest `uniq` tool in linux

I have large text file (1.5 G), I want to know what the fastest and more reliable tool in linux is. I usualy use: awk '!x[$0]++' file.txt But when I use htop command I see my memory usage is ...
3
votes
3answers
9k views

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": ...
3
votes
4answers
167 views

Identifying genes from a list of genes

I have a gene list file. Some thing like this SWT21 SSA1 NRP1 EFB1 TFC3 MDM10 I have another file which also contains the names of these genes in my list along with other ...
3
votes
3answers
184 views

Easily get a particular column from output without sed or awk

Is there a quicker way of getting a couple of column of values than futzing with sed and awk? For instance, if I have the output of ls -hal / and I want to just get the file and directory names and ...
3
votes
1answer
1k views

Awk - how to compare two numbers?

This example works fine: awk -v num1=5999 -v num2=5999 'BEGIN{ print (num2==num1) ? "equal" : "not equal" }' equal This example does not work well: awk -v num1=59558711052462309110012 -v ...
3
votes
2answers
3k views

How to apply the same awk action to different files?

I am new in awk and I do not know if it is possible to write an awk script that does this: I have hundred of data files that I have to sort. For each one I use the following one-liner: awk ...
3
votes
3answers
333 views

awk filtering live capture with file as pattern list

What I'm trying to do is quite simple. I'm generating output from tshark and redirecting it to awk with a pipe |. Since tshark gets live data, I want that awk search in every output for a pattern ...
3
votes
5answers
182 views

Merging Two Files columns in order

I want to merge two files. Files A.txt 001;abc;def;ghi;jkl;pqr 002;abc;def;ghi;jkl;pqr 003;abc;def;ghi;jkl;pqr 004;abc;def;ghi;jkl;pqr 005;abc;def;ghi;jkl;pqr . Second File B.txt 001;mno 002;mno ...
3
votes
2answers
718 views

Bash Script To Find Debian Release Number from /etc/debian_version

The following is what I'm currently using for testing the version of Debian. Instead of doing it this way, can I make it more succinct? #!/bin/bash VERSION=NONE if [ $(cat /etc/debian_version) == ...
3
votes
2answers
210 views

Join: Two files - but only append last two columns

Given the files: 1.txt 1, abc, 123, 456, 789 2, lmn, 123, 456, 789 3, pqr, 123, 456, 789 2.txt 1, abc, 123, 000, 000 3, lmn, 123, 000, 000 9, opq, 123, 000, 000 OUTPUT.txt ID, NAME, X, ...
3
votes
2answers
372 views

Search a string with last two characters as numeric

Suppose I have a file like this: adasfddfd09 dsassd90897 323sdsdsdsd sdddsdf56 dfdf45fdfdf I want to search only those strings which have last two characters numeric with sed, grep and awk.
3
votes
2answers
1k views

Extract file name from path in awk program

I have an awk script and I have passed a CSV file to it. awk -f script.awk /home/abc/imp/asgd.csv What am I doing is to get FILENAME within script.awk. FILENAME gives me the whole path. As I am in ...
3
votes
3answers
230 views

Rearrange text file by prepending data rows with header row

I have a file whose content looks like this: 2009 150 0 0 0 0000 75.316 0.0400390625 0.00007 0.00000 0.8980 76.216 0.0400390625 0.00007 1.00000 0.9046 77.217 0.0400390625 ...
3
votes
1answer
786 views

grep on line before the match plus the match

zzzzzzzzz aaaaaaaaa bbbbbbbbb & ccccccccc & ddddddddd hhhhhhhhh eeeeeeeee fffffffff & ggggggggg & in the above line, what i want is to grep/sed/awk (any method is fine) line that ...
3
votes
3answers
177 views

checking data in columns when a data or some may be missing or present?

I am not sure if this possible. say i have columns like : Team Colour Game Rainfall PlayerName XYZ Blue Cricket Yes Kapil suppose i need to search ...
3
votes
1answer
503 views

Output of `java -version` not matched by grep or awk

On my Linux machine, it isn't clear to me why if I do the following then I don't get only the version string ("1.5.0_32"). # java -version | grep version | awk '{print $NF}' java version "1.5.0_32" ...
3
votes
5answers
640 views

changing pattern of a text file

A text file has contents something like chair table pen desk Now i want it to be changed and stored in a variable say var as below ('chair','chair'),('table','table'),('pen','pen'),('desk','desk') ...
3
votes
3answers
2k views

Awk: check for length of field

In awk. I am working on Solaris 10, so it's probably an old(er) version of awk. I came up with this rudimentary one-liner that works, at least for my particular input. awk -F\; '$3 ~ /[ ...
3
votes
4answers
561 views

Find a string (like grep -q) within only a section of a file

I want to write some Bash that can verify that a string exists in a configuration file. I can't change the file format, it belongs to a different application. The file is subdivided into groups named ...
3
votes
2answers
2k views

How would I extract the user agent strings from a log file?

Currently I am running a command like this, to get the most requested content: grep "17\/Jul\/2011" other_vhosts_access.log | awk '{print $8}' | sort | uniq -c | sort -nr I want to now see the user ...
3
votes
3answers
132 views

Count records matching pattern with Awk

So I was told to do last > lastloggedin which creates a file that shows the classes last login since the last system reboot, and now I am asked to write an Awk script which is named myawk that ...
3
votes
3answers
408 views

AWK: Passing shell variables to awk

I am trying to pass a variable number of arguments from the shell script to a pattern recognition subset of a table. Here is my attempt so far: The file 'infile': ID,GROUP 1,GROUP2 ...
3
votes
4answers
345 views

extract the characters before () using grep

I need to extract the characters before the character '('. My file structure is: rose(good)ook daisy()no lilly(bad)fine sunflower()nice I need output file as in the form of: rose ...
3
votes
3answers
1k views

awk remove the rows based on another column value

I have a file like below which I have sorted based on the username field. UserID score UserName 1234 200 Jack 5678 150 Jill 8543 200 Jill 5678 100 John I am trying to remove ...
3
votes
2answers
104 views

Command in variable one are affecting output of other variable when displayed together in UNIX

I am having two file naming Extra_info and Class_info Class_info contains data : stuId stuName stuClass S01 jagdeep First S02 harsh First S03 Roop Second S04 Kali Third ...
3
votes
3answers
268 views

Print unique of who command

I want to show the online users using the who command but I want to have a unique output and not show any duplicates. I piped the output into awk but I am not very familiar with that, is this the ...
3
votes
4answers
1k views

Align text to center with padding on both sides

Is there an easy way to align text to the center with padding on both sides, with the column width being the longest line from the input? For example, this: aaa bbbb c ddddd ee will turn into this ...
3
votes
5answers
188 views

grep device name and look for next value :

I have this output from a find command: abc,10.11.13.14,def,1.2.3.4,geh,6.7.54.23 where abc,def and geh are device names and could be of any length and others are IP address belong to devices. ...
3
votes
3answers
2k views

Average rows with same first column

Given a file with two columns: Id ht 510 69 510 67 510 65 5l0 62 510 59 601 29 601 26 601 21 601 20 I need a way to coalesce all rows with the same ID into one that has an average height. In this ...