awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

4
votes
3answers
3k views

Diff, show modified line from right file only

I have file a and b and I would like to output lines of b that changed since it was cloned from a. Just the modified lines, no surrounding context, no diff offset marks. How can I do that using shell ...
4
votes
2answers
3k views

My awk program to change spaces into tabs doesn't work

How to write a shell script which uses awk to read in the data file students.txt and output the data in the tabbed format as shown: Surname Forename MSc Stream Date of Birth Smith John IT 15.01.1986 ...
3
votes
3answers
198 views

awk manipulation of file

I have a big file in this format, whose first few lines I am showing you including the header. gene c1 c2 c3 c4 c5 c6 c7 c8 G1*1 0 0 0 0 0 0 0 0 G2*2 0 0 0 0 1 ...
3
votes
1answer
516 views

Efficient way to print lines from a massive file using awk, sed, or something else? [duplicate]

If I had a plain-text file containing 8 million lines and I want to print out lines 4,000,010 to 4,000,000 to the screen, which would be more efficient: awk or sed? There is no pattern to the text, ...
2
votes
3answers
228 views

List all word pairs that appear in a line together in a file

How can I use awk to run through a file written as: Sarah,Masha,Helen Connor,Jessica,Jennifer,Candace Betty,Sarah,Helen John,Harold,Frank,Daisy and put out all pairs that appeared in a line ...
2
votes
4answers
794 views

Format a Date field of a .CSV file with multiple commas in a string field

I have a .CSV file (file.csv) whose data are all enclosed in double quotes. Sample format of the file is as below: column1,column2,column3,column4,column5,column6, column7, Column8, Column9, Column10 ...
2
votes
1answer
2k views

Diff the output of two `awk` commands

I'm trying to compute the difference between the output of two awk commands but my simple attempts at it seem to be failing. Here is what I'm trying: diff $(awk '{print $3}' f1.txt | sort -u) $(awk ...
2
votes
4answers
1k views

gawk join two TSV by columns (a'la sql join)

How to join, to tsv files, examples: a.tsv c 7 r z d 6 s w f 1 f f b 8 p y a 9 q x b.tsv a q a c r ccc b p bb 0 0 0 d s dddd Here I'd like to ...
1
vote
2answers
68 views

Compare and print data from different text files to one in Shell

How can I compare and print data from different text files to one in Shell. I have captures NAS details of three different boxes using SSH, now I need to combine all the three text files to one file ...
1
vote
4answers
160 views

Sed/awk/perl: reversing order of comma separated values, preserving other text

I have this text: LABEL1 .BYTE 01, 02, 03, 04, 05 .BYTE 01, 02, 03 I need to reverse only comma separated values order: LABEL1 .BYTE 05, 04, 03, 02, 01 .BYTE 03, 02, 01 I need ...
8
votes
2answers
226 views

Are there 2 ways to set awk vars via command line?

I noticed an O'Reilly awk example (1997) which assigned an awk variable by setting it on the command line after the program-text. It does work, but I can't find this syntax in man / info awk. Have I ...
8
votes
3answers
4k views

Why does awk do full buffering when reading from a pipe

I'm reading from a serial port connected to a gps device sending nmea strings. A simplified invocation to illustrate my point: $ awk '{ print $0 }' /dev/ttyPSC9 ...
7
votes
3answers
3k views

What are the exact differences between awk and cut with grep? [closed]

We know that we can get the second column of the line we want from a file using these two techniques: awk '/WORD/ { print $2 }' filename or grep WORD filename| cut -f 2 -d ' ' My questions are: ...
6
votes
7answers
500 views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
6
votes
2answers
645 views

How would I sort these directory names numerically?

I'm aware that I can somehow sort this output numerically (so cpu1/ follows cpu0/) ... I could probably get something to work eventually by splitting up the string various ways with awk, but is there ...
5
votes
5answers
207 views

Replace data at specific positions in txt file using data from another file

I have a text file in the below format: $data This is the experimental data good data This is good file datafile 1 4324 3673 6.2e+11 7687 67576 2 3565 8768 8760 5780 8778 "This is line ...
5
votes
5answers
729 views

Pattern matching across multiple lines

What is a quick and easy way to validate if the pattern "301 domaname.com 200" is present within BASH when spread across multiple lines of output? I was using something like the following: awk ...
5
votes
4answers
200 views

How to add each output of awk

How would you perform addition on the output of every line from awk? For example cat foo | awk '{print $1} 42 42 42 I'm looking for the answer to 42+42+42 (126). I'm trying to write a for loop ...
5
votes
2answers
937 views

Inotifywait for large number of files in a directory

What I want to do is, to monitor a directory (not recursive, just one) for new files created and append those files to one single big file as they are being written. The number of files that are ...
4
votes
3answers
105 views

awk separator preferences

I write ad hoc awk commands in cygwin to process tab delimited files. Pretty much every command starts with: BEGIN {FS="\t";OFS="\t";ORS="\r\n";} How can I make these separators the default to ...
4
votes
1answer
988 views

Pass colors from ls through pipe to awk 'print' statement

This is a follow-up to my question from yesterday, Show sum of file sizes in directory listing. Thanks to Zero Piraeus and a point in the right direction by Mauritz Hansen, I now have function ...
4
votes
2answers
536 views

AWK help for filling up rest of the columns with &nbsp in file

I'm trying to get AWK filling up following text file for "empty" columns". Basic idea of finding longest line with AWK NF and checking which line has no value on that column. Then adding x times N/A ...
4
votes
3answers
407 views

How to parse a file to extract 3 digits numbers kept in a “group number”

I am a beginner, and trying to write a shell script to parse a text file that is extracted from a standardization pdf file. I would like for each test group (identified by Group 0, Group 1... to get ...
3
votes
3answers
262 views

How can I delete all text between curly brackets in a multiline text file?

Example: This is { the multiline text file } that wants { to be changed } anyway. Should become: This is that wants anyway. I have found some similar threads in the forum, but they don't seem ...
3
votes
4answers
958 views

merge 2 rows based on the same column values

I have a file like below. 47196436 47723284 name1 1.77273 42672249 52856963 name2 1.06061 52856963 430695 name2 1.16667 55094959 380983 name3 1.55613 17926380 55584836 name4 1.02461 3213456 34211 ...
3
votes
2answers
1k views

What is the best way to find a list of several strings within a large text file

The short, general question is: In Unix/Linux, what is the best way to find a list of several (about 150) strings within a large text file? I am asking this to all Unix/Linux experts as a general ...
3
votes
1answer
288 views

grep list of names and information from bigger file

I have two files: one with list of names (500 entries) and other having some more information for each entry in A.txt and extra entries too. File A.txt (each line is starting with > (fasta format) ...
3
votes
3answers
2k views

Using sed to edit lines in a file with a variable

I need to edit lines in a file using sed. Now the problem is I am replacing a particular pattern with a combination of text and number. This number is a variable which keeps on incrementing for every ...
2
votes
2answers
63 views

How to wrap the comma delimited values to the next line without disturbing the other fields using awk or ksh

I have the following input, in which some of the values are comma delimited. I would like decouple each value and print it in the new line. I tried many ways in awk, but no luck. If I have three ...
2
votes
2answers
119 views

Merging two tables including multiple ocurrence of column identifiers

I would like to merge two tables: File 1: 1 today 2 tomorrow 3 red File 2: 1 a lot 1 sometimes 2 at work 2 at home 2 sometimes 3 new Desired output (file 3): 1 today ...
2
votes
2answers
265 views

Merging columns from two separate files

How to create a new file merging selective columns from two separate files using awk? Without messing up the elements orders of BOTH files. Exemple: File 3 may contain column 1,2,3 from File 1 and ...
2
votes
3answers
199 views

replace a character from a string that is in between first and second search using sed or awk

I need to replace _ (underscore) with ? (question mark) in a file containing email addresses. The file looks like below: EFT_020034-E015133 20140624 /ACC/EMAIL+test_1@mysite.com SHR ...
2
votes
3answers
2k views

Comparing text files using Bash and AWK

I have two text files, and I want to compare their corresponding values according to their rows and columns. By comparing, I mean to check if the values are equal and echo if the values are the same ...
2
votes
1answer
487 views

Replace semicolons to commas between the quotes only [duplicate]

Possible Duplicate: Remove comma between the quotes only I'm trying to use awk function to replace the semicolons between quotes to comma and it is 98% ok for me, because I want to keep the ...
2
votes
1answer
2k views

Time/Date grep for 15 mins

I need to grep a specific type of line in a specific time of the log, do you have any ideas? Here's my working script: cat *.log |grep -E '2011-06-30 (1[0-1]:[0-1][0-5]|10:16)'| grep -ach '0110 ...
1
vote
1answer
48 views

Linux log file structure change for usage with kst

My file (tcp dump from networked sensors) has this shape: 20:06:57.049686 IP (tos 0x0, ttl 64, id 26871, offset 0, flags [DF], proto UDP (17), length 68) e108-193.eduroam.tugraz.ac.at.34225 > ...
1
vote
2answers
91 views

Search for a string in a line and append related lines

I have a file containing some ID (identifier) and DS (description) lines. I'd like to keep the rows containing the string value in their ID line, as well as all the following (related) DS lines before ...
1
vote
2answers
213 views

Merge csv files with conditions

I have 2 CSV files for each date (csv_2014_4_15 and csv_2014_4_16) , with basic structure and couple of unique columns as below. id,name,created_at,updated_at,other columns 12, joe, 2013-1-1 18:30, ...
1
vote
1answer
267 views

Match first fields of two tab separated files and print matching values

I have two files. Match the 1st column of first file to 1st column of second file and print entire line of the second file. Second file is a <tab> separated file. I have tried many awk one ...
1
vote
2answers
178 views

Extract data in linux/unix

I have input data which I want to parse and extract values using awk/grep/sed: group-2 9 10 text color=black,from=ABCB11,fromid=4,order=2,thickness=3,to=ACE,toid=11,use=1,z=1 group-3 0 1 text ...
1
vote
4answers
3k views

Match multiple regular expressions from a single file using awk

I'm trying to parse a HTML file using shell scripting. There are 4 different regular expressions that i need to catch: name= , age= , classs= , marks=. Using grep "name=\|age=\|class=\|marks=" ...
1
vote
3answers
610 views

awk a specific column with unexpected line break

I can't seem to find a solution for this and it's driving me crazy. I know I can use awk to print a column(s). I'm having trouble printing a specific column though because of the way my file system is ...
1
vote
1answer
484 views

Reading two files into an IFS while loop — Is there a way to get a zero diff result in this case?

I have a text file full of several hundred lines of sequences like this: b 29. b 52. c 84. c 83. c 94. c 93. c 61. b 38. c 81. c 92. c 28. c 37. ...
1
vote
1answer
296 views

Need help processing a text file with awk to conform to CSV flat file format

I have the following problem. I collected data on reaction time from over 100 participants for an experiment I am running. Unfortunately, the separators between fields were not consistent, but after a ...
0
votes
3answers
76 views

Compare three file and print the highest value

I have three files. All of them having only one column. File1 M 0 M A File2 A 0 M A File3 M 0 A A compare each row in the three files, i.e, for first row (M,A,M) hence the maximum occurrence is ...
0
votes
2answers
5k views

Replacing one test string with another in particular column

I would like to replace "CC" with "C" and "AA" with A" in a particular column of Tab delimited file (using awk probably).