awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

4
votes
3answers
3k views

awk command for parsing a file

I have a following text file. I am showing you the first 3 lines. chrom st end gene strand c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 c12 c13 c14 chr6 3345 3543 geneA + 36 -23 ...
4
votes
4answers
344 views

Split the input file with line breaks

I am trying to insert line breaks on a file based on the same names which happens to be the second field in my file. So, my input file is like below. 17412193|name1|organization ...
4
votes
6answers
160 views

Extracting lines based on conditions

Each line in a comma-separated file has 5 fields. a,b,c,d,e f,g,c,i, j,k,c,m,n o,p,c,r,s t,u,c,w, x,y,z,aa,bb How can I extract the lines which have c in the 3rd field and their 5th field is NOT ...
4
votes
6answers
113 views

Only show the lines that are in all textfiles at least once

a.txt cat a.txt a b x c b.txt cat b.txt d e a f the q: SOMEMAGICK *.txt a Q: how can I show only the lines that are in all *.txt files?
4
votes
3answers
2k views

Show lines matching a pattern and the 4 lines before each

For example, from this file: CREATE SYNONYM I801XS07 FOR I8010.I801XT07 * ERROR at line 1: ORA-00955: name is already used by an existing object CREATE SYNONYM I801XS07 FOR ...
4
votes
3answers
307 views

Extracting unique piece of text from file with awk, sed or grep

From the output of pactl list sink-inputs, I need to grab the sink input number for VLC. Before that, I'm trying to extract the piece that contains the output for only VLC. All the methods that I ...
4
votes
2answers
815 views

Comma separated words to next line in new file

I have a input file like this (input.txt): Main1 one,two,three,four Main2 five,six,seven,eight I want the output like this in output file (output.txt): Main1 one Main1 two Main1 three Main1 four ...
4
votes
4answers
3k views

Print line after nth occurrence of a match

I am looking to display the line 4598 in the following file. Effectively I want to display the line AFTER the nth occurrence of a match. In this case, the line after the 3rd occurrence of <Car>. ...
4
votes
2answers
5k views

Processing two files using awk

I read Comparing two files using Unix and Awk. It is really interesting. I read and tested it, but I can't understand it completely and use it in other cases. I have two files. file1 has one field ...
4
votes
3answers
853 views

Easily get a particular column from output without sed or awk

Is there a quicker way of getting a couple of column of values than futzing with sed and awk? For instance, if I have the output of ls -hal / and I want to just get the file and directory names and ...
4
votes
4answers
1k views

Replacing pattern after nth match is found on each line?

I have a file containing lines: india;austria;japan;chile china;US;nigeria;mexico;russia I want to replace all the occurences of semicolon on each line with e.g. ;NEW;, but starting from the 2nd ...
4
votes
2answers
408 views

How do I populate end tags using sed, awk or any other command?

I have a txt file that looks like this: <sss>ss< or <firstword>anotherword< I want it to look like this <sss>ss</sss> or ...
4
votes
3answers
319 views

Rearrange text file by prepending data rows with header row

I have a file whose content looks like this: 2009 150 0 0 0 0000 75.316 0.0400390625 0.00007 0.00000 0.8980 76.216 0.0400390625 0.00007 1.00000 0.9046 77.217 0.0400390625 ...
4
votes
2answers
507 views

How can I get awk to print both matches and non-matches?

Basically, all I want to do is alter outputs if the input matches a filter, but the part being altered isn't what is being filtered (or I would use sed). The problem so far is that my awk is only ...
4
votes
2answers
5k views

How to apply the same awk action to different files?

I am new in awk and I do not know if it is possible to write an awk script that does this: I have hundred of data files that I have to sort. For each one I use the following one-liner: awk ...
4
votes
3answers
6k views

Get lines with maximum values in the column using awk, uniq and sort

I have a file with next format 2011-12-01 user1 access1 2011-12-01 user1 access2 2011-12-01 user2 access2 2011-12-01 user4 access2 2011-12-02 user1 access1 2012-01-01 user3 access1 2012-01-01 user4 ...
4
votes
3answers
150 views

Remove duplicate lines while keeping the order of the lines

[root@server]# awk '!seen[$0]++' out.txt > cleaned awk: (FILENAME=out.txt FNR=8547098) fatal error: internal error Aborted [root@server]# The ""server"" has: 8 GByte RAM + 16 GByte SWAP, x>300 ...
4
votes
2answers
3k views

Remove entire row in a file if first column is repeated

I have a file containing two columns and 10 million rows. The first column contains many repeated values, but there is a distinct value in column 2. I want to remove the repeated rows and want to keep ...
4
votes
4answers
352 views

Printing selective fields following 1st column instructions

Suppose I have a file in which I'd like to simultaneously print different awk commands following the first column instuctions, without messing up the original file (as it would happen with two ...
4
votes
2answers
1k views

Print lines if given column starts with a capital letter

I have a file like this: ID A56 DS /A56 DS AGE 56 And I'd like to print the whole line only if the second column starts with a capital letter. Expected output: ID A56 DS AGE 56 What I've ...
4
votes
2answers
8k views

Extract file name from path in awk program

I have an awk script and I have passed a CSV file to it. awk -f script.awk /home/abc/imp/asgd.csv What am I doing is to get FILENAME within script.awk. FILENAME gives me the whole path. As I am in ...
4
votes
2answers
340 views

Join: Two files - but only append last two columns

Given the files: 1.txt 1, abc, 123, 456, 789 2, lmn, 123, 456, 789 3, pqr, 123, 456, 789 2.txt 1, abc, 123, 000, 000 3, lmn, 123, 000, 000 9, opq, 123, 000, 000 OUTPUT.txt ID, NAME, X, ...
4
votes
4answers
863 views

Find a string (like grep -q) within only a section of a file

I want to write some Bash that can verify that a string exists in a configuration file. I can't change the file format, it belongs to a different application. The file is subdivided into groups named ...
4
votes
2answers
188 views

how to extract parts of file into separate files in linux using awk or grep or sed commands

I have text file as shown below: Input file1.txt ....................... cha21 1 3 5 nar cha21 2 3 6 piy cha23 2 3 5 ram cha23 3 ...
4
votes
3answers
261 views

Can tr commands be chained to avoid multiple tr processes in a pipeline?

I have a bunch of txt files, I'd like to output them lower-cased, only alphabetic and one word-per line, I can do it with several tr commands in a pipeline like this: tr -d '[:punct:]' ...
4
votes
3answers
10k views

How to replace the content of specific column with awk?

Given there is 40 columns in a record. I want to replace 35th column so that the 35th column will be replace with the content of 35th column and a "$" symbol. What came up my mind is like: awk ...
4
votes
3answers
2k views

awk remove the rows based on another column value

I have a file like below which I have sorted based on the username field. UserID score UserName 1234 200 Jack 5678 150 Jill 8543 200 Jill 5678 100 John I am trying to remove ...
4
votes
3answers
200 views

How to only output lines that are younger than 3 months?

cat a.txt 1387111124 ./asfjlasdf.txt 1348681215 ./akdfyxcv.txt So I need this: cat a.txt | SOMEMAGIC 1387111124 ./asfjlasdf.txt The output should only have the lines that are younger than 3 ...
4
votes
3answers
2k views

using awk to split a line on single spaces not multiples

I'm trying to split a line that I have no control over the format of. If parameter 7 and 8 are missing which is possible they will be replaced by a space so I would end up with, field1 field2 field3 ...
4
votes
2answers
5k views

Working with columns - awk and sed

I am trying to parse a text file that is generated by an expec script that pulls down some information from a switch. Here is a sample output: 192 0000.0000.0000 1/g23 Dynamic ...
4
votes
3answers
262 views

Get last part of string after hyphen

Is there a simple command line to extract the last part of a string separated by hyphens? E.g., I want to extract 123 from foo-bar-123.
4
votes
3answers
231 views

Extracting text using sed does not work as expected [duplicate]

I am just starting out using sed and I intend to use it to extract IP addresses from ping output. Here's what I am trying to achieve exactly: input text: ytmti (192.188.2.3) jjggy desired ...
4
votes
3answers
675 views

awk filtering live capture with file as pattern list

What I'm trying to do is quite simple. I'm generating output from tshark and redirecting it to awk with a pipe |. Since tshark gets live data, I want that awk search in every output for a pattern ...
4
votes
4answers
209 views

Combine fdisk and df in the same output

sudo fdisk -l gives : 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size ...
4
votes
2answers
64 views

Find holes in an enumeration in awk

I have a file that looks like this 1400a_comp 49472 2868636.1026583919660 GOOD 1400a_comp 49473 2868647.1026579919650 GOOD ...
4
votes
2answers
3k views

System calls, AWK, and Bringing in external inputs

awk '{ TEMPVAR="/usr/bin"; printf("%s", system("ls $TEMPVAR")); }' empty In this example I'm trying to bring in the variable TEMPVAR into the system call. How would I do this? What I'm aiming to ...
4
votes
2answers
175 views

Column mismatch and substituting

input.txt (tab-delimted) TTTTOTTT00000000008 RTTTT899 5.00E-28 TTTTOTTT00000000046 RTTTWRR 3.00E-31 TTTTOTTT00000000051 2.00E-11 TTTTOTTT00000000051 7.00E-12 TTTTOTTT00000000054 ...
4
votes
2answers
842 views

Why does this awk command not play as well with find as sed does?

I am very much a newbie at Unix/Linux command line stuff, and mostly get to where I'm going by copying, pasting, and modifying code I find on the internet. Just mentioning that in hopes people will ...
4
votes
3answers
145 views

How to select first occurrence between two patterns including them

How can I select first occurrence between two patterns including them. Preferably using sed or awk. I have: text something P1 something content1 content2 something P2 something text something P1 ...
4
votes
2answers
152 views

awk pairwise differences

I run into this problem a lot with data, if I have some integers, I frequently want to look at differences between adjacent integers, I usually solve this in Ruby or Python, but I think such a thing ...
4
votes
3answers
2k views

Change the text on a single line number in multiple text files

Suppose that I have ten bash shell scripts: script1.sh, script2.sh, ..., script10.sh. Initially, all ten files are identical. Now, I would like to make two changes in each script: In each file, ...
4
votes
4answers
3k views

How to echo an escaped string

How can I echo an escaped string that contains $ in Bourne Shell? user@server:~$ cat test.sh #!/bin/sh echo $1 user@server:~$ ./test.sh \$sad\$test $sad$test I want it to return an escaped ...
4
votes
3answers
294 views

How to view all the content in an awk array?

In my understanding, awk array is something like python dict. So I write down the code bellow to explore it: awk '{my_dict[$1] = $2} END { print my_dict}' zen And I got: awk: can't read value of ...
4
votes
2answers
319 views

grep for repeated values from pattern list

I have file A.txt which is a lists of unique ids (column 1) and its corresponding values(column 2): A.txt ABC1D_T1_B1 123.4 ABC2D_T1_B2 146.7 ABC3D_T1_B1 567.8 ABC4D_T0_B2 96.8 ABC1D_T0_B1 ...
4
votes
4answers
187 views

Replace a marker in a text file with arbitrary new text

I have a file which contains some text and - on a single line - a marker which indicates where new content should be added foo bar %SUBSTITUTE% foo The line line substitute should be replaced by a ...
4
votes
4answers
369 views

How do I remove every number that's surrounded by <>

I have tried this sed -i '' 's/[0-9]*<>/g' But it didn't work. Example file: <Number1> </Number8> output: <Number> </Number>
4
votes
3answers
3k views

GREP / SED or AWK: Print entire paragraph in a file on pattern match [duplicate]

I have a file with hundreds of paragraphs of around 15 lines each. I need to search for a pattern, say Occurance: 1. If this pattern is found in the para, I need to print the entire paragraph. Note ...
4
votes
2answers
369 views

Merge fields in a file

I have a file with 7 columns, a GFF file having chromosomal regions.I want to collapse the rows where REGION ="exon" to only one row in the file.The row has to be collapsed on the basis of regions ...
4
votes
1answer
3k views

How to find out common elements between two files?

For an example, I have 2 files having following info: File #1: 12 13 14 15 File #2: 12 1 13 2 14 2 15 6 16 7 17 8 Output File: 1 2 2 6 In the output file, I want only the second column ...
4
votes
5answers
2k views

extract fields from “tail -f” of a syslog stream

Example line from syslog file: Aug 1 10:25:50 10.10.10.1 id=firewall sn=XXXX time="2012-08-01 14:35:18 UTC" fw=x.x.x.x pri=6 c=1024 m=537 msg="Connection Closed" f=11 n=195273698 ...