awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

4
votes
3answers
684 views

pipe line with non-deterministic output

I have happened upon a command that sometimes works and sometimes does not, even when executed multiple times in rapid succession in a bash shell (I have not tested the behavior in other shells). The ...
4
votes
3answers
627 views

Replace “o” with “0” in large wordlist and save the original word?

I have a large text file with words. Every word is on a single line (typical wordlist). I want to replace all characters "o" with the number "0" in every word, and the new formed word to be pasted on ...
4
votes
3answers
2k views

Summing up an array inside of awk?

I have the following piece of code: sum1= sum2= declare -a a echo $temp | awk '{split($0,a,","); name=a[1] ; for(i=2;i<=4;i++) sum1+=a[i] ; for(i=5;i<=7;i++) sum2+=a[i] }' This code is ...
4
votes
5answers
2k views

How to remove all comments from a file?

I have a file with a few lines commented out: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I just want to print all the uncommented code: foo bar stuff morestuff ...
4
votes
6answers
5k views

Remove lines that do not start with a pattern from a given set of patterns

I have a file that contains data like this: report aaaaaaaa - .. -th bbbbbbbbb -to ccccccccc .. --. Question: I want to remove any line that does not start with following strings : report ...
4
votes
4answers
655 views

How to Prefix a column values with an apostrophe ( ' )?

I have a CSV file with multiple columns and 1000's of records, I need to prefix all the values of one of the columns (lets say 2nd column) with an apostrophe ' except in the first line or header line, ...
4
votes
5answers
13k views

removing the first and the last character of every line from command line

I am trying to remove the first and the last characters of everyline in a text file and save the resulting truncated version in a new file. Does anyone have an idea about how to do that efficiently ...
4
votes
4answers
1k views

shorten long lines in a log file

I keep a logfile of a program's output. The trouble is sometimes errors occur dumping very long lines of 7bit ascii data(encoded binary) I don't care to keep. These lines can be 200KB+ long before ...
4
votes
4answers
131 views

Current date in awk

Is there a constant variable in awk, that store today's date? if no, is there a way that can store today's date for daily use? let's say we have below file: boo,foo,2016-08-30 foo,boo,2016-07-31 ...
4
votes
2answers
777 views

Delete every Nth line in shell

I'm trying to delete a line after N lines using awk and I can't seem to get it right. The file format is like this YYYYYY XXXXXX XXXXXX YYYYYY XXXXXX XXXXXX The real example would be office3 3 1 ...
4
votes
7answers
628 views

How to split a filename list in 5GB sets?

I have a pretty big list of files which I like to group in smaller list of 5GB each for further processing. The file list is in the following form: "size" (from du) "tab" "absolute path", like this ...
4
votes
4answers
4k views

Redirecting Output of print to multiples files in awk

I want to print same lines to 2 different files. What I am doing right now is: print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file1.txt"; print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file2.txt"; ...
4
votes
3answers
1k views

How can I delete all text between curly brackets in a multiline text file?

Example: This is { the multiline text file } that wants { to be changed } anyway. Should become: This is that wants anyway. I have found some similar threads in the forum, but they don't seem ...
4
votes
3answers
3k views

awk command for parsing a file

I have a following text file. I am showing you the first 3 lines. chrom st end gene strand c1 c2 c3 c4 c5 c6 c7 c8 c9 c10 c11 c12 c13 c14 chr6 3345 3543 geneA + 36 -23 ...
4
votes
4answers
354 views

Split the input file with line breaks

I am trying to insert line breaks on a file based on the same names which happens to be the second field in my file. So, my input file is like below. 17412193|name1|organization ...
4
votes
2answers
5k views

Processing two files using awk

I read Comparing two files using Unix and Awk. It is really interesting. I read and tested it, but I can't understand it completely and use it in other cases. I have two files. file1 has one field ...
4
votes
6answers
113 views

Only show the lines that are in all textfiles at least once

a.txt cat a.txt a b x c b.txt cat b.txt d e a f the q: SOMEMAGICK *.txt a Q: how can I show only the lines that are in all *.txt files?
4
votes
3answers
2k views

Show lines matching a pattern and the 4 lines before each

For example, from this file: CREATE SYNONYM I801XS07 FOR I8010.I801XT07 * ERROR at line 1: ORA-00955: name is already used by an existing object CREATE SYNONYM I801XS07 FOR ...
4
votes
3answers
362 views

Extracting unique piece of text from file with awk, sed or grep

From the output of pactl list sink-inputs, I need to grab the sink input number for VLC. Before that, I'm trying to extract the piece that contains the output for only VLC. All the methods that I ...
4
votes
2answers
943 views

Comma separated words to next line in new file

I have a input file like this (input.txt): Main1 one,two,three,four Main2 five,six,seven,eight I want the output like this in output file (output.txt): Main1 one Main1 two Main1 three Main1 four ...
4
votes
4answers
3k views

Print line after nth occurrence of a match

I am looking to display the line 4598 in the following file. Effectively I want to display the line AFTER the nth occurrence of a match. In this case, the line after the 3rd occurrence of <Car>. ...
4
votes
3answers
927 views

Easily get a particular column from output without sed or awk

Is there a quicker way of getting a couple of column of values than futzing with sed and awk? For instance, if I have the output of ls -hal / and I want to just get the file and directory names and ...
4
votes
4answers
1k views

Replacing pattern after nth match is found on each line?

I have a file containing lines: india;austria;japan;chile china;US;nigeria;mexico;russia I want to replace all the occurences of semicolon on each line with e.g. ;NEW;, but starting from the 2nd ...
4
votes
6answers
194 views

Extracting lines based on conditions

Each line in a comma-separated file has 5 fields. a,b,c,d,e f,g,c,i, j,k,c,m,n o,p,c,r,s t,u,c,w, x,y,z,aa,bb How can I extract the lines which have c in the 3rd field and their 5th field is NOT ...
4
votes
2answers
417 views

How do I populate end tags using sed, awk or any other command?

I have a txt file that looks like this: <sss>ss< or <firstword>anotherword< I want it to look like this <sss>ss</sss> or ...
4
votes
2answers
603 views

Fastest `uniq` tool in linux

I have large text file (1.5 G), I want to know what the fastest and more reliable tool in linux is. I usualy use: awk '!x[$0]++' file.txt But when I use htop command I see my memory usage is ...
4
votes
3answers
327 views

Rearrange text file by prepending data rows with header row

I have a file whose content looks like this: 2009 150 0 0 0 0000 75.316 0.0400390625 0.00007 0.00000 0.8980 76.216 0.0400390625 0.00007 1.00000 0.9046 77.217 0.0400390625 ...
4
votes
2answers
519 views

How can I get awk to print both matches and non-matches?

Basically, all I want to do is alter outputs if the input matches a filter, but the part being altered isn't what is being filtered (or I would use sed). The problem so far is that my awk is only ...
4
votes
2answers
5k views

How to apply the same awk action to different files?

I am new in awk and I do not know if it is possible to write an awk script that does this: I have hundred of data files that I have to sort. For each one I use the following one-liner: awk ...
4
votes
3answers
6k views

Get lines with maximum values in the column using awk, uniq and sort

I have a file with next format 2011-12-01 user1 access1 2011-12-01 user1 access2 2011-12-01 user2 access2 2011-12-01 user4 access2 2011-12-02 user1 access1 2012-01-01 user3 access1 2012-01-01 user4 ...
4
votes
3answers
165 views

Remove duplicate lines while keeping the order of the lines

[root@server]# awk '!seen[$0]++' out.txt > cleaned awk: (FILENAME=out.txt FNR=8547098) fatal error: internal error Aborted [root@server]# The ""server"" has: 8 GByte RAM + 16 GByte SWAP, x>300 ...
4
votes
2answers
3k views

Remove entire row in a file if first column is repeated

I have a file containing two columns and 10 million rows. The first column contains many repeated values, but there is a distinct value in column 2. I want to remove the repeated rows and want to keep ...
4
votes
4answers
364 views

Printing selective fields following 1st column instructions

Suppose I have a file in which I'd like to simultaneously print different awk commands following the first column instuctions, without messing up the original file (as it would happen with two ...
4
votes
2answers
2k views

Print lines if given column starts with a capital letter

I have a file like this: ID A56 DS /A56 DS AGE 56 And I'd like to print the whole line only if the second column starts with a capital letter. Expected output: ID A56 DS AGE 56 What I've ...
4
votes
2answers
9k views

Extract file name from path in awk program

I have an awk script and I have passed a CSV file to it. awk -f script.awk /home/abc/imp/asgd.csv What am I doing is to get FILENAME within script.awk. FILENAME gives me the whole path. As I am in ...
4
votes
4answers
909 views

Find a string (like grep -q) within only a section of a file

I want to write some Bash that can verify that a string exists in a configuration file. I can't change the file format, it belongs to a different application. The file is subdivided into groups named ...
4
votes
2answers
216 views

how to extract parts of file into separate files in linux using awk or grep or sed commands

I have text file as shown below: Input file1.txt ....................... cha21 1 3 5 nar cha21 2 3 6 piy cha23 2 3 5 ram cha23 3 ...
4
votes
3answers
291 views

Can tr commands be chained to avoid multiple tr processes in a pipeline?

I have a bunch of txt files, I'd like to output them lower-cased, only alphabetic and one word-per line, I can do it with several tr commands in a pipeline like this: tr -d '[:punct:]' ...
4
votes
3answers
2k views

awk remove the rows based on another column value

I have a file like below which I have sorted based on the username field. UserID score UserName 1234 200 Jack 5678 150 Jill 8543 200 Jill 5678 100 John I am trying to remove ...
4
votes
3answers
205 views

How to only output lines that are younger than 3 months?

cat a.txt 1387111124 ./asfjlasdf.txt 1348681215 ./akdfyxcv.txt So I need this: cat a.txt | SOMEMAGIC 1387111124 ./asfjlasdf.txt The output should only have the lines that are younger than 3 ...
4
votes
2answers
2k views

grep one line before the match plus the match

zzzzzzzzz aaaaaaaaa bbbbbbbbb & ccccccccc & ddddddddd hhhhhhhhh eeeeeeeee fffffffff & ggggggggg & in the above line, what i want is to grep/sed/awk (any method is fine) line that ...
4
votes
3answers
2k views

using awk to split a line on single spaces not multiples

I'm trying to split a line that I have no control over the format of. If parameter 7 and 8 are missing which is possible they will be replaced by a space so I would end up with, field1 field2 field3 ...
4
votes
2answers
5k views

Working with columns - awk and sed

I am trying to parse a text file that is generated by an expec script that pulls down some information from a switch. Here is a sample output: 192 0000.0000.0000 1/g23 Dynamic ...
4
votes
3answers
323 views

Get last part of string after hyphen

Is there a simple command line to extract the last part of a string separated by hyphens? E.g., I want to extract 123 from foo-bar-123.
4
votes
3answers
237 views

Extracting text using sed does not work as expected [duplicate]

I am just starting out using sed and I intend to use it to extract IP addresses from ping output. Here's what I am trying to achieve exactly: input text: ytmti (192.188.2.3) jjggy desired ...
4
votes
3answers
702 views

awk filtering live capture with file as pattern list

What I'm trying to do is quite simple. I'm generating output from tshark and redirecting it to awk with a pipe |. Since tshark gets live data, I want that awk search in every output for a pattern ...
4
votes
4answers
218 views

Combine fdisk and df in the same output

sudo fdisk -l gives : 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size ...
4
votes
2answers
64 views

Find holes in an enumeration in awk

I have a file that looks like this 1400a_comp 49472 2868636.1026583919660 GOOD 1400a_comp 49473 2868647.1026579919650 GOOD ...
4
votes
2answers
4k views

System calls, AWK, and Bringing in external inputs

awk '{ TEMPVAR="/usr/bin"; printf("%s", system("ls $TEMPVAR")); }' empty In this example I'm trying to bring in the variable TEMPVAR into the system call. How would I do this? What I'm aiming to ...
4
votes
2answers
179 views

Column mismatch and substituting

input.txt (tab-delimted) TTTTOTTT00000000008 RTTTT899 5.00E-28 TTTTOTTT00000000046 RTTTWRR 3.00E-31 TTTTOTTT00000000051 2.00E-11 TTTTOTTT00000000051 7.00E-12 TTTTOTTT00000000054 ...