awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

5
votes
3answers
130 views

How to use awk through multiple files?

I want a solution for the general case (N folder) I'm using awk to process a file and extract its content and put it in a variable then echo it: This is the file: H1 H2 H3 H4 H5 H6 H7 H8 H9 not ...
0
votes
3answers
49 views

How to Print certain Fields if a condition is met

I have a big Input file as shown below. If "PRIMER_LEFT_NUM_RETURNED" is not equal to 0 then print the corresponding SEQUENCE_ID, SEQUENCE_TEMPLATE,PRIMER_LEFT_NUM_RETURNED, PRIMER_RIGHT_NUM_RETURNED, ...
3
votes
4answers
72 views

Need to remove - (Minus) sign at the end in number from many columns and insert it on columns removed

I have a huge csv file which has - (Minus) sign at the end of numbers. Now I need to move these - minus signs to begin of the numbers, i.e I need to remove the last character if it has '-' sign and ...
0
votes
1answer
103 views

Haskell, C++, AWK, Bash, Lisp … Which one is faster? [closed]

I was wondering, which of these programming languages are faster at handling big data files of only strings: Haskell, C++(Boost), AWK, Bash, Lisp... So far I have found the paper Haskell vs. Ada ...
0
votes
3answers
34 views

AWK remove one line?

I need some help with awk. This is my uptime uptime=`uptime|grep days|awk '{print$3,$4}'` But it displays now 23 days, I want to remove , How can I accomplish that? cheers and thanks
2
votes
7answers
82 views

How to replace the third character in number to other character

IPS parameter include diff IP address as: IPS="192.9.200.12-14 172.17.200.12-89 12.21.1.10-25 127.0.0.1-127" I want to replace the third “.” In the IP address number to “,” How to achieve this by ...
3
votes
2answers
97 views

Compare 2 tab delimited files and output differences with column header

I would like to compare 2 similar files on a common column. The files will have identical headers. file1.txt mem_id Date Time Building aa bb cc dd ee ff ...
4
votes
4answers
183 views

extract lines according to a particular column

I have a CSV file like CK,ck XYZ,xyz,xyzs ABC,abc,abcs PQR,pqr,pqrs LMN,lmn, IJK,ijk, I have to check if something is written in column 3 except whitespace, then display the whole line.the output ...
1
vote
2answers
418 views

Error: ambiguous redirect while transferring output to command

Output to redirect in script: 511@ubuntu:~/Unix/test$ ls -ltr |awk '{print $9}' default.txt dfah.txt fruit.txt fruit_prices.txt dfh.txt header.txt testfile.txt topoutput.txt Script written on shell ...
32
votes
4answers
2k views

Reverse grepping

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
25
votes
1answer
22k views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
1
vote
4answers
43 views

Loop through array

I have records in list.txt (INPUT) as List of animals SET 1=Dog 2=Crow 3=Snake Pet,bird,reptile List of Countries SET 1=France 2=Singapore 3=Columbia Europe,Asia,SouthAmerica List of Parts SET ...
1
vote
1answer
74 views

Why is the output of this paste command split on separate lines with indents?

I was trying to merge files using either of these commands: paste data_{1..2}.txt > 1_2.txt paste <(awk '{print $1}' data_2.txt ) <(awk '{print $1}' data_2.txt) > 1_2.txt But the ...
4
votes
3answers
602 views

Replace “o” with “0” in large wordlist and save the original word?

I have a large text file with words. Every word is on a single line (typical wordlist). I want to replace all characters "o" with the number "0" in every word, and the new formed word to be pasted on ...
2
votes
4answers
312 views

Only cat from specific line X (with a pattern) to other specific line Y (with a pattern)

A little extended problem from "cat line x to line y on a huge file": I have a huge file (2-3 GB). I'd like to cat/print only from the line having "foo:" to the line having "goo:". Assume that "foo:" ...
-1
votes
2answers
51 views

Searching out a pattern from Logs

The format of the logs is in this manner: Junk Junk Junk : Junk Junk Junk:IP/Port Junk Junk I want to search using the IP and the output should be only the IP/Port part of each line in that log ...
0
votes
0answers
87 views

wget: Retrieving a list of URLs when modifying input data file on the fly

This issue is currently driving me up the wall. It just does not work as it should. I have a file inp with audio samples to download where I preserved the internal ID number by parsing some other ...
1
vote
2answers
65 views

awk: for-loop with break option

I am trying to use awk to manipulate data.I have a data file with two columns and I would like to find the row that contains an specific value. But there is more than one row that contains this value ...
0
votes
0answers
58 views

Can sed and awk take in multiple files?

grep can search in multiple files, including those files recursively under a specified directory. This makes grep able to find which file contains a pattern. Can awk and sed also? It seems that they ...
1
vote
1answer
49 views

awk: Columns are not printed when concatenating strings is passed as command string

I've found some anomaly when writing a script. The following examples works as expected: $ echo 123 | awk '{print $1 456}' 123456 $ sh -c "echo 123 | awk '{print $1}'" 123 But the following ...
5
votes
3answers
312 views

find length of longest line in all text files in a directory

I know how to get length of the longest line in a text file with awk awk ' { if ( length > L ) { L=length} }END{ print L}' file.txt but how can I get the length of the longest line of all files ...
0
votes
1answer
78 views

How to copy rows of different files and paste column-wise?

I want to create a new file by copying rows of each file and pasting columnwise according to their file name orders. Here is the sample input and required output files below. file1.txt 1234 5678 ...
0
votes
5answers
119 views

print only rows that has latest non empty cell that starts with the word “Failed” on a CSV file in Linux [closed]

I want to print the table only if non-empty latest column that starts with the word "Failed"on a CSV file in Linux.. My input - Name,Mon,Tue,Wed,Thu,Fri,Mon Linux,Successfull Fullbackup; Failed ...
5
votes
3answers
158 views

Remove duplicate fields in a given column

I'd like to remove from a given column ($2 in the example) the duplicate fields (comma separated). Input file: A 1,2,3,4 B 4,5,6,3 C 2,15 Expected output: A 1,2,3,4 B 5,6 C ...
0
votes
2answers
47 views

joining 2 files on 2 columns but at different column numbers using awk

I have 2 files which has to be joined on $1,$2 of 1st file and $1,$8 of 2nd file using awk.Can anyone please provide the solution for this as this is an urgent requirement. Thanks in Advance file1: ...
1
vote
2answers
65 views

grep contents of files based on a list of files

Running CentOS 6.5, qemu-kvm. I have a script which builds a list of running VM's. I want to pull information from those .xml files of only the VM's that are running. cat /dev/null > ...
1
vote
3answers
38 views

Concatenate Sequences under a Heading

I have a file like File1. With a heading for example ">Contig1" and I want to concatenate all the sequences under the respective heading. File1: >Contig1 ACCCCATACATCCAAA NNNNNNNNNNCATACA ...
1
vote
2answers
60 views

Passing print pattern as a variable to awk

I am trying to use awk to print variables from various columns in a line delimited by , .The columns and pattern which I want to print is stored in another variable b. The pattern a and print patter ...
1
vote
3answers
214 views

Trouble in adding a column with awk

I'm trying to add a column to a csv file by using: awk -F "," '{print $0"new_col"}' file.csv but I got : new_col8,0.405,0.101 new_col4,0.374,0.100 new_col23,0.342,0.100 new_col130,0.298,0.100 ...
3
votes
3answers
57 views

Removal of lines with more than 'N' fields?

I am working on mac with sed, perl, awk, bash.. I have a large-ish (10GB) text file which has 13 fields (columns) of TAB delimited data. Unfortunately some of these lines have extraneous TABs , so I ...
4
votes
5answers
571 views

Multi-line grep beween two tags, only show the last match

I'm having a program that intermingles data segments into log output: log message log message ----BEGIN INLINE DATA---- data data data -----END INLINE DATA----- log messge I'm looking for something ...
3
votes
4answers
124 views

How to remove text matching specific patterns from a file

I want to compare two files to check for differences between them, fileA and fileB. fileA is like a template file and fileB is the file which I want to compare to it. Anytime I find a difference I ...
0
votes
1answer
27 views

How to print 1st and 4th variable only if 4th variable is > or equal to 25

Networking: [Test] => 10.729519791085 GAMES: [Real] => 10 Fashion: [Test] => 27.729519791085 Movies: [Real] => 5 Food: [Test] ...
2
votes
2answers
159 views

Delete rows in a csv file based on a specific column value

I have a csv file with the following headers - Interaction_ID, PDB_ID, First_Residue, First_Chain, Second_Residue, Third_Residue, Third_Chain & Pattern: ...
3
votes
4answers
210 views

Count how many elements in the nth column in a whole file

Input file: A 1,2,3,4 #length($2)=4 B 1,2 #length($2)=2 C 9,8,7,6,5,4 #length($2)=6 Expected output: 12 #4+2+6 A method like: awk -F '[\t,]' '{print length($2)}' but ...
0
votes
3answers
1k views

Select/extract columns based on the values on a certain row

Here is an example text file: A B C D E F G 1 2 3 4 5 6 7 2 3 4 5 6 7 8 3 4 5 6 7 8 9 I would like to extract specific columns based on the values on the 3rd row, that is, 2 3 4 5 6 7 8. Let's say, ...
3
votes
3answers
149 views

Parse multiple sections of data into seperate files

I have a concatenated log file with multiple logs inside that I'm trying to parse out into individual log files (I will later rename them to the date/time of each). Each log is separated by "--- LOG ...
0
votes
3answers
59 views

Replace specific text only in specific fields using nawk

My input data file abc.txt has this "20131008","0004","0568","98" I ran this command to replace the text nawk -F, -vOFS=',' '{gsub("0568","0808",$3); print }' abc.txt In the output the ...
0
votes
2answers
5k views

Replacing one test string with another in particular column

I would like to replace "CC" with "C" and "AA" with A" in a particular column of Tab delimited file (using awk probably).
5
votes
4answers
236 views

How to replace all the first occurences in matched lines with incremental number?

I have a file like this ... 1562 first part 1563 H col3 H col4 1564 H col3 H col4 ... 3241 H col3 H col4 3242 third part ... I want to replace only the first H in every line to H#, with ...
2
votes
1answer
33 views

Get each column which contains multiple spaces?

My file: subnet 172.25.10.0 netmask 255.255.255.0 { option domain-name-servers 172.25.10.9, 8.8.8.8; default-lease-time 300; max-lease-time 7500; range dynamic-bootp 172.25.10.10 172.25.10.30 ; ...
3
votes
6answers
936 views

Convert string into date time stamp in gawk or awk

I want to convert following string (20140805234656) into date time stamp (2014-08-05 23:46:56).I am new to gawk and I don't know the exact syntax,how can I put - at every 5,8 and : at every 14,17 and ...
4
votes
6answers
305 views

want the last line when duplicate awk?

I have the file: key value blah blah blah blah blahblah man1 boy1 blah blah man1 boy2 man1 boy1 I do this to remove duplicate lines: awk '/man1/ { print $1,$2} ' file | awk '!x[$0]++' and the ...
0
votes
5answers
57 views

re-arrange values in one column without affecting other columns using awk or sed

I have a csv file in following format: abc,124,123,2014-08-13,abc def,124,123,2014-08-13,abc ghi,124,123,2014-08-13,abc Now I need to modify only 5th column (2014-08-13) to produce output as given ...
3
votes
1answer
64 views

Handling dynamically changing column positions and splitting file

I have a unique problem. I have a file with following columns PersonId, AB1, AB2, AB3, DG1_B1,DG1_B2,DG2 1,0,1,0,3,5,7 2,0,3,1,3,7,4 I want to split this and create multiple files based on the ...
2
votes
4answers
233 views

Insert new line after non matching pattern

I have following output from script host1:101 host1:102 host2:101 host2:102 host2:103 host4:101 host5:101 host5:102 host5:103 host5:104 etc.. I wanted to insert a new line after each host group. ...
2
votes
1answer
44 views

Move matched line above a different matched line using sed

I have something like dsadjakldjska safdsfdsfsf fdsfsf**STGN**fdsfsd fdsfkjldsf fdsff fdsfs**IRV**fdsfs and I want to get dsadjakldjska safdsfdsfsf fdsfkjldsf fdsff fdsfsf**STGN**fdsfsd ...
2
votes
2answers
71 views

Memory allocation for sparse array in awk

I have searched but didnt reach to any conclusion that when i defined an sparse array does it reserve all the contiguous memory upto maximum index or it allocate the memory at that particular index ...
0
votes
2answers
102 views

Delete every number that comes before “=”

i have a very, very, very long list. Like over 830,000 characters. I will of course perform said command on a copy. There are multiple lines, like this hey1=lol hey2=lol hey3=lol I want that to ...
4
votes
3answers
410 views

pipe line with non-deterministic output

I have happened upon a command that sometimes works and sometimes does not, even when executed multiple times in rapid succession in a bash shell (I have not tested the behavior in other shells). The ...