awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

-1
votes
1answer
35 views

search for a character in one column and replace value in another column of matched records

I have a requirement to find value 2 in column 4 and wherever it matches replace 6th column with '-'sign in the beginning suppose if i have csv file - file1 like this abc,0,def,1234,ijk ...
2
votes
1answer
47 views

Getting matched fasta file

list.txt 58759__len__2903 58759__len__2903 673957__len__1655 673957__len__1655 3566454__len__1744 seq.fasta >58759__len__2903 TTTTCCGTAGAGGAGATCCCTATTTTTAGGTTTGTAAGAGATCATTTT ...
-1
votes
1answer
57 views

Match and combine records in Bash [duplicate]

Consider two csv files: $ cat given.csv 123,John 246,Paul 369,George 987,Ringo $ cat family.csv 246,McCartney 123,Lennon 987,Starr 369,Harrison Note that the lines are not ordered! How might I use ...
2
votes
1answer
42 views

Substitute one field with other if its entry starts with specific letter

If I have a file something like: 223 898 teeth 223 899 mouth 223 900 r9 445 800 round My question is how can I use awk command to substitute column 2 with column 3, only for the row in which column ...
0
votes
2answers
52 views

Convert two rows to two columns

I would like to convert every two rows to two columns using awk. Any help appreciated. input.txt: # Query: gi|11465907|ref|NC_001872.1| Chlamydomonas eugametos genome, complete genome # 0 hits found ...
2
votes
2answers
33 views

Gather job usage information on cluster

I am running simulations on a linux cluster shared by hundreds of other users. There is a job scheduler installed on the cluster that determines who runs where, and one of the scheduler commands ...
8
votes
2answers
274 views

Reindexing a large CSV file

I went through the answers in this helpful thread, but my problem seems to be different enough that I can't think of good answer (at least with sed). I have a large CSV file (200+ GB) with rows that ...
2
votes
3answers
101 views

Use AWK to pad date and time formats

I would like to use AWK to pad the date and time fields in my report. Here is my current .csv that is actually tab delimited. AA 9/7/2014 2:30:38 PM AA0000011111 08 Y A Jane, Doe AA ...
2
votes
3answers
46 views

Merging two tables including multiple ocurrence of column identifiers and unique lines

I asked a similar question yesterday (Merging two tables including multiple ocurrence of column identifiers) but ran into a problem with unique lines. I would like to merge two tables based on column ...
4
votes
6answers
153 views

Sed — Replace first k instances of a word in the file

I want to replace only the first k instances of a word. How can I do this? Eg. Say file foo.txt contains 100 instances occurrences of word 'linux' . I need to replace first 50 occurrences only.
2
votes
2answers
105 views

Merging two tables including multiple ocurrence of column identifiers

I would like to merge two tables: File 1: 1 today 2 tomorrow 3 red File 2: 1 a lot 1 sometimes 2 at work 2 at home 2 sometimes 3 new Desired output (file 3): 1 today ...
0
votes
4answers
88 views

Use SED or AWK to move rows to columns

I have a report that is generated in rows and I need to move it to columns. The number of rows per record will always be the same. This will be in a BASH script on Linux, so I would like to use ...
1
vote
1answer
40 views

Take the subset of the file related to the time stamp

I have some json file in dictionary format. The lines looks like here: {"a":"1", "b":"2", "c":"3", "time":1334572551435} {"a":"1", "b":"2", "c":"4", "time":1334575352456} {"a":"2", "b":"2", "c":"7", ...
1
vote
1answer
59 views

Sed / awk script for a non-standard apache access log ( Oracle Weblogic )

I have hit a wall with my limited sed scripting abilities, and i wonder if any of you guys could help me out. I have a non-standard apache access log with the following format: #Version: 1.0 ...
1
vote
4answers
116 views

How to split text file into records by first non-blank charecter using awk and print only records containing string

I have a text file with information separated in blocks. I would like to separate these blocks into records so that I can handle them separately. This is my text file (file.txt): Alarm Stats: ...
1
vote
4answers
60 views

Parsing complex text file using Unix commands

I have the following text structure which I would like to parse: >Cluster 423 0 56aa, >HWI-ST1448:257:C3V2HACXX:1:1106:19087:2550.1... at 92.86% 1 64aa, ...
3
votes
2answers
195 views

awk: Extracting a fixed number of rows where the last row number may vary

This question may look like a duplicate, but only at first glance. Of course, I would no longer need help in how to code a one-liner that extracts a fixed number of continuous lines (e. g. 5 in this ...
2
votes
2answers
57 views

Use awk to find first occurrence

I have a log file that gets updated when my script runs. The script will insert "Start of script" text when it starts and "End of script" text when it finishes running. I am trying to capture the ...
0
votes
3answers
45 views

How to Print certain Fields if a condition is met

I have a big Input file as shown below. If "PRIMER_LEFT_NUM_RETURNED" is not equal to 0 then print the corresponding SEQUENCE_ID, SEQUENCE_TEMPLATE,PRIMER_LEFT_NUM_RETURNED, PRIMER_RIGHT_NUM_RETURNED, ...
5
votes
5answers
102 views

Replace data at specific positions in txt file using data from another file

I have a text file in the below format: $data This is the experimental data good data This is good file datafile 1 4324 3673 6.2e+11 7687 67576 2 3565 8768 8760 5780 8778 "This is line ...
5
votes
3answers
102 views

How to use awk through multiple files?

I want a solution for the general case (N folder) I'm using awk to process a file and extract its content and put it in a variable then echo it: This is the file: H1 H2 H3 H4 H5 H6 H7 H8 H9 not ...
0
votes
4answers
60 views

Adding a character to every other text line

I have a huge (ca. 20G) text file which contains millions of passages (a.k.a. paragraphs) with headers. Headers and paragraphs are always one line each, e.g., Sunshine This is a sunny day. ...
3
votes
4answers
51 views

Need to remove - (Minus) sign at the end in number from many columns and insert it on columns removed

I have a huge csv file which has - (Minus) sign at the end of numbers. Now I need to move these - minus signs to begin of the numbers, i.e I need to remove the last character if it has '-' sign and ...
0
votes
3answers
30 views

AWK remove one line?

I need some help with awk. This is my uptime uptime=`uptime|grep days|awk '{print$3,$4}'` But it displays now 23 days, I want to remove , How can I accomplish that? cheers and thanks
0
votes
1answer
82 views

Haskell, C++, AWK, Bash, Lisp … Which one is faster? [closed]

I was wondering, which of these programming languages are faster at handling big data files of only strings: Haskell, C++(Boost), AWK, Bash, Lisp... So far I have found the paper Haskell vs. Ada ...
1
vote
2answers
126 views

Error: ambiguous redirect while transferring output to command

Output to redirect in script: 511@ubuntu:~/Unix/test$ ls -ltr |awk '{print $9}' default.txt dfah.txt fruit.txt fruit_prices.txt dfh.txt header.txt testfile.txt topoutput.txt Script written on shell ...
3
votes
2answers
68 views

Compare 2 tab delimited files and output differences with column header

I would like to compare 2 similar files on a common column. The files will have identical headers. file1.txt mem_id Date Time Building aa bb cc dd ee ff ...
4
votes
4answers
159 views

extract lines according to a particular column

I have a CSV file like CK,ck XYZ,xyz,xyzs ABC,abc,abcs PQR,pqr,pqrs LMN,lmn, IJK,ijk, I have to check if something is written in column 3 except whitespace, then display the whole line.the output ...
2
votes
7answers
78 views

How to replace the third character in number to other character

IPS parameter include diff IP address as: IPS="192.9.200.12-14 172.17.200.12-89 12.21.1.10-25 127.0.0.1-127" I want to replace the third “.” In the IP address number to “,” How to achieve this by ...
1
vote
4answers
32 views

Loop through array

I have records in list.txt (INPUT) as List of animals SET 1=Dog 2=Crow 3=Snake Pet,bird,reptile List of Countries SET 1=France 2=Singapore 3=Columbia Europe,Asia,SouthAmerica List of Parts SET ...
1
vote
1answer
63 views

Why is the output of this paste command split on separate lines with indents?

I was trying to merge files using either of these commands: paste data_{1..2}.txt > 1_2.txt paste <(awk '{print $1}' data_2.txt ) <(awk '{print $1}' data_2.txt) > 1_2.txt But the ...
4
votes
3answers
599 views

Replace “o” with “0” in large wordlist and save the original word?

I have a large text file with words. Every word is on a single line (typical wordlist). I want to replace all characters "o" with the number "0" in every word, and the new formed word to be pasted on ...
-1
votes
2answers
39 views

Searching out a pattern from Logs

The format of the logs is in this manner: Junk Junk Junk : Junk Junk Junk:IP/Port Junk Junk I want to search using the IP and the output should be only the IP/Port part of each line in that log ...
2
votes
4answers
274 views

Only cat from specific line X (with a pattern) to other specific line Y (with a pattern)

A little extended problem from "cat line x to line y on a huge file": I have a huge file (2-3 GB). I'd like to cat/print only from the line having "foo:" to the line having "goo:". Assume that "foo:" ...
0
votes
0answers
54 views

wget: Retrieving a list of URLs when modifying input data file on the fly

This issue is currently driving me up the wall. It just does not work as it should. I have a file inp with audio samples to download where I preserved the internal ID number by parsing some other ...
0
votes
0answers
57 views

Can sed and awk take in multiple files?

grep can search in multiple files, including those files recursively under a specified directory. This makes grep able to find which file contains a pattern. Can awk and sed also? It seems that they ...
1
vote
1answer
32 views

awk: Columns are not printed when concatenating strings is passed as command string

I've found some anomaly when writing a script. The following examples works as expected: $ echo 123 | awk '{print $1 456}' 123456 $ sh -c "echo 123 | awk '{print $1}'" 123 But the following ...
0
votes
1answer
60 views

How to copy rows of different files and paste column-wise?

I want to create a new file by copying rows of each file and pasting columnwise according to their file name orders. Here is the sample input and required output files below. file1.txt 1234 5678 ...
1
vote
2answers
58 views

awk: for-loop with break option

I am trying to use awk to manipulate data.I have a data file with two columns and I would like to find the row that contains an specific value. But there is more than one row that contains this value ...
1
vote
2answers
63 views

grep contents of files based on a list of files

Running CentOS 6.5, qemu-kvm. I have a script which builds a list of running VM's. I want to pull information from those .xml files of only the VM's that are running. cat /dev/null > ...
1
vote
3answers
36 views

Concatenate Sequences under a Heading

I have a file like File1. With a heading for example ">Contig1" and I want to concatenate all the sequences under the respective heading. File1: >Contig1 ACCCCATACATCCAAA NNNNNNNNNNCATACA ...
0
votes
2answers
45 views

joining 2 files on 2 columns but at different column numbers using awk

I have 2 files which has to be joined on $1,$2 of 1st file and $1,$8 of 2nd file using awk.Can anyone please provide the solution for this as this is an urgent requirement. Thanks in Advance file1: ...
1
vote
2answers
52 views

Passing print pattern as a variable to awk

I am trying to use awk to print variables from various columns in a line delimited by , .The columns and pattern which I want to print is stored in another variable b. The pattern a and print patter ...
0
votes
5answers
114 views

print only rows that has latest non empty cell that starts with the word “Failed” on a CSV file in Linux [closed]

I want to print the table only if non-empty latest column that starts with the word "Failed"on a CSV file in Linux.. My input - Name,Mon,Tue,Wed,Thu,Fri,Mon Linux,Successfull Fullbackup; Failed ...
3
votes
3answers
50 views

Removal of lines with more than 'N' fields?

I am working on mac with sed, perl, awk, bash.. I have a large-ish (10GB) text file which has 13 fields (columns) of TAB delimited data. Unfortunately some of these lines have extraneous TABs , so I ...
0
votes
1answer
27 views

How to print 1st and 4th variable only if 4th variable is > or equal to 25

Networking: [Test] => 10.729519791085 GAMES: [Real] => 10 Fashion: [Test] => 27.729519791085 Movies: [Real] => 5 Food: [Test] ...
5
votes
3answers
126 views

Remove duplicate fields in a given column

I'd like to remove from a given column ($2 in the example) the duplicate fields (comma separated). Input file: A 1,2,3,4 B 4,5,6,3 C 2,15 Expected output: A 1,2,3,4 B 5,6 C ...
3
votes
4answers
187 views

Count how many elements in the nth column in a whole file

Input file: A 1,2,3,4 #length($2)=4 B 1,2 #length($2)=2 C 9,8,7,6,5,4 #length($2)=6 Expected output: 12 #4+2+6 A method like: awk -F '[\t,]' '{print length($2)}' but ...
3
votes
3answers
129 views

Parse multiple sections of data into seperate files

I have a concatenated log file with multiple logs inside that I'm trying to parse out into individual log files (I will later rename them to the date/time of each). Each log is separated by "--- LOG ...
2
votes
2answers
77 views

Delete rows in a csv file based on a specific column value

I have a csv file with the following headers - Interaction_ID, PDB_ID, First_Residue, First_Chain, Second_Residue, Third_Residue, Third_Chain & Pattern: ...