awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

3
votes
1answer
156 views

How to print incremental count of occurrences of unique values in column 1

I'm trying to come up with an solution to this problem, I need to incrementally count and then print the counts of the unique values in column 1 of a tab delimited text file. Here is an example: ...
0
votes
1answer
239 views

Avoid conversion to scientific notation in awk

I have a csv file that contains hundreds of columns. I would like to subtract the value of a specific column from a static number and then print the entire line. The values of that column are long ...
1
vote
2answers
127 views

Appending an input file with awk after a pattern (looping?)

I am trying to take a text file called 'input', of the form line1 line2 line3 PATTERN x y z x y z lineN and use awk to place a number (1) after each "x y z" line. I need the lines before the ...
4
votes
1answer
2k views

remove lines where a field's value is less than or equal to 3 - sed or awk?

I need to remove every line that has a value of 2 or less in the 8th field (column). My data looks like this: 12-31 Airport 189 379 41 49.70946503 -124.91377258 2 2880 30.8 01-01 ...
3
votes
5answers
609 views

Pick the latest file based upon the sequence number given in the file name

I have a requirement, I have few files in a directory, I need to pick the oldest one for processing. For Exp: I have below files in /tmp directory: Sample_0000237826-001_xyz.dat ...
0
votes
0answers
11 views

use php variables for awk regex commands [duplicate]

trying to implement autosuggest using php. autosuggest list generated using awk $input= $_GET["term"]; $suggest=array(); exec('awk \'BEGIN{FS=","}/.$input./{print $2,"-cost-",$1}\' ...
2
votes
3answers
170 views

Find the number FN with awk

I have one document, and this document has several columns, like this: ,Flag2,,Flag4,Flag5,,,,,Flag10,Flag11,Flag12,Flag13 If I use awk to know the total number of fields, knowing the separator ...
1
vote
3answers
54 views

Separating names separated by “| character” to individual pairs

I want to separate the names in the same line into separate individual pairs. TMPRSS2|pp9284 AADAT Sample1 ERG TMPRSS2|pp9284 Sample2 TMPRSS2|pp9284 ETV1 Sample3 PDE4A ...
0
votes
3answers
738 views

Awk/sed command to cut on multiple delimiters [closed]

I have input file name as: Sample_0000860156-001-out_20150224150524.xml.gz I want only 0000860156-001 from the file_name to append to my target file name.
1
vote
1answer
526 views

Print lines between two patterns matching a condition in awk

How can I print lines between two patterns which meet a certain condition? For instance for an input file partially containing the following: Time %MEM %CPU %SWAP 00:05:02 7.3 ...
5
votes
4answers
6k views

Redirecting Output of print to multiples files in awk

I want to print same lines to 2 different files. What I am doing right now is: print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file1.txt"; print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file2.txt"; ...
0
votes
1answer
42 views

Printing out range of a pattern with awk

Im trying to print out a section of a txt file using awk. Basically I want to print out the paragraph in the a txt file that begins with certain words all up to a blank line.Here's what I have so far: ...
2
votes
3answers
3k views

Trouble in adding a column with awk

I'm trying to add a column to a csv file by using: awk -F "," '{print $0"new_col"}' file.csv but I got : new_col8,0.405,0.101 new_col4,0.374,0.100 new_col23,0.342,0.100 new_col130,0.298,0.100 ...
2
votes
3answers
461 views

how to use awk/cut to get column data which has space

I have data in below format: 16 SQL*Plus vilconv1 dox-conv2 16 TOAD background query session Disha WORKGROUP\AD now I want to ...
1
vote
2answers
222 views

Last line is wrong when splitting a file with awk

I have a data file main.txt that looks like this 20130826,aaaaaaaaaaaaaa,bbbbbbb 20130826,sdfasdfasdfas,sdfasdfasd 20130826,dfasdfas,asdf2323 ... 20130827,sfasdfasdfasd,sdfasdfwea I used the ...
2
votes
2answers
234 views

Expose awk over tcp (inetd, socat, etc.)

I was playing around with awk, and I wanted to expose it over a tcp socket. At first I tried, but failed with micro-inetd. Thinking that maybe the issue could be in the way that carriage ...
3
votes
3answers
14k views

Insert text in specific lines of a file?

How can insert some text in specific lines of a file? What is most simplest method that I can use? (bash, sed, awk?) What I want to do, might be simple, but I don't know where I should start. It's ...
0
votes
1answer
88 views

Space removal after a pattern in SED

I have a file instrument.xml bash-4.1$ more instrument.xml <instr instr_id="CPT-2380950-011-001-826" valor_ch="" instr_type="CurrentAccount" instr_name="CPT-2380950-011-001-826" qn_type="1" ...
4
votes
4answers
258 views

Replace a marker in a text file with arbitrary new text

I have a file which contains some text and - on a single line - a marker which indicates where new content should be added foo bar %SUBSTITUTE% foo The line line substitute should be replaced by a ...
25
votes
12answers
9k views

Remove duplicate $PATH entries with awk command

I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my path environment. I was told that it is possible to achieve this with a one line ...
0
votes
3answers
54 views

Split the file comparing two fields in the files

Have two files which need to compare the first two columns. Sample input file1: |CID|SID|order|orderdate|confirmdate |151244127820|7177638911583| 2|2015-02-04 07:14:44|2015-02-04 07:15:32 ...
2
votes
1answer
70 views

how can I add an extra character after a word search [closed]

I have the following field names demelog voyapro naisjdf naismc decoide decoccm travide travccm equiccm mariccm Below is a sample of my file. There are more columns before the first fck=83... ...
2
votes
2answers
151 views

Delete rows from a simplified CSV where some columns matching specific pattern

I have the following simplified CSV file (no separators or newlines embedded in fields): ...
1
vote
2answers
1k views

extract blocks of text from xml file

I have a file with these contents: <People> <Person> <Name>Joe Blogs</Name> <Address>55 Oxford St</Address> ...lots of other properties ...
2
votes
3answers
788 views

How to insert bash variables in awk?

How to insert bash variables in awk (for example i need to do in in some for loop - like this: in first iteration use awk to search string by first column, next by second column and so on with using ...
1
vote
3answers
95 views

Filter csv file based on extended column values

I have the following csv file: ID,PDBID,FirstResidue,SecondResidue,ThirdResidue,FourthResidue,Pattern RZ_AUTO_1,4tov,1404,1405,1518,1519,CG/AA Canonical ribose-zipper ...
0
votes
2answers
92 views

what are considered old and powerfull commands? [closed]

I came across the terms awk and sed, awk goes once through all lines and performs a task whenever a line meets a certain condition, sed can manipulate a stream of input before it goes further to the ...
0
votes
3answers
229 views

How can I output sample rows from a .csv?

I'm doing a lot of data merge printing and would like to be able to output, say, 10 samples from a large database. My goal would be to extract the 10 rows spread evenly from the supplied .csv ...
5
votes
3answers
451 views

Can tr commands be chained to avoid multiple tr processes in a pipeline?

I have a bunch of txt files, I'd like to output them lower-cased, only alphabetic and one word-per line, I can do it with several tr commands in a pipeline like this: tr -d '[:punct:]' ...
1
vote
2answers
77 views

How to match multiple lines starting with a TAB, and the line before the 1st one in a group?

I've got some third party log files I'm trying to pull errors out of on the command line. The logs look like this: time=1 time=2 time=3 at com.test.com.... at com.test.com.... at ...
0
votes
2answers
294 views

awk print a match only if last record before other match

I want to use awk for printing a match of some pattern, only if it is the last line of the file before some constant text. For example, i want to print the value of "c" only if it is the last line ...
2
votes
2answers
1k views

Delete rows in a simplified csv file (one newline per row) based on a specific column value

I have a csv file with the following headers - Interaction_ID, PDB_ID, First_Residue, First_Chain, Second_Residue, Third_Residue, Third_Chain & Pattern: ...
-1
votes
1answer
2k views

search for a character in one column and replace value in another column of matched records

I have a requirement to find value 2 in column 4 and wherever it matches replace 6th column with '-'sign in the beginning suppose if i have csv file - file1 like this abc,0,def,1234,ijk ...
1
vote
2answers
150 views

Remove a line if regex exists only in 1st column of each line in a csv file

I have the following comma-delimited data (spaces used for visual clarity): 1 2014 23 2352 19.24 7.18 1 2014 23 2353 19.78 4.65 24 2014 23 2354 20.26 25.34 1 2014 23 2355 26.65 42.76 1 2014 23 ...
2
votes
2answers
48 views
10
votes
4answers
1k views

Output part of each line to a separate file

I have a file like this: a AGTACTTCCAGGAACGGTGCACTCTCC b ATGGATTTTTGGAGCAGGGAGATGGAATAGGAGCATGCTCCAT c ATATTAAATGGATTTTTGGAGCAGGGAGATGGAATAGGAGCATGCTCCATCCACTCCACAC d ...
0
votes
1answer
109 views

Run unix command on awk field [duplicate]

I want to run the basename command on a certain awk field. echo "1 /this/is/a/path" | awk '{print $1" "system("/usr/bin/basename " $2)}' but the output always produces a 0 from the system command. ...
0
votes
1answer
61 views

Filter by column value and write to file

I have a table like this: BB1520_v1.0_0103 exon 3329 3601 BB1520_v1.0_0103 exon 3650 4040 BB1520_v1.0_0089 exon 1 862 BB1520_v1.0_0089 exon 1312 ...
0
votes
1answer
198 views

Reading a string till a key word and replacing from there with another string

I am new to UNIX and am practising shell scripting. I have an application where I need string searching and replacement. There are some lines in a file 1 and file 2. I have a search key word. If the ...
1
vote
1answer
126 views

When will awk treat newline character as `;` and when not?

I'm reading the famous awk tutorial from http://www.grymoire.com/Unix/Awk.html. In which it mentioned To be precise, in the original AWK you can insert a new line character after the curly braces, ...
3
votes
3answers
358 views

Why awk understand FS=“*” but not for FS=“-*-”?

I got a test file, its content is: a -*- b I used awk 'BEGIN {FS="*"} {print $2}' test, it prints out - b Correct! But when I use awk 'BEGIN {FS="-*-"} {print $2}' test, I got: * I know that ...
3
votes
2answers
452 views

how to run awk twice across the same file

I have a log file with user_id's and another column with contest results. I would like to: find all the user_ids of the users who won given those user_id's, return ALL the log entries for those ...
1
vote
1answer
103 views

Comparing two files using awk language

I have two files named file1 and file2. I want to compare with the NR of first files with the first field of file2 i.e. $1. Like this: if(file1.NR==file2.$1){ then do some calculations}. Please help ...
1
vote
1answer
148 views

setenv from awk in tcsh

I am trying to set environment variables one per line from awk: $ <source command> awk '/inst/ { n = split($0,a,"/"); cmd = "setenv " a[n-1] "_ver" $0; system(cmd) }' However the out I get is ...
4
votes
1answer
173 views

Why does awk '{print “Hi, Judy”}' print “Hi, Judy” repeatedly when pressing Return?

For my understanding, usually awk follows a file or an output. Then it will exec its code on everyline of that file or output. In my case, my command awk '{print "Hi, Judy"}' didn't follow by any ...
1
vote
1answer
159 views

Searching file based on data range

I previously posted requesting help with counting occurrences of a string. I'm now hoping to search for the occurrence of a string within a range of values and print out a similarly formatted file ...
-1
votes
2answers
29 views

Need to merge mulitple unix txt lines [closed]

Hi I need to merge multiple lines too one line in txt using awk,have tried different things but can't get the result I'm after! An example; need it to go from this: 11 01 30 31 33 37 39 39 30 30 30 ...
2
votes
1answer
81 views

How to remove duplicate lines while keeping the order AND ignoring case?

[user@notebook ~]$ printf '1\n5\n3\n3\n4\n2\nA\nB\na\n' 1 5 3 3 4 2 A B a [user@notebook ~]$ [user@notebook ~]$ printf '1\n5\n3\n3\n4\n2\nA\nB\na\n' | awk '!seen[$0]++' 1 5 3 4 2 A B a [user@notebook ...
0
votes
2answers
34 views

How to select specific sets or sturture from a file, if struture contains specific string?

I want to extract the sets with having specific string, for example: file abc.txt contains abc 123 abcd 1234 abcde 12345 abc 123 abcd 123 abcde 12345 abc 123 abcd 1234 abcde 12345 Now from the ...
1
vote
2answers
239 views

Use awk to replace a column without changing format [closed]

I have a PDB file (see below) and I need to change the A in column 5 to a B, AND I need to change it without changing the format of the original file. I was able to change the A to a B using this awk ...