awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

4
votes
2answers
62 views

How to print matching value of patterned field separator in AWK

In the below command, cat abc.txt|awk -F"[0-9]+." '{print FS $3}' Here FS prints [0-9]+.. However, I want the awk to print the matched value by that field separator e.g. "99." . Any idea how can ...
0
votes
2answers
37 views

Multiply a column using awk

How do I multiply a column by an integer (say 3) and replace the older value by new one, using awk? Input: Data 9390.900391 10573.089844 80.000000 200.000000 2.700000 Data ...
1
vote
2answers
84 views

Formatting ifconfig using sed/awk

I'm using ifconfig on OpenSUSE. When I run ifconfig eth0 I get eth0 Link encap:Ethernet HWaddr CE:FD:75:DF:A5:6D inet addr:172.16.4.177 Bcast:172.16.5.255 Mask:255.255.254.0 inet6 ...
1
vote
2answers
140 views

Multiple File Redirection with awk and find

I have some files in a directory. And I want to add some lines at top and end of file using awk. Example: My awk command: awk 'BEGIN { print "line1\nline2" } { print $0 } END { print "line3\nline4" ...
2
votes
3answers
175 views

How can I insert the contents of a file into a string in bash

I would like to append the contents of a multi-line text file after a particular line in a string. For example, if the file file.txt contains line 1 line 2 I'd like do so something like printf ...
1
vote
1answer
185 views

Compare two Arrays in KSH and output the difference

I am not extremely familiar with KSH (Actually just started using it) and I am having problems with trying to create a script that will essentially compare two arrays that have been stored and then ...
-4
votes
1answer
194 views

Merge two csvfiles by column header

I have two csv files: success.csv "ID","Legacy ID","Field1", "Field2" 1,1111,google,news 2,2222,yahoo,news error.csv "Legacy ID","Field1","Field2","Message" 3333,aol,news,"Failed to upload ...
3
votes
4answers
156 views

How to swap entire html blocks in multiple files with sed

I have tons of html pages with a simlar content as follows: <div class="wrapper"> <div class="aaa"> ...lot of html1 here like div's/spans etc.. </div> <!-- end aaa --> ...
3
votes
5answers
412 views

Awk - output the second line of a number of .dat files to one file

I have searched through previously asked questions and can't find an answer to what I am trying to do. note, I am new to linux. I have multiple files something like: (in reality i have 80) file1.dat ...
2
votes
6answers
6k views

Extract substring using regular expression on a Unix file

I have file with below contents . /ABC/RTE/AD_900_VOP_123/OPP /ABC/RTE/TRE/AD_900_VOP_145/BBB /ABC/RTE/AN_900_VFP_124/FBF /ABC/RTE/HD_900_FOP_153/WEW /ABD/RDV/AD_900_VOP_123/OPP ...
2
votes
2answers
115 views

Extracting values from a file keyed by multiple keys

Consider a file with key=value pairs, and each key is optionally a concatenation of multiple keys. In other words, many keys can map to one value. The reason behind this is that each key is a ...
3
votes
1answer
42 views

Filter out inconsistent data blocks

I would like to filter out rows that has inconsistent geno values in line-part blocks. In case of duplicate consistent values , I would just keep the first row. As example, 'gf345 part1' has more ...
0
votes
1answer
36 views

For value falls within a range get corresponding value

I am thinking to use awk to search a value from input file in a reference file and get a corresponding value from reference file. Both reference and input files can be sorted based on $3 and $2 ...
1
vote
1answer
263 views

Awk compare 2 files, print match together with non match lines filled with 0

I have two files. I want to match the 1st column of first file to 1st column of second file. If there is a match I'd like to print the 2st column of second file, if not 0. I have tried awk (awk ...
1
vote
2answers
54 views

splitting a file by column number

I have a file with 10,671 columns. I am trying to split it into 10 smaller files, each with 1000 columns until all columns have been printed, named file_transposed01, file_transposed02, ... ...
3
votes
4answers
574 views

Aggregate and group text file in perl or bash

I have a big text file(with 5m lines) in this format(4 columns, separated by ;): string1; string2; string3; userId The first 3 strings (SHA1s) form a single ID, called appId (so it can be ...
0
votes
2answers
42 views

Rearranging data in file with awk

I have a table ID Comment Statement 1 hi hello 2 hi hello 1 cya bye 1 k ok 2 kk ok How would one go about producing a result ...
2
votes
3answers
74 views

Rename mp3 file names to a pattern

I have a list of like names likes so: /A/B/C/D/E/F/filename_album1-003.mp3 its more like album\d+ I would like to change it to something like /A/B/C/D/E/F/filename_song-3.mp3 song is just a ...
2
votes
2answers
97 views

Filtering a file on a string

I have a file which looks like this a 0 gene1 56 0 6S32M12S * b 256 gene2 56 0 6S32M12S * c 256 gene3 55 0 6S27M17S * d 16 gene4 110 9 19S25M6S * e 272 ...
0
votes
1answer
31 views

putting headers back after awk command

I have file like this. Its called a sam file @HD VN:1.0 SO:unsorted @SQ SN:RNU2-1 LN:388 @SQ SN:RNU6-1100P LN:304 SRR959756.23 0 RNU2-1 56 0 6S32M12S * 0 0 ...
1
vote
2answers
66 views

How to make a filter more efficient

I've created a script to compare a list of users from a MSSQL database against a list of disabled AD users. The tool we use for AD authentication returns data in the following format (with plenty ...
1
vote
1answer
158 views

awk in ssh in su in a command substitution

I am creating a script that will ssh to a host and print all the user accounts and when they will expire. On a host I can run awk -F':' '{ print $1}' /etc/passwd and it will give me a list of all ...
1
vote
1answer
63 views

Get file names in awk result

I have many files in a location where I need to list file names in which 32th column has value more than 3000. This is the command that gives me those records: awk -F, '$32 > 3000' * Now can ...
3
votes
1answer
122 views

awk causing memory leak?

I am using awk to process a huge data file (about 100 million lines). This is my expression- tac <myfile> | awk -F',' 'OFS=\",\" ...
2
votes
1answer
199 views

Merge some tab-delimited files

I have 100 files with 57,816 rows each.I want to do an outer join of these files by merging on basis of the common column. I am using R programming for this but this is very slow. fileList <- ...
4
votes
5answers
11k views

removing the first and the last character of every line from command line

I am trying to remove the first and the last characters of everyline in a text file and save the resulting truncated version in a new file. Does anyone have an idea about how to do that efficiently ...
3
votes
4answers
170 views

pattern match and capturing in unix

I have a file like this: A1-1915435_1_MIMAT0000069_MirBase_miR-16_microRNA_1_22-A1-1915435_1_ENSG00000206737_ENST00000384010_U1_snRNA_73_115 ...
3
votes
3answers
150 views

Extract lines between and including 2 patterns

I have a file with data structured as follows 1, p1, p2, p3, p4, p5, p6, p7 2, p9, p10,, 2, q1, q2, q3, q4, q5, q6, q7 2, q9, q10,, 2, r1, r2, r3, r4, r5, r6, r7 2, r9, r10,, 1, s1, s2, s3, s4, s5, ...
1
vote
1answer
48 views

Analysing nginx error log

I am looking for a way to the following. Here is one line from nginx's error log. 2014/12/29 21:17:57 [error] 30078#0: *311826 openat() ...
4
votes
5answers
236 views

how to split file and lines

I have a text file like : 1_01 { ; quite good spirals reset=2000 type=mandel passes=1 corners=-0.6014129278/-0.5990935452/0.427747516/0.429487053 params=0/0 float=y ...
5
votes
1answer
162 views

How to calculate percent of every column?

I'm trying to transform some data to a percent of the total of each column, very similar to this thread except that I need to do this for every column: Calculate and divide by total with AWK Data ...
3
votes
2answers
125 views

Comparing two files in unix and awk

I have to compare two files, file1 and file2. Each file has 56 columns separated by |. First column is the employee number in the file, I will check whether same employee number is present in the ...
16
votes
5answers
26k views

How to manipulate a CSV file with sed or awk?

How can I do the following to a CSV file using sed or awk? Delete a column Duplicate a column Move a column I have a big table with over 200 rows, and I'm not that familiar with sed.
2
votes
7answers
1k views

Print first and last line of all files in folder

I have a bunch of log files that get overwritten (file.log.1, file.log.2 etc). When I copy them from the device making them onto my local machine I lose the original time stamps. So I'd like to put ...
2
votes
1answer
118 views

How to count the number of occurrences of a number larger than x, from every column?

I'm trying to figure out how to count the number of occurrences of values that are greater than some number (i.e. 0.1) in each column and then print that value out for each column. Here would be some ...
4
votes
2answers
588 views

Use sed or awk to join lines in a certain pattern?

Not sure on how to turn this input: A B C apartner bpartner cpartner into this output: A apartner B bpartner c cpartner I have this pattern in a 80 line file that needs to be 40 line. Also, how ...
1
vote
1answer
93 views

awk array using number as value did not work

I have a reference file: Refrence File Dpse\GA30012 FBgn0000447 chr2 26607738 26607962 -1 Dpse\GA19764 FBgn0085819 chrX 28571020 28571736 -1 Dpse\ttk FBgn0000100 chr2 ...
5
votes
4answers
169 views

Increment numeric pattern in file

I have a File that contains only the following line: BOF (REF-REQ0001 SW_requirement) EOF I need to generate a second file that would contain the following: BOF (REF-REQ0001 SW_requirement) ...
6
votes
3answers
12k views

Adding a Column of values in a tab delimited file

How can I add a Column of values in a file which has a certain number of rows. I have a input file like this: Input file: SPATA17 1 217947738 LYPLAL1 1 219383905 FAM47E 4 77192838 SHROOM3 4 ...
3
votes
4answers
154 views

tab separation of the file

I have a file with seq likes this ATCGTTTCTCAGCCTTTTGGCAAGACCAAGTGTAGTATCTGTTCTTATCAGATATTGGTATCTGCAGGCACTCCTGGAACCAAT ...
3
votes
3answers
235 views

awk manipulation of file

I have a big file in this format, whose first few lines I am showing you including the header. gene c1 c2 c3 c4 c5 c6 c7 c8 G1*1 0 0 0 0 0 0 0 0 G2*2 0 0 0 0 1 ...
-2
votes
4answers
239 views

How to ignore the lines starts with # using grep / awk

cat /etc/oratab #test1:/opt/oracle/app/oracle/product/11.2.0.4:N +ASM2:/grid/oracle/app/oracle/product/11.2.0.4:N # line added by Agent test2:/opt/oracle/app/oracle/product/11.2.0.4:N ...
3
votes
3answers
572 views

Calculate and divide by total with AWK

Given the following data file... foo 10 bar 20 oof 50 rab 20 ... how would I print column two as a percent of the total of column two? In other words, I want... foo 10 10% ...
-1
votes
2answers
131 views

understanding sed, awk and curl interaction extracting ssh URL information from github page

I needed to pull every public repo in a user's GitHub account. To do this I got the API output of https://api.github.com/users/SOMEUSER/repos This is what I came up with. To extract the ssh_url line ...
2
votes
3answers
298 views

Calculate and round numbers up in bash

I'm struggling with a slight variation on: How to calculate by taking first field and How to round floating point numbers in shell? I have a file that looks like (space between columns): 1533 ...
0
votes
2answers
44 views

Regex gone Wrong

I am using a Bash script to use awk to look for this: (0010,0080) The awk will generally look for any numeric value seperated by a comma in between parenthesis. I have done everything I can think of ...
-2
votes
1answer
71 views

Making use of awk on strings [closed]

I'm interested in listing in the terminal the lines from a csv file where the first character is 'A' . Is there a prompt way to do that?
0
votes
3answers
246 views

kill several processes using awk tools

I want to stop several processes using awk.My command is like this: sudo ps -ef|grep wget |grep -v grep|awk '{print $2;sudo kill -STOP $2}' I want to stop all processes which is running wget. $2 of ...
1
vote
1answer
133 views

Counting the instances of specific words in a text, using awk [duplicate]

If I have a file words_of_interest.txt with one word per line, is there a way to use awk (or some other *nix tools) to obtain the number of times each of these words occurs in another text file ...
4
votes
3answers
508 views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...