awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

2
votes
2answers
97 views

Filtering a file on a string

I have a file which looks like this a 0 gene1 56 0 6S32M12S * b 256 gene2 56 0 6S32M12S * c 256 gene3 55 0 6S27M17S * d 16 gene4 110 9 19S25M6S * e 272 ...
0
votes
1answer
31 views

putting headers back after awk command

I have file like this. Its called a sam file @HD VN:1.0 SO:unsorted @SQ SN:RNU2-1 LN:388 @SQ SN:RNU6-1100P LN:304 SRR959756.23 0 RNU2-1 56 0 6S32M12S * 0 0 ...
1
vote
2answers
66 views

How to make a filter more efficient

I've created a script to compare a list of users from a MSSQL database against a list of disabled AD users. The tool we use for AD authentication returns data in the following format (with plenty ...
1
vote
1answer
144 views

awk in ssh in su in a command substitution

I am creating a script that will ssh to a host and print all the user accounts and when they will expire. On a host I can run awk -F':' '{ print $1}' /etc/passwd and it will give me a list of all ...
1
vote
1answer
63 views

Get file names in awk result

I have many files in a location where I need to list file names in which 32th column has value more than 3000. This is the command that gives me those records: awk -F, '$32 > 3000' * Now can ...
3
votes
1answer
120 views

awk causing memory leak?

I am using awk to process a huge data file (about 100 million lines). This is my expression- tac <myfile> | awk -F',' 'OFS=\",\" ...
2
votes
1answer
193 views

Merge some tab-delimited files

I have 100 files with 57,816 rows each.I want to do an outer join of these files by merging on basis of the common column. I am using R programming for this but this is very slow. fileList <- ...
4
votes
5answers
11k views

removing the first and the last character of every line from command line

I am trying to remove the first and the last characters of everyline in a text file and save the resulting truncated version in a new file. Does anyone have an idea about how to do that efficiently ...
3
votes
4answers
169 views

pattern match and capturing in unix

I have a file like this: A1-1915435_1_MIMAT0000069_MirBase_miR-16_microRNA_1_22-A1-1915435_1_ENSG00000206737_ENST00000384010_U1_snRNA_73_115 ...
3
votes
3answers
149 views

Extract lines between and including 2 patterns

I have a file with data structured as follows 1, p1, p2, p3, p4, p5, p6, p7 2, p9, p10,, 2, q1, q2, q3, q4, q5, q6, q7 2, q9, q10,, 2, r1, r2, r3, r4, r5, r6, r7 2, r9, r10,, 1, s1, s2, s3, s4, s5, ...
1
vote
1answer
46 views

Analysing nginx error log

I am looking for a way to the following. Here is one line from nginx's error log. 2014/12/29 21:17:57 [error] 30078#0: *311826 openat() ...
4
votes
5answers
229 views

how to split file and lines

I have a text file like : 1_01 { ; quite good spirals reset=2000 type=mandel passes=1 corners=-0.6014129278/-0.5990935452/0.427747516/0.429487053 params=0/0 float=y ...
5
votes
1answer
158 views

How to calculate percent of every column?

I'm trying to transform some data to a percent of the total of each column, very similar to this thread except that I need to do this for every column: Calculate and divide by total with AWK Data ...
3
votes
2answers
123 views

Comparing two files in unix and awk

I have to compare two files, file1 and file2. Each file has 56 columns separated by |. First column is the employee number in the file, I will check whether same employee number is present in the ...
16
votes
5answers
26k views

How to manipulate a CSV file with sed or awk?

How can I do the following to a CSV file using sed or awk? Delete a column Duplicate a column Move a column I have a big table with over 200 rows, and I'm not that familiar with sed.
2
votes
7answers
1k views

Print first and last line of all files in folder

I have a bunch of log files that get overwritten (file.log.1, file.log.2 etc). When I copy them from the device making them onto my local machine I lose the original time stamps. So I'd like to put ...
2
votes
1answer
110 views

How to count the number of occurrences of a number larger than x, from every column?

I'm trying to figure out how to count the number of occurrences of values that are greater than some number (i.e. 0.1) in each column and then print that value out for each column. Here would be some ...
4
votes
2answers
586 views

Use sed or awk to join lines in a certain pattern?

Not sure on how to turn this input: A B C apartner bpartner cpartner into this output: A apartner B bpartner c cpartner I have this pattern in a 80 line file that needs to be 40 line. Also, how ...
1
vote
1answer
93 views

awk array using number as value did not work

I have a reference file: Refrence File Dpse\GA30012 FBgn0000447 chr2 26607738 26607962 -1 Dpse\GA19764 FBgn0085819 chrX 28571020 28571736 -1 Dpse\ttk FBgn0000100 chr2 ...
5
votes
4answers
164 views

Increment numeric pattern in file

I have a File that contains only the following line: BOF (REF-REQ0001 SW_requirement) EOF I need to generate a second file that would contain the following: BOF (REF-REQ0001 SW_requirement) ...
6
votes
3answers
12k views

Adding a Column of values in a tab delimited file

How can I add a Column of values in a file which has a certain number of rows. I have a input file like this: Input file: SPATA17 1 217947738 LYPLAL1 1 219383905 FAM47E 4 77192838 SHROOM3 4 ...
3
votes
4answers
152 views

tab separation of the file

I have a file with seq likes this ATCGTTTCTCAGCCTTTTGGCAAGACCAAGTGTAGTATCTGTTCTTATCAGATATTGGTATCTGCAGGCACTCCTGGAACCAAT ...
3
votes
3answers
235 views

awk manipulation of file

I have a big file in this format, whose first few lines I am showing you including the header. gene c1 c2 c3 c4 c5 c6 c7 c8 G1*1 0 0 0 0 0 0 0 0 G2*2 0 0 0 0 1 ...
-2
votes
4answers
209 views

How to ignore the lines starts with # using grep / awk

cat /etc/oratab #test1:/opt/oracle/app/oracle/product/11.2.0.4:N +ASM2:/grid/oracle/app/oracle/product/11.2.0.4:N # line added by Agent test2:/opt/oracle/app/oracle/product/11.2.0.4:N ...
3
votes
3answers
532 views

Calculate and divide by total with AWK

Given the following data file... foo 10 bar 20 oof 50 rab 20 ... how would I print column two as a percent of the total of column two? In other words, I want... foo 10 10% ...
-1
votes
2answers
126 views

understanding sed, awk and curl interaction extracting ssh URL information from github page

I needed to pull every public repo in a user's GitHub account. To do this I got the API output of https://api.github.com/users/SOMEUSER/repos This is what I came up with. To extract the ssh_url line ...
2
votes
3answers
287 views

Calculate and round numbers up in bash

I'm struggling with a slight variation on: How to calculate by taking first field and How to round floating point numbers in shell? I have a file that looks like (space between columns): 1533 ...
0
votes
2answers
43 views

Regex gone Wrong

I am using a Bash script to use awk to look for this: (0010,0080) The awk will generally look for any numeric value seperated by a comma in between parenthesis. I have done everything I can think of ...
-2
votes
1answer
71 views

Making use of awk on strings [closed]

I'm interested in listing in the terminal the lines from a csv file where the first character is 'A' . Is there a prompt way to do that?
0
votes
3answers
238 views

kill several processes using awk tools

I want to stop several processes using awk.My command is like this: sudo ps -ef|grep wget |grep -v grep|awk '{print $2;sudo kill -STOP $2}' I want to stop all processes which is running wget. $2 of ...
1
vote
1answer
131 views

Counting the instances of specific words in a text, using awk [duplicate]

If I have a file words_of_interest.txt with one word per line, is there a way to use awk (or some other *nix tools) to obtain the number of times each of these words occurs in another text file ...
4
votes
3answers
472 views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
2
votes
1answer
41 views

comparing 2 lists with additional data to be printed

I have a file lets say file1.txt ENSG00000223116 AL157931.1 ENSG00000233440 HMGA1P6 ENSG00000207157 RNY3P4 ENSG00000229483 LINC00362 ENSG00000252952 RNU6-58P I have another file say ...
1
vote
2answers
40 views

Parse command output [duplicate]

I want to only get the value associated to "cached" column of the free -m command. What is the most efficient way to achieve it? Something with sed or awk?
0
votes
2answers
71 views

Using variable with awk -v in a shell script

I am modifying/re-writing some of the scripts written by former employees of my company and in one of the shell scripts I found the following line. b=`benchmark=30;grep "Waiting for timer" wk.txt | ...
4
votes
1answer
74 views

Insert missing data within blocks

I`m a biologist and I am trying to solve this problem, please guide me on where to start. I do not know which forum to post this in, please let me know if this place is not appropriate. I have blocks ...
1
vote
7answers
136 views

repeat a line, splitting one field

I have a tab delimited file in which in every line I have this: K00001;K00004;K00008 0 0 34 0 0 0 0 0 0 0 0 0 0 0 0 0 36 0 0 52 0 0 0 6 0 I would ...
2
votes
5answers
12k views

awk-Printing column value without new line and adding comma

input.txt EN1 EN2 EN3 EN4 EN5 output EN1,EN2,EN3,EN4,EN5 I have tried awk.But it is not printing with comma awk 'BEGIN { OFS = ","} { printf $1}' input.txt I have GNU Awk 4.0.0 version
-1
votes
2answers
38 views

Extract 2 fields from 2 different lines

I have the below lines in my file # Employee Id : ABC123, # Employee Name: Vivek # Employee Type: Temp Join Date: 31-jan-2014 Buz Code: 2098 # Employee Comments: # ...
1
vote
1answer
35 views

how to move a header to the last column, using awk or sed

how to move a header to the last column, using awk or sed input file look like this: Line 1.000N x y z 23.88 44.66 56.6 23.81 41.66 53.6 Line 81.000N x y z 13.88 34.66 56.6 13.81 ...
8
votes
6answers
496 views

How to append Line to previous Line?

I have a Log file which need to be parsed and analysed. File contains something similar like below: File: 20141101 server contain dump 20141101 server contain nothing {uekdmsam ikdas jwdjamc ...
1
vote
2answers
130 views

how to compare values in two columns in two different files, echoing full lines where absolute value of difference is < a small maximum value?

I want to do a line-by-line comparison of values in, say, column 6 of two files, and then echo the lines and their full contents, piping them into a combined file, say the line from one file and then ...
2
votes
1answer
85 views

Specify lines in awk?

Im going to use this awk script to delete duplicate lines from my text file: awk '!x[$0]++' However, I want to specify which lines that they are to be run on. There could be 10000 duplicates, but ...
0
votes
2answers
93 views

delete multiple users

I am the root user and I am setting up a menu for another user to use. This other user will only get this menu. There are two options that are interlinked: the first option is to search users. The ...
2
votes
2answers
40 views

Create small record files from a big group of record file

I have a file like this : Mainfile: Record|1111|ABC text in between for record 1 text in between for record 1 Record|2222|XYZ text in between for record 2 Record|3333|XYZ ...
0
votes
1answer
55 views

Replacing code between two JavaScript comments

I need to set a value of JavaScript variable from bash. Variable lives in index.html file and I'd like to use unix SED command to do that. Inside index.html I have it lake this: <script> ...
2
votes
7answers
812 views

How to print lines with blank 5th field in CSV

I have to check if the fifth field is empty in a CSV file. This is my file: 1,abc,543,87,DATA,fsg; 1,abc,543,87,,fsg; 1,abc,543,87,DATA,fsg; 1,abc,543,88,,fsg; 1,abc,543,,DATA,fsg; As you can ...
4
votes
2answers
232 views

grep for repeated values from pattern list

I have file A.txt which is a lists of unique ids (column 1) and its corresponding values(column 2): A.txt ABC1D_T1_B1 123.4 ABC2D_T1_B2 146.7 ABC3D_T1_B1 567.8 ABC4D_T0_B2 96.8 ABC1D_T0_B1 ...
1
vote
1answer
36 views

awk how to set parameter $2 become $b?

In a Bash script: #!/usr/bin/bash #grep() #{ #grep -q #} if [[ -z $1 ]]; then echo "usage: pkill -signal process name " fi if [[ -z $2 ]]; then echo "error: not enough parameters " exit fi ...
2
votes
2answers
52 views

Combine output from df -h /dev/xvda1 into a single line

I want to convert the 2 line output from df -h into a single, meaningful line i.e. Take [root@ip-10-42-1-246 tmp]# df -h /dev/xvda1 Filesystem Size Used Avail Use% Mounted on /dev/xvda1 ...