awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

3
votes
1answer
152 views

awk causing memory leak?

I am using awk to process a huge data file (about 100 million lines). This is my expression- tac <myfile> | awk -F',' 'OFS=\",\" ...
3
votes
4answers
184 views

pattern match and capturing in unix

I have a file like this: A1-1915435_1_MIMAT0000069_MirBase_miR-16_microRNA_1_22-A1-1915435_1_ENSG00000206737_ENST00000384010_U1_snRNA_73_115 ...
3
votes
3answers
190 views

Extract lines between and including 2 patterns

I have a file with data structured as follows 1, p1, p2, p3, p4, p5, p6, p7 2, p9, p10,, 2, q1, q2, q3, q4, q5, q6, q7 2, q9, q10,, 2, r1, r2, r3, r4, r5, r6, r7 2, r9, r10,, 1, s1, s2, s3, s4, s5, ...
1
vote
1answer
66 views

Analysing nginx error log

I am looking for a way to the following. Here is one line from nginx's error log. 2014/12/29 21:17:57 [error] 30078#0: *311826 openat() ...
1
vote
2answers
115 views

Remove a line if regex exists only in 1st column of each line in a csv file

I have the following comma-delimited data (spaces used for visual clarity): 1 2014 23 2352 19.24 7.18 1 2014 23 2353 19.78 4.65 24 2014 23 2354 20.26 25.34 1 2014 23 2355 26.65 42.76 1 2014 23 ...
4
votes
5answers
269 views

how to split file and lines

I have a text file like : 1_01 { ; quite good spirals reset=2000 type=mandel passes=1 corners=-0.6014129278/-0.5990935452/0.427747516/0.429487053 params=0/0 float=y ...
3
votes
2answers
160 views

Comparing two files in unix and awk

I have to compare two files, file1 and file2. Each file has 56 columns separated by |. First column is the employee number in the file, I will check whether same employee number is present in the ...
3
votes
4answers
187 views

How to swap entire html blocks in multiple files with sed

I have tons of html pages with a simlar content as follows: <div class="wrapper"> <div class="aaa"> ...lot of html1 here like div's/spans etc.. </div> <!-- end aaa --> ...
2
votes
7answers
2k views

Print first and last line of all files in folder

I have a bunch of log files that get overwritten (file.log.1, file.log.2 etc). When I copy them from the device making them onto my local machine I lose the original time stamps. So I'd like to put ...
2
votes
1answer
196 views

How to count the number of occurrences of a number larger than x, from every column?

I'm trying to figure out how to count the number of occurrences of values that are greater than some number (i.e. 0.1) in each column and then print that value out for each column. Here would be some ...
5
votes
1answer
210 views

How to calculate percent of every column?

I'm trying to transform some data to a percent of the total of each column, very similar to this thread except that I need to do this for every column: Calculate and divide by total with AWK Data ...
5
votes
4answers
198 views

Increment numeric pattern in file

I have a File that contains only the following line: BOF (REF-REQ0001 SW_requirement) EOF I need to generate a second file that would contain the following: BOF (REF-REQ0001 SW_requirement) ...
3
votes
4answers
159 views

tab separation of the file

I have a file with seq likes this ATCGTTTCTCAGCCTTTTGGCAAGACCAAGTGTAGTATCTGTTCTTATCAGATATTGGTATCTGCAGGCACTCCTGGAACCAAT ...
-2
votes
4answers
579 views

How to ignore the lines starts with # using grep / awk

cat /etc/oratab #test1:/opt/oracle/app/oracle/product/11.2.0.4:N +ASM2:/grid/oracle/app/oracle/product/11.2.0.4:N # line added by Agent test2:/opt/oracle/app/oracle/product/11.2.0.4:N ...
3
votes
3answers
248 views

awk manipulation of file

I have a big file in this format, whose first few lines I am showing you including the header. gene c1 c2 c3 c4 c5 c6 c7 c8 G1*1 0 0 0 0 0 0 0 0 G2*2 0 0 0 0 1 ...
1
vote
1answer
97 views

awk array using number as value did not work

I have a reference file: Refrence File Dpse\GA30012 FBgn0000447 chr2 26607738 26607962 -1 Dpse\GA19764 FBgn0085819 chrX 28571020 28571736 -1 Dpse\ttk FBgn0000100 chr2 ...
3
votes
3answers
1k views

Calculate and divide by total with AWK

Given the following data file... foo 10 bar 20 oof 50 rab 20 ... how would I print column two as a percent of the total of column two? In other words, I want... foo 10 10% ...
-1
votes
2answers
222 views

understanding sed, awk and curl interaction extracting ssh URL information from github page

I needed to pull every public repo in a user's GitHub account. To do this I got the API output of https://api.github.com/users/SOMEUSER/repos This is what I came up with. To extract the ssh_url line ...
0
votes
2answers
50 views

Regex gone Wrong

I am using a Bash script to use awk to look for this: (0010,0080) The awk will generally look for any numeric value seperated by a comma in between parenthesis. I have done everything I can think of ...
-2
votes
1answer
72 views

Making use of awk on strings [closed]

I'm interested in listing in the terminal the lines from a csv file where the first character is 'A' . Is there a prompt way to do that?
2
votes
3answers
378 views

Calculate and round numbers up in bash

I'm struggling with a slight variation on: How to calculate by taking first field and How to round floating point numbers in shell? I have a file that looks like (space between columns): 1533 ...
0
votes
3answers
337 views

kill several processes using awk tools

I want to stop several processes using awk.My command is like this: sudo ps -ef|grep wget |grep -v grep|awk '{print $2;sudo kill -STOP $2}' I want to stop all processes which is running wget. $2 of ...
1
vote
1answer
150 views

Counting the instances of specific words in a text, using awk [duplicate]

If I have a file words_of_interest.txt with one word per line, is there a way to use awk (or some other *nix tools) to obtain the number of times each of these words occurs in another text file ...
2
votes
1answer
42 views

comparing 2 lists with additional data to be printed

I have a file lets say file1.txt ENSG00000223116 AL157931.1 ENSG00000233440 HMGA1P6 ENSG00000207157 RNY3P4 ENSG00000229483 LINC00362 ENSG00000252952 RNU6-58P I have another file say ...
1
vote
2answers
57 views

Parse command output [duplicate]

I want to only get the value associated to "cached" column of the free -m command. What is the most efficient way to achieve it? Something with sed or awk?
0
votes
2answers
97 views

Using variable with awk -v in a shell script

I am modifying/re-writing some of the scripts written by former employees of my company and in one of the shell scripts I found the following line. b=`benchmark=30;grep "Waiting for timer" wk.txt | ...
4
votes
3answers
979 views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
4
votes
1answer
75 views

Insert missing data within blocks

I`m a biologist and I am trying to solve this problem, please guide me on where to start. I do not know which forum to post this in, please let me know if this place is not appropriate. I have blocks ...
-1
votes
2answers
45 views

Extract 2 fields from 2 different lines

I have the below lines in my file # Employee Id : ABC123, # Employee Name: Vivek # Employee Type: Temp Join Date: 31-jan-2014 Buz Code: 2098 # Employee Comments: # ...
1
vote
7answers
170 views

repeat a line, splitting one field

I have a tab delimited file in which in every line I have this: K00001;K00004;K00008 0 0 34 0 0 0 0 0 0 0 0 0 0 0 0 0 36 0 0 52 0 0 0 6 0 I would ...
1
vote
1answer
40 views

how to move a header to the last column, using awk or sed

how to move a header to the last column, using awk or sed input file look like this: Line 1.000N x y z 23.88 44.66 56.6 23.81 41.66 53.6 Line 81.000N x y z 13.88 34.66 56.6 13.81 ...
1
vote
2answers
184 views

how to compare values in two columns in two different files, echoing full lines where absolute value of difference is < a small maximum value?

I want to do a line-by-line comparison of values in, say, column 6 of two files, and then echo the lines and their full contents, piping them into a combined file, say the line from one file and then ...
2
votes
1answer
86 views

Specify lines in awk?

Im going to use this awk script to delete duplicate lines from my text file: awk '!x[$0]++' However, I want to specify which lines that they are to be run on. There could be 10000 duplicates, but ...
0
votes
2answers
146 views

delete multiple users

I am the root user and I am setting up a menu for another user to use. This other user will only get this menu. There are two options that are interlinked: the first option is to search users. The ...
4
votes
2answers
287 views

grep for repeated values from pattern list

I have file A.txt which is a lists of unique ids (column 1) and its corresponding values(column 2): A.txt ABC1D_T1_B1 123.4 ABC2D_T1_B2 146.7 ABC3D_T1_B1 567.8 ABC4D_T0_B2 96.8 ABC1D_T0_B1 ...
2
votes
2answers
42 views

Create small record files from a big group of record file

I have a file like this : Mainfile: Record|1111|ABC text in between for record 1 text in between for record 1 Record|2222|XYZ text in between for record 2 Record|3333|XYZ ...
2
votes
7answers
2k views

How to print lines with blank 5th field in CSV

I have to check if the fifth field is empty in a CSV file. This is my file: 1,abc,543,87,DATA,fsg; 1,abc,543,87,,fsg; 1,abc,543,87,DATA,fsg; 1,abc,543,88,,fsg; 1,abc,543,,DATA,fsg; As you can ...
1
vote
1answer
49 views

awk how to set parameter $2 become $b?

In a Bash script: #!/usr/bin/bash #grep() #{ #grep -q #} if [[ -z $1 ]]; then echo "usage: pkill -signal process name " fi if [[ -z $2 ]]; then echo "error: not enough parameters " exit fi ...
2
votes
2answers
61 views

Combine output from df -h /dev/xvda1 into a single line

I want to convert the 2 line output from df -h into a single, meaningful line i.e. Take [root@ip-10-42-1-246 tmp]# df -h /dev/xvda1 Filesystem Size Used Avail Use% Mounted on /dev/xvda1 ...
2
votes
2answers
2k views

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
0
votes
4answers
83 views

Print last character in the given path

I am trying to concat "comnp.dat_CY*" files into one file. File name, source path and destination path are the parameters. I am trying to find the last character in the path to check if it is "/" or ...
-5
votes
3answers
579 views

How do you print the current month, day and year using awk and date?

I can't seem to grab the month, day and year using awk '{print $2, $3, $6}' date Does anyone know how to?
3
votes
1answer
48 views

create a realtime machine by assigning CPU cores to critical processes

I'm using an Udoo board for running two critical processes. It can run Ubuntu ARM edition and it has a Quadcore processor. I need to set the affinity of all the processes afinity running and the ones ...
2
votes
2answers
176 views

How to base awk selection on contents of 1 field?

How can I select all the records that have '2' as the second field? My data is: $ cat numbers.txt 1 2 3 4 5 6 7 8 2 4 6 8 10 12 14 16 3 6 9 12 15 18 21 24 My awk is: awk '$2 - /^2$/ {print}' ...
3
votes
2answers
136 views

Read a two-character column as two separate columns

I have a file with genotype data. The second column has both alleles for a particular genetic variant concatenated, as below. rs969931 CA 1.000 2.000 2.000 2.000 2.000 2.000 1.000 ...
4
votes
3answers
1k views

awk + print lines from the first line until match word

I want to print all lines from file until the match word please advice how to do that with awk for example I want to print all lines until word PPP remark the first line chuld be diff from AAA ( ...
1
vote
1answer
54 views

Extracting trace information using gawk

I am new to awk programming and am using gawk to extract trace information from a mrt file in order to use it further for analysis. I have successfully been able to extract trace information from a ...
0
votes
2answers
58 views

Substituting \ for \\ in column

I exported a column of thousands of salts from Mysql for hashcat. Every instance of the character \ was replaced with a double character \\ in the exported column. Is there a way to either sed or awk ...
0
votes
4answers
91 views

sed remove last space only on end of line,combined with awk

I have those line Siouxsie Sioux Siouxsie Sioux ... Sylvia Tobias Sylvia Tobias ... Sydney Urshan Sydney Urshan ... Club Drummer (uncredited) Club Drummer (uncredited) ... ...
0
votes
1answer
67 views

Issue finding lines using grep in UNIX [closed]

I am trying to use a .txt file with around 5000 patterns (spaced with a line) to search through another file of 18000 lines for any matches. So far I've tried every form of grep and awk I can find on ...