awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

0
votes
1answer
32 views

putting headers back after awk command

I have file like this. Its called a sam file @HD VN:1.0 SO:unsorted @SQ SN:RNU2-1 LN:388 @SQ SN:RNU6-1100P LN:304 SRR959756.23 0 RNU2-1 56 0 6S32M12S * 0 0 ...
2
votes
2answers
101 views

Filtering a file on a string

I have a file which looks like this a 0 gene1 56 0 6S32M12S * b 256 gene2 56 0 6S32M12S * c 256 gene3 55 0 6S27M17S * d 16 gene4 110 9 19S25M6S * e 272 ...
1
vote
1answer
224 views

awk in ssh in su in a command substitution

I am creating a script that will ssh to a host and print all the user accounts and when they will expire. On a host I can run awk -F':' '{ print $1}' /etc/passwd and it will give me a list of all ...
1
vote
2answers
67 views

How to make a filter more efficient

I've created a script to compare a list of users from a MSSQL database against a list of disabled AD users. The tool we use for AD authentication returns data in the following format (with plenty ...
1
vote
1answer
72 views

Get file names in awk result

I have many files in a location where I need to list file names in which 32th column has value more than 3000. This is the command that gives me those records: awk -F, '$32 > 3000' * Now can ...
2
votes
1answer
259 views

Merge some tab-delimited files

I have 100 files with 57,816 rows each.I want to do an outer join of these files by merging on basis of the common column. I am using R programming for this but this is very slow. fileList <- ...
3
votes
1answer
139 views

awk causing memory leak?

I am using awk to process a huge data file (about 100 million lines). This is my expression- tac <myfile> | awk -F',' 'OFS=\",\" ...
3
votes
4answers
174 views

pattern match and capturing in unix

I have a file like this: A1-1915435_1_MIMAT0000069_MirBase_miR-16_microRNA_1_22-A1-1915435_1_ENSG00000206737_ENST00000384010_U1_snRNA_73_115 ...
3
votes
3answers
170 views

Extract lines between and including 2 patterns

I have a file with data structured as follows 1, p1, p2, p3, p4, p5, p6, p7 2, p9, p10,, 2, q1, q2, q3, q4, q5, q6, q7 2, q9, q10,, 2, r1, r2, r3, r4, r5, r6, r7 2, r9, r10,, 1, s1, s2, s3, s4, s5, ...
1
vote
1answer
58 views

Analysing nginx error log

I am looking for a way to the following. Here is one line from nginx's error log. 2014/12/29 21:17:57 [error] 30078#0: *311826 openat() ...
1
vote
2answers
101 views

Remove a line if regex exists only in 1st column of each line in a csv file

I have the following comma-delimited data (spaces used for visual clarity): 1 2014 23 2352 19.24 7.18 1 2014 23 2353 19.78 4.65 24 2014 23 2354 20.26 25.34 1 2014 23 2355 26.65 42.76 1 2014 23 ...
4
votes
5answers
248 views

how to split file and lines

I have a text file like : 1_01 { ; quite good spirals reset=2000 type=mandel passes=1 corners=-0.6014129278/-0.5990935452/0.427747516/0.429487053 params=0/0 float=y ...
3
votes
2answers
141 views

Comparing two files in unix and awk

I have to compare two files, file1 and file2. Each file has 56 columns separated by |. First column is the employee number in the file, I will check whether same employee number is present in the ...
3
votes
4answers
168 views

How to swap entire html blocks in multiple files with sed

I have tons of html pages with a simlar content as follows: <div class="wrapper"> <div class="aaa"> ...lot of html1 here like div's/spans etc.. </div> <!-- end aaa --> ...
2
votes
7answers
1k views

Print first and last line of all files in folder

I have a bunch of log files that get overwritten (file.log.1, file.log.2 etc). When I copy them from the device making them onto my local machine I lose the original time stamps. So I'd like to put ...
2
votes
1answer
153 views

How to count the number of occurrences of a number larger than x, from every column?

I'm trying to figure out how to count the number of occurrences of values that are greater than some number (i.e. 0.1) in each column and then print that value out for each column. Here would be some ...
5
votes
1answer
181 views

How to calculate percent of every column?

I'm trying to transform some data to a percent of the total of each column, very similar to this thread except that I need to do this for every column: Calculate and divide by total with AWK Data ...
5
votes
4answers
177 views

Increment numeric pattern in file

I have a File that contains only the following line: BOF (REF-REQ0001 SW_requirement) EOF I need to generate a second file that would contain the following: BOF (REF-REQ0001 SW_requirement) ...
3
votes
4answers
155 views

tab separation of the file

I have a file with seq likes this ATCGTTTCTCAGCCTTTTGGCAAGACCAAGTGTAGTATCTGTTCTTATCAGATATTGGTATCTGCAGGCACTCCTGGAACCAAT ...
-2
votes
4answers
384 views

How to ignore the lines starts with # using grep / awk

cat /etc/oratab #test1:/opt/oracle/app/oracle/product/11.2.0.4:N +ASM2:/grid/oracle/app/oracle/product/11.2.0.4:N # line added by Agent test2:/opt/oracle/app/oracle/product/11.2.0.4:N ...
3
votes
3answers
242 views

awk manipulation of file

I have a big file in this format, whose first few lines I am showing you including the header. gene c1 c2 c3 c4 c5 c6 c7 c8 G1*1 0 0 0 0 0 0 0 0 G2*2 0 0 0 0 1 ...
1
vote
1answer
96 views

awk array using number as value did not work

I have a reference file: Refrence File Dpse\GA30012 FBgn0000447 chr2 26607738 26607962 -1 Dpse\GA19764 FBgn0085819 chrX 28571020 28571736 -1 Dpse\ttk FBgn0000100 chr2 ...
3
votes
3answers
757 views

Calculate and divide by total with AWK

Given the following data file... foo 10 bar 20 oof 50 rab 20 ... how would I print column two as a percent of the total of column two? In other words, I want... foo 10 10% ...
-1
votes
2answers
164 views

understanding sed, awk and curl interaction extracting ssh URL information from github page

I needed to pull every public repo in a user's GitHub account. To do this I got the API output of https://api.github.com/users/SOMEUSER/repos This is what I came up with. To extract the ssh_url line ...
0
votes
2answers
46 views

Regex gone Wrong

I am using a Bash script to use awk to look for this: (0010,0080) The awk will generally look for any numeric value seperated by a comma in between parenthesis. I have done everything I can think of ...
-2
votes
1answer
71 views

Making use of awk on strings [closed]

I'm interested in listing in the terminal the lines from a csv file where the first character is 'A' . Is there a prompt way to do that?
2
votes
3answers
340 views

Calculate and round numbers up in bash

I'm struggling with a slight variation on: How to calculate by taking first field and How to round floating point numbers in shell? I have a file that looks like (space between columns): 1533 ...
0
votes
3answers
284 views

kill several processes using awk tools

I want to stop several processes using awk.My command is like this: sudo ps -ef|grep wget |grep -v grep|awk '{print $2;sudo kill -STOP $2}' I want to stop all processes which is running wget. $2 of ...
1
vote
1answer
143 views

Counting the instances of specific words in a text, using awk [duplicate]

If I have a file words_of_interest.txt with one word per line, is there a way to use awk (or some other *nix tools) to obtain the number of times each of these words occurs in another text file ...
2
votes
1answer
42 views

comparing 2 lists with additional data to be printed

I have a file lets say file1.txt ENSG00000223116 AL157931.1 ENSG00000233440 HMGA1P6 ENSG00000207157 RNY3P4 ENSG00000229483 LINC00362 ENSG00000252952 RNU6-58P I have another file say ...
1
vote
2answers
45 views

Parse command output [duplicate]

I want to only get the value associated to "cached" column of the free -m command. What is the most efficient way to achieve it? Something with sed or awk?
0
votes
2answers
83 views

Using variable with awk -v in a shell script

I am modifying/re-writing some of the scripts written by former employees of my company and in one of the shell scripts I found the following line. b=`benchmark=30;grep "Waiting for timer" wk.txt | ...
4
votes
3answers
712 views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
4
votes
1answer
74 views

Insert missing data within blocks

I`m a biologist and I am trying to solve this problem, please guide me on where to start. I do not know which forum to post this in, please let me know if this place is not appropriate. I have blocks ...
-1
votes
2answers
39 views

Extract 2 fields from 2 different lines

I have the below lines in my file # Employee Id : ABC123, # Employee Name: Vivek # Employee Type: Temp Join Date: 31-jan-2014 Buz Code: 2098 # Employee Comments: # ...
1
vote
7answers
146 views

repeat a line, splitting one field

I have a tab delimited file in which in every line I have this: K00001;K00004;K00008 0 0 34 0 0 0 0 0 0 0 0 0 0 0 0 0 36 0 0 52 0 0 0 6 0 I would ...
1
vote
1answer
38 views

how to move a header to the last column, using awk or sed

how to move a header to the last column, using awk or sed input file look like this: Line 1.000N x y z 23.88 44.66 56.6 23.81 41.66 53.6 Line 81.000N x y z 13.88 34.66 56.6 13.81 ...
1
vote
2answers
167 views

how to compare values in two columns in two different files, echoing full lines where absolute value of difference is < a small maximum value?

I want to do a line-by-line comparison of values in, say, column 6 of two files, and then echo the lines and their full contents, piping them into a combined file, say the line from one file and then ...
2
votes
1answer
86 views

Specify lines in awk?

Im going to use this awk script to delete duplicate lines from my text file: awk '!x[$0]++' However, I want to specify which lines that they are to be run on. There could be 10000 duplicates, but ...
0
votes
2answers
123 views

delete multiple users

I am the root user and I am setting up a menu for another user to use. This other user will only get this menu. There are two options that are interlinked: the first option is to search users. The ...
4
votes
2answers
262 views

grep for repeated values from pattern list

I have file A.txt which is a lists of unique ids (column 1) and its corresponding values(column 2): A.txt ABC1D_T1_B1 123.4 ABC2D_T1_B2 146.7 ABC3D_T1_B1 567.8 ABC4D_T0_B2 96.8 ABC1D_T0_B1 ...
2
votes
2answers
41 views

Create small record files from a big group of record file

I have a file like this : Mainfile: Record|1111|ABC text in between for record 1 text in between for record 1 Record|2222|XYZ text in between for record 2 Record|3333|XYZ ...
2
votes
7answers
1k views

How to print lines with blank 5th field in CSV

I have to check if the fifth field is empty in a CSV file. This is my file: 1,abc,543,87,DATA,fsg; 1,abc,543,87,,fsg; 1,abc,543,87,DATA,fsg; 1,abc,543,88,,fsg; 1,abc,543,,DATA,fsg; As you can ...
1
vote
1answer
41 views

awk how to set parameter $2 become $b?

In a Bash script: #!/usr/bin/bash #grep() #{ #grep -q #} if [[ -z $1 ]]; then echo "usage: pkill -signal process name " fi if [[ -z $2 ]]; then echo "error: not enough parameters " exit fi ...
2
votes
2answers
57 views

Combine output from df -h /dev/xvda1 into a single line

I want to convert the 2 line output from df -h into a single, meaningful line i.e. Take [root@ip-10-42-1-246 tmp]# df -h /dev/xvda1 Filesystem Size Used Avail Use% Mounted on /dev/xvda1 ...
2
votes
2answers
1k views

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
0
votes
4answers
77 views

Print last character in the given path

I am trying to concat "comnp.dat_CY*" files into one file. File name, source path and destination path are the parameters. I am trying to find the last character in the path to check if it is "/" or ...
-5
votes
3answers
409 views

How do you print the current month, day and year using awk and date?

I can't seem to grab the month, day and year using awk '{print $2, $3, $6}' date Does anyone know how to?
3
votes
1answer
45 views

create a realtime machine by assigning CPU cores to critical processes

I'm using an Udoo board for running two critical processes. It can run Ubuntu ARM edition and it has a Quadcore processor. I need to set the affinity of all the processes afinity running and the ones ...
2
votes
2answers
168 views

How to base awk selection on contents of 1 field?

How can I select all the records that have '2' as the second field? My data is: $ cat numbers.txt 1 2 3 4 5 6 7 8 2 4 6 8 10 12 14 16 3 6 9 12 15 18 21 24 My awk is: awk '$2 - /^2$/ {print}' ...