awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
2answers
45 views

splitting a file by column number

I have a file with 10,671 columns. I am trying to split it into 10 smaller files, each with 1000 columns until all columns have been printed, named file_transposed01, file_transposed02, ... ...
3
votes
4answers
511 views

Aggregate and group text file in perl or bash

I have a big text file(with 5m lines) in this format(4 columns, separated by ;): string1; string2; string3; userId The first 3 strings (SHA1s) form a single ID, called appId (so it can be ...
0
votes
2answers
41 views

Rearranging data in file with awk

I have a table ID Comment Statement 1 hi hello 2 hi hello 1 cya bye 1 k ok 2 kk ok How would one go about producing a result ...
2
votes
3answers
63 views

Rename mp3 file names to a pattern

I have a list of like names likes so: /A/B/C/D/E/F/filename_album1-003.mp3 its more like album\d+ I would like to change it to something like /A/B/C/D/E/F/filename_song-3.mp3 song is just a ...
2
votes
2answers
89 views

Filtering a file on a string

I have a file which looks like this a 0 gene1 56 0 6S32M12S * b 256 gene2 56 0 6S32M12S * c 256 gene3 55 0 6S27M17S * d 16 gene4 110 9 19S25M6S * e 272 ...
0
votes
1answer
29 views

putting headers back after awk command

I have file like this. Its called a sam file @HD VN:1.0 SO:unsorted @SQ SN:RNU2-1 LN:388 @SQ SN:RNU6-1100P LN:304 SRR959756.23 0 RNU2-1 56 0 6S32M12S * 0 0 ...
1
vote
2answers
66 views

How to make a filter more efficient

I've created a script to compare a list of users from a MSSQL database against a list of disabled AD users. The tool we use for AD authentication returns data in the following format (with plenty ...
1
vote
1answer
91 views

awk in ssh in su in a command substitution

I am creating a script that will ssh to a host and print all the user accounts and when they will expire. On a host I can run awk -F':' '{ print $1}' /etc/passwd and it will give me a list of all ...
1
vote
1answer
59 views

Get file names in awk result

I have many files in a location where I need to list file names in which 32th column has value more than 3000. This is the command that gives me those records: awk -F, '$32 > 3000' * Now can ...
3
votes
1answer
105 views

awk causing memory leak?

I am using awk to process a huge data file (about 100 million lines). This is my expression- tac <myfile> | awk -F',' 'OFS=\",\" ...
2
votes
1answer
142 views

Merge some tab-delimited files

I have 100 files with 57,816 rows each.I want to do an outer join of these files by merging on basis of the common column. I am using R programming for this but this is very slow. fileList <- ...
4
votes
5answers
10k views

removing the first and the last character of every line from command line

I am trying to remove the first and the last characters of everyline in a text file and save the resulting truncated version in a new file. Does anyone have an idea about how to do that efficiently ...
3
votes
4answers
161 views

pattern match and capturing in unix

I have a file like this: A1-1915435_1_MIMAT0000069_MirBase_miR-16_microRNA_1_22-A1-1915435_1_ENSG00000206737_ENST00000384010_U1_snRNA_73_115 ...
3
votes
3answers
122 views

Extract lines between and including 2 patterns

I have a file with data structured as follows 1, p1, p2, p3, p4, p5, p6, p7 2, p9, p10,, 2, q1, q2, q3, q4, q5, q6, q7 2, q9, q10,, 2, r1, r2, r3, r4, r5, r6, r7 2, r9, r10,, 1, s1, s2, s3, s4, s5, ...
1
vote
1answer
35 views

Analysing nginx error log

I am looking for a way to the following. Here is one line from nginx's error log. 2014/12/29 21:17:57 [error] 30078#0: *311826 openat() ...
4
votes
5answers
175 views

how to split file and lines

I have a text file like : 1_01 { ; quite good spirals reset=2000 type=mandel passes=1 corners=-0.6014129278/-0.5990935452/0.427747516/0.429487053 params=0/0 float=y ...
5
votes
1answer
132 views

How to calculate percent of every column?

I'm trying to transform some data to a percent of the total of each column, very similar to this thread except that I need to do this for every column: Calculate and divide by total with AWK Data ...
3
votes
2answers
114 views

Comparing two files in unix and awk

I have to compare two files, file1 and file2. Each file has 56 columns separated by |. First column is the employee number in the file, I will check whether same employee number is present in the ...
16
votes
5answers
25k views

How to manipulate a CSV file with sed or awk?

How can I do the following to a CSV file using sed or awk? Delete a column Duplicate a column Move a column I have a big table with over 200 rows, and I'm not that familiar with sed.
2
votes
7answers
586 views

Print first and last line of all files in folder

I have a bunch of log files that get overwritten (file.log.1, file.log.2 etc). When I copy them from the device making them onto my local machine I lose the original time stamps. So I'd like to put ...
2
votes
1answer
70 views

How to count the number of occurrences of a number larger than x, from every column?

I'm trying to figure out how to count the number of occurrences of values that are greater than some number (i.e. 0.1) in each column and then print that value out for each column. Here would be some ...
4
votes
2answers
556 views

Use sed or awk to join lines in a certain pattern?

Not sure on how to turn this input: A B C apartner bpartner cpartner into this output: A apartner B bpartner c cpartner I have this pattern in a 80 line file that needs to be 40 line. Also, how ...
1
vote
1answer
93 views

awk array using number as value did not work

I have a reference file: Refrence File Dpse\GA30012 FBgn0000447 chr2 26607738 26607962 -1 Dpse\GA19764 FBgn0085819 chrX 28571020 28571736 -1 Dpse\ttk FBgn0000100 chr2 ...
5
votes
4answers
139 views

Increment numeric pattern in file

I have a File that contains only the following line: BOF (REF-REQ0001 SW_requirement) EOF I need to generate a second file that would contain the following: BOF (REF-REQ0001 SW_requirement) ...
6
votes
3answers
10k views

Adding a Column of values in a tab delimited file

How can I add a Column of values in a file which has a certain number of rows. I have a input file like this: Input file: SPATA17 1 217947738 LYPLAL1 1 219383905 FAM47E 4 77192838 SHROOM3 4 ...
3
votes
4answers
146 views

tab separation of the file

I have a file with seq likes this ATCGTTTCTCAGCCTTTTGGCAAGACCAAGTGTAGTATCTGTTCTTATCAGATATTGGTATCTGCAGGCACTCCTGGAACCAAT ...
3
votes
3answers
228 views

awk manipulation of file

I have a big file in this format, whose first few lines I am showing you including the header. gene c1 c2 c3 c4 c5 c6 c7 c8 G1*1 0 0 0 0 0 0 0 0 G2*2 0 0 0 0 1 ...
-2
votes
4answers
99 views

How to ignore the lines starts with # using grep / awk

cat /etc/oratab #test1:/opt/oracle/app/oracle/product/11.2.0.4:N +ASM2:/grid/oracle/app/oracle/product/11.2.0.4:N # line added by Agent test2:/opt/oracle/app/oracle/product/11.2.0.4:N ...
3
votes
3answers
347 views

Calculate and divide by total with AWK

Given the following data file... foo 10 bar 20 oof 50 rab 20 ... how would I print column two as a percent of the total of column two? In other words, I want... foo 10 10% ...
-1
votes
2answers
82 views

understanding sed, awk and curl interaction extracting ssh URL information from github page

I needed to pull every public repo in a user's GitHub account. To do this I got the API output of https://api.github.com/users/SOMEUSER/repos This is what I came up with. To extract the ssh_url line ...
2
votes
3answers
157 views

Calculate and round numbers up in bash

I'm struggling with a slight variation on: How to calculate by taking first field and How to round floating point numbers in shell? I have a file that looks like (space between columns): 1533 ...
0
votes
2answers
41 views

Regex gone Wrong

I am using a Bash script to use awk to look for this: (0010,0080) The awk will generally look for any numeric value seperated by a comma in between parenthesis. I have done everything I can think of ...
-2
votes
1answer
69 views

Making use of awk on strings [closed]

I'm interested in listing in the terminal the lines from a csv file where the first character is 'A' . Is there a prompt way to do that?
0
votes
3answers
161 views

kill several processes using awk tools

I want to stop several processes using awk.My command is like this: sudo ps -ef|grep wget |grep -v grep|awk '{print $2;sudo kill -STOP $2}' I want to stop all processes which is running wget. $2 of ...
1
vote
1answer
109 views

Counting the instances of specific words in a text, using awk [duplicate]

If I have a file words_of_interest.txt with one word per line, is there a way to use awk (or some other *nix tools) to obtain the number of times each of these words occurs in another text file ...
6
votes
4answers
19k views

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": ...
4
votes
3answers
217 views

Insert text from file inline after matching pattern in another file

I'm trying to take the contents of a file and insert it after a matching pattern in another file using sed. My question is very similar to this question, but I wish to insert the contents of a file ...
2
votes
1answer
40 views

comparing 2 lists with additional data to be printed

I have a file lets say file1.txt ENSG00000223116 AL157931.1 ENSG00000233440 HMGA1P6 ENSG00000207157 RNY3P4 ENSG00000229483 LINC00362 ENSG00000252952 RNU6-58P I have another file say ...
1
vote
2answers
37 views

Parse command output [duplicate]

I want to only get the value associated to "cached" column of the free -m command. What is the most efficient way to achieve it? Something with sed or awk?
0
votes
2answers
61 views

Using variable with awk -v in a shell script

I am modifying/re-writing some of the scripts written by former employees of my company and in one of the shell scripts I found the following line. b=`benchmark=30;grep "Waiting for timer" wk.txt | ...
4
votes
1answer
73 views

Insert missing data within blocks

I`m a biologist and I am trying to solve this problem, please guide me on where to start. I do not know which forum to post this in, please let me know if this place is not appropriate. I have blocks ...
1
vote
7answers
122 views

repeat a line, splitting one field

I have a tab delimited file in which in every line I have this: K00001;K00004;K00008 0 0 34 0 0 0 0 0 0 0 0 0 0 0 0 0 36 0 0 52 0 0 0 6 0 I would ...
2
votes
5answers
11k views

awk-Printing column value without new line and adding comma

input.txt EN1 EN2 EN3 EN4 EN5 output EN1,EN2,EN3,EN4,EN5 I have tried awk.But it is not printing with comma awk 'BEGIN { OFS = ","} { printf $1}' input.txt I have GNU Awk 4.0.0 version
-1
votes
2answers
36 views

Extract 2 fields from 2 different lines

I have the below lines in my file # Employee Id : ABC123, # Employee Name: Vivek # Employee Type: Temp Join Date: 31-jan-2014 Buz Code: 2098 # Employee Comments: # ...
1
vote
1answer
31 views

how to move a header to the last column, using awk or sed

how to move a header to the last column, using awk or sed input file look like this: Line 1.000N x y z 23.88 44.66 56.6 23.81 41.66 53.6 Line 81.000N x y z 13.88 34.66 56.6 13.81 ...
8
votes
6answers
462 views

How to append Line to previous Line?

I have a Log file which need to be parsed and analysed. File contains something similar like below: File: 20141101 server contain dump 20141101 server contain nothing {uekdmsam ikdas jwdjamc ...
1
vote
2answers
88 views

how to compare values in two columns in two different files, echoing full lines where absolute value of difference is < a small maximum value?

I want to do a line-by-line comparison of values in, say, column 6 of two files, and then echo the lines and their full contents, piping them into a combined file, say the line from one file and then ...
2
votes
1answer
84 views

Specify lines in awk?

Im going to use this awk script to delete duplicate lines from my text file: awk '!x[$0]++' However, I want to specify which lines that they are to be run on. There could be 10000 duplicates, but ...
0
votes
2answers
64 views

delete multiple users

I am the root user and I am setting up a menu for another user to use. This other user will only get this menu. There are two options that are interlinked: the first option is to search users. The ...
2
votes
2answers
39 views

Create small record files from a big group of record file

I have a file like this : Mainfile: Record|1111|ABC text in between for record 1 text in between for record 1 Record|2222|XYZ text in between for record 2 Record|3333|XYZ ...