awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

2
votes
1answer
41 views

How to limit printed output based on number of occurences (AWK)

So I am trying to track down individuals on our network surfing porn via https. The pattern is pretty easy to see, 15+ hits in a row for images without a referrer in the HTTP header on our Bro logs. ...
3
votes
2answers
78 views

Using find -exec how to print name of file in front of each line?

Similar to, but beyond: find . -iname "somepatter*txt" -print -exec cat {} \; How might the file name from each file be printed in front of each line with the results from find?
3
votes
3answers
85 views

How do I create uniform columns from text containing both tabs and spaces?

In the simplest form, let's say I have an auto-generated file called file.txt. The contents of file.txt appears as follows: Source Destination Maximum To Maximum ...
1
vote
2answers
48 views

How do I stream edit progress-like output?

I'm having trouble editing some progress output. In this case it's Postgres' pg_basebackup, but it's similar to other progress output like curl and wget. Take for example the output of this: ...
0
votes
4answers
238 views

How to sum up values of each two rows across their line in linux

I have a data like this: input.txt 1 0000100101000000 1 0000010100000000 2 1110000001000000 2 1111000000001000 3 0000000111111111 3 1111111100000000 4 8888345500000000 4 0000000000000000 and I ...
-1
votes
0answers
15 views

How to accumulate values of each two rows across a line in linux? [duplicate]

I have a data like this: 1 0000100101000000 1 0000010100000000 2 1110000001000000 2 1111000000001000 3 0000000111111111 3 1111111100000000 and I want to sum up the values within eachtwo rows with ...
4
votes
5answers
402 views

Sum of filesize of a list of files

I have a text-file that contains a list of filenames (one filename per line). Now I would like to calculate the size of all these files. I think I will have to do a ls -la on every line of the file ...
1
vote
3answers
184 views

Help with awk / sed shell script

I have to make a script using the info in the following table (fake info) AnimalNumber,DOB,Gender,Breed,Date-moved-in IE161289240602,04/02/2010,M,AAX,20/07/2011, ...
6
votes
5answers
294 views

Find 2nd Occurrence of string from the end of file

I have a log file that looks like this, Another thousand lines above this I 10/03/15 12:04AM 42 [Important] 4th to last I 10/03/15 04:31AM 42 (534642712) [1974,2106,258605080,0,0,32817,30711] I ...
2
votes
1answer
109 views

How can I append an incremental count to every line of a text file?

I have a large document with over 500000 lines like this car milk dog And I want to count every line and add the number in the end of line and the final document to look like this car1 milk1 dog1 ...
1
vote
3answers
47 views

awk script to delete blocks in json

I have a newline delimited JSON file with entries like this: {"id":"eprints.ulster.ac.uk/view/year/2015.html","title":"Items where Year is 2015 - Ulster Institutional ...
1
vote
4answers
88 views

Find the smallest numbers in the second column corresponding to index values in first column

I have some very large tables and I need to extract specific rows. I am illustrating the task using a simple example. Say, I have weighed a number of apples, bananas and oranges. I need to extract the ...
0
votes
2answers
71 views

print field that contain a certain value

is it possible to print the field that contain a certain value? Consider below file: foo,boo,123,1234312,ABC foo,boo,ABC,bla,bla There is a field that contain ABC but it is not a fixed with all ...
4
votes
1answer
71 views

Printing more than one field

Is there a way to print multiple fields with a single line command? I want to print $3 to $NF, I can do it like below: awk -F[:] '{print $3,$4,$5 .... $NF}' Yet it is not doable with large ...
1
vote
2answers
75 views

Compare two dates with different format [closed]

I have two files, each file contain a different date's format as below: File1: 2014-NOV-0620:59:59 File2: 2014-12-31 How can I compare those two dates in awk?
1
vote
1answer
140 views

awk - how to ignore wildcard matches?

I want to process a csv file based on 3rd column value and save the matching lines to a separate file. #!/bin/bash awk -F, '$3 ~ /0/ {print}' ./inputfile.csv > ./out/output.csv What I want is ...
0
votes
3answers
52 views

Reformatting a table with awk

I have to reformat a number of very long tables as follows Original format: John Smith,Jones,Taylor Janet Williams,Brown,Wilson Desired format: John Smith John Jones John Taylor Janet Williams ...
1
vote
2answers
108 views

Awk condition along with header

I have a dummy employee.txt file as follows to play around with awk: 100 Thomas Manager Sales $5,000 200 Jason Developer Technology $5,500 300 Sanjay Sysadmin Technology $7,000 ...
0
votes
0answers
20 views

awk - how to reorder the contents of a file based on another pattern input file

I have a huge file with a list of names/characters in each line. The good thing is each line is unique, not sorted, but the order makes some sense and same needs to be retained. This is the file to be ...
1
vote
1answer
28 views

Read out File during awk

My script continuously reads a file with awk and evaluates input data. Now I need to read and write the first line of the output file. There is the information, what in which row is (°C of first ...
0
votes
1answer
38 views

how to extract some rows in a data file by comparing with another data file? (merging) [duplicate]

I have 2 different file : first file: id1 19501112 id2 19831207 1d4 20001212 id6 20000101 id10 20061213 id11 20081212 id12 20100109 id13 20101218 id14 20101212 id15 20111111 id16 20150919 id20 ...
4
votes
5answers
428 views

How to split a file based on information within a column in linux

I would like to separate those rows where the value in second column is smaller than a specific value. example input: id1 19501112 id2 19831207 id3 20001212 id6 20000101 id10 20061213 id11 20081212 ...
5
votes
4answers
238 views

how to split a file into separate files based on the column headers in original file?

I would like to split a file to different files based on the information in the first line. For example, I have: Input: 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 4 30 30 30 30 0 2 2 0 2 ...
4
votes
3answers
265 views

How to convert a single column file into a single row?

I would like to convert my one colum of many rows into one line: example : 1 1 1 1 1 2 2 2 3 3 4 and I want : 11111222334 any suggestion please?
1
vote
2answers
30 views

extract part of the text from all matching lines

I am running a for loop against set of servers to find the cert expiry. Output of the cert command will be below for each server: line 1 line 2 line 3 Valid From: Tuesday, 29 September 2015 10:40:13 ...
1
vote
2answers
22 views

Duplication of each line while changing some value within each copy of a line differently in linux

I Would like to have each line in a file repeated a two number of times while in in first copy of line any value of 3 be changed into 2 and any value of 4 be changed into 1, and in second copy of each ...
0
votes
2answers
55 views

select and move the contents to another file

I have a file with the below contents. All I need to do is filter the ALTER script that has UNIQUE keyword in it and move it to a different file. I tried using sed -n '/ALTER TABLE/,/UNIQUE("ID");/p' ...
0
votes
0answers
24 views

How to restore sbatch array job output file names to their original names

In order to submit array jobs using Slurm, I've renamed hundreds of files with incremental numbers using the following command: COUNT=1 for file in ./*; do mv "$file" "${COUNT}" let COUNT++ ...
0
votes
0answers
33 views

Issue with netstat script

I am trying to run the following script to capture the netstat data and put it in to log file. But this is giving error while running the netstat command, however if i directly run the netstat ...
1
vote
2answers
103 views

Find and replace a string if particular pattern is found in a line

All I need to do is, search for CCSID in this file, wherever it finds CCSID, the CHAR in that line should be replaced with NCHAR and VARCHAR in that line should be replaced with NVARCHAR2. I tried ...
2
votes
2answers
58 views

Exchanging One Column From File1 with files from File2

I have two files. The first file has 11 columns, for example: 1 2 3 4 5 6 7 8 9 10 11 The second has 10 columns and might look like this: 11 22 33 44 55 66 77 88 99 100 What I want to do is look ...
0
votes
1answer
106 views

How to delete the lines that starts with a specific word and end with a specific word

Eg: CREATE TABLE COMM.ACL_ENTRY ( ACL_OID CHAR(18) FOR BIT DATA NOT NULL , USER_DESC_TYPE FOR COLUMN USER_00001 BIGINT NOT NULL , USER_DESC_OID FOR COLUMN USER_00002 ...
1
vote
1answer
43 views

Create new text file with sum of third column of several others

I have several different text files of the format: x y z x y z x y z I want to create a new file which sums the z values in each, and then has the summed value in place of z i.e. x y ...
0
votes
1answer
53 views

Need help in writing awk to extract cpu utilization from sar command

I am trying to extract the values of the %user,%nice etc from the output of the sar command. Command : sar -P ALL 1 1 Output of this : Linux 2.6.32-358.el6.x86_64 (ftizsldapp009.ftiz.cummins.com) ...
0
votes
1answer
69 views

sed html parsing

I have to parse HTML to change text between two brackets that are not HTML code back to < or >. Here an example of my HTML code that I have to replace: ...
0
votes
1answer
55 views

Check if two lines start with the same character, if so the output average, if not, print actual value [closed]

I'd like to check if two rows start with the same number in the 1st column, if that happens, then the average of the 2nd column should be displayed. File example: 01 21 6 10% 93.3333% 01 ...
0
votes
1answer
59 views

Compare first column of 2 files based on the second column using the delimiter ;

I need to compare below 2 files. The head_file: 2345;int builder( 1000;char type::edit::display <test>( 250;void unamed_string_tree* The ref_file: 2000;int builder( ...
3
votes
4answers
342 views

How do I remove first character in specific column using sed

I have file with the next structure: 999963803|081439403|FFF|33 999964211|081823904|FFF|33 999970948|081765738|FFF|33 999971835|081736346|FFF|33 999972780|082127527|FFF|33 I need remove the first ...
0
votes
1answer
108 views

Compare columns between different files

I have multiple files ( about 20 files with 30000 lines and 32 columns) and I need to keep only the lines that start with the same string. I found these cases that are quite similar to what I need but ...
0
votes
1answer
18 views

Is there a way to get all the NEWS.gz which is/was this year in /usr/share/doc/ of $PACKAGENAME.

If people go into /usr/share/doc/$PACKAGENAME (you can use any valid package name which you have installed on your system) you will see that the directory has various files in it, at the very least ...
1
vote
3answers
92 views

bash: grepping contents of column which is dynamic

I have data as following : What i need to get is the OS_NATIVE_NAME of a particular DEVICE. I know one way is to print its column value using: awk '{print $*col_num*}' But the column number ...
-1
votes
4answers
71 views

How to break a string into smaller substrings with a condition on their lengths?

I have a file that contains some genetic sequences like this one: ...
3
votes
3answers
77 views

How to remove duplicate lines that begin with a pattern and the next line after that?

I want to remove duplicate lines that begin with > and the next line after that. For example: >1 ACCGGTTTCCTTGAAATT >2 AACCTTCCGGTTAATT >3 AACCTTCCGGTTAATT >1 ACCGGTTTCCTTGAAATT ...
1
vote
1answer
65 views

Changing the sign (+ or -) of a number based on non-matching columns

I'm having a bit of an issue with an awk script which should probably be pretty simple. I have files which looks like this, which have thousands of rows, with columns 2 and 3 consisting of G,C,T and ...
3
votes
4answers
376 views

how to change one column and add other columns in a row by awk

I have a file MDD.TXT with following records. file : MDD.TXT 0|ABC|055309|20120929053309|11965150 0|ABC|061434|20120929063434|11155537 0|ABC|070228|20120929073228|11965150 ...
0
votes
1answer
38 views

Please explain this awk statement [duplicate]

Yesterday I was googling how to merge two files and came across an awk snippet. I need a simple merge, so sort -u is not the way to go, but the code below works. Could some one please explain what ...
0
votes
3answers
44 views

How to get filename in awk

I have a zip file with below name and record: Filename : OMS_Profile_20150922.list.Z Sample Record: 1234568194|301776931|301344831|301745924|4000247920|5|1|14| ...
6
votes
4answers
297 views

Count number of a substring repetition in a string

I have a file which contains a gene sequence such as: ...
3
votes
2answers
232 views

split a large file to new file with unique file names

I appreciate your help if some one could help me with a script that could split a file into unique file names. I can do it with sed command eg, sed -n '/scaffold135_/w 135-scaf.txt' input file.txt ...
3
votes
7answers
110 views

How to replace line with file contents and around the content with some text?

I would like to combine VPN certificates into a single file. This is a template file: # some unrelated stuff ca ca.crt # other unrelated stuff And I want to replace line ca ca.crt with the ...