awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

11
votes
4answers
24k views

How to extract logs between two time stamps

I want to extract all logs between two timestamps. Some lines may not have the timestamp, but I want those lines also. In short, I want every line that falls under two time stamps. My log structure ...
3
votes
1answer
266 views

How to print incremental count of occurrences of unique values in column 1

I'm trying to come up with an solution to this problem, I need to incrementally count and then print the counts of the unique values in column 1 of a tab delimited text file. Here is an example: ...
1
vote
3answers
50 views

Iterate through subdirectories and execute an awk script against certain files

I have a directory named ../words that contains a number of subdirectories(words/blabla,words/blabla), in turn each subdirectory contains two types of files (.*1.txt) and (.*2.txt) . What I need, it ...
1
vote
1answer
61 views

Script to Create multipath.conf Alias

I found a script to create alias for multipath for Nimble volumes and I modified it to work in freenas the content of it is #/bin/bash # This script will scan the system for attached Nimble Volumes ...
0
votes
3answers
56 views

Print lines after pattern in several files

I have a file with the same pattern several times. Something like: time 12:00 12:32 23:22 time 10:32 1:32 15:45 I want to print the lines after the pattern, in ...
20
votes
9answers
24k views

How to print certain columns by name?

I have the following file: id name age 1 ed 50 2 joe 70 I want to print just the id and age columns. Right now I just use awk: cat file.tsv | awk '{ print $1, $3 }' However, this ...
6
votes
3answers
229 views

How can I display 3 lines of text at a time onto 1 line?

I have a log file that displays data 3 lines at a time. Like this: 1 data 2 data 3 data 1 data 2 data 3 data 1 data 2 data 3 data I'd like to take each 3 lines and ...
3
votes
2answers
119 views

How to extract several data from a file and store it in a csv file?

I have a file in the following format : 19-08-02 Name appel ok hope local merge (mk) juin nov sept ...
1
vote
2answers
33 views

Effective way of changing the format of numeric file data

So I have a big file containing the values of a matrix with the folowing format: (0,0) 0.6666 (0,1) 0.0000 (0,2) 1.3333 ... And I want to convert it to: 0 0 0.6666e+00 0 1 0.0000e+00 0 2 ...
0
votes
2answers
110 views

Convert unix timestamp to hh:mm:ss:SSS (where SSS is milliseconds) in AWK

How can I convert unix timestamp to hh:mm:ss:SSS (where SSS is milliseconds) in AWK. For example: echo 1456478048306 > time cat time | awk ....... > readable_time_format
0
votes
1answer
31 views

pattern regex for apache rewrite

I have a bunch of URLs counting to thousands i need to redirect and there are thre main patterns in URL first pattern There is a dash - starting the title url and want to remove that dash e..g ...
4
votes
3answers
116 views

Delete consecutive lines in CSV with duplicate values in one field, but keep the last line

I have a long CSV file with two columns, which includes runs of consecutive duplicates like this: ... 1500,1533 1554,1678 1554,1703 1554,1728 1593,1766 ... I need to delete all of these duplicates ...
0
votes
1answer
35 views

How to sum numbers that have matching strings

I currently have a script that results in this file : Score1 5 Score1 9 Score1 4 Score2 6 Score2 11 Score3 10 Score3 1 Score3 2 ... And then a number of other scores, in the hundreds. I want to add ...
2
votes
2answers
82 views

List PDF files with missing metadata

I have had a few tasks where I needed to make reports on file changes on directories from different servers and have managed to make the reports using simple Unix commands. I used: ls -R -lT ...
0
votes
4answers
51 views

Extract ranges of lines from a file, saving each range to a separate file

I have a file that looks like this 740*02/01/2016*00:00* EJ LOG COPIED OK AUTO INIT COPY DRIVE NOT CONFIGURED E1EF3901 [020t*741*02/01/2016*05:45* *TRANSACTION STARTED* [020t CARD INSERTED ...
0
votes
0answers
59 views

Need some help on awk/grep command on shell script

This is in reference to Need some help on grep command on shell script I have a log file and I am trying to write a shell script to find pause/play happening or not (Success /Failure). The script ...
1
vote
2answers
63 views

Adding a number as a suffix to multiple columns

I have data that looks like: chr1 enhancer_ 875557 876449 . enhancer_ chr1 enhancer_ 876970 877622 . enhancer_ chr1 enhancer_ 995719 996320 . enhancer_ chr1 enhancer_ ...
0
votes
2answers
25 views

can't get an “awk for loop” to work, while using basename

I have many xyz files that I want to run through awk. I just want to create a new file with an "_s" appended, for each one. Having a little trouble getting the basename syntax correct. Here is what I ...
0
votes
3answers
69 views

Count all lines between a delimiter

I want to count the number of addresses attached to each device in the below example. Preferably using awk or sed. /dev/1 addr1 sometext sometext sometext addr2 sometext sometext sometext addr3 ...
5
votes
4answers
280 views

Join two files on two common fields

I have two files file1.txt 78Z|033333157|0000001|PERD1|2150421|D|0507020|3333333311 78Z|033333157|0000001|PERD0|2160208|A|1900460|3333333311 78Z|033333157|0000001|RSAB1|2150421|D|0507070|3333333311 ...
0
votes
1answer
36 views

How can I replace rows containing ONLY number ' 0' with blank rows using AWK?

I am new to AWK, I am looking for a command that would replace the rows containing ONLY number 0 in their columns, with 'blank' lines. e.g. I have below input: Taj_Mahal ...
0
votes
0answers
21 views

Order Columns in CSV file

I am trying to order all columns by column name in a csv file. What I have is something like this: name ,adress ,mobile-number Ane ,USA ,12121212 Joane ,England ,234234 and the output I ...
2
votes
2answers
61 views

Print lines of a file between two matching patterns [duplicate]

How can I print all the lines between two lines starting with one pattern for the first line and ending with another pattern for the last line? Update I guess it was a mistake to mention that this ...
0
votes
2answers
56 views

awk FS with back slashes

I use the command find . -maxdepth 1 -not -type d which generates output like ./filename.1.out I pipe the find command output to awk. The goal is to split on either the literal ./ or .. I have it ...
-2
votes
1answer
61 views

How to find the word before a match

I have the following log file. I need to know if there are any fault / suspect condition using a shell script. I need to find fault and check the previous word, if it is greater than 0 then there is ...
-1
votes
3answers
35 views

Transposing rows into columns in absence of few rows using AWK [duplicate]

I have a text file containing below: title1 A1 title3 A3 title4 A4 title5 A5 title1 B1 title2 B2 title5 B5 title1 C1 title2 C2 title4 C4 title5 C5 title1 D1 title2 D2 title3 D3 I would ...
0
votes
0answers
39 views

Standard deviation of number next to a pattern

I have a file with lots of tags with a number next to them, set out like this: <Score>4 ... other data <Score>2 ... other data <Score>3 and so on... I have used grep to get the ...
2
votes
3answers
47 views

Copy column with file name as column header

I wanted to organize my dataset by copying the a column from a number of text file and append the file name in the first row of the new files. Here is sample of my data file1 a a b b 1 2 ...
3
votes
2answers
77 views

Find each line matching a pattern but print only the line above it

I need to find a string and need to print the line above it. case1: The same line won't have the more than one matching pattern ie) consider a file containing $cat > para returns between the ...
1
vote
3answers
5k views

Find common elements in a given column from two files and output the column values from each file

I have two files with tab-seperated values that look like this: file1: A 1 B 3 C 1 D 4 file2: E 1 B 3 C 2 A 9 I would like to find rows between files 1 and 2 where the ...
-2
votes
1answer
18 views

Remove some elements of a string and replace with others

How can I convert this : RS.PK.03.01.4200.03.014.01 to man03140101? What needs to be done? Text before 5th "." has to be removed : RS.PK.03.01.4200. The remaining text should be converted ...
0
votes
6answers
78 views

Merge multiple lines in same file based on column 1

I'm still learning programming and I've tried many things but just cannot get the correct format. I have a tab delimited file with 17 columns and many (around 50.000) rows. The file is already sorted ...
3
votes
1answer
46 views

Replacing a & in XML correctly?

I want to replace a '&' with &amp; Except the following occurance of XML special characters. " &quot; ' &apos; < &lt; > &gt; & &amp; I tried the ...
1
vote
2answers
47 views

Split file into multiple files based on pattern [duplicate]

I have a large file that's a couple hundred lines long. This file is partitioned into many parts by a specific identifier, lets say 'ABC'. This line 'ABC' appears 6 times so I want 6 output files. I'm ...
0
votes
1answer
18 views

awk: detect content separated by arbitrary character instead of spaces? [duplicate]

Is there a way to make awk detect content separated by commas (or any other character) instead of spaces? e.g. a file (.csv) contains a list of contacts in the format ...
3
votes
1answer
88 views

Print some lines before and after pattern match excluding lines matching another patterns

I use awk on Solaris to print 5 lines before and after a pattern match. Currently, the following one-liner does the job: /usr/xpg4/bin/awk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print ...
11
votes
5answers
3k views

How do you check if a file exists within awk? [-d 'filename'] failing

I'm trying to generate a list of users who have a home directory set which does not exist. It seems I should be able to do this with awk, but something is wrong with my syntax. It keeps telling me ...
3
votes
2answers
54 views

How can I do a binary search of a text file for a specific character string?

I have a text file that contains a large number of records, each on a single line. Some of the records have special characters that have been corrupted, and I'm attempting to find those by looking for ...
1
vote
3answers
76 views

Parse file with Awk only when first row matches pattern

I need to interrogate the header of a CSV file, and if a column exists proceed with the data rows. Context is when the data contains columns depending on when and what emitted it. Hoping for a "pure" ...
0
votes
0answers
15 views

Loop through directory and sum value next to specific pattern producing an average

I want to loop through all the files in the directory. the files are set out like this: <Overall>4 other data <Overall>2 other data ...... I have the code: for file in .dat; do awk ...
2
votes
4answers
103 views

Sum value next to specific pattern

I have a file with multiple tags with a number next to it e.g. <Overall>4 other <tags> and data <Overall>2 other <tags> and data <Overall>3 How would I search through ...
2
votes
4answers
765 views

How to calculate the sum of the data that have the same ID in the first column?

I want to sum the values of the rows of my data that have the same ID in the first column. My data looks like data.txt Id a b c d e 1 1 2 3 4 5 1 2 3 4 5 ...
1
vote
2answers
217 views

How can I reformat tabular data using AWK?

I have a file contain following output. ora.abc.db 1 ONLINE ONLINE serverA Open 2 ONLINE ONLINE serverB Open ora.xyz.db 1 ...
0
votes
1answer
28 views

How to add a condition within a for loop during substitution if string is null

I am trying to add a condition in this code, that if there is null string in translation file for either string or repl[string] for example, my file input_chk.txt has following details: input_chk.txt ...
1
vote
1answer
27 views

insert string for missing value

I've run into some problems with editing a text file I've received due to some missing values in a specific column for some rows. For example, here is what the file looks like (mostly): ID10 dad ...
2
votes
2answers
43 views

Reformat date to unix time stamp in csv table

I have a .csv file containing the date and time in the format 01/20/2016 23:53:01 in the first column. My columns are separated by semicolons, i.e. 01/21/2016 03:03:01;18616;0;1 01/21/2016 ...
3
votes
4answers
69 views

display lines with not more than 4 words in the second column

I have a symbol table of the form M07UP49A0870I422.wav <s> haraa keelaa <bn> </s> M07UP49A0870I423.wav <s> <horn> jau <babble> </s> ...
1
vote
1answer
40 views

Regex matching on awk in command line

I'm trying to get the filename from unrar command verbose and deleting them all through command line. The unrar command output is similar to this: Extracting BucketSort.cpp ...
1
vote
2answers
68 views

How to replace multiple blank lines with a specific non-blank line?

The sample input is 1 2 3 4 5 6 7 8 9 The expected output is 1 2 3 --- 4 5 --- 6 7 8 --- 9
4
votes
2answers
258 views

split a large file to new file with unique file names

I need to split a file into unique file names. I can do it with sed command eg, sed -n '/scaffold135_/w 135-scaf.txt' input file.txt but it's time consuming so I need a smart way to do it faster. ...