awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
1answer
56 views

printing in awk

I have a file number.txt with following record. Number.txt: 123456,2711448,1,14-Feb-15 09:24:25,14-Jul-15 09:24:25,120,20150814163821,13-Aug-15 09:24:25,,,12-Sep-15 09:24:25,,,12-Oct-15 09:24:25,,,...
0
votes
1answer
3k views

shell script to print rows if there is a value in column 2

1 2 3 4 3 1 4 2 1 4 3 2 4 2 1 3 How can I print all rows, on which there is a 2 in the second column.
0
votes
2answers
276 views

AWK, SED or GREP for extracting data from HTML file

Can anyone tell me which is best to use to extract the following data from a html file which was received with curl . <script>document.getElementById("test-summary").innerHTML = "<strong>...
8
votes
4answers
841 views

Splitting file for every 10000 numbers ( not lines )

I have a file that looks like the following: chr19 61336212 + 0 0 CG CGT chr19 61336213 - 0 0 CG CGG chr19 61336218 + ...
1
vote
1answer
49 views

add pairwise values looking up from a file

I have several variables (col1) which have 3 values (col 3,4,5) for some levels (col2) mango123 L1 1 2 3 mango123 L2 4 5 6 squash378 L1 7 8 9 squash378 L5 10 12 13 squash378 L15 0 1 3 ...
3
votes
3answers
303 views

Multiple column matching and adjusting with awk

I have a problem that's giving me a bit of a headache, but it doesn't tend to take much as a relative beginner. I have a file that looks like this ID A1 A2 A3 1 A G A 2 T G A ...
6
votes
3answers
450 views

Print random lines respecting the order of the source file

I have a large text file (~500K lines) with short sentences (couple of words long). Additionally, there is some XML markup in most of the lines. Finally, the text file has been sorted before the ...
4
votes
1answer
235 views

awk/nawk alternative on SunOs and Linux

I have a script that basically checks the Java version on the box it's running on and does whatever based on the version number. My problem is that I want to use the same command on both SunOs and ...
1
vote
1answer
111 views

Awkward awk exit code

I'm writing some awk scripts for data validation. These are intended to be selected and invoked by another proces which needs to see the exit code from the awk script to determine if the content was ...
2
votes
2answers
68 views

filter based on a field value in awk

I want to print the countries which which have to after ~:. welcome~:to~:Germany welcome~:no please~:Italy welcome~:to~:Brazil welcome~:not ok~:China Note: I do not know what will be there other ...
1
vote
1answer
57 views

Compare records value with each others.

Please consider below file: foo,boo,900 foo,boo,900 foo,boo,850 I need to compare the a field ($3) with the next record, if the difference is equal or more than 50, then print the record. i.e ...
2
votes
4answers
357 views

Fill empty lines in specific column with values

I have data of the following format: 2342 1 6473 2 7463 2 3647 2734 1 9343 The lines that do not have a value in the 2nd column are empty in that spot (i.e. there is no space, tab etc.). I am ...
1
vote
3answers
60 views

Remove words (letters followed by space) from a specific column

I have a file and the format is as follows: cat dog AHF123432 | 123432 | dhfshfjdh lion AFG23412 |23412 | dfshjhfjdhj I need to remove those words from first column which contains only alphabets i....
9
votes
2answers
16k views

Conditional block vs conditional statement (if)

Say I have a file: PRO 1 GLN 5.55112e-17 ILE -6.245e-17 THR 5.55112e-17 I want every line that has a number unequal to 1 in the second column to change it to 0 and keep the rest. If I use if (i.e ...
0
votes
1answer
618 views

Timestamp comparison in awk

Considering below file, how can I compare two timestamps, noting that the time part is negligible. foo,boo,2038-01-18T12:00:00+02:00 foo,boo,2015-09-12T01:31:24+03:00 foo,boo,2015-08-04T03:15:38+03:...
1
vote
2answers
316 views

Difference between dates by awk

I have the below file: 1,2,3,4,5,2015-10-13,7,8,2037-01-15 1,2,3,4,5,2015-09-05,7,8,2040-01-15 1,2,3,4,5,2015-09-15,7,8,2016-06-15 1,2,3,4,5,2015-10-13,7,8,2015-07-08 1,2,3,4,5,2015-11-02,7,8,2015-...
4
votes
1answer
190 views

In AWK, is it possible to specify a range for operation?

Example: Given a comma separated file "foo" with 50 fields per line, I want to print $0 if $10 != the range from 1001 to 1010. I can do it using below code: awk -F, '{if($10!=1001 && $10!=...
0
votes
1answer
90 views

How do I add a \n to the end of the record on file using AWK

I have a file that can contain 1 or many lines oracle1.txt oracle2.txt oracle3.txt How do I using AWK put a \n after the last line only so it will show oracle1.txt oracle2.txt oracle3.txt\n I ...
0
votes
1answer
51 views

How to get a X line from file1 and Y line from file2 using shell command?

In file 1, 'Server restoration is complete now. It will be operational within few seconds. Please keep all the log file once the restoration is complete.' In file 2, 'Restoration was successfully ...
7
votes
3answers
507 views

awk does not end

I have the following code in a file named awktest1.awk: #!/bin/awk -f BEGIN{print "start"} {print $2, "\t", $5} END{print "end"} employee.txt where employee.txt contains the following data: 100 ...
-1
votes
2answers
178 views

Using awk to select column data from separate files

I am not an experienced Linux user. I am trying to select column data from 2 separate files and write to a third file using awk. I have tried to paste the files together i.e. paste file1 file2 and ...
0
votes
2answers
126 views

To add line numbers in every line using shell command?

My file, PSS-A (Primary A) PSS-B (Primary B) PSS-C (Primary C) PSS-D (Primary D) PSS-E (Primary E) PSS-F (Primary F) PSS-G (Primary G) PSS-H (Primary H) PSS-I (Primary I) SPARE (...
1
vote
1answer
21 views

Removing Zeroes before and after data sample

I have several data samples that I need to grop the zeroes before and after the data sample. However there are zeroes between the data sample that I must keep for obvious reasons. How can I do this ...
0
votes
2answers
3k views

How to detect and remove newline character within a column in a csv file?

I have a large (2 Million rows) csv file exported from a SQL Server database. I don't have access to the database, and there's some newline character within a column, which makes it difficult to ...
3
votes
3answers
184 views

Organize entries as a spreadsheet (csv) using awk

I have information for entries (N= 1000) saved on a computer in a delimited text file/per entry ( every entry has one text file). The information recorded in the same format as the following: =======...
22
votes
9answers
5k views

How to delete every second line from a file?

File: Data inserted into table. Total count 13 No error occurred Data inserted into table. Total count 45 No error occurred Data inserted into table. Total count 14 No error occurred Data inserted ...
5
votes
3answers
420 views

Join: Two files - but only append last two columns

Given the files: 1.txt 1, abc, 123, 456, 789 2, lmn, 123, 456, 789 3, pqr, 123, 456, 789 2.txt 1, abc, 123, 000, 000 3, lmn, 123, 000, 000 9, opq, 123, 000, 000 OUTPUT.txt ID, NAME, X, 1A,...
4
votes
2answers
140 views

Find if any line of a file is a subset of data in another file

I have a file file1.txt with the following data: MIME_HTML_ONLY SUSPICIOUS_RECIPS SORTED_RECIPS HFILTER_HELO_5 and another file file2.txt with lines as: HFILTER_FROM_BOUNCE FORGED_OUTLOOK_HTML ...
1
vote
1answer
76 views

how to make awk print 3 items on the same line then switch to a new line

I am trying to parse the output of my top results to pickout the timestamp, used Mem and used Swap: top - 12:06:52 up 3:36, 37 users, load average: 0.00, 0.02, 0.00 Tasks: 563 total, 1 running, ...
3
votes
4answers
6k views

How to take the absolute value using awk?

If I have below two dates: 2015-09-12,2015-08-13 And I need to get the number of days between them, I will use the below code: awk -F'[-,]' '{print 360*($4-$1)+30*($5-$2)+($6-$3)}' The output ...
0
votes
1answer
187 views

find and replace with the value in another file

I have two files with different formats with columns tab spaced. I have to compare the columns column1, column2 of file1 with file2. If they matches, I need to replace the value in column6 of file1 ...
2
votes
2answers
274 views

search for exact string in unix

I have a file, xyz.txt, which has one string per line. How do I search xyz.txt for an exact match? I get an input string, eg. defabc, from another app. If defabc is found, it should return true, else ...
1
vote
2answers
289 views

Using awk to store a binary number in text file as string and then convert to hexadecimal

I'm using awk to parse a text file. The text file has four fields of byte data and appears as such: 11110000 10100000 10110000 10010000 I want to read the fields and convert them from binary to ...
1
vote
2answers
59 views

Apply same field from different records to an operation.

Considering below file: 4,5,8,3,41,86,32, 0,0,0,0,38,0,8,8, 9,8,8,9,42,8,9,8, 1,1,7,1,11,2,5,1, How can I compare each $5 with the same field from the next record. Ex for the first two records: ...
-1
votes
2answers
114 views

How to remove line break if line character count less than n chars [closed]

Could anyone help me to concatenate the line with next line if the line char count is less than x chars.
1
vote
1answer
56 views

Get the filename that the record come from

I have a huge number of .gz files, which being uncompressed using zcat together. I am looking to get from where each record came from, as below example: File1: 1,2,3,4,5,6,7,NEW,9,10,11,12 13,14,...
6
votes
3answers
12k views

Filter a .CSV file based on the 5th column values of a file and print those records into a new file

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , ...
1
vote
1answer
41 views

Form a output by combining multiple lines

I have a log file as below: /export/home/got/logs/o2877612.01:job_1432787863184_159408 Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: /user/got/...
2
votes
1answer
108 views

Using portion of a field in awk

How to use portion of a field for awk script? The input file: 123456789,0,0,4,1,6,8,6 987654321,0,0,8,4,6,9,8 251335549,8,9,6,3,1,8,8 897465419,8,1,8,9,4,7,9 I need to select the 3rd character ...
0
votes
1answer
81 views

AWK summary file from a filtered list

I have 2 files, the first being a file of records that tell me which rows from the second file I am interested in (I'll call this the FILTER_FILE). This file can contain duplicate lines and I only ...
2
votes
1answer
279 views

how to split multiline records by awk?

How to split multi line records by awk, i.e below file: 027,027,0,3,,1,0,1,1,0,0,0,0,0,0,0,0,6,1,,1.211100,1,2015-08-03,2015-07-04,,2015-11-01,0,0,2015-11-01,1,1,,0,0,0,0,0,131,0,0,0,0,0,2015-06-01,...
1
vote
1answer
120 views

How to add hostname, date, and fix disk size

I'm collecting disk usage data and want to add the date, time, and hostname to this command. How can I do it in this format: YYYY-MM-DD HH:MM:SS,HOSTNAME,DISK_USAGE_DATA Command I have so far: ...
5
votes
1answer
1k views

Grep on single line [duplicate]

I have a script that streams a pattern in a single line (no linebreaks). I want to grep stock_ticker in that line and output it as soon as I have found one. Now the script is never ending and is ...
1
vote
1answer
43 views

Using awk to edit numbers in a column

lets say I have a basic text file that looks like this: 91 N 91 H 1.48 95 N 95 H -0.26 97 N 97 H 1.79 98 N 98 H -0.76 I want to change columns 1 and 3 so that the integer values are n-3 like this: ...
8
votes
7answers
1k views

How to efficiently split up a large text file wihout splitting multiline records?

I have a big text file (~50Gb when gz'ed). The file contains 4*N lines or N records; that is every record consists of 4 lines. I would like to split this file into 4 smaller files each sized roughly ...
-1
votes
1answer
51 views

How to add a word at the first of every lines? [duplicate]

File1: Data will be inserted on Thursday Messages has been sent at 6pm Office hours is from 9am to 6pm All the employees are present Output what I want: DMS Data will be inserted on ...
1
vote
3answers
542 views

Why do awk split() make first field to be last element in array?

I'm probably missing something really simple here, but when I say echo 'The quick brown fox jumped over the lazy dog.' | \ awk '{ split($0, WORDS, " "); for ( WORD in WORDS ) { ...
2
votes
1answer
201 views

Format sar samples live with paste and awk

I'm combining live sar samples with paste and trying to format the output with awk live. It works as expected to format the output, but it doesn't do the formatting live on each sample and instead ...
2
votes
1answer
106 views

Show nth occurrence of a record

Using awk, how to show the nth occurrence of a record and delete the other occurrence, for example" Input File: 79933,2 79933,4 79933,5 55251,1 49494,4 49494,2 49494,2 49494,1 in this example, ...
2
votes
3answers
660 views

Get last set of non-empty lines

I'm running a cron job which should get only the last result of iostat -d 1 2. This requires some parsing: What is the simplest way to get the last set of non-empty lines from standard input to ...