awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

3
votes
3answers
93 views

Organize entries as a spreadsheet (csv) using awk

I have information for entries (N= 1000) saved on a computer in a delimited text file/per entry ( every entry has one text file). The information recorded in the same format as the following: ...
21
votes
9answers
3k views

How to delete every second line from a file?

File: Data inserted into table. Total count 13 No error occurred Data inserted into table. Total count 45 No error occurred Data inserted into table. Total count 14 No error occurred Data inserted ...
5
votes
3answers
353 views

Join: Two files - but only append last two columns

Given the files: 1.txt 1, abc, 123, 456, 789 2, lmn, 123, 456, 789 3, pqr, 123, 456, 789 2.txt 1, abc, 123, 000, 000 3, lmn, 123, 000, 000 9, opq, 123, 000, 000 OUTPUT.txt ID, NAME, X, ...
4
votes
2answers
66 views

Find if any line of a file is a subset of data in another file

I have a file file1.txt with the following data: MIME_HTML_ONLY SUSPICIOUS_RECIPS SORTED_RECIPS HFILTER_HELO_5 and another file file2.txt with lines as: HFILTER_FROM_BOUNCE FORGED_OUTLOOK_HTML ...
1
vote
1answer
31 views

how to make awk print 3 items on the same line then switch to a new line

I am trying to parse the output of my top results to pickout the timestamp, used Mem and used Swap: top - 12:06:52 up 3:36, 37 users, load average: 0.00, 0.02, 0.00 Tasks: 563 total, 1 running, ...
2
votes
4answers
168 views

How to take the absolute value using awk?

If I have below two dates: 2015-09-12,2015-08-13 And I need to get the number of days between them, I will use the below code: awk -F'[-,]' '{print 360*($4-$1)+30*($5-$2)+($6-$3)}' The output ...
0
votes
1answer
70 views

find and replace with the value in another file

I have two files with different formats with columns tab spaced. I have to compare the columns column1, column2 of file1 with file2. If they matches, I need to replace the value in column6 of file1 ...
2
votes
2answers
74 views

search for exact string in unix

I have a file, xyz.txt, which has one string per line. How do I search xyz.txt for an exact match? I get an input string, eg. defabc, from another app. If defabc is found, it should return true, else ...
1
vote
2answers
44 views

Using awk to store a binary number in text file as string and then convert to hexadecimal

I'm using awk to parse a text file. The text file has four fields of byte data and appears as such: 11110000 10100000 10110000 10010000 I want to read the fields and convert them from binary to ...
1
vote
2answers
46 views

Apply same field from different records to an operation.

Considering below file: 4,5,8,3,41,86,32, 0,0,0,0,38,0,8,8, 9,8,8,9,42,8,9,8, 1,1,7,1,11,2,5,1, How can I compare each $5 with the same field from the next record. Ex for the first two records: ...
-1
votes
2answers
37 views

How to remove line break if line character count less than n chars [closed]

Could anyone help me to concatenate the line with next line if the line char count is less than x chars.
1
vote
1answer
31 views

Get the filename that the record come from

I have a huge number of .gz files, which being uncompressed using zcat together. I am looking to get from where each record came from, as below example: File1: 1,2,3,4,5,6,7,NEW,9,10,11,12 ...
6
votes
3answers
8k views

Filter a .CSV file based on the 5th column values of a file and print those records into a new file

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , ...
1
vote
1answer
32 views

Form a output by combining multiple lines

I have a log file as below: /export/home/got/logs/o2877612.01:job_1432787863184_159408 Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: ...
2
votes
1answer
95 views

Using portion of a field in awk

How to use portion of a field for awk script? The input file: 123456789,0,0,4,1,6,8,6 987654321,0,0,8,4,6,9,8 251335549,8,9,6,3,1,8,8 897465419,8,1,8,9,4,7,9 I need to select the 3rd character ...
0
votes
1answer
45 views

AWK summary file from a filtered list

I have 2 files, the first being a file of records that tell me which rows from the second file I am interested in (I'll call this the FILTER_FILE). This file can contain duplicate lines and I only ...
2
votes
1answer
107 views

how to split multiline records by awk?

How to split multi line records by awk, i.e below file: ...
1
vote
3answers
33 views

Using awk to merge 2 files

I would like to merge 2 files based on column 1, 2, 3. I have tried the awk command below but is not working. awk 'NR==FNR {h[$1FS$2FS$3]=$4; next}{k=$1FS$2FS$3; if (k in h) print $1,$2,$3,$4,h[k] ...
1
vote
1answer
34 views

How to add hostname, date, and fix disk size

I'm collecting disk usage data and want to add the date, time, and hostname to this command. How can I do it in this format: YYYY-MM-DD HH:MM:SS,HOSTNAME,DISK_USAGE_DATA Command I have so far: ...
3
votes
1answer
60 views

Grep on single line [duplicate]

I have a script that streams a pattern in a single line (no linebreaks). I want to grep stock_ticker in that line and output it as soon as I have found one. Now the script is never ending and is ...
0
votes
1answer
24 views

Using awk to edit numbers in a column

lets say I have a basic text file that looks like this: 91 N 91 H 1.48 95 N 95 H -0.26 97 N 97 H 1.79 98 N 98 H -0.76 I want to change columns 1 and 3 so that the integer values are n-3 like this: ...
7
votes
7answers
418 views

How to efficiently split up a large text file wihout splitting multiline records?

I have a big text file (~50Gb when gz'ed). The file contains 4*N lines or N records; that is every record consists of 4 lines. I would like to split this file into 4 smaller files each sized roughly ...
4
votes
5answers
2k views

How to remove all comments from a file?

I have a file with a few lines commented out: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I just want to print all the uncommented code: foo bar stuff morestuff ...
-1
votes
1answer
44 views

How to add a word at the first of every lines? [duplicate]

File1: Data will be inserted on Thursday Messages has been sent at 6pm Office hours is from 9am to 6pm All the employees are present Output what I want: DMS Data will be inserted on ...
1
vote
3answers
166 views

Why do awk split() make first field to be last element in array?

I'm probably missing something really simple here, but when I say echo 'The quick brown fox jumped over the lazy dog.' | \ awk '{ split($0, WORDS, " "); for ( WORD in WORDS ) { ...
2
votes
1answer
41 views

Format sar samples live with paste and awk

I'm combining live sar samples with paste and trying to format the output with awk live. It works as expected to format the output, but it doesn't do the formatting live on each sample and instead ...
2
votes
1answer
72 views

Show nth occurrence of a record

Using awk, how to show the nth occurrence of a record and delete the other occurrence, for example" Input File: 79933,2 79933,4 79933,5 55251,1 49494,4 49494,2 49494,2 49494,1 in this example, ...
2
votes
3answers
504 views

Get last set of non-empty lines

I'm running a cron job which should get only the last result of iostat -d 1 2. This requires some parsing: What is the simplest way to get the last set of non-empty lines from standard input to ...
17
votes
11answers
4k views

There must be a better way to replace single newlines only?

I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
2
votes
3answers
61 views

Extract specific value from blocks of data

I have the blocks of data which are differentiated by the global Ids as below. I need to extract the last block of data with the highest id value, in the below example, id="82". <gc type="global" ...
1
vote
3answers
25 views

Extract specific value from block of data

I have the following block of data and need to extract the percent value in XYZ FREEBYTES, i.e. captured value should be equal to 64. As far as I know, I can grep and cut the data. However, the digits ...
1
vote
2answers
34 views

Append mth and nth columns of a file with the columns of another file

I have 2 files having different number of columns and with huge number of rows: bash-3.00$ cat fileA a1,a2,a3,a4,a5,a6,a7,a8,a9 q,w,,,,y,u,, a,z,,,,q,n,, ......................... z,p,,,,w,e,, ...
1
vote
1answer
56 views

error in awk/sed usage of file manipulation line-by-line [duplicate]

I have a requirement of appending the string " 1" at the end of every line of the file. After the execution of below two commands, an empty file is created. awk '{ print $0 " 1" }' < file.txt > ...
2
votes
4answers
119 views

Multiple files - awk command filter

I use a script say; filter1.sh which contains awk command; bzip2 -dc File_1.tsv.bz2 | awk -F '\t' 'BEGIN {OFS=FS} { if (($7) > 50) print $0 }' > File_1.tsv bzip2 -dc File_2.tsv.bz2 | awk -F ...
0
votes
1answer
32 views

Help with a AWK function

I have problems with AWK command. When I used this code is: $ awk '{a[NR]=$1} END {for (i=0;i<NR;i++) {B=a[i+1];A=a[i];C=(B-A);D=int(C/16)} {for (j=0;j<=D;j++) ...
2
votes
2answers
50 views

Awk - keep word “example” and “EXAMPLE” with one input

I am attempting to to only keep customers that contain certain words. Here is an example < newname.txt awk 'BEGIN {FS=","} $7~/USA|United States/' > usa.txt The problem I am having is if they ...
4
votes
6answers
196 views

Extracting lines based on conditions

Each line in a comma-separated file has 5 fields. a,b,c,d,e f,g,c,i, j,k,c,m,n o,p,c,r,s t,u,c,w, x,y,z,aa,bb How can I extract the lines which have c in the 3rd field and their 5th field is NOT ...
3
votes
3answers
215 views

awk comparison using arrays

I have the following file: 6180,6180,0,1,,1,0,1,1,0,0,0,0,0,0,0,0,4326,4326,,0.440000, 6553,6553,0,1,,1,0,1,1,0,0,0,0,1,0,1,0,4326,4326,,9.000000, ...
2
votes
2answers
22 views

How to replace value for a given condition in specific column of file

I have file separetd by pipe | and I want that when column 6 contain letter I print 0758000 in column 7 and when the column 6 contain letter A print 0800000 in column 7, I can not find how to do it!!! ...
1
vote
3answers
150 views

How to pass variables to awk command with conditions?

I have a while loop which reads data from a file with city names: city_name: COIMBATORE MADURAI PONDICHERRY SALEM TRIPUR TIRUCHI TIRUNELVELI TUTUCORIN VELLORE and using an awk command within the ...
3
votes
3answers
1k views

Bash Script To Find Debian Release Number from /etc/debian_version

The following is what I'm currently using for testing the version of Debian. Instead of doing it this way, can I make it more succinct? #!/bin/bash VERSION=NONE if [ $(cat /etc/debian_version) == ...
2
votes
4answers
126 views

Convert multilines to single line

I have multiline log file and I want to convert it to single line log. Multiline example: 6/13/2015 12:00:47 AM - { 562} START Web 6/13/2015 12:00:47 AM - Requested Web connection from ...
0
votes
4answers
191 views

How to extract column name (header) from a CSV file which contains the max value in a row?

I am trying to extract the column name with the maximum value in each row using bash script i.e., the column header value or the value from the same column in the first row. I am using the following ...
0
votes
1answer
50 views

Why the number is rounded over ssh? [duplicate]

localhost:~ $ df -k / | awk '{print $3/1024/1024/1024}' 0 0.00618061 localhost:~ $ ssh localhost "df -k / | awk '{print $3/1024/1024/1024}'" 01 01
2
votes
2answers
43 views

Grouping lines into heterogeneous subsets

I have file with n lines.  (Each line refers to a “question”, and therefore they are labeled Q.1, Q.2, Q.3, ..., Q.n.)  Each line (question) has a “Marks” attribute, which has the value 2, 3, 4, 5, or ...
0
votes
1answer
75 views

How can I find and replace only in a range of columns?

I have a text file with 9267 lines. I want to find "A" with 5 spaces after it and replace it with "A" with 5 spaces before and after it on all lines, but only between columns 67 and 88. Text outside ...
5
votes
4answers
162 views

Remove lines from file up to a pattern, unless the pattern doesn't exist

I am attempting to write a filter using something like sed or awk to do the following: If a given pattern does not exist in the input, copy the entire input to the output If the pattern exists in ...
0
votes
1answer
41 views

How to replace the content of specific column with awk? Tab Delimited File

Here's a similar question but I ran into a problem that was not addressed in the question. How to replace the content of specific column with awk? I want to do something similar, however my file is ...
1
vote
1answer
21 views

AWK: where is my error?

I have a file with :-delimited fields. If the 3rd field of a line is greater than 22, that line should be printed to file44. The file looks like this: 1:2:3:4 1:2:3:411 1:2:33:42 1:2:3:44444 ...
2
votes
3answers
66 views

Read a column from a file and append a specific column of another file

I have two files as such: fileA: Name,Site Name,Product UPEI,Product Name,NMS Flag,Product Model, SW FileB: NE_Name SW ABC 4.4 ASD 4.3 ... ... Note that ...