awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
2answers
56 views

join two files based on column when there is no one-to-one correspondence in bash script (awk, grep , sed)

file1.txt 112|9305|/inst.exe 112|9305|/lkj.exe 112|9305|/dje.jar 112|9305|/ind.pdf 112|9306|/ma.exe 112|9306|/ngg.pdf 112|9307|/jhhh.dat 112|9312|/ee.dat 112|9312|/qwq.dll file2.txt 117|9305|www....
3
votes
5answers
493 views

How to use awk or sed to convert rows to columns

I need to transpose a file. Input file: 1/1/1111 1 2 3 4 2/2/2222 5 6 7 8 Output: 1/1/1111 1 2 3 4 2/2/2222 5 6 7 8
0
votes
2answers
1k views

Bash to join columns from multiple files

I have around 20 files in different folders for which I have created a separate file called pathtofiles.in which lists path information: /home/users/gray_wolf/unix/File_1.tsv /home/users/gray_wolf/...
0
votes
1answer
51 views

How to get a X line from file1 and Y line from file2 using shell command?

In file 1, 'Server restoration is complete now. It will be operational within few seconds. Please keep all the log file once the restoration is complete.' In file 2, 'Restoration was successfully ...
0
votes
1answer
81 views

Array with two awk input files

I have two files, one with NF=7 and the other has NF=47 I need to check the number of occurrences of $2 in the file with NF=7. If the count is <=2, then I need to make an array from the same file ...
-1
votes
2answers
191 views

Using awk to select column data from separate files

I am not an experienced Linux user. I am trying to select column data from 2 separate files and write to a third file using awk. I have tried to paste the files together i.e. paste file1 file2 and ...
0
votes
2answers
132 views

To add line numbers in every line using shell command?

My file, PSS-A (Primary A) PSS-B (Primary B) PSS-C (Primary C) PSS-D (Primary D) PSS-E (Primary E) PSS-F (Primary F) PSS-G (Primary G) PSS-H (Primary H) PSS-I (Primary I) SPARE (...
7
votes
3answers
519 views

awk does not end

I have the following code in a file named awktest1.awk: #!/bin/awk -f BEGIN{print "start"} {print $2, "\t", $5} END{print "end"} employee.txt where employee.txt contains the following data: 100 ...
2
votes
2answers
101 views

Delimiter is being changed with fields

I have a file with values organized in columns, and separated by commas, as in the file below: 324,01,1,113333600000,1,, 016,01,1,134954200000,1,, 770,01,1,109069200000,1,, 853,01,1,111518800000,1,, ...
1
vote
3answers
72 views

Sum values in 5th column that correspond to same field in 2nd column

Considering below file: 0,2,,,10 0,2,,,15 0,1,,,984 0,2,,,9 1,14,,,5 Using awk, I need to calculate the total value in $5 per each $2. The desired output would look like below: 2,34 1,984 14,5
1
vote
2answers
10k views

How to remove a column or multiple columns from file using shell command?

My file, ARCHIVE B1_NAME B2_NAME B3_NAME ELEMENT INFO_NAM WERT PROCID -------- -------- -------- -------- -------- -------- ---- ------ ...
1
vote
1answer
21 views

Removing Zeroes before and after data sample

I have several data samples that I need to grop the zeroes before and after the data sample. However there are zeroes between the data sample that I must keep for obvious reasons. How can I do this ...
0
votes
2answers
3k views

How to detect and remove newline character within a column in a csv file?

I have a large (2 Million rows) csv file exported from a SQL Server database. I don't have access to the database, and there's some newline character within a column, which makes it difficult to ...
3
votes
3answers
191 views

Organize entries as a spreadsheet (csv) using awk

I have information for entries (N= 1000) saved on a computer in a delimited text file/per entry ( every entry has one text file). The information recorded in the same format as the following: =======...
1
vote
1answer
79 views

how to make awk print 3 items on the same line then switch to a new line

I am trying to parse the output of my top results to pickout the timestamp, used Mem and used Swap: top - 12:06:52 up 3:36, 37 users, load average: 0.00, 0.02, 0.00 Tasks: 563 total, 1 running, ...
4
votes
2answers
149 views

Find if any line of a file is a subset of data in another file

I have a file file1.txt with the following data: MIME_HTML_ONLY SUSPICIOUS_RECIPS SORTED_RECIPS HFILTER_HELO_5 and another file file2.txt with lines as: HFILTER_FROM_BOUNCE FORGED_OUTLOOK_HTML ...
5
votes
4answers
6k views

How to take the absolute value using awk?

If I have below two dates: 2015-09-12,2015-08-13 And I need to get the number of days between them, I will use the below code: awk -F'[-,]' '{print 360*($4-$1)+30*($5-$2)+($6-$3)}' The output ...
2
votes
2answers
293 views

search for exact string in unix

I have a file, xyz.txt, which has one string per line. How do I search xyz.txt for an exact match? I get an input string, eg. defabc, from another app. If defabc is found, it should return true, else ...
1
vote
2answers
60 views

Apply same field from different records to an operation.

Considering below file: 4,5,8,3,41,86,32, 0,0,0,0,38,0,8,8, 9,8,8,9,42,8,9,8, 1,1,7,1,11,2,5,1, How can I compare each $5 with the same field from the next record. Ex for the first two records: ...
1
vote
1answer
56 views

Get the filename that the record come from

I have a huge number of .gz files, which being uncompressed using zcat together. I am looking to get from where each record came from, as below example: File1: 1,2,3,4,5,6,7,NEW,9,10,11,12 13,14,...
-1
votes
2answers
116 views

How to remove line break if line character count less than n chars [closed]

Could anyone help me to concatenate the line with next line if the line char count is less than x chars.
1
vote
1answer
42 views

Form a output by combining multiple lines

I have a log file as below: /export/home/got/logs/o2877612.01:job_1432787863184_159408 Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: /user/got/...
2
votes
1answer
110 views

Using portion of a field in awk

How to use portion of a field for awk script? The input file: 123456789,0,0,4,1,6,8,6 987654321,0,0,8,4,6,9,8 251335549,8,9,6,3,1,8,8 897465419,8,1,8,9,4,7,9 I need to select the 3rd character ...
2
votes
1answer
289 views

how to split multiline records by awk?

How to split multi line records by awk, i.e below file: 027,027,0,3,,1,0,1,1,0,0,0,0,0,0,0,0,6,1,,1.211100,1,2015-08-03,2015-07-04,,2015-11-01,0,0,2015-11-01,1,1,,0,0,0,0,0,131,0,0,0,0,0,2015-06-01,...
1
vote
3answers
123 views

Using awk to merge 2 files

I would like to merge 2 files based on column 1, 2, 3. I have tried the awk command below but is not working. awk 'NR==FNR {h[$1FS$2FS$3]=$4; next}{k=$1FS$2FS$3; if (k in h) print $1,$2,$3,$4,h[k] ;...
1
vote
1answer
144 views

How to add hostname, date, and fix disk size

I'm collecting disk usage data and want to add the date, time, and hostname to this command. How can I do it in this format: YYYY-MM-DD HH:MM:SS,HOSTNAME,DISK_USAGE_DATA Command I have so far: ...
0
votes
1answer
195 views

find and replace with the value in another file

I have two files with different formats with columns tab spaced. I have to compare the columns column1, column2 of file1 with file2. If they matches, I need to replace the value in column6 of file1 ...
1
vote
2answers
309 views

Using awk to store a binary number in text file as string and then convert to hexadecimal

I'm using awk to parse a text file. The text file has four fields of byte data and appears as such: 11110000 10100000 10110000 10010000 I want to read the fields and convert them from binary to ...
1
vote
1answer
45 views

Using awk to edit numbers in a column

lets say I have a basic text file that looks like this: 91 N 91 H 1.48 95 N 95 H -0.26 97 N 97 H 1.79 98 N 98 H -0.76 I want to change columns 1 and 3 so that the integer values are n-3 like this: ...
1
vote
2answers
349 views

Difference between dates by awk

I have the below file: 1,2,3,4,5,2015-10-13,7,8,2037-01-15 1,2,3,4,5,2015-09-05,7,8,2040-01-15 1,2,3,4,5,2015-09-15,7,8,2016-06-15 1,2,3,4,5,2015-10-13,7,8,2015-07-08 1,2,3,4,5,2015-11-02,7,8,2015-...
0
votes
1answer
84 views

AWK summary file from a filtered list

I have 2 files, the first being a file of records that tell me which rows from the second file I am interested in (I'll call this the FILTER_FILE). This file can contain duplicate lines and I only ...
22
votes
9answers
5k views

How to delete every second line from a file?

File: Data inserted into table. Total count 13 No error occurred Data inserted into table. Total count 45 No error occurred Data inserted into table. Total count 14 No error occurred Data inserted ...
-1
votes
1answer
52 views

How to add a word at the first of every lines? [duplicate]

File1: Data will be inserted on Thursday Messages has been sent at 6pm Office hours is from 9am to 6pm All the employees are present Output what I want: DMS Data will be inserted on ...
2
votes
1answer
107 views

Show nth occurrence of a record

Using awk, how to show the nth occurrence of a record and delete the other occurrence, for example" Input File: 79933,2 79933,4 79933,5 55251,1 49494,4 49494,2 49494,2 49494,1 in this example, ...
2
votes
1answer
214 views

Format sar samples live with paste and awk

I'm combining live sar samples with paste and trying to format the output with awk live. It works as expected to format the output, but it doesn't do the formatting live on each sample and instead ...
1
vote
3answers
572 views

Why do awk split() make first field to be last element in array?

I'm probably missing something really simple here, but when I say echo 'The quick brown fox jumped over the lazy dog.' | \ awk '{ split($0, WORDS, " "); for ( WORD in WORDS ) { ...
1
vote
2answers
68 views

Append mth and nth columns of a file with the columns of another file

I have 2 files having different number of columns and with huge number of rows: bash-3.00$ cat fileA a1,a2,a3,a4,a5,a6,a7,a8,a9 q,w,,,,y,u,, a,z,,,,q,n,, ......................... z,p,,,,w,e,, PS1:...
1
vote
3answers
77 views

Extract specific value from block of data

I have the following block of data and need to extract the percent value in XYZ FREEBYTES, i.e. captured value should be equal to 64. As far as I know, I can grep and cut the data. However, the digits ...
2
votes
3answers
78 views

Extract specific value from blocks of data

I have the blocks of data which are differentiated by the global Ids as below. I need to extract the last block of data with the highest id value, in the below example, id="82". <gc type="global" ...
1
vote
1answer
82 views

error in awk/sed usage of file manipulation line-by-line [duplicate]

I have a requirement of appending the string " 1" at the end of every line of the file. After the execution of below two commands, an empty file is created. awk '{ print $0 " 1" }' < file.txt > ...
0
votes
1answer
39 views

Help with a AWK function

I have problems with AWK command. When I used this code is: $ awk '{a[NR]=$1} END {for (i=0;i<NR;i++) {B=a[i+1];A=a[i];C=(B-A);D=int(C/16)} {for (j=0;j<=D;j++) {if(C!=...
2
votes
4answers
331 views

Multiple files - awk command filter

I use a script say; filter1.sh which contains awk command; bzip2 -dc File_1.tsv.bz2 | awk -F '\t' 'BEGIN {OFS=FS} { if (($7) > 50) print $0 }' > File_1.tsv bzip2 -dc File_2.tsv.bz2 | awk -F ...
3
votes
3answers
281 views

awk comparison using arrays

I have the following file: 6180,6180,0,1,,1,0,1,1,0,0,0,0,0,0,0,0,4326,4326,,0.440000, 6553,6553,0,1,,1,0,1,1,0,0,0,0,1,0,1,0,4326,4326,,9.000000, 1297,1297,0,0,,0,0,1,0,0,0,0,0,1,0,1,0,1707,1707,,7....
2
votes
2answers
329 views

How to replace value for a given condition in specific column of file

I have file separetd by pipe | and I want that when column 6 contain letter I print 0758000 in column 7 and when the column 6 contain letter A print 0800000 in column 7, I can not find how to do it!!! ...
5
votes
1answer
1k views

Grep on single line [duplicate]

I have a script that streams a pattern in a single line (no linebreaks). I want to grep stock_ticker in that line and output it as soon as I have found one. Now the script is never ending and is ...
1
vote
3answers
4k views

How to pass variables to awk command with conditions?

I have a while loop which reads data from a file with city names: city_name: COIMBATORE MADURAI PONDICHERRY SALEM TRIPUR TIRUCHI TIRUNELVELI TUTUCORIN VELLORE and using an awk command within the ...
2
votes
4answers
176 views

Convert multilines to single line

I have multiline log file and I want to convert it to single line log. Multiline example: 6/13/2015 12:00:47 AM - { 562} START Web 6/13/2015 12:00:47 AM - Requested Web connection from 123.125....
0
votes
1answer
347 views

How can I find and replace only in a range of columns?

I have a text file with 9267 lines. I want to find "A" with 5 spaces after it and replace it with "A" with 5 spaces before and after it on all lines, but only between columns 67 and 88. Text outside ...
2
votes
2answers
58 views

Grouping lines into heterogeneous subsets

I have file with n lines.  (Each line refers to a “question”, and therefore they are labeled Q.1, Q.2, Q.3, ..., Q.n.)  Each line (question) has a “Marks” attribute, which has the value 2, 3, 4, 5, or ...
0
votes
1answer
55 views

Why the number is rounded over ssh? [duplicate]

localhost:~ $ df -k / | awk '{print $3/1024/1024/1024}' 0 0.00618061 localhost:~ $ ssh localhost "df -k / | awk '{print $3/1024/1024/1024}'" 01 01