awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
1answer
19 views

Better way to apply formula to specific column of inputfile, and pass-through all other columns as-is

I regularly need to apply simple formulas to input data give in textual tabular form. In the beginning, I wrote specific little functions for each use-case. Then I re-factored the "common code" into ...
4
votes
2answers
82 views

How Do I Remove Duplicate Words With Suffixes?

How do I compare a list of unsorted words of variable length and remove duplicate words that have a suffix? Example word list: iron curl curled railroad curler ...
-3
votes
0answers
27 views

How to do calculation on current line based on the output of previous line in unix? [on hold]

My input file is like: Sales,week,INV,purchase,EOH 1000,w1,500,200,700 1500,w2,400,100,1200 1300,w3,300,50,1050 900,w4,400,100,600 I need o/p file like, for week 1: f1,f2,f3,f4 1000,500,200,700 ...
-3
votes
0answers
27 views

Unix Line calculation in some fields of a file [on hold]

My input file is comma delimited with structure like: f1 f2 f3 f4 f5=f1-f3+f4 1000 wk1 500 200 700 1500 wk2 400 100 1200 1300 wk3 300 50 1050 900 wk4 400 100 600 ...
0
votes
1answer
41 views

sum of numbers located in different rows using awk

The following lines sum numbers located in the row #2 and row #3 , row #4 and row #5, row #6 and row #7 and so on (so, sum in a consecutive way). awk ' NR==1{$1=$1; print; next} !(NR%2){split($0,a);...
1
vote
3answers
47 views

Getting substrings in between delimiters multiple times from one string without looping

I have looked all over for a solution to this, but I couldn't find one. I am using qstat -x to pass a large string of job information. There is only whitespace in this output in certain areas. The ...
1
vote
2answers
8k views

How to remove a column or multiple columns from file using shell command?

My file, ARCHIVE B1_NAME B2_NAME B3_NAME ELEMENT INFO_NAM WERT PROCID -------- -------- -------- -------- -------- -------- ---- ------ ...
1
vote
1answer
32 views

Calculate hits within a timespan from log file [closed]

I need to use UNIX utilities that implement out-of-core divide-and-conquer algorithms so to decrease the amount of virtual memory used (cap of 1 mb). I have a large data set that is an apache log file....
1
vote
2answers
67 views

how to use \n( and )\n as delimiter in awk

I have a file which contains below data 7[label = "ScanStep: T_b0\n(T.a = 1)\na\nb\nc\nd\ne\ndob\ntimestamp1\nUnc: Integer(8)\nUnc: Integer(8)\nUnc: Varchar(80)\nUnc: Numeric(10,2)\nUnc: Varbinary(80)...
1
vote
1answer
61 views

Need to strip off the comments in double quotes

I have test case which contains string message in double quotes and I want to strip off data in double quotes. I used awk command to strip off comment section data but it doesn't work awk '/^"/ { ...
1
vote
3answers
42 views

Reformatting negative numbers with sed or awk

I have a text file that contains some negative numbers except, they are formatted as: 00000012- With the hyphen after the number. I am processing the file using awk to reformat it into SQL insert ...
-7
votes
0answers
39 views

How to get specific information out of /etc/passwd and display it into a file? [on hold]

Alright so I posted a previous POST which I figured out half my problem which I had was to get information like the user, hom dirs and bins to show and display out of /etc/passwd, this was done by: ...
2
votes
2answers
83 views

BASH: Using awk to filter unique lines results in 0 length array

Note: Thank you to Jeff Schaller and steeldriver. But as neither posted as an answer, I'm not sure how to mark as solved. I now have a better understanding of pipes/subshells. I'm pretty sure I once ...
0
votes
2answers
42 views

How to add a column of numbers to a array data file

I have a data file with two columns of numbers: input file 4.182 4.1843 4.184 4.2648 4.2281 4.0819 4.2204 4.1676 4.0482 4.1683 4.0156 4.2895 4.4504 5.2369 4.3776 4.4979 4.3797 4.1372 4....
14
votes
1answer
705 views

Why does awk print the whole line when I want it to print a variable?

When I type: echo "foo bar" | gawk -v var="content" '{print $var}' I expect the output to be content. Why is it foo bar?
0
votes
1answer
22 views

How to multiply two columns in a file by a constant number

I have a file like this. input data 4.2394 4.4569 4.2427 4.1011 4.2879 4.1237 4.2106 4.4844 4.2373 4.1071 4.1322 4.0502 4.3103 4.4255 4.4342 4.5262 I need to multiply each element by a constant ...
3
votes
3answers
39 views

awk to cut portion of a field and still print entire line

I have this line: 08:30:02.480507 IP 192.168.100.150.65119 > 192.168.100.151:53 59865 [1au] A? click.em.redbox.com. (48) I need to cut out the ".480507" in the timestamp so the line looks like ...
1
vote
3answers
80 views

Find and Replace with command line

my log.txt file includes ... mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 ... how can I replace mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 with mynetworks = 127.0....
2
votes
1answer
29 views

Merging specific columns of 2 files based on field match

I know I'm not the first one asking this but my code still does not work: File 1: gi|1283| tRNAscan exon 87020 88058 . - . transcript_id "Parent=tRNA-Tyr5.r01"; gi|3283| tRNAscan exon 97020 97058 . +...
-2
votes
1answer
50 views

filtering pattern in a file using sed or grep

I'm new to Linux. I'd like to store all the unique usernames present in the extended.log file into a new file using the commands awk, grep, and/or sed. The following are the field-names in my file, ...
0
votes
1answer
24 views

Setting Threshold on log Output

I have to check the log & monitor the I/O output, the log is generated per second I seems a hard job to monitor the log which get generated every second, I am looking for some awk command which ...
0
votes
2answers
45 views

Read a file using shell script and make 2 new files based on Name of item

I have below file with me :- ====== 20160606:034441 ====== Mango(Test) TestName MangoT Row 0 Season N Name Safeda Location Delhi ====== 20160606:034441 ====== Mango(Result) TestName ...
1
vote
2answers
49 views

Make the 1st column as header and list its corresponding values using shell script

I have a large file contents are as shown below :- Quantity 20589 Quantity 12297 Quantity 100346 Quantity 0 Quantity 141999 Quantity 23662 Quantity 551071 Quantity 72917 ...
2
votes
1answer
111 views

Read from and append to file at the same time while preserving end-of-line

Conceptually I have an easy task... I have [loosely] structured data in a file: Testing: debug, default CXXFLAGS <100's of additional output lines> Testing: release, default CXXFLAGS <100's ...
0
votes
4answers
44 views

Find a text in a file and Insert a text after it

I have a text file. The text file contains the below content. ServerName ServerAdmin webmaster@localhost DocumentRoot I need to insert a text www.mydomain.com, after the ServerName. Is there any ...
2
votes
2answers
105 views

Join seven files with awk line-by-line

I have seven (or eight and so on) files with same number of lines. file1 1.001 1.002 1.003 1.004 file2 2.001 2.002 2.003 2.004 file3 3.001 3.002 3.003 3.004 etc. Desired output: 1.001;2....
0
votes
2answers
2k views

Sum two columns from different files

I have two files like this: File1: -54.005 -67.405 0.1 -54.05 -68.7833 0.2 -54.05 -68.7333 0.3 File2: -54.005 -67.4050 70 12.7 -54.05 -68.7833 167 12.5 -54.05 -68.7333 205 12.6 ...
3
votes
2answers
346 views

grep: How to find Closing Bracket?

One part of /usr/share/X11/xkb/symbols/us starts with xkb_symbols "dvorak" { and ends with the closing curly bracket }; which line-number I want to find. partial alphanumeric_keys xkb_symbols "...
0
votes
3answers
3k views

How to convert Log file to CSV using bash

Is it possible to turn this log entry into a properly formatted csv file using sed/awk? I've tried multiple ways to do this using sed/awk. Alas I couldn't do it. Before I do it in perl. is this ...
1
vote
1answer
44 views

Make an IF condition out of result we get from remote server

I need to check for files availability in remote server, and based on the files availability i need to make an if condition. I have tried the below commands in a shell script to list the files in the ...
5
votes
3answers
7k views

How to apply the same awk action to different files?

I am new in awk and I do not know if it is possible to write an awk script that does this: I have hundred of data files that I have to sort. For each one I use the following one-liner: awk 'ORS=NR%3?...
-8
votes
2answers
48 views

how to print in a single line with pattern matched

student:xxxxx student:yyyyy student:zzzzz student:kkkkk student:. teacher:aaaaa teacher:bbbbb teacher:lllll teacher:. student:sssss student:mmmmm student:. output should be, student:xxxxx yyyyy ...
0
votes
0answers
21 views

awk - how to pass dynamic value in pattern? [duplicate]

I am passing following date pattern to retrieve log details for particular date from the application log file. awk '/2016-06-15/{print $0}' $application.log >> $LOG_FILE I want date value as ...
2
votes
1answer
53 views

mysterious awk script joins lines

I have a text file that has blocks of consecutive non empty lines separated by a blank line. I want to join the lines in a block (similar to the "J" key in the vi editor). Here I found the following ...
2
votes
2answers
72 views

Grep for dates older than 14 days in a file [duplicate]

I'm trying to find a way to grep how many messages in a file are older than 14 days and have a value of number of results return Example going with today's date of 20160616. $grep 'Put Date' filename:...
0
votes
2answers
316 views

Find line within file matching pattern and make a substitution

I'm trying to simplify my life here but I'm stuck. I have a file containing multiple lines. Each line has a user and a number. I'd like to know a bash one-liner to search for a user and change the ...
25
votes
12answers
12k views

How can I write to the second line of a file from the command line?

I have an external program that produces an output file (largish, 20K lines possible). I need to insert a new line between the existing line 1 and line 2. I've been looking at awk and sed - I use ...
1
vote
1answer
64 views

stat(FILENAME) in awk

Title say it all. How can I get file attribute with awk ? or Is there an equivalent to c/php stat() function ? obviously when awk'ing a pipe, file attribute would be undefined. I know I can use ...
1
vote
2answers
27 views

Insert text inside stanza at the end of the data string

I have a file I need to echo text into at the end of the file from the command line, the problem is the data is in a stanza, all within curly braces. This is the end of the file: text-string1,text-...
3
votes
2answers
72 views

Save the matches of grep as an array

I have a file with the following contents: FILETYPE=A:B:C:D How can I grep out A, B, C and D and save them as an array in a ksh script on Linux? I tried FILETYPES=`grep "FILETYPE" ${CONF_FILE} | ...
1
vote
2answers
55 views

Creating a text file from other text files

I have a text file like this word_1 root_of_word_A word_2 root_of_word_B word_3 root_of_word_C word_4 root_of_word_A And other document like this root_of_word_A root_of_word_1 root_of_word_B ...
-1
votes
1answer
29 views

How to record numbers of array elements based on selection comparision criteria

I have many files which has many 2x2 arrays. In the present example I show only one file (file1) Input file(file1): 1: 6.1703 541.631 46.0391 2: 6.1930 537.446 45.9239 3: 6.1931 177.171 288.579 ...
1
vote
2answers
45 views

print multiple words seperated by space

I have a text file and the data inside has format like (name age country): michael jordans 25 US adam smith 30 UK chris wood ABC 22 Aus if I use command: cat text.txt | awk {'print $1'} --> it ...
8
votes
2answers
17k views

Extract file name from path in awk program

I have an awk script and I have passed a CSV file to it. awk -f script.awk /home/abc/imp/asgd.csv What am I doing is to get FILENAME within script.awk. FILENAME gives me the whole path. As I am in ...
2
votes
2answers
64 views

How to get the last occurrence of lines between two patterns from a file?

I have a log file which reports on the output of a process, I'd like to extract all lines from between the last occurrence of two patterns. The patterns will be along the lines of; Summary process ...
4
votes
2answers
40 views

Is it possible to print an iterated range of arguments via cli from a csv? (see example)

I have hundreds of logs spread across a few thousand VM's and I'm trying to create a few scripts to quickly scan logs and I have figured out the majority of the presentation and wanted to know if ...
-5
votes
3answers
97 views

How to add all the numbers of a specific column in UNIX without using awk command?

Let us take a sample file with the contents below, an expense report containing items and their prices: $ cat file Item1,200 Item2,500 Item345,900 Item20896,800 Item111,600 Now, a friend challenged ...
0
votes
1answer
31 views

difference between [-0-9] and [-0-9 ] pattern in awk command

What is difference between the following commands: awk '/[-0-9]/ {print $0}' data_file and: awk '/[-0-9 ]/ {print $0}' data_file the second command differs by a space in [-0-9] pattern from the ...
0
votes
2answers
1k views

Bash to join columns from multiple files

I have around 20 files in different folders for which I have created a separate file called pathtofiles.in which lists path information: /home/users/gray_wolf/unix/File_1.tsv /home/users/gray_wolf/...
2
votes
2answers
126 views

Multiplying numbers in-place in a file

I have ASCII file contain numbers as follow: 1 2 3 5 6 9 7 8 9 . . . . . . . . . . . . . . . etc I want to upgrade this file by multiplying each value by "3" to get : 3 6 9 15 18 27 21 24 27 ...