awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

15
votes
4answers
14k views

Delete the first n bytes of files

I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way. I want to delete the first 31 bytes of each file ...
10
votes
12answers
3k views

What's a good way to filter a text file to remove empty lines?

I have a .csv file (on a mac) that has a bunch of empty lines, e.g.: "1", "2", "lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum lorem ipsum ...
-1
votes
2answers
35 views

awk print all matching lines including duplicates [duplicate]

How can I print all matching lines between two files including the duplicates sourcefile: SNP value RS1 0.55 RS2 0.52 RS3 0.25 RS4 0.22 queryfile: SNP RS1 RS2 RS2 RS3 RS4 RS4 Desired output: ...
0
votes
2answers
38 views

Remove lines in a file where pattern only occurs once

I have a CSV file. It is ordered by an identifier the first field. The identifier in the first field can appear in either 1 or in 2 lines in the field. I want to remove all lines in the file where ...
25
votes
11answers
12k views

How can I write to the second line of a file from the command line?

I have an external program that produces an output file (largish, 20K lines possible). I need to insert a new line between the existing line 1 and line 2. I've been looking at awk and sed - I use ...
0
votes
1answer
19 views

Check if stdout is attached to a terminal from an AWK script

Is there any way I can check if an AWK script's stdout is attached to a terminal, just like test -t 1 in a shell?
0
votes
1answer
38 views

Awk to get same column with maximum filed

I have two files. I have to get matching name from file1, print the entire content of file1, and add a fourth column (exp) in file 1 using a second file with max value file1 name value1 ...
1
vote
1answer
37 views

Awk field printing within the ternary operator

If you're familiar with DISA STIGs, the test for RHEL-06-000518 is /bin/rpm -Va 2>/dev/null \ | /bin/grep '^.M' What I want to do is pull just the file name out of that test, and make pretty ...
1
vote
1answer
18 views

Awk problem: an 'if column conditional' replaces the column entry!

I'm trying to get a grip on awk. Suppose I have an input: this header fred blah 1 blank fred blah 1 blank huh blah 2 blank fred blah 3 blank fred blah 3 blank and after I use the following awk ...
0
votes
2answers
53 views

get day from date with append 0 in 1 to 9 digit

i am using awk command to get creation date of file and from that i m fetching day,month and year but for 1-9 digit i am getting output like 2016041 but i want output 20160401. D=$(ls -ltr $line | ...
1
vote
1answer
65 views

Break up large log files

I am trying to break a large log file into smaller files based on date. The file is of the following form, where some lines may not have a date. Those lines should be included with the previous ...
2
votes
1answer
59 views

POSIX Awk sort array

I would like to sort an array with POSIX Awk, however I have discovered that: asort() asorti() PROCINFO["sorted_in"] are all GNU extensions. Can this be done with without GNU Awk?
9
votes
6answers
47k views

How to use a shell command to only show the first column and last column in a text file?

I need some help to figure out how to use the sed command to only show the first column and last column in a text file. Here is what I have so far for column 1: cat logfile | sed 's/\|/ /'|awk ...
0
votes
2answers
164 views

Printing all lines of a file with multiple columns in a shell script

I have a file with 4 columns with the below format: name;user_id;dob;date_of_joining I want to print all the variables in below output via shell script The employee "$name" is having "$user_id" ...
0
votes
1answer
47 views

Ternary operator and print

What is wrong with below command awk 'BEGIN {a = 5;b=6; (a==b)? print "a==b" : print "a!=b"}' Output : awk: BEGIN {a = 5;b=6; (a==b)? print "a==b" : print "a!=b"} awk: ...
0
votes
3answers
17 views

find muli lines in file from other file

i have text file that have multi lines i wanna search for lines containing some text and add # at the start of those lines ex: file1 that i wanna change it acnet 6801/tcp # ...
2
votes
1answer
77 views

Why do newer version of awk print large integers instead of decimals?

I often use the awk (gawk) calculator: calc(){ awk "BEGIN{ print $* }" ;} In Ubuntu 14.04, it output big numbers in standard form, e.g. $ calc 56777654409823*4674990588446667952594759939814064128 ...
0
votes
1answer
29 views

print only lines that starts with digit from specific column

I have a file which has second column as hostname of all hosts in my network: Host example.com (x.x.x.x) is up (0.0041s latency). And so on, but many lines doesn't start with hostname in second ...
3
votes
1answer
41 views

Combining every two column values in file

I have a file with (with only one line) with multiple columns say 4 (in this case) : A B C D And I want the output as every possible combination of column values taking 2 at a time like: AB AC AD ...
0
votes
1answer
44 views

How to make sed as fast as grep when replacing values

If I search with grep in a 30 gig database for some values it finds all of them in no time, if I run the same command with sed to replace those values it takes ages. Why is grep so much faster than ...
2
votes
1answer
45 views

How to convert a timezone in logs as they are being tailed

I would like to manipulate the format of a datestamp from a log, in realtime, as it is being tailed. The current format is 2016-04-06T23:19:20.878Z. I would like to convert it to our local timezone ...
1
vote
2answers
50 views

Define a variable to awk pattern matching from bash

I am trying to grab 100 lines after the text "time: X" with X in {0,40,80,...,200}. Here is what I have so far: #!/bin/bash start=1 end=5 for i in $(seq $start $end);do j=$(($i*40)) awk '/time: ...
0
votes
1answer
31 views

Extract two lines following a line with a specific value

I have a text file and I would like to extract the two lines of data following a line containing only "Name" as a value. Name 032 Express intent to engage in diplomatic cooperation (such ...
2
votes
1answer
45 views

Better way to run a series of seven identical commands in sed? [duplicate]

In the post Sed - Replace first k instances of a word in a file, a user suggests: sed 's/old/new/;s/old/new/;s/old/new/;s/old/new/;s/old/new/' filename as a method of replacing the first k ...
3
votes
2answers
104 views

Filter files generated by `find` by parsed output of `file` command

I'm writing a quick tool to inspect the contents of a node.js node_modules folder or python virtualenv for native dependencies. As a quick first approximation to this I wrote the following command. ...
2
votes
1answer
41 views

Bash script to summarize “ last -a ”

I would like to make a shell script such that it runs the " last -a " command and summarizes it's data in the form : userID : number of login sessions Host or ip1 - number of ...
-3
votes
1answer
34 views

How do i create a sed script to prompt the user to replace a number in a file? [duplicate]

I'm new in Unix, and I'm learning about sed command. I need to write a very small script for searching a number in a file and replacing it. I tried doing grep, and than deleting the specific phone ...
3
votes
1answer
54 views

A common way of combining two files in Solaris and Linux

I am trying to write a script that would combine two files into one and should work in both Solaris 5.10 and a Linux box. File1: dit710 dit710 dit720 dit720 File2: FacA D0000000000000001 FacA ...
1
vote
0answers
17 views

Find and redirect to capture pattern filename

I'm looking a way to do something like: find /path -name "201[2-6]*" | awk -F"/" '{print $5"/"$6"/"$7"/"$8",0,0,0"}' > archives_201\1.csv where archives_201\1.csv could be archives_201[2-6].csv ...
0
votes
1answer
47 views

Column comparison using awk [duplicate]

I would like to compare the first column of file1 with the second column of file2 and only if they match it should display any matching lines in file2 only as the output. The columns are separated by ...
1
vote
1answer
40 views

Split large log file into pieces based on date

I am trying to work out a Linux command to split a large log file into pieces based on date. Using ...
0
votes
3answers
33 views

How can I designate range within an column when making an awk file?

I have this data file which I want to sort out by making an awk file. I want to find the average rate within certain range of ages. For example, from 10<age<20and 20<age<30. And I can't ...
4
votes
4answers
14k views

Using sed/awk to remove anything after first space

aaaaaaaa 09 bbbbbbbb 90 ccccccccccccccc 89 ddddd 09 Using sed/awk/replace, in the above text I want to remove anything that comes after the first space in each line. For example the output ...
3
votes
3answers
97 views

Replace multiple lines with a string when the line numbers are stored in a file

I have a file with few million lines, all the same. just for an example: Known Known Known Known Known Known ... I have another file with few thousand line numbers, for an example: 3 5 6 ... I ...
1
vote
5answers
163 views

How to print nth line after match / empty lines

I'm trying to write a short script that finds empty lines and then prints the nth line after the empty lines. For foo1 foo2 foo3 bar1 bar2 bar3 spam1 spam2 spam3 eggs1 eggs2 eggs3 printing ...
0
votes
1answer
16 views

Separate values based on record

Using awk is it doable to separate values based on a certain record? Let's say we have following file: ACCJSH PREMIUM I5001 7 5976 7 @# 0.2 5195 5231 0.2 @# 39 5197 39 @# If we take the @# for ...
-1
votes
2answers
29 views

Need help in removing XML Tags with empty properties

Input File: <ServiceArea type="STATE" value="DC"> <DaylightSavingsUsed value="true"/> <FemtoSipProxyAddress value="10.13.123.100"/> <TermAuthAAAPSK ...
3
votes
3answers
127 views

Add mtime to grep -c output and sort the output by mtime

I have a directory full of logs named in the following style: info.log00001 info.log00002 info.log00003 ... info.log09999 info.log My current output (using grep -c) I need to analyze the ...
1
vote
2answers
61 views

Tar files starting with the same pattern

I have some log files named : 2016-02-10_03-52.log 2016-02-10_04-43.log 2016-02-10_02-13.log ... 2016-03-15_07-03.log 2016-03-15_09-08.log Basically the pattern is : YYYY-MM-DD_.log I would ...
0
votes
1answer
33 views

How can i remove duplicates from the output of tshark output [duplicate]

sudo tshark -i ppp0 'tcp port 80 \ and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' \ -R'http.request.method == "GET" && http.request.uri contains ...
0
votes
2answers
51 views

Help in making multiple lines as single lines

I have below 4 lines: EXTRACT ETEST Last Started 2016-03-20 20:56 Status RUNNING Checkpoint Lag 00:00:02 (updated 00:00:04 ago) EXTRACT PTEST Initialized 2016-03-31 06:25 Status ...
-1
votes
1answer
52 views

Organize a file [closed]

I want to thank the great help and support. Well, I have the following file: name id alello chr pos snp s1 215 a 1 11 a215 s1 216 a 1 11 a216 s1 217 ...
2
votes
2answers
42 views

How To Use Awk To Print Between Two Positional Parameters

Is there a way to print everything after a positional parameters using awk? What I would like the following command to print is "p4 p5 p6 p7" echo p1 p2 p3 p4 p5 p6 p7 | awk '{print $4...}' ...
5
votes
2answers
237 views

Case-insensitive search in awk

I need to search for a keyword using awk, but I want to perform a case-insensitive (non case sensitive) search. I think the best approach is to capitalize both the search term ("key word") and the ...
1
vote
2answers
44 views

Compare 1st column of the 1st file and 2nd column of the 2nd file and get the the output as following [duplicate]

File1: Excel file (.xls) UN ID St M1 M2 SE DOF PV PA FC 17127159 0 -5.9 297.3 765.7 0.22 4 0.003 0.00389231 2.57536 17127163 2 ...
0
votes
1answer
54 views

Compare 1st column of the 1st file and 2nd column of the 2nd file and matching

File1: Excel file (.xls) UN ID St M1 M2 SE DOF PV PA FC 17127159 0 -5.9 297.3 765.7 0.22 4 0.003 0.00389231 2.57536 17127163 2 ...
0
votes
1answer
39 views

How to compare (using greater than and less than symbols) strings using awk [duplicate]

This is my input file: PA43171-7,1,3,/vobs/atlas-idc/src/utils/logger/IDCLogger/IDC.cpp,48,12:01:16.086‌​.660, Valid Sensor Reading,Date = 08-19-2014 , Time = 12:01:16 Voltage Sensor=11 , V$ $=11 , ...
0
votes
2answers
86 views

how does this work in awk even without print?

bash-3.00# cat > b1.txt AAA,Apples,123 BBB,Bananas,124 CCC,Carrot,125 bash-3.00# cat > b2.txt Store1|AAA|123|11 Store2|BBB|124|23 Store3|CCC|125|57 Store4|DDD|126|38 bash-3.00# awk -F"[,|]" ...
3
votes
2answers
4k views

Merging 2 files with based on field match

I want to create a file that contains columns from two input files. File1 is like: aa 32 bb 15 cc 78 File2 is: fa 19 bc 23 cc 50 de 28 aa 45 bb 31 The task is, read through File1, if the 1st field ...
0
votes
2answers
93 views

Curl and Grep to Parsing Content

I need to parse the output of a curl command: curl --user test:test http://192.168.1.1/security/pkm.html | egrep '@company|config.pkm.password' This returns: <input type="text" ...