awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

4
votes
4answers
2k views

Split long output into two columns

Is there a simple utility or script to columnate the output from one of my scripts? I have data in some form: A aldkhasdfljhaf B klajsdfhalsdfh C salkjsdjkladdag D lseuiorlhisnflkc E sdjklfhnslkdfhn ...
3
votes
2answers
804 views

Split log file by time range

I have a log file and would like to cut it into copies of 15min. In the log file the lines start with: dd mmm yyyy hh:mm:ss,xxx e.g. 12 Feb 2013 16:05:02,xxx log text... and the file rotates ...
3
votes
3answers
525 views

awk floating point arithmetic - surprising result

I've been trying to get awk to do some trivial arithmetic, which involves carrying some values from one line to the next. Here is a minimal example pair, for comparison. The first example is expected ...
0
votes
4answers
59 views

How to remove line based on Delimeter in perl / Shell?

Can anyone help to remove the line in file based on delimeter(Comma), Incase if line contain less number of columns or Bad records , Need to delete those. Input File: a,b,c,d a,b,d,f c,d a,v,b,h ...
1
vote
1answer
81 views

string search using awk

I have file-1 which has list of transaction id . File -1 201503301616|9b8791b3-f860-409f-aad0-24debf834c1b|10.233.18.29.677220869928387637969 ...
0
votes
0answers
40 views

awk & grep to search for all strings in file1 on file2 where strings can have pipe [duplicate]

Please ignore - this post is a duplicate of this one. I have file-1 which has list of transaction id . File -1 ...
0
votes
2answers
56 views

Find and replace values with counters [closed]

Sorry my earlier question was bit confusing... I have a file that has values like: location 100 SlotNumber ..some lines inbetween location 150 SlotNumber ..some lines inbetween location 160 ...
3
votes
1answer
41 views

Extract first word after pattern that includes `[…]`

I have an issue when I try to extract the first word after grep matches my pattern . Let me show you an example : cat test.txt [profile dev] xxx xxx [profile prod] xxx xxx now I want to extract ...
2
votes
4answers
99 views

How to highlight given string in given place?

How can we modify this: <a href="http://foo.bar1">asfdlksafbar1qsasadf</a><br> <a href="http://foo.bar2">svasfbar2saldkfj</a><br> <a ...
0
votes
3answers
79 views

awk to remove string from a variable

HP-UX *** B.11.31 U ia64 ***unlimited-user license I want to move a file in the current directory called abc.txt to a directory called ./begperl. While I do so I want the moved version of the file to ...
1
vote
5answers
2k views

How can I output the difference between 2 files?

My file consists of the the following; roughly: username:username:username:username:username The above line continues to about 600 characters. I use the awk command in order to use it as an ...
63
votes
2answers
111k views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
0
votes
2answers
32 views

Select lines and format

Is there a way to format the following output so that only lines 1 and the respective lines that state the "rule" name print out? Depending on the configuation, the "rule" line could vary. It could be ...
1
vote
2answers
52 views

Script to hang up Asterisk channels

I am trying to create a script for Asterisk. We use a lot of channspy. Sometimes the channspy stays opened and didn't hangup even if the softphone is hangup (I don't understand why). So I thought of ...
0
votes
2answers
40 views

list the “middle” part of an input in the end of the output in awk

This is regarding a hostfile. INPUT: 1.2.3.4 foooo00001 foo001 # AIX bar0003 NOSLS NOSLS NOCOMMENT 2.3.4.1 foooo00002 foo002 fooo0002 # Linux bar0003 NOSLS NOSLS NOCOMMENT OUTPUT: echo "$INPUT" ...
4
votes
3answers
157 views

Get last part of string after hyphen

Is there a simple command line to extract the last part of a string separated by hyphens? E.g., I want to extract 123 from foo-bar-123.
5
votes
6answers
5k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" ...
2
votes
1answer
51 views

How to select lines and reformat command output?

Is there a way to format the following output so that only lines 1 and 4 print out? It would be best if the first line and fourth line could come out on the same line like this: sw pool test(no ...
1
vote
5answers
59 views

keep out line break

I have some files with this structure: 2015-03-25 17:08:17 sysUpTimeInstance 93474;^M 1.ValueforState=2500 I want to replace the line break, and leave the third line with the second line, I mean, ...
0
votes
2answers
98 views

How to delete specific lines within a certain area?

I want to delete some specific lines in the YAML front matter: --- Author: aaaa Title: bbbb Time: cccc Place: dddd ... Lets say I wanted to delete only the lines in the front matter that contained ...
6
votes
2answers
12k views

Join two files with matching columns

File1.txt id No gi|371443199|gb|JH556661.1| 7907290 gi|371443198|gb|JH556662.1| 7573913 gi|371443197|gb|JH556663.1| 7384412 gi|371440577|gb|JH559283.1| ...
-2
votes
1answer
48 views

how to print new word after two lines using awk

I need to insert a blank line after two lines in a file that i did with cat rat hat pot NOT lot big big big sit sit sit awk '{print};(NR%2==0){print ""}' cat rat hat pot NOT ...
1
vote
1answer
52 views

sort and uniq in awk

I know there are "sort" and "uniq" out there, however, today's question is about how to utilise AWK to do that kind of a job. Say if I have a list of anything really (ips, names, or numbers) and I ...
1
vote
1answer
35 views

awk script to rearrange similar rows

I want rearrange about 5 million rows (with 300 columns) into groups. Data looks like the following: where there were various experiments (column 2) conducted at different locations (column headers ...
0
votes
4answers
42 views

AWK print regex pattern

I am trying to print a regex pattern for the following piece of mail log. In particular I am trying to get the ID between the square brackets (see the second line for reference). Mar 29 03:48:13 ...
8
votes
5answers
5k views

How to remove multiple blank lines from a file?

I have some text-files I use to take notes in - just plain text, usually just using cat >> file. Occasionally I use a blank line or two (just return - the new-line character) to specify a new ...
3
votes
4answers
7k views

Compare two text files and find matching lines

I have two files A and B. A looks like this (4 to 6 lines): GAGA CAGA GGGT TATT file B is a really big file with thousands of lines, here is a short example: AAATGTCAAGAGACAGAAATGTCAAGAGGGT ...
0
votes
3answers
39 views

How to display only the last line of a listing ?

Supposed I make a listing and sort the files by its temporal attribute: ls -ltr -rwxrwxrwx 1 bla bla 4096 Feb 01 20:10 foo1 -rwxrwxrwx 1 bla bla 4096 Feb 01 20:12 foo2 . . . -rwxrwxrwx 1 bla bla ...
0
votes
2answers
63 views

Print awk's current new line character

How can I find what new line character is in my file when using awk? I know the default is \n but this is what manual says. I want to see it with my own eyes. I have just started learning awk and ...
1
vote
1answer
48 views

Write text on last line of document [duplicate]

If I have this document aaa bbb ccc And I want to add " ddd" on the last line. My document should look like this: aaa bbb ccc ddd How can I do this? If I prefer awk or sed but ...
1
vote
4answers
120 views

bash division of numbers in text file

I have two numbers in test.txt file, I would like a way to divide the second number by the first number and assign the result to a variable or file in a bash script that I can use. I have used awk ...
6
votes
5answers
223 views

Using awk to identify the number identical columns

I am new to all this, and this is probably a rather simple question, but I am stuck: I have a large number of individual files that contain six columns each (number of rows can vary). As a simple ...
2
votes
2answers
88 views

How to print from a given range of ip addresses every class-c with the last octect replaced by an asterisk

example: I have a range from 10.0.100.0 to 10.0.224.0 and I want the output like this: 10.0.100.* 10.0.101.* 10.0.102.* and so on
3
votes
2answers
148 views

Nested 'awk' in a 'while' loop, parse two files line by line and compare column values

I need some help with a combination of awk & while loop. I have two simple files with columns (normal ones are very large), one representing simple intervals for an ID=10(of coding ...
10
votes
3answers
6k views

How to sort the string which combined with string + numeric using bash script?

This is the data what I want to sort. But sort treats the numeric to string, the data it no sorted as I expected. /home/files/profile1 /home/files/profile10 /home/files/profile11 ...
-1
votes
1answer
32 views

AWK and Bash Scripting [duplicate]

How to use a value of a variable in awk? Something like this: filename = "test.txt" ls -l | awk '{ if ($9 == filename) print("File exists")}' I can't use $ in awk to access the value of ...
5
votes
1answer
58 views

How to use ^#$ as record separator in awk?

How do you tell awk to use a # character by itself in a line as record separator? The problem is you can't say RS="^#$" because ^ matches the beginning of the file, not the beginning of a line, and ...
0
votes
2answers
78 views

sed command to replace a string from one file with entire contents of another file [duplicate]

I want to write a sed (or awk) command to replace a string from one file with the entire contents of another file. Note that second file from which I want to get the content has more than one line. I ...
1
vote
1answer
63 views

awk with variables in condition and in output redirection file

I would please like some help with this command because I didn't find anything in documentation that can cover everything I want. I have some variables that are global, so I would prefer to keep them ...
4
votes
3answers
146 views

sum columns based on matching fields

I have encountered a problem that exceeds my basic unix knowledge and would really appreciate some help. I have a large file in the following format: 2 1019 0 12 2 1019 3 0 2 1021 0 2 ...
1
vote
1answer
56 views

awk pattern with variable in a bash script

I have problem with command awk in bash I want to find pattern with variable, but it doesn't work. Can you tell me what's wrong with this line? awk -F" " "/$PWD/ {print $1,$3}" file.txt
1
vote
3answers
137 views

how to delete line from the XML file

how to delete only the line from xml file ( with sed/awk or perl one liner line ) that start with: <units> and ended with </Networks> as the follwoing <units><unit ip= ...
1
vote
1answer
19 views

Awk - matching on 2 columns for differents lines

Given this file: 92157768877;Sof_deme_Fort_Email_am_%yyyy%%mm%%dd%;EMAIL;20/02/2015;1;0;0 92157768877;Sof_trav_Fort_Email_am_%yyyy%%mm%%dd%;EMAIL;20/02/2015;1;0;0 ...
1
vote
2answers
54 views

How to read only the second last line of a file

I have a client that updates/writes into myfile.csv arbitrarily. I've written the following code, I left the MySQL snippet out: while true do awk_variables_value_array=`tail -n2 myfile.csv | awk ...
1
vote
1answer
27 views

awk adds extra comma at several places

i want to add a timestamp with awk to a CSV-file. To generate the timestamp works, but the result CSV-file have some extra comma at several places. How can I fix the problem? origin-CSV (1st line): ...
1
vote
1answer
104 views

awk directory of files

Given a directory full of .sam files, for each file: if column3=blah and 451000 =< column4 <= 468999, write line to file_ribos.sam otherwise, write line to file_non_ribos.sam Example Input ...
1
vote
1answer
61 views

Awk sum csv columns based on fields

I have N files like so: file1.txt Header1,Header2,Header3,Header4,Header5 A,B,RANDOM,1,2 C,D,RANDOM,3,4 fileN.txt Header1,Header2,Header3,Header4,Header5 A,B,RANDOM,1,2 C,D,RANDOM,3,4 They all ...
2
votes
5answers
180 views

How to reverse the lines between a pattern into another file

I have the following in the input file #Start para 0 hello dq 1 world dq 2 welcomes dq 3 you dq #Start para 0 how tq 1 are tq 2 you tq #Start para 0 say dq 1 hello dq 2 to dq 3 the dq 4 world dq I ...
2
votes
2answers
34 views

How to Save variables in a script that can be shared between two runs of awk against the same input file in the script? [closed]

I have a log file, whose formatting is very consistent and which I control. It produces pipe-delimited fields of constant length. Two fields are relevant to the filtering process I wish to create, ...
1
vote
2answers
39 views

How do I update the dates (by 30 days) in the files going to that specific location in unix?

How do I update the dates (by 30 days) in the files going to that specific location in unix? Can awk or sed help ? File format is as below - Thomson: xyz Format: 1 Character_set: ASCII8 File_type: ...