awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

4
votes
3answers
150 views

Remove duplicate lines while keeping the order of the lines

[root@server]# awk '!seen[$0]++' out.txt > cleaned awk: (FILENAME=out.txt FNR=8547098) fatal error: internal error Aborted [root@server]# The ""server"" has: 8 GByte RAM + 16 GByte SWAP, x>300 ...
0
votes
2answers
88 views

Awk one-liner to print column of SQL table?

Suppose I have table like: This is first line name minimum maximum config_value ----------------------------------- ----------- ----------- ------------ some ...
1
vote
1answer
129 views

Output different lines when comparing two files

I have two text files with the same number of newline terminated strings. The lines in the two files correspond to each other. The lines in each file individually might be repeated. I would like to ...
3
votes
3answers
677 views

awk floating point arithmetic - surprising result

I've been trying to get awk to do some trivial arithmetic, which involves carrying some values from one line to the next. Here is a minimal example pair, for comparison. The first example is expected ...
1
vote
4answers
96 views

How to remove line based on Delimeter in perl / Shell?

Can anyone help to remove the line in file based on delimeter(Comma), Incase if line contain less number of columns or Bad records , Need to delete those. Input File: a,b,c,d a,b,d,f c,d a,v,b,h ...
3
votes
4answers
258 views

How do I extract the first integer from text string in a column of a tab-delimited file?

I work in Medical Genetics and often have delimited text files where in one column (ex. column 5) there is a text string with a "mutation" in our jargon: c.2458C>T or c.45_46delAA or ...
0
votes
0answers
40 views

awk & grep to search for all strings in file1 on file2 where strings can have pipe [duplicate]

Please ignore - this post is a duplicate of this one. I have file-1 which has list of transaction id . File -1 ...
1
vote
1answer
105 views

string search using awk

I have file-1 which has list of transaction id . File -1 201503301616|9b8791b3-f860-409f-aad0-24debf834c1b|10.233.18.29.677220869928387637969 ...
0
votes
2answers
76 views

Find and replace values with counters [closed]

Sorry my earlier question was bit confusing... I have a file that has values like: location 100 SlotNumber ..some lines inbetween location 150 SlotNumber ..some lines inbetween location 160 ...
0
votes
3answers
129 views

awk to remove string from a variable

HP-UX *** B.11.31 U ia64 ***unlimited-user license I want to move a file in the current directory called abc.txt to a directory called ./begperl. While I do so I want the moved version of the file to ...
3
votes
1answer
146 views

Extract first word after pattern that includes `[…]`

I have an issue when I try to extract the first word after grep matches my pattern . Let me show you an example : cat test.txt [profile dev] xxx xxx [profile prod] xxx xxx now I want to extract ...
0
votes
2answers
38 views

Select lines and format

Is there a way to format the following output so that only lines 1 and the respective lines that state the "rule" name print out? Depending on the configuation, the "rule" line could vary. It could be ...
1
vote
2answers
49 views

list the “middle” part of an input in the end of the output in awk

This is regarding a hostfile. INPUT: 1.2.3.4 foooo00001 foo001 # AIX bar0003 NOSLS NOSLS NOCOMMENT 2.3.4.1 foooo00002 foo002 fooo0002 # Linux bar0003 NOSLS NOSLS NOCOMMENT OUTPUT: echo "$INPUT" ...
2
votes
2answers
128 views

Script to hang up Asterisk channels

I am trying to create a script for Asterisk. We use a lot of channspy. Sometimes the channspy stays opened and didn't hangup even if the softphone is hangup (I don't understand why). So I thought of ...
4
votes
3answers
262 views

Get last part of string after hyphen

Is there a simple command line to extract the last part of a string separated by hyphens? E.g., I want to extract 123 from foo-bar-123.
2
votes
1answer
57 views

How to select lines and reformat command output?

Is there a way to format the following output so that only lines 1 and 4 print out? It would be best if the first line and fourth line could come out on the same line like this: sw pool test(no ...
1
vote
5answers
72 views

keep out line break

I have some files with this structure: 2015-03-25 17:08:17 sysUpTimeInstance 93474;^M 1.ValueforState=2500 I want to replace the line break, and leave the third line with the second line, I mean, ...
-2
votes
1answer
97 views

how to print new word after two lines using awk

I need to insert a blank line after two lines in a file that i did with cat rat hat pot NOT lot big big big sit sit sit awk '{print};(NR%2==0){print ""}' cat rat hat pot NOT ...
1
vote
1answer
258 views

sort and uniq in awk

I know there are "sort" and "uniq" out there, however, today's question is about how to utilise AWK to do that kind of a job. Say if I have a list of anything really (ips, names, or numbers) and I ...
1
vote
1answer
68 views

awk script to rearrange similar rows

I want rearrange about 5 million rows (with 300 columns) into groups. Data looks like the following: where there were various experiments (column 2) conducted at different locations (column headers ...
0
votes
4answers
71 views

AWK print regex pattern

I am trying to print a regex pattern for the following piece of mail log. In particular I am trying to get the ID between the square brackets (see the second line for reference). Mar 29 03:48:13 ...
1
vote
3answers
53 views

How to display only the last line of a listing ?

Supposed I make a listing and sort the files by its temporal attribute: ls -ltr -rwxrwxrwx 1 bla bla 4096 Feb 01 20:10 foo1 -rwxrwxrwx 1 bla bla 4096 Feb 01 20:12 foo2 . . . -rwxrwxrwx 1 bla bla ...
0
votes
2answers
74 views

Print awk's current new line character

How can I find what new line character is in my file when using awk? I know the default is \n but this is what manual says. I want to see it with my own eyes. I have just started learning awk and ...
1
vote
1answer
58 views

Write text on last line of document [duplicate]

If I have this document aaa bbb ccc And I want to add " ddd" on the last line. My document should look like this: aaa bbb ccc ddd How can I do this? If I prefer awk or sed but ...
2
votes
2answers
124 views

How to print from a given range of ip addresses every class-c with the last octect replaced by an asterisk

example: I have a range from 10.0.100.0 to 10.0.224.0 and I want the output like this: 10.0.100.* 10.0.101.* 10.0.102.* and so on
-1
votes
1answer
79 views

AWK and Bash Scripting [duplicate]

How to use a value of a variable in awk? Something like this: filename = "test.txt" ls -l | awk '{ if ($9 == filename) print("File exists")}' I can't use $ in awk to access the value of ...
5
votes
1answer
114 views

How to use ^#$ as record separator in awk?

How do you tell awk to use a # character by itself in a line as record separator? The problem is you can't say RS="^#$" because ^ matches the beginning of the file, not the beginning of a line, and ...
0
votes
2answers
119 views

How to delete specific lines within a certain area?

I want to delete some specific lines in the YAML front matter: --- Author: aaaa Title: bbbb Time: cccc Place: dddd ... Lets say I wanted to delete only the lines in the front matter that contained ...
4
votes
2answers
473 views

Nested 'awk' in a 'while' loop, parse two files line by line and compare column values

I need some help with a combination of awk & while loop. I have two simple files with columns (normal ones are very large), one representing simple intervals for an ID=10(of coding ...
0
votes
2answers
298 views

sed command to replace a string from one file with entire contents of another file [duplicate]

I want to write a sed (or awk) command to replace a string from one file with the entire contents of another file. Note that second file from which I want to get the content has more than one line. I ...
1
vote
1answer
141 views

awk with variables in condition and in output redirection file

I would please like some help with this command because I didn't find anything in documentation that can cover everything I want. I have some variables that are global, so I would prefer to keep them ...
6
votes
3answers
446 views

sum columns based on matching fields

I have encountered a problem that exceeds my basic unix knowledge and would really appreciate some help. I have a large file in the following format: 2 1019 0 12 2 1019 3 0 2 1021 0 2 ...
1
vote
1answer
100 views

awk pattern with variable in a bash script

I have problem with command awk in bash I want to find pattern with variable, but it doesn't work. Can you tell me what's wrong with this line? awk -F" " "/$PWD/ {print $1,$3}" file.txt
2
votes
3answers
265 views

how to delete line from the XML file

how to delete only the line from xml file ( with sed/awk or perl one liner line ) that start with: <units> and ended with </Networks> as the follwoing <units><unit ip= ...
1
vote
4answers
192 views

bash division of numbers in text file

I have two numbers in test.txt file, I would like a way to divide the second number by the first number and assign the result to a variable or file in a bash script that I can use. I have used awk ...
1
vote
2answers
140 views

How to read only the second last line of a file

I have a client that updates/writes into myfile.csv arbitrarily. I've written the following code, I left the MySQL snippet out: while true do awk_variables_value_array=`tail -n2 myfile.csv | awk ...
1
vote
1answer
34 views

awk adds extra comma at several places

i want to add a timestamp with awk to a CSV-file. To generate the timestamp works, but the result CSV-file have some extra comma at several places. How can I fix the problem? origin-CSV (1st line): ...
1
vote
1answer
132 views

awk directory of files

Given a directory full of .sam files, for each file: if column3=blah and 451000 =< column4 <= 468999, write line to file_ribos.sam otherwise, write line to file_non_ribos.sam Example Input ...
1
vote
1answer
31 views

Awk - matching on 2 columns for differents lines

Given this file: 92157768877;Sof_deme_Fort_Email_am_%yyyy%%mm%%dd%;EMAIL;20/02/2015;1;0;0 92157768877;Sof_trav_Fort_Email_am_%yyyy%%mm%%dd%;EMAIL;20/02/2015;1;0;0 ...
2
votes
1answer
218 views

Awk sum csv columns based on fields

I have N files like so: file1.txt Header1,Header2,Header3,Header4,Header5 A,B,RANDOM,1,2 C,D,RANDOM,3,4 fileN.txt Header1,Header2,Header3,Header4,Header5 A,B,RANDOM,1,2 C,D,RANDOM,3,4 They all ...
2
votes
5answers
190 views

How to reverse the lines between a pattern into another file

I have the following in the input file #Start para 0 hello dq 1 world dq 2 welcomes dq 3 you dq #Start para 0 how tq 1 are tq 2 you tq #Start para 0 say dq 1 hello dq 2 to dq 3 the dq 4 world dq I ...
1
vote
2answers
45 views

How do I update the dates (by 30 days) in the files going to that specific location in unix?

How do I update the dates (by 30 days) in the files going to that specific location in unix? Can awk or sed help ? File format is as below - Thomson: xyz Format: 1 Character_set: ASCII8 File_type: ...
0
votes
1answer
142 views

Using “case” with awk

Some time ago, I used this part of my awk script: case "AlarmLevel.0" : alarmAlertLevel=$2 break But now, the version of information has changed, and the level is dynamic, I mean now, the ...
3
votes
2answers
40 views

How to Save variables in a script that can be shared between two runs of awk against the same input file in the script? [closed]

I have a log file, whose formatting is very consistent and which I control. It produces pipe-delimited fields of constant length. Two fields are relevant to the filtering process I wish to create, ...
3
votes
3answers
301 views

Splitting a sequence into fixed width

I have a file like this which is a two-column tab-separated file. CTGCAGTTTCCCCAAATGTGGGAAACTTGACTGTATAATTTGTGGCAGTGGTA a1 GATTTCCCCAAATGTGGGAAACTCACTCGGCAGGCGTTGATA a2 I want to get an output ...
1
vote
3answers
58 views

How to do find and replace in one field based on the value in the other field?

I have a file like this. IMS00|20140101|Unix|Jan14 IMS01|20140201|Unix|Feb14 IMS02|20140301|Unix|Mar14 IMS03|20140101|Unix|Jan14 IMS04|20140201|Unix|Feb14 My criteria are, if the column 1 value is ...
2
votes
1answer
32 views

AWK get first line

I'm trying to add a column for each line together, and I do it like this: awk '{s+=$2} END {print s}' Now, when I have the final sum, s, I want to subtract the second column of the first line, like ...
0
votes
1answer
203 views

/usr/bin/awk: Argument list too long

I am trying to run the following bash script: #!/bin/bash file=$1 filename=${file%%.*} line1=$(sed -n 1~2p ${file}) seqs=$(grep -v '^>' ${file}) pos=$(echo "${line1}" | awk -F"[__]" ...
3
votes
1answer
116 views

How to add number to awk array elements?

If I have the following bash variable: $ echo "${pos}" 201 719 744 205 354 ...the following produces... !#bin/bash ( IFS=: awk -v str2="$pos" -v sep="[$IFS]" ' BEGIN { ...
-3
votes
1answer
83 views

Using AWK to define variables in a shell

I've got a live MQTT stream being filtered out and dumped into a CSV file. I use the fourth parameter ($4) and fifth parameter ($5) in the CSV file to locate the information I want. I give this ...