awk - pattern-directed scanning and processing language
2
votes
3answers
164 views
Bash commands/script to remove a line from CSV with duplicate in column
I have a lot CSV files that I have combined. However, there are duplicates, but the entire line is not duplicated. I do have a column that I want to use as the criteria to search for a duplicate. ...
3
votes
1answer
41 views
How to edit files in-place while performing operation on the values matching the searched pattern?
Given a list files containing random text and many datetimes in ISO format in it (e.g "2012-07-02T10:47:24+02:00"), how can I find all these datetimes and add 2 hours to each one ?
Ideally a solution ...
12
votes
9answers
1k views
There must be a better way to replace single newlines only?
I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
4
votes
3answers
200 views
Print a line only if the next line does NOT contain a particular match
I am trying to search a log file for logged activities that did not complete. For example, I log a "Starting activity for ID 1234..." and if successful, the next line will be "Activity 1234 ...
4
votes
4answers
668 views
Text between two tags
I wanna retrieve whatever is between these two tags – <tr> </tr> – from an html doc.
Now I don't have any specific html requirements that would warrant for an html parser. I just plain ...
11
votes
1answer
1k views
Making a progressbar with “dialog” from rsync output
I'm looking for a way to filter/redirect rsync output in a manner where it can be fed to the "dialog --gauge" command, so I can get a nice looking progressbar during file sync.
Currently I have only ...
3
votes
4answers
102 views
Replacing pattern after nth match is found on each line?
I have a file containing lines:
india;austria;japan;chile
china;US;nigeria;mexico;russia
I want to replace all the occurences of semicolon on each line with e.g. ;NEW;, but starting from the 2nd ...
5
votes
7answers
4k views
How to print only last column?
echo -e 'one two three\nfour five six\nseven eight nine'
one two three
four five six
seven eight nine
how can I do some "MAGIC" do get this output?:
three
six
nine
UPDATE:
I don't need it in this ...
0
votes
1answer
152 views
How can I find and filter a specific column in a .csv file? [duplicate]
I have .csv files with the following structure:
cat,dog,mouse,pig,bird,cow,...
21,34,54,566,78,1,...
23,10,12,569,56,4,...
32,20,13,123,56,3,...
34,30,44,322,66,2,...
I want to filter the column ...
0
votes
0answers
47 views
how to parse this data and count the matching patterns? [closed]
Gi1/0/12
Gi1/0/13
Gi1/0/14
Gi1/0/15
Gi1/0/16
Gi1/0/17
Gi1/0/18
Gi1/0/19
Gi1/0/20
Gi1/0/21
Gi1/0/22
Gi1/0/23
Fa2/0/13
Fa2/0/14
Fa2/0/15
Fa2/0/16
Fa2/0/17
Fa2/0/18
Fa2/0/19
Fa2/0/20
Fa2/0/21
Fa2/0/22
...
1
vote
1answer
72 views
How to parse this output and put it into an array?
i have expect script whose output looks like :
I would like to extract all words like Gi/02/12 etc in an array or a list.
VLAN Name Status Ports
---- ...
1
vote
2answers
64 views
Match ids and print original file
I have two files
Original: (5000 entries)
Chr Position
chr1 879108
chr1 881918
chr1 896874
…
and a file with allele frequencies (2000 entries)
Chr Position MAF
chr1 881918 0.007
...
1
vote
3answers
84 views
Is it mandatory to have a variable in awk's first statement?
I am having some hard time with awk today. If I try to do:
df|awk '{print $2; $some=$2; print $some}'
It works as expected and I get the size of the disks twice but if I do:
df|awk '{$some=$2; ...
2
votes
2answers
114 views
Extract Data between comma
I have an output file which contains different numbers seperated by commas. But the issue is I do not know the number of commas.
I would in turn like this data to be grep on another file. For ...
2
votes
2answers
101 views
changing pattern by removing spaces?
file :
hi hello,new york, , ,brazil site ,brazil
january,month is feb , , ,indiana jones
task:
delete all horizontal space (spaces and tab) between any two comas (like , ,) if ...
1
vote
1answer
262 views
AWK how to add all columns beginning with specific row
I have this output:
Count Process
238 0
1 1124027
1 1124065
1 1124066
1 1124069
1 1124072
1 1124073
1 1547365
1 1547366
I wanted to add all Count ...
2
votes
1answer
123 views
shell variable in awk is not being passed to all the lines but just for the very first line of input?
inputfile:is2.txt
10.39.5.41,A1,B1
10.39.5.41,A2,B2
10.39.5.41,A3,B3
10.39.5.41,A4,B4
10.39.5.41,A5,B5
10.39.5.41,A6,B6
script :
#!/bin/bash
second_column="OOOOOOO" # OOOOOOO will be added to ...
7
votes
5answers
2k views
Remove comma between the quotes only in a comma delimited file
I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row
123,"ABC, DEV 23",345,534.202,NAME
I need to ...
1
vote
4answers
67 views
How to find what device a file is on (and use that in a script)?
I want to find out what device my file is on so that I can use it in a script. I can get this far:
$ df .
Filesystem 512-blocks Used Available Capacity Mounted on
/dev/disk0s2 498438976 ...
1
vote
2answers
107 views
sed pattern matching with deletion over a line
I want to delete only some spaces(say 4) after the line$#*425 with space
I am not able to complete it please suggest
echo "line$#*425 with space " | sed 's/ /\1 \2/g'
i am thinking of ...
3
votes
2answers
152 views
AWK for data smoothing
I need some help to smooth some data with awk olny of column 4 and next.
This is a data example:
Date;time;Time_ms;A;B;C;D
23.11.2012;15:03:00;41236627696;1;2;2;3
...
1
vote
2answers
81 views
Print fields from file only if line starts with a specific word
I have the following file:
…
LINK=dummy 172.17.100 => 10.218.11 [PATH=/etc]
FILE=hosts 172.17.100 => 10.218.11 [PATH=/etc]
FILE=network 172.17.100 => 10.218.11 ...
1
vote
0answers
137 views
Processing shell script options with awk
I'm looking for way to process shell script arguments that is cleaner and more "self documenting" than getopt/getopts.
It would need to provide...
Full support of long options with or without a ...
1
vote
0answers
107 views
How to awk printf additional string with backslash
I have a csv that I want to add a preceding column to and a trailing column. The trailing column is a date (manual string right now). Every time I try to get that date, it ends up replacing some text ...
1
vote
1answer
233 views
How to calculate average requests per seconds using awk or sed on Apache logs
I am trying to calculate the average requests per seconds for a specific period of time of my Apache logs.
I have been able to narrow the results down to the period I am interested in, as well as ...
2
votes
2answers
213 views
System calls, AWK, and Bringing in external inputs
awk '{ TEMPVAR="/usr/bin"; printf("%s", system("ls $TEMPVAR")); }' empty
In this example I'm trying to bring in the variable TEMPVAR into the system call. How would I do this?
What I'm aiming to ...
3
votes
2answers
121 views
AWK : Converting Row Formated Data into Columnar
I have a dataset as such:
Name: Jim Bean
Vice: Dice
ID: AFDSDFDSFDSFASFA
LoginTime: 12343314
Name: Bob Dylon
Vice: Trumpets
ID: AFD232SFDSFASFA
LoginTime: 12343314
...
5
votes
2answers
173 views
what is the meaning of 1 at the end of awk script
I was reading this awk script
awk -F"=" '{OFS="=";gsub(",",";",$2)}1'
I want to know what is the function of 1 at the end of it
2
votes
1answer
251 views
How can I use awk to (sometimes) remove the first character of a column?
I run a command and pipe it through awk as below :
mycommand | awk '{print $1,tolower($3),$4}
This gives me the following output:
fred.o: t .abcdefg
fred.o: u .rstuvwxy
fred.o: t .defghi
...
0
votes
0answers
58 views
Co Record Verification
I have a task requiring some awk to verify a few records quickly.
Let' say:
A1,A2
B1,B2
C1,C2
C2,C1
A1,C1
A1,B1
B1,A1
Which would be the best way to check for reciprocity only between the A# the ...
2
votes
2answers
57 views
sort a file based on length of the column/row
I need to sort a file based on the number of chars in the first column.
I have no idea on how to go about this. (On Linux, so sed/awk/sort is available).
An example:
.abs is bla bla 12
.abc is bla ...
2
votes
1answer
144 views
Parse csv using awk [duplicate]
Possible Duplicate:
Is there a robust command line tool for processing csv files?
I am trying to parse the second line in a file (which contains commas within double quotes)
"100 Watt ...
1
vote
0answers
128 views
Conky - string formatting
I have conky the record:
${color lightblue} Down: ${downspeed wlan0} Up: ${upspeed wlan0}
Conky prints link speed, but the text moves:
Down: OB Up: OB
Down: 60B Up: 60B
Down: 148B Up: 148B
...
3
votes
3answers
151 views
How can awk cut certain fields and add to the end of each line?
I would like to convert a list containing bunch of svn URLs of issues:
cat list.txt
//svn.server.address/repos/project/module1/branches/issue-001-name1
...
1
vote
2answers
275 views
Parsing string by awk and get only elements without pipes or semi colons
I would like get this string line
AUGUSTYN|Stanisław|3589238
without | and :
I tried something like that:
cat baza|grep "AUGUSTYN" -n|awk -F '|' '{print $1,$2,$3,$4}'`
(baza is my file with ...
3
votes
1answer
83 views
No output from inotifywait | awk
I'm attempting to use part of a one-liner found here: Script to monitor folder for new files?
When I try the following procedure I get no output whatsoever and I cannot figure out why.
In terminal ...
1
vote
2answers
262 views
Help inserting a new line of text after matching a line of text (sed)?
I need to insert a hidden HTML input tag into any form tag within a bunch of HTML files. I assume this is possible with sed, but need help forming the command.
My idea is to search for any instance ...
2
votes
2answers
293 views
Data pivot with awk
I'm trying to pivot a file using awk. This is an example of my input File:
VarName;TimeString;VarValue;Validity;Time_ms
A;23.11.201215:03:53;1;1;41236627696,7593
...
2
votes
2answers
93 views
Awk: expanding first field along the column
I have a task which requires some awk scripting:
Lets say I have a file:
A,under1
,under2
,under3
,under4
,under5
B,new1
,new2
,new3
C,sm1
,sm2
How could I make it look like this using awk?
...
0
votes
2answers
98 views
Errors when searching for special characters in awk
How can I search for a pattern in awk if the text contains special characters?
billinAddress ||
when I search, it is showing an error on pipe sign ||. How can I search for the pipe sign || in awk?
...
1
vote
1answer
266 views
Validate Unix Timestamp using BASH
My google-fu has failed me so I'm hoping someone here can point me in the right direction. I'm in the middle of doing some data validation efforts and I suspect the system is recording some invalid ...
3
votes
4answers
920 views
removing the first and the last character of every line from command line
I am trying to remove the first and the last characters of everyline in a text file and save the resulting truncated version in a new file. Does anyone have an idea about how to do that efficiently ...
0
votes
2answers
369 views
Replace matches with multiline string using sed [duplicate]
Possible Duplicate:
Insert a multiline string into another string
I have a file called foo.txt containing:
foo
bar
baz
I want to replace every occurence of bar with a multiline string, ...
1
vote
3answers
260 views
linux/solaris + verify duplicate valid IP address from file
what the best way to find duplicate IP from file ( I have ksh script in this script I need to write function that check for duplicate IP )
for example if IP - 192.1926.23.52 exists twice in file - ...
2
votes
3answers
268 views
Awk to output read file
I havent stepped into awk yet but it is on my plate for learning this coming year. I asked a previous question that led me to awk and it works. However, I am unsure how to input all file types and ...
6
votes
3answers
1k views
AWK: wrap lines to 72 characters
$ awk 'length > 72' {HOW TO PRINT THE LINEs IN PCS?} msg
ie I want it to add \n after 72 chars and continue, so initially you may need to remove all single \ns and the add them. It may be easier ...
2
votes
1answer
371 views
Bash variable in Awk script
I have this flat file database(ff_servers.db) with following contents:
192.168.154.2 Alaska hp
192.168.157.3 Colorado dell
192.168.156.3 hawaii hp
From command line, I could run:
awk ...
3
votes
1answer
80 views
Are there awk versions that provide syntax for computing aggregations?
From time to time I find myself writing awk scripts that compute some simple statistics. For example computing a histogram, the average of a value, the standard deviation or even the variance ...
...
1
vote
1answer
154 views
Sed and Awk learning material [closed]
I'm looking for some learning material for sed and awk, can anyone make any suggestions?
Please don’t refer to man and help pages.
-O-
2
votes
1answer
145 views
Using awk, how to get desired output from multiple file sources
I have two files:
file1:
A,D
B,E
C,F
file2:
G,H,I
J,K,L
Using awk, I want to print:
A,D
B,E,I
C,F,L
How can I use awk to achieve this?
