awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

2
votes
4answers
1k views

Compare two text files and find matching lines

I have two files A and B. A looks like this (4 to 6 lines): GAGA CAGA GGGT TATT file B is a really big file with thousands of lines, here is a short example: AAATGTCAAGAGACAGAAATGTCAAGAGGGT ...
1
vote
1answer
251 views

syntax error in conditional expression while executing the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
3
votes
2answers
213 views

Only print rows that have 2 or more fields?

I've got a text file and I'm trying to figure out how to filter out any lines with data in only 1 field, leaving lines with 2 fields. I came across the following code: awk '$NF > 1' but I can't ...
0
votes
2answers
70 views

search for first pattern and copy the string inside the ( ) into the 3rd field in lines where the second pattern found

Here are my spec: first search for "#firstpattern(" copy the content inside the ( ) to the 3rd field inside the ( ) of all lines where the "secondpattern" found. remove the entire line with the ...
6
votes
2answers
638 views

Counting number of times each IP address appears in log file

I have a file in the format as follows: $ cat file.txt 27.33.65.2 27.33.65.2 58.161.137.7 121.50.198.5 184.173.187.1 184.173.187.1 184.173.187.1 What's the best way to parse the file file.txt into ...
2
votes
1answer
192 views

solaris awk problems with date and if statements

I have daily files that come in via FTP with incorrect dates in the first column of the file. I have figured out how to deduct one day to derive the correct date and print this to a new file. The main ...
1
vote
1answer
152 views

awk date identification from input file

I currently have daily files that come in via FTP with incorrect dates in the first column of the file. I have figured out how to deduct one day to derive the correct date and print this to a new ...
1
vote
1answer
118 views

replace a pattern found with a number plus 1 on the same line

I have a file where each line contains a keyword (HELLO) and one or more numerical fields designated A1, A2, etc. The value in the numerical field is enclosed in brackets after the field name and is ...
6
votes
3answers
1k views

How to subtract rows (lines) with AWK

I'm trying to figure out how I can use AWK to subtract lines. For example, imagine the input file is: 30 20 The output would be: 10 Now, as a test I am trying to calculate the "Used" memory ...
-3
votes
2answers
74 views

How to get a device details only using awk commands

Friends, I have fdisk -l output in one file called test1.txt. Now I want to get the device name only using awk. How to get a device name? I want output like this: /dev/sda1 /dev/sda2 /dev/sda3 ...
2
votes
5answers
239 views

Can the string “abcedf” be matched to string “bafcde” in a single line command?

I am planning to implement an indexing structure in my program. For example, if I have 100 rows in the table, I will number these rows from 1 to 100 in another column by appending an _ to the end of ...
4
votes
3answers
3k views

Adding a Column of values in a tab delimited file

How can I add a Column of values in a file which has a certain number of rows. I have a input file like this: Input file: SPATA17 1 217947738 LYPLAL1 1 219383905 FAM47E 4 77192838 SHROOM3 4 ...
1
vote
4answers
85 views

How can I list the second to last word found in a sequence of statements in awk

My file is written like: apples,oranges,berries,cherries beer,wine,shots chocolate,icecream,cake,muffins cherries,guacamole,lychee I need to output the second to last word in lines that contain ...
1
vote
4answers
235 views

Pattern Matching and Delete the whole line

I want to delete all the Lines of File 1, if Column1 of File1 matches exactly with Column 1 File2. File 1: r001:21:10 21 AAAAAATTTGC * = XM:21 r002:21:10 21 YAAAATTTGC * ...
0
votes
2answers
83 views

How to extract submatches from a string and write those to a file?

I want to parse a string for specific submatches and output those - one submatch per line - into a file. Essentially I would like to extract the value of each href-attribute: $ cat Fin <a ...
1
vote
3answers
744 views

How can I remove the text from a line after a certain character with awk

How can I use awk to remove all text after a certain character ";" that appears on every line of my text file? (I then need to run for loops on the text) Jenny,Sarah,John;North Dakota ...
2
votes
3answers
163 views

List all word pairs that appear in a line together in a file

How can I use awk to run through a file written as: Sarah,Masha,Helen Connor,Jessica,Jennifer,Candace Betty,Sarah,Helen John,Harold,Frank,Daisy and put out all pairs that appeared in a line ...
4
votes
2answers
138 views

How can I delete all unordered lines from a text file?

Overview Consider an ordered list interspersed with unordered elements, e.g.: Alligator Ant Falcon <-- Baboon Badger Armadillo <-- Caiman Cat How can this list be processed so that all ...
4
votes
2answers
340 views

Replace string with sequential index

Can someone suggest an elegant way to accomplish this? Input: test instant () test instant () ... test instant () //total 1000 lines output should be: test instant1 () test ...
0
votes
3answers
167 views

removing spaces from first column

My input file has positions in first column with different number of spaces (or no space) 16504 16516 1650811 16520 1651 16524 16516111 16528 165204 16532 ...
2
votes
1answer
144 views

Awk regular expression syntax with number of repetition - different handling between gawk 3 and gawk 4

I am having trouble understanding or explaining why the following expression fails in gawk 3.1.x and yet works in gawk 4.1.x: (Minimum working example) echo ";#ADCDE#" | awk '/#.{5}#$/' -> produces ...
2
votes
2answers
70 views

/etc/shadow & gawk

I ran this small script on /etc/shadow: awk '{ FS = ":" } { print $1, $2 }' /etc/shadow It does what it should on all lines except for the root's record I.e. it just prints the entire record. Know ...
2
votes
3answers
73 views

How to optimize this oneliner? (if second column empty, put the line in another file)

I have a little "oneliner": while read ONELINE; do if [[ -n "`echo $ONELINE | awk '{print $2}'`" ]]; then echo "$ONELINE" >> LIST-ok.txt else echo "$ONELINE" >> ...
2
votes
1answer
213 views

Awk field separator bug?

This is the expected way for the awk field separator to work: $ echo 'fooXbar' | awk 'BEGIN {FS="X"} {print $1}' foo $ echo 'fooXbar' | awk 'BEGIN {FS="X"} {print $2}' bar $ but if the FS is ...
1
vote
3answers
77 views

parsing file from ls -laLR

ls -laLR /home/tools > all_site1.txt I need the state of all folders and files in subfolders of /home/tools. With this command I put all in this file. What I need now is a solution how to parse ...
5
votes
3answers
89 views

Increment numbers inside strings by an offset

I want to search a text file for strings like these. [task0] [task1] ... [task1613] and increment each by a given offset, in this case by 3640. So, with the example I gave above, I'd get the ...
2
votes
3answers
301 views

How to show CPU time for processes via top without 'root' procs

Okay, So I've been attempting this for about three hours now without success. How do you search/display/use the top command (or ps command if it works...) to output a list of all procs, sorted by CPU ...
0
votes
2answers
741 views

How do I multiply and sum column data using awk and or sed?

I have several columns of data. I always have the same number of rows (say 5). In the 2nd column, I want to multipy the first value by 5, then the second value by 4, the third value by 3, etc. I then ...
3
votes
2answers
87 views

Command in variable one are affecting output of other variable when displayed together in UNIX

I am having two file naming Extra_info and Class_info Class_info contains data : stuId stuName stuClass S01 jagdeep First S02 harsh First S03 Roop Second S04 Kali Third ...
2
votes
4answers
2k views

awk + print awk output in one line

When I run the following hasys command with awk , I get output with machines names as the following: hasys -display|grep Shutdown | awk '{print $1}' machine1a machine1b please advice how to ...
4
votes
4answers
589 views

Print line after nth occurrence of a match

I am looking to display the line 4598 in the following file. Effectively I want to display the line AFTER the nth occurrence of a match. In this case, the line after the 3rd occurrence of <Car>. ...
2
votes
1answer
169 views

What is an array in awk?

I'm very new to unix and am trying to learn basic awk scripting - during which I keep coming across the word "array", and "associative arrays". I have no clue as to what this is, and don't tell me to ...
4
votes
5answers
93 views

Reformat each two-line sequence into two columns

I would like to tab my printout: 1 2 3 4 5 6 7 8 to 1 2 5 6 3 4 7 8 How to adjust it using simple command line (awk or sed)??
2
votes
4answers
120 views

Count rows with specific integer in a column

I have 6 columns, each with multiple rows. I want to count the number of rows which have the integer 4 or 5 in the fifth column. A1 jhfj jdhfjkhd kdkfjjh 5 jhsdjkfh A2 ujhf jhdfhsd dsfkks 4 ...
0
votes
4answers
75 views

What are alternative tools for searching files in full content?

I realized that most of the text related tools I use, grep/egrep/awk/sed are line-oriented. I guess that is a simplified approach to search in files but I was wondering if there is another set of ...
1
vote
2answers
123 views

comparing user given time for log file time

Apologies straight away for bad-styled-question-and-heading.....learning unix and its command these days, m trying to figure out a solution for below scenario : Lets say i have a log file which ...
1
vote
1answer
83 views

pcregrep fails due to segmentation fault

I am extracting the HTML source of a web page using the below command. curl http://www.linkedin.com/pub/ramesh-venkataraman/63/621/b42 -s | w3m -dump -T text/html > foo.txt The foo.txt has the ...
1
vote
1answer
121 views

Is Perl ever a better tool than awk for text processing [closed]

This question was inspired by a discussion on stackoverflow - see in particular the comment thread under Ed Morton's answer. I figured this was the site where I could get a definitive answer: In ...
1
vote
1answer
139 views

How to get the JSON STRING from the given string

I have a string given below. I just need to get the first accountHeader (bolded) JSON string, I tried everything.. It didn't work. Can you help me to get the data? <START ...
2
votes
3answers
507 views

Print Column if values is between two other values

I am fairly new to data analysis and I have a problem with filtering some of my values. I have data arranged in a file with four columns like this: A 1 10 5 B 10 100 120 C ...
6
votes
4answers
582 views

number parsing in awk

I have a text file with numbers in second column and the numbers have thousand-separator and decimal-separator: 2014-01 2,277.40 2014-02 2,282.20 2014-03 3,047.90 2014-04 4,127.60 ...
0
votes
4answers
464 views

Removing certain fields from a CSV file

I have a large CSV file (~15GB) which a client has sent over as an export of a table. The format is a follows: field, ... , field, field which sometimes contains commas, another field which sometimes ...
2
votes
3answers
3k views

How to merge two files based on the matching of two columns?

I have file1 likes: 0 AFFX-SNP-000541 NA 0 AFFX-SNP-002255 NA 1 rs12103 0.6401 1 rs12103_1247494 0.696 1 rs12142199 0.7672 And a file2: 0 AFFX-SNP-000541 1 0 ...
2
votes
2answers
472 views

Do I need to encapsulate awk variables in quotes in order to sanitize them?

As per an answer on stackoverflow, it's my understanding that encapsulating bash variables in double-quotes is a fairly safe way of sanitizing user input. What about awk variables? For example, if ...
3
votes
2answers
379 views

Add thousands separator in a number

In python re.sub(r"(?<=.)(?=(?:...)+$)", ",", stroke ) To split a number by triplets, e.g.: echo 123456789 | python -c 'import sys;import re; print re.sub(r"(?<=.)(?=(?:...)+$)", ",", ...
2
votes
1answer
58 views

Why does AWK not like dots in one of two substitutions?

I'm trying to use AWK in a bash script to look for a placeholder string in a template file and replace it with the contents of a variable that (can) contain various special characters. Example: ...
2
votes
2answers
3k views

Use awk to split line into array and use that array's values in calling shell

First of all, my apologies for the somewhat clumsily worded title. If you can think of a better one, please feel free to edit. [EDIT: done] What I am trying to do is use awk inside a bash script and ...
3
votes
1answer
86 views

Reorder Multiple Line Blocks with Sed

I am trying to reorder generated output as follows; given a tex fragment as follows, I want to move the /^Constant/ down below /Dummies/, with the line before and after /^Constant/. Further, I would ...
-2
votes
3answers
84 views

Deleting Lines, which match a particular Identifier from another file

I have 2 files. File 1, has an identifier (eg. D7MHBF:11:1449:1988) and every new entry starts with @. It has few more fields, which are not important in our analysis. File 2 consists of a column of ...
4
votes
2answers
64 views

awk pairwise differences

I run into this problem a lot with data, if I have some integers, I frequently want to look at differences between adjacent integers, I usually solve this in Ruby or Python, but I think such a thing ...