awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

2
votes
3answers
288 views

Text to yaml conversion

I have a tab separated text file in this format: your-email your-order-id PayPal-transaction-id your-first-name your-second-name s@ess.com 12345 54321 sooky spooky t@tee.com 23456 ...
6
votes
2answers
1k views

Inotifywait for large number of files in a directory

What I want to do is, to monitor a directory (not recursive, just one) for new files created and append those files to one single big file as they are being written. The number of files that are ...
2
votes
3answers
2k views

Easy way to parse syslog date format

I'd like to write a script that analyses the last week's syslog. But my syslog logs in a stupid date format, e.g. "May 22". If it logged as 2013-05-22, this task would be trivial. Is there a way ...
1
vote
1answer
42 views

Matching hexadecimals with gawk

I have a binary file with the following content (printed in hexadecimal): $ xxd -p virus.com 5669727573b440bb0100b90500ba0000cd21 I can match the first 5 letters (which are the alphanumeric string ...
5
votes
2answers
244 views

awk + paste for cleaning up PATH?

I have seen this code in .cshrc init files on a few machines. I went through a few awk tutorials in trying to understand how it works, but I am still unable to decrypt it. setenv PATH `echo $PATH | ...
1
vote
3answers
694 views

awk a specific column with unexpected line break

I can't seem to find a solution for this and it's driving me crazy. I know I can use awk to print a column(s). I'm having trouble printing a specific column though because of the way my file system is ...
2
votes
3answers
63 views

Replace certain strings with value from former line

I have the following text: Source IP,URL 1.1.1.1,example1.com Blank,example2.com 2.2.2.2,example3.com Blank,example4.com Blank,example5.com Blank,example6.com Blank,example7.com 7.7.7.7,example8.com ...
3
votes
3answers
190 views

sed + how to remove character/s that start or ended on each number

How do I remove the . character(s) that start in the beginning of each number or end on each number? Remark – perl one liner also good alternative for sed. Example input: .23.12.44.5. .233.3.3.3 ...
3
votes
3answers
203 views

checking data in columns when a data or some may be missing or present?

I am not sure if this possible. say i have columns like : Team Colour Game Rainfall PlayerName XYZ Blue Cricket Yes Kapil suppose i need to search ...
2
votes
2answers
110 views

confuse with awk user define function

I have solved Projecteuler Problem 3, but the thing is that I am not able to create the function in awk. I have tried this working code (without function): #!/usr/bin/awk -f BEGIN{ ...
1
vote
1answer
711 views

Using awk to parse AIX's lsuser output in stanza format

I'm creating a simple awk script to parse AIX's lsuser -f ALL output and return information in one line. A sample of the output is something like this. root: id=0 pgrp=something ...
2
votes
6answers
193 views

How can I separate these two outputs using awk command

Here's my output: MessageID=3990592283244651750-30192b51.13df831d93a.7eb4;EsbTiming=2013-04-12T01:07:46.099 I want only the time. How can I separate them?
0
votes
1answer
266 views

moving columns across rows

I am dealing with a txt file and trying to keep the first column/field in place, but shift the 2nd column down by 1 row. I tried this, but it gave the error unmatched .'s. awk '{for(i=1; i<NR; ...
4
votes
3answers
7k views

How can I use SED or AWK to replace placeholders in a template file with variable content that contains special characters?

CentOS 6.3 I'm trying to get a small script to send an email containing a copy of email headers in the body (for the purpose of internal reporting). The template file contains the following: ...
4
votes
3answers
2k views

Remove string from a particular field using awk/sed

I have a file (>80,000 lines) that looks likes this: chr1 GTF2GFF chromosome 1 249213345 . . . ID=chr1;Name=chr1 chr1 GTF2GFF gene 11874 14408 . + . ...
0
votes
1answer
1k views

comparing lines with awk vs while read line

I have two files one with 17k lines and another one with 4k lines. I wanted to compare position 115 to position 125 with each line in the second file and if there is a match, write the entire line ...
3
votes
2answers
313 views

Automated partition selection in bash script

I'm trying to write a bash script to run from Linux on a CD or flashdrive that can mount Windows and change stuff, but I don't know how to go about detecting which partition/drive is Windows. My best ...
2
votes
3answers
3k views

Using bash shell function inside AWK

Is it possible to use bash function inside AWK somehow? Example file (string, int, int, int) Mike 247808 247809 247810 Trying to convert values from decimal to hexadecimal. Function defined ...
4
votes
1answer
1k views

Pass colors from ls through pipe to awk 'print' statement

This is a follow-up to my question from yesterday, Show sum of file sizes in directory listing. Thanks to Zero Piraeus and a point in the right direction by Mauritz Hansen, I now have function ...
3
votes
1answer
673 views

Output of `java -version` not matched by grep or awk

On my Linux machine, it isn't clear to me why if I do the following then I don't get only the version string ("1.5.0_32"). # java -version | grep version | awk '{print $NF}' java version "1.5.0_32" ...
7
votes
4answers
4k views

How to remove multiple blank lines from a file?

I have some text-files I use to take notes in - just plain text, usually just using cat >> file. Occasionally I use a blank line or two (just return - the new-line character) to specify a new ...
1
vote
3answers
5k views

awk + print line only if the first field start with string as Linux1

how to print the line in case the first field start with Linux1 for example: echo Linux1_ver2 12542 kernel-update | awk '{if ($1 ~ Linux1 ) print $0;}' the target is to print the line , ...
1
vote
2answers
295 views

How to count the number of words and print the lines that match exactly a given pattern?

I want to count the number of words and print the matched pattern lines which matches exactly with following pattern: abc-ERROR: The input File contains: # abc-ERROR: xyxxkkfgfr # def-Error: ...
9
votes
9answers
26k views

Show sum of file sizes in directory listing

The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current ...
1
vote
2answers
60 views

Trimming one part if equal to another part?

I use an e-mail quote attribution string like this: On dd mmm yyyy hh:mm +hhmm, from info@example.org (Example Dot Org): where info@example.org is the sender's e-mail address and Example Dot Org is ...
5
votes
2answers
664 views

Gawk: Passing arrays to functions

Stuck with GNU awk 3.1.6 and think I've worked around its array bugs but still have what looks like a scope problem in a 600-line awk program. Need to verify understanding of array scope in awk to ...
1
vote
1answer
1k views

awk - gensub function

I need to insert contents of variable to parameter function gensub in awk, specifically instead of regular expression's parameter. Can you help me with it? I have: gawk ' BEGIN { a = ...
3
votes
2answers
571 views

cumulative totals with shifted column

Apologies in advance since I am sure this is an easy problem. Nevertheless, here goes. cat blah.txt aa+2 bb+4 cc+10 dd+31 I would like to process blah.txt to produce the folowing: aa+0 bb+2 cc+6 ...
1
vote
1answer
344 views

Safely convert unicode strings to printable characters

I have many HTML files containing mixed unicode strings like \303\243 and printable characters like %s. What I'd like to do is converting the unicode strings into printable characters in a safe way. ...
0
votes
2answers
277 views

Reordering strings in linux

How can I change lines around using the command line? e.g. I have these lines: Acct-Status-Type = Start User-Name = "37XXXXXXX" Event-Timestamp = "Apr 12 2013 15:56:55 AMT" I need to ...
0
votes
1answer
309 views

Regex awk to Cisco Interface

I would like to know if have some Sed/Grep or Awk regex to parse Cisco interface section, with specific attribute, like bellow. Content of file.txt ! interface FastEthernet0/1 no ip unreachables ! ...
0
votes
1answer
197 views

replacing text and deleting text using awk in file using

I have a script and I want to do some replace and delete operations on it. I used sed to extract values between words SendCommands and end like this sed -n '/SendCommands/,/end/p' ddtt As awk was ...
2
votes
1answer
1k views

Vim - how to increase each number in visual block?

I have the following SQL: update am.PERMISSIONS set PRM_ORDER = 35 PRM_VISIBLE = b'1' where PRM_ID = 3; update am.PERMISSIONS set PRM_ORDER = [35] PRM_VISIBLE = b'1' where PRM_ID = 7; update ...
1
vote
2answers
144 views

How do I get a field in bash's job list using awk?

I wanted to write a function to retrieve a field from the list of background jobs in bash. For example to get a specific argument. Let's say I send vim to the background, so "jobs" displays this ...
0
votes
3answers
597 views

Sorting IP address according to the second field in file

in my ksh script I need to add the following task ( OS - linux/solaris ) I have the following file more test.txt /etc/backup/app 172.1.120 172.110.120 /etc/backup/app 172.1.120.12 ...
-3
votes
1answer
162 views

How to transform two delimited ASCII files

These are 2 common delimited, text file transformation problems I'd like to find scripts to do (I dont know if awk or perl is best for these tasks). csh is the default shell on my Solaris10 unix ...
0
votes
3answers
1k views

Select/extract columns based on the values on a certain row

Here is an example text file: A B C D E F G 1 2 3 4 5 6 7 2 3 4 5 6 7 8 3 4 5 6 7 8 9 I would like to extract specific columns based on the values on the 3rd row, that is, 2 3 4 5 6 7 8. Let's say, ...
5
votes
1answer
10k views

Replacing string based on line number

I have a situation where i want to replace a particular string in many files Replace a string AAA with another string BBB but there are lot of strings starting with AAA or ending in AAA ,and i want ...
1
vote
2answers
130 views

Problem using awk

I have a problem using awk. Print, from every file given as a parameter, the number of line that has the length at least 10. Also, print the content of that line(s), except the fist 10 characters. At ...
2
votes
3answers
343 views

awk, sed, grep, perl… which to print out in this case?

I have this syntax in a file (http response times from analog): <thead><tr><th class="x">seconds</th><th class="R">reqs</th><th ...
7
votes
7answers
809 views

Slick one-liner to convert a list like “1: 2, 3, 4, 5” to “1.2, 1.3, 1.4, 1.5”

Let's say I have a file that looks something like this: 23: a, b, c, d 24: b, d, f 25: c, g and I want to get output like this: 23.a 23.b 23.c 23.d 24.b 24.d 24.f 25.c 25.g Of course it's not ...
3
votes
1answer
64 views

How to edit files in-place while performing operation on the values matching the searched pattern?

Given a list files containing random text and many datetimes in ISO format in it (e.g "2012-07-02T10:47:24+02:00"), how can I find all these datetimes and add 2 hours to each one ? Ideally a solution ...
2
votes
3answers
2k views

Bash commands/script to remove a line from CSV with duplicate in column

I have a lot CSV files that I have combined. However, there are duplicates, but the entire line is not duplicated. I do have a column that I want to use as the criteria to search for a duplicate. ...
1
vote
3answers
611 views

Newbie struggles with grep, sed, awk to filter html

I am a beginner linux user, trying to teach myself how to use linux tools on a cygwin install. I decided to make up a project to try to teach myself the basics of shell scripting and simultaneously ...
3
votes
4answers
1k views

Replacing pattern after nth match is found on each line?

I have a file containing lines: india;austria;japan;chile china;US;nigeria;mexico;russia I want to replace all the occurences of semicolon on each line with e.g. ;NEW;, but starting from the 2nd ...
1
vote
1answer
3k views

How can I find and filter a specific column in a .csv file? [duplicate]

I have .csv files with the following structure: cat,dog,mouse,pig,bird,cow,... 21,34,54,566,78,1,... 23,10,12,569,56,4,... 32,20,13,123,56,3,... 34,30,44,322,66,2,... I want to filter the column ...
1
vote
1answer
123 views

How to parse this output and put it into an array?

i have expect script whose output looks like : I would like to extract all words like Gi/02/12 etc in an array or a list. VLAN Name Status Ports ---- ...
1
vote
3answers
362 views

Is it mandatory to have a variable in awk's first statement?

I am having some hard time with awk today. If I try to do: df|awk '{print $2; $some=$2; print $some}' It works as expected and I get the size of the disks twice but if I do: df|awk '{$some=$2; ...
1
vote
2answers
80 views

Match ids and print original file

I have two files Original: (5000 entries) Chr Position chr1 879108 chr1 881918 chr1 896874 … and a file with allele frequencies (2000 entries) Chr Position MAF chr1 881918 0.007 ...
-1
votes
1answer
329 views

Search low values in mainframe files

I am working with mainframe files and it seems that the low values (x'00') exists in the records in the file which throws off my import for fixed block. Does anyone know how to eliminate those ...