awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

22
votes
12answers
4k views

Remove duplicate $PATH entries with awk command

I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my path environment. I was told that it is possible to achieve this with a one line ...
5
votes
4answers
225 views

How to replace m part of Lines with n part of Lines?

I have a file "test" like this: RewiteEngine On RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.7$ [NC] RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.8$ [NC] RewriteCond %{REMOTE_ADDR} ...
0
votes
2answers
126 views

Rearrange the file content

I have a file with content as 123 OneTwoThree 234 TwoThreeFour 345 ThreeFourFive I want to convert this file into something like this OneTwoThree 123 TwoThreeFour 234 ThreeFourFive 345 Here ...
2
votes
5answers
286 views

extracting/copy ranges of columns from large data file

I was trying to prepare data files from one large file by extracting columns and pasting into a new file. The number of columns of the given data is very large. I want to prepare a number of data ...
0
votes
4answers
291 views

Extract value between double quotes

My query is to extract the value between double quotes "". Sample input is : 10.219.41.68 - - - [11/Jun/2014:10:23:04 -0400] Sec:0 MicSec:1797 "GET /balancer-manager HTTP/1.1" 200 28980 "-" ...
5
votes
1answer
335 views

Delete an array in awk

In awk, I can clear an array with a loop, making it an empty array, which is equivalent to deleting it. for (key in array) delete array[key]; Is there a simpler way? Can I completely delete an ...
0
votes
3answers
84 views

awk pattern matching and merge files

I have 3 files 1.csv and 2.csv and 3.csv 1.csv TELECOM_DEVELOPMENT_AFGHANISTAN_COMPANY,AFGHANISTAN,Alphanumeric_A_MSISDN_blocking,1 ...
5
votes
2answers
173 views

How to use shell to derive an IPv6 address from a MAC address?

We know that we can use the MAC address to create an interface identifier, e.g. for a link-local IPv6 address which should be unique in the Network. The image shows the way to do this: My ...
5
votes
6answers
304 views

Remove duplicates values within a field

How to remove duplicate (or more) values within a selected field in a file, keeping only one copy? Exemple Input file: A 1,2,3,45,1,8,2,3 B 5,6,6,6,6,6,2,3,7 Expected output: A ...
7
votes
3answers
3k views

What are the exact differences between awk and cut with grep? [closed]

We know that we can get the second column of the line we want from a file using these two techniques: awk '/WORD/ { print $2 }' filename or grep WORD filename| cut -f 2 -d ' ' My questions are: ...
2
votes
2answers
87 views

How to escape spaces etc in passed variable, for system call to cp in awk

As the title suggests, this problem has enough layers of complexity to make simple escaping characters difficult for me. I have a bash script which, for a large part, has an embedded awk script that ...
2
votes
6answers
96 views

Printing specific section of a line when a trigger value is present

I'm parsing data in the following format: prop1=value1:prop2=value2:prop3=value3+prop1=value4:prop2=value5 parts of the string are delimited by + properties can appear in any order the desired ...
3
votes
1answer
64 views

grep upto matching brackets

I have a file like this:- pin(ABC) { a b c d e f { abc } } pin(XYZ) { g h i j k { j k { ...
1
vote
2answers
49 views

awk - read program AND input from the standard input?

Is it possible for awk to read the program and the input from the standard input? I would like to be able to pipe a file to the following function. process_data () { awk -f - <<EOF {print} ...
1
vote
2answers
276 views

Extracting text string from file, assigning them to variables then writing them to another file?

Basically, its a game server (minecraft) and I have a situation where I am attempting to automate new world generation. Basically, I have a file that gets generated happily which has this snippet in ...
0
votes
2answers
57 views

remove duplicate entry from file1 if entry present in file2

I want to delete duplicate entry e from file1 if e is also present in file2. Input file1: x1 y1 x2 y2 x3 y3 x4 y4 y1 x1 x5 y5 y3 x3 x6 y6 x5 y5 Input file2: y1 x1 y2 x2 y3 x3 y4 x4 ...
3
votes
3answers
49 views

Patterns and file processing

Let's say I have to perform these actions from an input file: extract nth field from a line starting with a given pattern (in the exemple: 2nd field of the line starting with pattern 'name') ...
0
votes
2answers
176 views

AWK command failing for large file

I have a .csv file of 6.5 GB. I need to sort the files with 1st, 2nd, 3rd, 4th, 5th and 13th column and need to get the unique records. Also I need to get the duplicate records and redirect it to some ...
1
vote
2answers
121 views

rename a batch of files after reading from a source file

I have a directory full of over a thousand files and the names are bit long (and no pattern found in the naming style). I would like to simplify the filenames. E.g.: ...
0
votes
2answers
108 views

How match different column in file 1 and file2 with awk?

I have the following two files. The first file is : 3184 2014-07-28 04:15 global.Remote-Access 10.111.8.25 81.245.6.25 tcp 3268 3035 2014-07-28 04:16 global.Remote-Access 10.111.8.12 81.245.6.25 ...
1
vote
2answers
74 views

How can I sort a .txt file by two columns?

Feb 7 domainserver dovecot[37495]: auth(default): od(tjones): lookup failed for user: tjones Above is a sample output of what I'd like to sort. I'd like to sort by date on the first column, but ...
0
votes
1answer
107 views

grepping of common lines between two file based on specific columns values in case of multi matching

I want to match column3 of file1 & column2 of file2 , after that i want to print all column of matching lines in a single line, i know how to do it but the problem is that if there are not unique ...
0
votes
3answers
63 views

Finding and replacing text inside awk block

I want to find and replace a value inside awk block. Here is a sample input file to my script P1 10,9:11/18013013582 ,10:1 ,167:0 ,487:5/E.164 11,9:15/310410532169026 ,10:60 ,167:0 ,487:4/IMSI ...
3
votes
4answers
2k views

Split long output into two columns

Is there a simple utility or script to columnate the output from one of my scripts? I have data in some form: A aldkhasdfljhaf B klajsdfhalsdfh C salkjsdjkladdag D lseuiorlhisnflkc E sdjklfhnslkdfhn ...
0
votes
2answers
477 views

how to remove all double quotes from csv except specific field

I want to remove all double quotes from my csv but not the fourth field (because the four fields represent PATH of file) Please advice how to implement this by sed or awk or perl one liner , etc ...
3
votes
1answer
84 views

awk + gsub + how to work with all kind of characters

The following awk command target is remove spaces and unnecessary quotes from CSV file and only from single words between separators awk 'gsub(/("[ ]+|[ ...
3
votes
2answers
1k views

Replace a pattern in a file with a long string

I'm writing a little shell script to generate a directory listing. To make the output easily customizable, the script just builds a HTML-table and than should replace a specific token in a template ...
2
votes
2answers
78 views

Replace column entry with sum of other column entries

Let's say I have a file that contains these lines: 02.03.14 14.50 14.50 0.00 Desc 02.03.14 17.00 0.00 17.00 Desc 01.03.14 2.82 1.68 calc Desc 02.03.14 1.04 0.00 ...
1
vote
1answer
48 views

Combining awk statements

I'd like to combine those awk statements: awk -F'\t' 'NF>1 && $NF!=""' #prints every row with more than two fields awk '$1 ~ /^[0-9]/' #prints every row starting with a number awk ...
4
votes
4answers
182 views

Printing selective fields following 1st column instructions

Suppose I have a file in which I'd like to simultaneously print different awk commands following the first column instuctions, without messing up the original file (as it would happen with two ...
1
vote
2answers
89 views

Search for a string in a line and append related lines

I have a file containing some ID (identifier) and DS (description) lines. I'd like to keep the rows containing the string value in their ID line, as well as all the following (related) DS lines before ...
4
votes
2answers
377 views

Print lines if given column starts with a capital letter

I have a file like this: ID A56 DS /A56 DS AGE 56 And I'd like to print the whole line only if the second column starts with a capital letter. Expected output: ID A56 DS AGE 56 What I've ...
0
votes
2answers
249 views

How to print a string at the end of line using awk?

I am very new to awk. I am trying to add a new calculated column. But awk insert the column to the next line rather to the end of the current line. Here is an example of my data: ...
2
votes
2answers
69 views

Format output to a specific line length

So I have some output that has very long lines, and I want to have it formatted to lines no more than 80 columns wide, but I don't want to split words because they are in the column limit. I tried ...
12
votes
4answers
15k views

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string 'bar' in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
2
votes
3answers
361 views

Aggregate and group text file in bash

I have a big text file(with 5m lines) in this format(4 columns, separated by ;): string1; string2; string3; userId The first 3 strings (SHA1s) form a single ID, called appId (so it can be ...
1
vote
2answers
69 views

How to determine the newly closed file within a continuous audit log rotation?

I am attempting to find the best way to determine when the second file (of a matching criteria) is created. The context is an audit log rotation. Given a directory where audit logs are created every ...
1
vote
2answers
83 views

Assignment in braces vs outside braces in awk

From this question, I answer with this solution: awk 'FNR==NR{a[FNR]=$2;next}{$NF=a[FNR]}1' file2 file1 It works as expected. But the first solution I thought is: $ awk ...
2
votes
2answers
193 views

Merging columns from two separate files

How to create a new file merging selective columns from two separate files using awk? Without messing up the elements orders of BOTH files. Exemple: File 3 may contain column 1,2,3 from File 1 and ...
3
votes
1answer
112 views

Is there a way to pass a variable from an awk statement to a bash function as an argument? [closed]

I am trying to look a for some specific values from (such as 1 or 4 and 2 or 3) a text file using an awk statement in a bash script. If this value is found in the file (within the awk statement), then ...
6
votes
4answers
4k views

How to extract logs between two time stamps

I want to extract all logs between two timestamps. Some lines may not have the timestamp, but I want those lines also. In short, I want every line that falls under two time stamps. My log structure ...
3
votes
5answers
463 views

Print last element of each row

How can one print the element after the last tab in a file? Exemple: File1 A 3 8 6 7 B 4 6 2 3 6 8 c 1 9 would return: 7 8 9
4
votes
3answers
1k views

awk with if statements

I am trying to print from a file using awk, but my output is empty. Here is my code so far accountNum=$1 while read -r LINE || [[ -n $LINE ]] ; do awk -F',' '{ if($1==accountNum) { print ...
6
votes
1answer
2k views

Removing leading zeros from DATE

I made an alias of the date command to display date in the following format: 2013.06.14.12.10.02 using this command: alias date = date +"%Y.%m.%d.%H.%M.%S" Everything works great, except I ...
0
votes
2answers
1k views

Awk printf number in width and round it up

I need to printf a number out, but with given width and rounded (with awk!) %10s I have this and somehow I need to connect the %d but everything I do, ends up with too much parametres for awk ...
6
votes
2answers
635 views

Number of comma-separated fields in a text file

I'm trying to build an awk statement to read this file: A 1,2,3 * A 4,5,6 ** B 1 B 4,5 * and build a file like this: A 1,2,3 * 3 1 0.333 A 4,5,6 ** 3 2 0.666 B ...
2
votes
1answer
148 views

Backslash with pipe in system command from awk

I want to invoke shell command from awk so I am using system() but when I am passing expression in I am getting error at backslash. Here is my expression echo "select * from abc limit 1;"| ...
5
votes
1answer
46 views

Count duplicate rows with constraint

I have the following file: A B 1990 A C 2001 A C 2013 A B 2001 B C 2013 A D 2001 A D 2014 I need to count duplicate pairs (in the 1st and 2nd column) and to each such pair assign the lowest value ...
0
votes
1answer
59 views

Check values in a text file

Is there a simple way to check if values in the second column of a file ( File 2) exist in the second column of another file (File 1), for example by printing a star next to the concerned line? If ...
3
votes
1answer
41 views

Calculate intervals

How to explicit the intervals between numerical values in a file? (better if an awk one-liner) Input: A 1-3 B 5-7 Output A 1,2,3 B 5,6,7 Code I've tried so far: sed -i 's/-/\t/g' ...