awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

5
votes
4answers
245 views

How to replace all the first occurences in matched lines with incremental number?

I have a file like this ... 1562 first part 1563 H col3 H col4 1564 H col3 H col4 ... 3241 H col3 H col4 3242 third part ... I want to replace only the first H in every line to H#, with ...
2
votes
1answer
35 views

Get each column which contains multiple spaces?

My file: subnet 172.25.10.0 netmask 255.255.255.0 { option domain-name-servers 172.25.10.9, 8.8.8.8; default-lease-time 300; max-lease-time 7500; range dynamic-bootp 172.25.10.10 172.25.10.30 ; ...
3
votes
6answers
1k views

Convert string into date time stamp in gawk or awk

I want to convert following string (20140805234656) into date time stamp (2014-08-05 23:46:56).I am new to gawk and I don't know the exact syntax,how can I put - at every 5,8 and : at every 14,17 and ...
4
votes
6answers
311 views

want the last line when duplicate awk?

I have the file: key value blah blah blah blah blahblah man1 boy1 blah blah man1 boy2 man1 boy1 I do this to remove duplicate lines: awk '/man1/ { print $1,$2} ' file | awk '!x[$0]++' and the ...
3
votes
1answer
74 views

Handling dynamically changing column positions and splitting file

I have a unique problem. I have a file with following columns PersonId, AB1, AB2, AB3, DG1_B1,DG1_B2,DG2 1,0,1,0,3,5,7 2,0,3,1,3,7,4 I want to split this and create multiple files based on the ...
0
votes
5answers
57 views

re-arrange values in one column without affecting other columns using awk or sed

I have a csv file in following format: abc,124,123,2014-08-13,abc def,124,123,2014-08-13,abc ghi,124,123,2014-08-13,abc Now I need to modify only 5th column (2014-08-13) to produce output as given ...
2
votes
1answer
50 views

Move matched line above a different matched line using sed

I have something like dsadjakldjska safdsfdsfsf fdsfsf**STGN**fdsfsd fdsfkjldsf fdsff fdsfs**IRV**fdsfs and I want to get dsadjakldjska safdsfdsfsf fdsfkjldsf fdsff fdsfsf**STGN**fdsfsd ...
2
votes
2answers
79 views

Memory allocation for sparse array in awk

I have searched but didnt reach to any conclusion that when i defined an sparse array does it reserve all the contiguous memory upto maximum index or it allocate the memory at that particular index ...
2
votes
4answers
276 views

Insert new line after non matching pattern

I have following output from script host1:101 host1:102 host2:101 host2:102 host2:103 host4:101 host5:101 host5:102 host5:103 host5:104 etc.. I wanted to insert a new line after each host group. ...
2
votes
6answers
896 views

Handling comma in string values in a CSV file [duplicate]

I have a comma-separated file which has numeric and string columns. String columns are quoted and can have comma in between the quotes. How do I identify the columns with FS =","? Sample records ...
4
votes
3answers
429 views

pipe line with non-deterministic output

I have happened upon a command that sometimes works and sometimes does not, even when executed multiple times in rapid succession in a bash shell (I have not tested the behavior in other shells). The ...
0
votes
2answers
103 views

Delete every number that comes before “=”

i have a very, very, very long list. Like over 830,000 characters. I will of course perform said command on a copy. There are multiple lines, like this hey1=lol hey2=lol hey3=lol I want that to ...
1
vote
2answers
101 views

cygwin awk output to windows line delimeters

I could find the way to print zip entries' list and forward to a txt file. unzip -l my.zip | awk '{print $4}' > my.txt When I open the file my.txt in windows' notepad the line delimiters ignored ...
5
votes
3answers
604 views

How to understand the command awk '{for(x=1;$x;++x)print $x}'?

I know this command awk '{for(x=1;$x;++x)print $x}' will print out all columns in a line. wouldn't this ++x change x to 2, and thus print $2 first? As I understood based on this: ...
2
votes
2answers
172 views

awk character class mystery

I seem to be unable to use a character class for an awk regular expression, almost exactly as described here : user@host:~$ awk -W version mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan ...
-2
votes
2answers
265 views

Extract certain column from CSV files [closed]

I have a directory with a group of CSV files. All files have the same column headers. I need to extract values in a certain column from all files. The common part in files names is ...
0
votes
2answers
129 views

Rearrange the file content

I have a file with content as 123 OneTwoThree 234 TwoThreeFour 345 ThreeFourFive I want to convert this file into something like this OneTwoThree 123 TwoThreeFour 234 ThreeFourFive 345 Here ...
5
votes
6answers
318 views

Remove duplicates values within a field

How to remove duplicate (or more) values within a selected field in a file, keeping only one copy? Exemple Input file: A 1,2,3,45,1,8,2,3 B 5,6,6,6,6,6,2,3,7 Expected output: A ...
2
votes
2answers
101 views

How to escape spaces etc in passed variable, for system call to cp in awk

As the title suggests, this problem has enough layers of complexity to make simple escaping characters difficult for me. I have a bash script which, for a large part, has an embedded awk script that ...
0
votes
3answers
95 views

awk pattern matching and merge files

I have 3 files 1.csv and 2.csv and 3.csv 1.csv TELECOM_DEVELOPMENT_AFGHANISTAN_COMPANY,AFGHANISTAN,Alphanumeric_A_MSISDN_blocking,1 ...
2
votes
5answers
356 views

extracting/copy ranges of columns from large data file

I was trying to prepare data files from one large file by extracting columns and pasting into a new file. The number of columns of the given data is very large. I want to prepare a number of data ...
1
vote
2answers
51 views

awk - read program AND input from the standard input?

Is it possible for awk to read the program and the input from the standard input? I would like to be able to pipe a file to the following function. process_data () { awk -f - <<EOF {print} ...
5
votes
4answers
226 views

How to replace m part of Lines with n part of Lines?

I have a file "test" like this: RewiteEngine On RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.7$ [NC] RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.8$ [NC] RewriteCond %{REMOTE_ADDR} ...
1
vote
4answers
488 views

Extract value between double quotes

My query is to extract the value between double quotes "". Sample input is : 10.219.41.68 - - - [11/Jun/2014:10:23:04 -0400] Sec:0 MicSec:1797 "GET /balancer-manager HTTP/1.1" 200 28980 "-" ...
1
vote
2answers
362 views

Extracting text string from file, assigning them to variables then writing them to another file?

Basically, its a game server (minecraft) and I have a situation where I am attempting to automate new world generation. Basically, I have a file that gets generated happily which has this snippet in ...
6
votes
1answer
484 views

Delete an array in awk

In awk, I can clear an array with a loop, making it an empty array, which is equivalent to deleting it. for (key in array) delete array[key]; Is there a simpler way? Can I completely delete an ...
3
votes
1answer
75 views

grep upto matching brackets

I have a file like this:- pin(ABC) { a b c d e f { abc } } pin(XYZ) { g h i j k { j k { ...
0
votes
2answers
59 views

remove duplicate entry from file1 if entry present in file2

I want to delete duplicate entry e from file1 if e is also present in file2. Input file1: x1 y1 x2 y2 x3 y3 x4 y4 y1 x1 x5 y5 y3 x3 x6 y6 x5 y5 Input file2: y1 x1 y2 x2 y3 x3 y4 x4 ...
1
vote
2answers
191 views

rename a batch of files after reading from a source file

I have a directory full of over a thousand files and the names are bit long (and no pattern found in the naming style). I would like to simplify the filenames. E.g.: ...
1
vote
2answers
83 views

How can I sort a .txt file by two columns?

Feb 7 domainserver dovecot[37495]: auth(default): od(tjones): lookup failed for user: tjones Above is a sample output of what I'd like to sort. I'd like to sort by date on the first column, but ...
0
votes
1answer
118 views

grepping of common lines between two file based on specific columns values in case of multi matching

I want to match column3 of file1 & column2 of file2 , after that i want to print all column of matching lines in a single line, i know how to do it but the problem is that if there are not unique ...
0
votes
2answers
123 views

How match different column in file 1 and file2 with awk?

I have the following two files. The first file is : 3184 2014-07-28 04:15 global.Remote-Access 10.111.8.25 81.245.6.25 tcp 3268 3035 2014-07-28 04:16 global.Remote-Access 10.111.8.12 81.245.6.25 ...
3
votes
3answers
49 views

Patterns and file processing

Let's say I have to perform these actions from an input file: extract nth field from a line starting with a given pattern (in the exemple: 2nd field of the line starting with pattern 'name') ...
0
votes
2answers
656 views

how to remove all double quotes from csv except specific field

I want to remove all double quotes from my csv but not the fourth field (because the four fields represent PATH of file) Please advice how to implement this by sed or awk or perl one liner , etc ...
3
votes
1answer
110 views

awk + gsub + how to work with all kind of characters

The following awk command target is remove spaces and unnecessary quotes from CSV file and only from single words between separators awk 'gsub(/("[ ]+|[ ...
2
votes
2answers
78 views

Replace column entry with sum of other column entries

Let's say I have a file that contains these lines: 02.03.14 14.50 14.50 0.00 Desc 02.03.14 17.00 0.00 17.00 Desc 01.03.14 2.82 1.68 calc Desc 02.03.14 1.04 0.00 ...
1
vote
1answer
53 views

Combining awk statements

I'd like to combine those awk statements: awk -F'\t' 'NF>1 && $NF!=""' #prints every row with more than two fields awk '$1 ~ /^[0-9]/' #prints every row starting with a number awk ...
4
votes
4answers
221 views

Printing selective fields following 1st column instructions

Suppose I have a file in which I'd like to simultaneously print different awk commands following the first column instuctions, without messing up the original file (as it would happen with two ...
4
votes
2answers
477 views

Print lines if given column starts with a capital letter

I have a file like this: ID A56 DS /A56 DS AGE 56 And I'd like to print the whole line only if the second column starts with a capital letter. Expected output: ID A56 DS AGE 56 What I've ...
1
vote
2answers
91 views

Search for a string in a line and append related lines

I have a file containing some ID (identifier) and DS (description) lines. I'd like to keep the rows containing the string value in their ID line, as well as all the following (related) DS lines before ...
0
votes
2answers
330 views

How to print a string at the end of line using awk?

I am very new to awk. I am trying to add a new calculated column. But awk insert the column to the next line rather to the end of the current line. Here is an example of my data: ...
32
votes
4answers
2k views

Reverse grepping

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
2
votes
2answers
88 views

Format output to a specific line length

So I have some output that has very long lines, and I want to have it formatted to lines no more than 80 columns wide, but I don't want to split words because they are in the column limit. I tried ...
2
votes
3answers
398 views

Aggregate and group text file in bash

I have a big text file(with 5m lines) in this format(4 columns, separated by ;): string1; string2; string3; userId The first 3 strings (SHA1s) form a single ID, called appId (so it can be ...
3
votes
1answer
126 views

Is there a way to pass a variable from an awk statement to a bash function as an argument? [closed]

I am trying to look a for some specific values from (such as 1 or 4 and 2 or 3) a text file using an awk statement in a bash script. If this value is found in the file (within the awk statement), then ...
1
vote
2answers
77 views

How to determine the newly closed file within a continuous audit log rotation?

I am attempting to find the best way to determine when the second file (of a matching criteria) is created. The context is an audit log rotation. Given a directory where audit logs are created every ...
3
votes
5answers
537 views

Print last element of each row

How can one print the element after the last tab in a file? Exemple: File1 A 3 8 6 7 B 4 6 2 3 6 8 c 1 9 would return: 7 8 9
1
vote
2answers
89 views

Assignment in braces vs outside braces in awk

From this question, I answer with this solution: awk 'FNR==NR{a[FNR]=$2;next}{$NF=a[FNR]}1' file2 file1 It works as expected. But the first solution I thought is: $ awk ...
2
votes
2answers
263 views

Merging columns from two separate files

How to create a new file merging selective columns from two separate files using awk? Without messing up the elements orders of BOTH files. Exemple: File 3 may contain column 1,2,3 from File 1 and ...
4
votes
3answers
2k views

awk with if statements

I am trying to print from a file using awk, but my output is empty. Here is my code so far accountNum=$1 while read -r LINE || [[ -n $LINE ]] ; do awk -F',' '{ if($1==accountNum) { print ...