awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
2answers
31 views

Remove duplicate fields in a given column

I'd like to remove from a given column ($2 in the example) the duplicate fields (comma separated). Input file: A 1,2,3,4 B 4,5,6,3 C 2,15 Expected output: A 1,2,3,4 B 5,6 C ...
3
votes
4answers
164 views

Count how many elements in the nth column in a whole file

Input file: A 1,2,3,4 #length($2)=4 B 1,2 #length($2)=2 C 9,8,7,6,5,4 #length($2)=6 Expected output: 12 #4+2+6 A method like: awk -F '[\t,]' '{print length($2)}' but ...
-4
votes
0answers
34 views

Can you please explain this syntax for me [on hold]

Please can you explain this syntax $ awk 'BEGIN{FS="FS";}'
3
votes
3answers
105 views

Parse multiple sections of data into seperate files

I have a concatenated log file with multiple logs inside that I'm trying to parse out into individual log files (I will later rename them to the date/time of each). Each log is separated by "--- LOG ...
2
votes
2answers
21 views

Delete rows in a csv file based on a specific column value

I have a csv file with the following headers - Interaction_ID, PDB_ID, First_Residue, First_Chain, Second_Residue, Third_Residue, Third_Chain & Pattern: ...
3
votes
4answers
59 views

How to remove text matching specific patterns from a file

I want to compare two files to check for differences between them, fileA and fileB. fileA is like a template file and fileB is the file which I want to compare to it. Anytime I find a difference I ...
1
vote
3answers
33 views

Trouble in adding a column with awk

I'm trying to add a column to a csv file by using: awk -F "," '{print $0"new_col"}' file.csv but I got : new_col8,0.405,0.101 new_col4,0.374,0.100 new_col23,0.342,0.100 new_col130,0.298,0.100 ...
0
votes
3answers
41 views

Replace specific text only in specific fields using nawk

My input data file abc.txt has this "20131008","0004","0568","98" I ran this command to replace the text nawk -F, -vOFS=',' '{gsub("0568","0808",$3); print }' abc.txt In the output the ...
4
votes
2answers
80 views

find length of longest line in all text files in a directory

I know how to get length of the longest line in a text file with awk awk ' { if ( length > L ) { L=length} }END{ print L}' file.txt but how can I get the length of the longest line of all files ...
5
votes
4answers
206 views

How to replace all the first occurences in matched lines with incremental number?

I have a file like this ... 1562 first part 1563 H col3 H col4 1564 H col3 H col4 ... 3241 H col3 H col4 3242 third part ... I want to replace only the first H in every line to H#, with ...
-1
votes
2answers
14 views

Design the Code to combine the splitted files based on serial,pin,denomination [on hold]

source is a directory contain two files s1 and s2. These files contain some data in columns: serial,pin,denomination datas. This should be moved to another directory called destination with ...
2
votes
1answer
22 views

Get each column which contains multiple spaces?

My file: subnet 172.25.10.0 netmask 255.255.255.0 { option domain-name-servers 172.25.10.9, 8.8.8.8; default-lease-time 300; max-lease-time 7500; range dynamic-bootp 172.25.10.10 172.25.10.30 ; ...
3
votes
6answers
201 views

Convert string into date time stamp in gawk or awk

I want to convert following string (20140805234656) into date time stamp (2014-08-05 23:46:56).I am new to gawk and I don't know the exact syntax,how can I put - at every 5,8 and : at every 14,17 and ...
4
votes
6answers
279 views

want the last line when duplicate awk?

I have the file: key value blah blah blah blah blahblah man1 boy1 blah blah man1 boy2 man1 boy1 I do this to remove duplicate lines: awk '/man1/ { print $1,$2} ' file | awk '!x[$0]++' and the ...
3
votes
1answer
38 views

Handling dynamically changing column positions and splitting file

I have a unique problem. I have a file with following columns PersonId, AB1, AB2, AB3, DG1_B1,DG1_B2,DG2 1,0,1,0,3,5,7 2,0,3,1,3,7,4 I want to split this and create multiple files based on the ...
0
votes
5answers
47 views

re-arrange values in one column without affecting other columns using awk or sed

I have a csv file in following format: abc,124,123,2014-08-13,abc def,124,123,2014-08-13,abc ghi,124,123,2014-08-13,abc Now I need to modify only 5th column (2014-08-13) to produce output as given ...
2
votes
1answer
27 views

Move matched line above a different matched line using sed

I have something like dsadjakldjska safdsfdsfsf fdsfsf**STGN**fdsfsd fdsfkjldsf fdsff fdsfs**IRV**fdsfs and I want to get dsadjakldjska safdsfdsfsf fdsfkjldsf fdsff fdsfsf**STGN**fdsfsd ...
2
votes
2answers
41 views

Memory allocation for sparse array in awk

I have searched but didnt reach to any conclusion that when i defined an sparse array does it reserve all the contiguous memory upto maximum index or it allocate the memory at that particular index ...
2
votes
4answers
66 views

Insert new line after non matching pattern

I have following output from script host1:101 host1:102 host2:101 host2:102 host2:103 host4:101 host5:101 host5:102 host5:103 host5:104 etc.. I wanted to insert a new line after each host group. ...
2
votes
6answers
415 views

Handling comma in string values in a CSV file [duplicate]

I have a comma-separated file which has numeric and string columns. String columns are quoted and can have comma in between the quotes. How do I identify the columns with FS =","? Sample records ...
4
votes
3answers
336 views

pipe line with non-deterministic output

I have happened upon a command that sometimes works and sometimes does not, even when executed multiple times in rapid succession in a bash shell (I have not tested the behavior in other shells). The ...
0
votes
2answers
100 views

Delete every number that comes before “=”

i have a very, very, very long list. Like over 830,000 characters. I will of course perform said command on a copy. There are multiple lines, like this hey1=lol hey2=lol hey3=lol I want that to ...
1
vote
2answers
27 views

cygwin awk output to windows line delimeters

I could find the way to print zip entries' list and forward to a txt file. unzip -l my.zip | awk '{print $4}' > my.txt When I open the file my.txt in windows' notepad the line delimiters ignored ...
5
votes
3answers
519 views

How to understand the command awk '{for(x=1;$x;++x)print $x}'?

I know this command awk '{for(x=1;$x;++x)print $x}' will print out all columns in a line. wouldn't this ++x change x to 2, and thus print $2 first? As I understood based on this: ...
2
votes
2answers
160 views

awk character class mystery

I seem to be unable to use a character class for an awk regular expression, almost exactly as described here : user@host:~$ awk -W version mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan ...
-3
votes
2answers
48 views

Extract certain column from CSV files [closed]

I have a directory with a group of CSV files. All files have the same column headers. I need to extract values in a certain column from all files. The common part in files names is ...
0
votes
2answers
110 views

Rearrange the file content

I have a file with content as 123 OneTwoThree 234 TwoThreeFour 345 ThreeFourFive I want to convert this file into something like this OneTwoThree 123 TwoThreeFour 234 ThreeFourFive 345 Here ...
5
votes
6answers
187 views

Remove duplicates values within a field

How to remove duplicate (or more) values within a selected field in a file, keeping only one copy? Exemple Input file: A 1,2,3,45,1,8,2,3 B 5,6,6,6,6,6,2,3,7 Expected output: A ...
2
votes
2answers
51 views

How to escape spaces etc in passed variable, for system call to cp in awk

As the title suggests, this problem has enough layers of complexity to make simple escaping characters difficult for me. I have a bash script which, for a large part, has an embedded awk script that ...
0
votes
3answers
47 views

awk pattern matching and merge files

I have 3 files 1.csv and 2.csv and 3.csv 1.csv TELECOM_DEVELOPMENT_AFGHANISTAN_COMPANY,AFGHANISTAN,Alphanumeric_A_MSISDN_blocking,1 ...
2
votes
5answers
155 views

extracting/copy ranges of columns from large data file

I was trying to prepare data files from one large file by extracting columns and pasting into a new file. The number of columns of the given data is very large. I want to prepare a number of data ...
1
vote
2answers
35 views

awk - read program AND input from the standard input?

Is it possible for awk to read the program and the input from the standard input? I would like to be able to pipe a file to the following function. process_data () { awk -f - <<EOF {print} ...
5
votes
4answers
206 views

How to replace m part of Lines with n part of Lines?

I have a file "test" like this: RewiteEngine On RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.7$ [NC] RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.8$ [NC] RewriteCond %{REMOTE_ADDR} ...
0
votes
4answers
56 views

Extract value between double quotes

My query is to extract the value between double quotes "". Sample input is : 10.219.41.68 - - - [11/Jun/2014:10:23:04 -0400] Sec:0 MicSec:1797 "GET /balancer-manager HTTP/1.1" 200 28980 "-" ...
1
vote
2answers
98 views

Extracting text string from file, assigning them to variables then writing them to another file?

Basically, its a game server (minecraft) and I have a situation where I am attempting to automate new world generation. Basically, I have a file that gets generated happily which has this snippet in ...
5
votes
1answer
74 views

Delete an array in awk

In awk, I can clear an array with a loop, making it an empty array, which is equivalent to deleting it. for (key in array) delete array[key]; Is there a simpler way? Can I completely delete an ...
3
votes
1answer
48 views

grep upto matching brackets

I have a file like this:- pin(ABC) { a b c d e f { abc } } pin(XYZ) { g h i j k { j k { ...
-2
votes
0answers
22 views

remove duplicates from file1 if entry present in file2 [duplicate]

Input file1: x1 y1 x2 y2 x3 y3 x4 y4 y1 x1 x5 y5 y3 x3 x6 y6 x5 y5 Input file2: y1 x1 y2 x2 y3 x3 y4 x4 x1 y1 y5 x5 x3 y3 y6 x6 x5 y5 Desired output: x1 y1 x2 y2 x3 y3 x4 ...
0
votes
2answers
43 views

remove duplicate entry from file1 if entry present in file2

I want to delete duplicate entry e from file1 if e is also present in file2. Input file1: x1 y1 x2 y2 x3 y3 x4 y4 y1 x1 x5 y5 y3 x3 x6 y6 x5 y5 Input file2: y1 x1 y2 x2 y3 x3 y4 x4 ...
1
vote
2answers
25 views

rename a batch of files after reading from a source file

I have a directory full of over a thousand files and the names are bit long (and no pattern found in the naming style). I would like to simplify the filenames. E.g.: ...
1
vote
2answers
39 views

How can I sort a .txt file by two columns?

Feb 7 domainserver dovecot[37495]: auth(default): od(tjones): lookup failed for user: tjones Above is a sample output of what I'd like to sort. I'd like to sort by date on the first column, but ...
0
votes
1answer
53 views

grepping of common lines between two file based on specific columns values in case of multi matching

I want to match column3 of file1 & column2 of file2 , after that i want to print all column of matching lines in a single line, i know how to do it but the problem is that if there are not unique ...
0
votes
2answers
59 views

How match different column in file 1 and file2 with awk?

I have the following two files. The first file is : 3184 2014-07-28 04:15 global.Remote-Access 10.111.8.25 81.245.6.25 tcp 3268 3035 2014-07-28 04:16 global.Remote-Access 10.111.8.12 81.245.6.25 ...
3
votes
3answers
44 views

Patterns and file processing

Let's say I have to perform these actions from an input file: extract nth field from a line starting with a given pattern (in the exemple: 2nd field of the line starting with pattern 'name') ...
0
votes
2answers
68 views

how to remove all double quotes from csv except specific field

I want to remove all double quotes from my csv but not the fourth field (because the four fields represent PATH of file) Please advice how to implement this by sed or awk or perl one liner , etc ...
3
votes
1answer
21 views

awk + gsub + how to work with all kind of characters

The following awk command target is remove spaces and unnecessary quotes from CSV file and only from single words between separators awk 'gsub(/("[ ]+|[ ...
2
votes
2answers
69 views

Replace column entry with sum of other column entries

Let's say I have a file that contains these lines: 02.03.14 14.50 14.50 0.00 Desc 02.03.14 17.00 0.00 17.00 Desc 01.03.14 2.82 1.68 calc Desc 02.03.14 1.04 0.00 ...
1
vote
1answer
45 views

Combining awk statements

I'd like to combine those awk statements: awk -F'\t' 'NF>1 && $NF!=""' #prints every row with more than two fields awk '$1 ~ /^[0-9]/' #prints every row starting with a number awk ...
4
votes
4answers
125 views

Printing selective fields following 1st column instructions

Suppose I have a file in which I'd like to simultaneously print different awk commands following the first column instuctions, without messing up the original file (as it would happen with two ...
4
votes
2answers
190 views

Print lines if given column starts with a capital letter

I have a file like this: ID A56 DS /A56 DS AGE 56 And I'd like to print the whole line only if the second column starts with a capital letter. Expected output: ID A56 DS AGE 56 What I've ...