awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

0
votes
3answers
46 views

Replace specific text only in specific fields using nawk

My input data file abc.txt has this "20131008","0004","0568","98" I ran this command to replace the text nawk -F, -vOFS=',' '{gsub("0568","0808",$3); print }' abc.txt In the output the ...
5
votes
3answers
132 views

find length of longest line in all text files in a directory

I know how to get length of the longest line in a text file with awk awk ' { if ( length > L ) { L=length} }END{ print L}' file.txt but how can I get the length of the longest line of all files ...
5
votes
4answers
221 views

How to replace all the first occurences in matched lines with incremental number?

I have a file like this ... 1562 first part 1563 H col3 H col4 1564 H col3 H col4 ... 3241 H col3 H col4 3242 third part ... I want to replace only the first H in every line to H#, with ...
2
votes
1answer
25 views

Get each column which contains multiple spaces?

My file: subnet 172.25.10.0 netmask 255.255.255.0 { option domain-name-servers 172.25.10.9, 8.8.8.8; default-lease-time 300; max-lease-time 7500; range dynamic-bootp 172.25.10.10 172.25.10.30 ; ...
3
votes
6answers
279 views

Convert string into date time stamp in gawk or awk

I want to convert following string (20140805234656) into date time stamp (2014-08-05 23:46:56).I am new to gawk and I don't know the exact syntax,how can I put - at every 5,8 and : at every 14,17 and ...
4
votes
6answers
290 views

want the last line when duplicate awk?

I have the file: key value blah blah blah blah blahblah man1 boy1 blah blah man1 boy2 man1 boy1 I do this to remove duplicate lines: awk '/man1/ { print $1,$2} ' file | awk '!x[$0]++' and the ...
3
votes
1answer
46 views

Handling dynamically changing column positions and splitting file

I have a unique problem. I have a file with following columns PersonId, AB1, AB2, AB3, DG1_B1,DG1_B2,DG2 1,0,1,0,3,5,7 2,0,3,1,3,7,4 I want to split this and create multiple files based on the ...
0
votes
5answers
52 views

re-arrange values in one column without affecting other columns using awk or sed

I have a csv file in following format: abc,124,123,2014-08-13,abc def,124,123,2014-08-13,abc ghi,124,123,2014-08-13,abc Now I need to modify only 5th column (2014-08-13) to produce output as given ...
2
votes
1answer
32 views

Move matched line above a different matched line using sed

I have something like dsadjakldjska safdsfdsfsf fdsfsf**STGN**fdsfsd fdsfkjldsf fdsff fdsfs**IRV**fdsfs and I want to get dsadjakldjska safdsfdsfsf fdsfkjldsf fdsff fdsfsf**STGN**fdsfsd ...
2
votes
2answers
52 views

Memory allocation for sparse array in awk

I have searched but didnt reach to any conclusion that when i defined an sparse array does it reserve all the contiguous memory upto maximum index or it allocate the memory at that particular index ...
2
votes
4answers
92 views

Insert new line after non matching pattern

I have following output from script host1:101 host1:102 host2:101 host2:102 host2:103 host4:101 host5:101 host5:102 host5:103 host5:104 etc.. I wanted to insert a new line after each host group. ...
2
votes
6answers
457 views

Handling comma in string values in a CSV file [duplicate]

I have a comma-separated file which has numeric and string columns. String columns are quoted and can have comma in between the quotes. How do I identify the columns with FS =","? Sample records ...
4
votes
3answers
351 views

pipe line with non-deterministic output

I have happened upon a command that sometimes works and sometimes does not, even when executed multiple times in rapid succession in a bash shell (I have not tested the behavior in other shells). The ...
0
votes
2answers
100 views

Delete every number that comes before “=”

i have a very, very, very long list. Like over 830,000 characters. I will of course perform said command on a copy. There are multiple lines, like this hey1=lol hey2=lol hey3=lol I want that to ...
1
vote
2answers
37 views

cygwin awk output to windows line delimeters

I could find the way to print zip entries' list and forward to a txt file. unzip -l my.zip | awk '{print $4}' > my.txt When I open the file my.txt in windows' notepad the line delimiters ignored ...
5
votes
3answers
529 views

How to understand the command awk '{for(x=1;$x;++x)print $x}'?

I know this command awk '{for(x=1;$x;++x)print $x}' will print out all columns in a line. wouldn't this ++x change x to 2, and thus print $2 first? As I understood based on this: ...
2
votes
2answers
163 views

awk character class mystery

I seem to be unable to use a character class for an awk regular expression, almost exactly as described here : user@host:~$ awk -W version mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan ...
-3
votes
2answers
80 views

Extract certain column from CSV files [closed]

I have a directory with a group of CSV files. All files have the same column headers. I need to extract values in a certain column from all files. The common part in files names is ...
0
votes
2answers
117 views

Rearrange the file content

I have a file with content as 123 OneTwoThree 234 TwoThreeFour 345 ThreeFourFive I want to convert this file into something like this OneTwoThree 123 TwoThreeFour 234 ThreeFourFive 345 Here ...
5
votes
6answers
212 views

Remove duplicates values within a field

How to remove duplicate (or more) values within a selected field in a file, keeping only one copy? Exemple Input file: A 1,2,3,45,1,8,2,3 B 5,6,6,6,6,6,2,3,7 Expected output: A ...
2
votes
2answers
60 views

How to escape spaces etc in passed variable, for system call to cp in awk

As the title suggests, this problem has enough layers of complexity to make simple escaping characters difficult for me. I have a bash script which, for a large part, has an embedded awk script that ...
0
votes
3answers
59 views

awk pattern matching and merge files

I have 3 files 1.csv and 2.csv and 3.csv 1.csv TELECOM_DEVELOPMENT_AFGHANISTAN_COMPANY,AFGHANISTAN,Alphanumeric_A_MSISDN_blocking,1 ...
2
votes
5answers
178 views

extracting/copy ranges of columns from large data file

I was trying to prepare data files from one large file by extracting columns and pasting into a new file. The number of columns of the given data is very large. I want to prepare a number of data ...
1
vote
2answers
38 views

awk - read program AND input from the standard input?

Is it possible for awk to read the program and the input from the standard input? I would like to be able to pipe a file to the following function. process_data () { awk -f - <<EOF {print} ...
5
votes
4answers
211 views

How to replace m part of Lines with n part of Lines?

I have a file "test" like this: RewiteEngine On RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.7$ [NC] RewriteCond %{REMOTE_ADDR} !^192\.168\.1\.8$ [NC] RewriteCond %{REMOTE_ADDR} ...
0
votes
4answers
84 views

Extract value between double quotes

My query is to extract the value between double quotes "". Sample input is : 10.219.41.68 - - - [11/Jun/2014:10:23:04 -0400] Sec:0 MicSec:1797 "GET /balancer-manager HTTP/1.1" 200 28980 "-" ...
1
vote
2answers
120 views

Extracting text string from file, assigning them to variables then writing them to another file?

Basically, its a game server (minecraft) and I have a situation where I am attempting to automate new world generation. Basically, I have a file that gets generated happily which has this snippet in ...
5
votes
1answer
104 views

Delete an array in awk

In awk, I can clear an array with a loop, making it an empty array, which is equivalent to deleting it. for (key in array) delete array[key]; Is there a simpler way? Can I completely delete an ...
3
votes
1answer
50 views

grep upto matching brackets

I have a file like this:- pin(ABC) { a b c d e f { abc } } pin(XYZ) { g h i j k { j k { ...
0
votes
2answers
48 views

remove duplicate entry from file1 if entry present in file2

I want to delete duplicate entry e from file1 if e is also present in file2. Input file1: x1 y1 x2 y2 x3 y3 x4 y4 y1 x1 x5 y5 y3 x3 x6 y6 x5 y5 Input file2: y1 x1 y2 x2 y3 x3 y4 x4 ...
1
vote
2answers
46 views

rename a batch of files after reading from a source file

I have a directory full of over a thousand files and the names are bit long (and no pattern found in the naming style). I would like to simplify the filenames. E.g.: ...
1
vote
2answers
48 views

How can I sort a .txt file by two columns?

Feb 7 domainserver dovecot[37495]: auth(default): od(tjones): lookup failed for user: tjones Above is a sample output of what I'd like to sort. I'd like to sort by date on the first column, but ...
0
votes
1answer
72 views

grepping of common lines between two file based on specific columns values in case of multi matching

I want to match column3 of file1 & column2 of file2 , after that i want to print all column of matching lines in a single line, i know how to do it but the problem is that if there are not unique ...
0
votes
2answers
67 views

How match different column in file 1 and file2 with awk?

I have the following two files. The first file is : 3184 2014-07-28 04:15 global.Remote-Access 10.111.8.25 81.245.6.25 tcp 3268 3035 2014-07-28 04:16 global.Remote-Access 10.111.8.12 81.245.6.25 ...
3
votes
3answers
46 views

Patterns and file processing

Let's say I have to perform these actions from an input file: extract nth field from a line starting with a given pattern (in the exemple: 2nd field of the line starting with pattern 'name') ...
0
votes
2answers
111 views

how to remove all double quotes from csv except specific field

I want to remove all double quotes from my csv but not the fourth field (because the four fields represent PATH of file) Please advice how to implement this by sed or awk or perl one liner , etc ...
3
votes
1answer
35 views

awk + gsub + how to work with all kind of characters

The following awk command target is remove spaces and unnecessary quotes from CSV file and only from single words between separators awk 'gsub(/("[ ]+|[ ...
2
votes
2answers
76 views

Replace column entry with sum of other column entries

Let's say I have a file that contains these lines: 02.03.14 14.50 14.50 0.00 Desc 02.03.14 17.00 0.00 17.00 Desc 01.03.14 2.82 1.68 calc Desc 02.03.14 1.04 0.00 ...
1
vote
1answer
46 views

Combining awk statements

I'd like to combine those awk statements: awk -F'\t' 'NF>1 && $NF!=""' #prints every row with more than two fields awk '$1 ~ /^[0-9]/' #prints every row starting with a number awk ...
4
votes
4answers
143 views

Printing selective fields following 1st column instructions

Suppose I have a file in which I'd like to simultaneously print different awk commands following the first column instuctions, without messing up the original file (as it would happen with two ...
4
votes
2answers
226 views

Print lines if given column starts with a capital letter

I have a file like this: ID A56 DS /A56 DS AGE 56 And I'd like to print the whole line only if the second column starts with a capital letter. Expected output: ID A56 DS AGE 56 What I've ...
1
vote
2answers
81 views

Search for a string in a line and append related lines

I have a file containing some ID (identifier) and DS (description) lines. I'd like to keep the rows containing the string value in their ID line, as well as all the following (related) DS lines before ...
0
votes
2answers
80 views

How to print a string at the end of line using awk?

I am very new to awk. I am trying to add a new calculated column. But awk insert the column to the next line rather to the end of the current line. Here is an example of my data: ...
30
votes
4answers
2k views

Reverse grepping

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
1
vote
2answers
42 views

Format output to a specific line length

So I have some output that has very long lines, and I want to have it formatted to lines no more than 80 columns wide, but I don't want to split words because they are in the column limit. I tried ...
2
votes
3answers
162 views

Aggregate and group text file in bash

I have a big text file(with 5m lines) in this format(4 columns, separated by ;): string1; string2; string3; userId The first 3 strings (SHA1s) form a single ID, called appId (so it can be ...
3
votes
1answer
78 views

Is there a way to pass a variable from an awk statement to a bash function as an argument? [closed]

I am trying to look a for some specific values from (such as 1 or 4 and 2 or 3) a text file using an awk statement in a bash script. If this value is found in the file (within the awk statement), then ...
1
vote
2answers
54 views

How to determine the newly closed file within a continuous audit log rotation?

I am attempting to find the best way to determine when the second file (of a matching criteria) is created. The context is an audit log rotation. Given a directory where audit logs are created every ...
3
votes
5answers
364 views

Print last element of each row

How can one print the element after the last tab in a file? Exemple: File1 A 3 8 6 7 B 4 6 2 3 6 8 c 1 9 would return: 7 8 9
1
vote
2answers
72 views

Assignment in braces vs outside braces in awk

From this question, I answer with this solution: awk 'FNR==NR{a[FNR]=$2;next}{$NF=a[FNR]}1' file2 file1 It works as expected. But the first solution I thought is: $ awk ...