awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

5
votes
6answers
192 views

Remove duplicates values within a field

How to remove duplicate (or more) values within a selected field in a file, keeping only one copy? Exemple Input file: A 1,2,3,45,1,8,2,3 B 5,6,6,6,6,6,2,3,7 Expected output: A ...
7
votes
3answers
790 views

What are the exact differences between awk and cut with grep? [closed]

We know that we can get the second column of the line we want from a file using these two techniques: awk '/WORD/ { print $2 }' filename or grep WORD filename| cut -f 2 -d ' ' My questions are: ...
2
votes
2answers
52 views

How to escape spaces etc in passed variable, for system call to cp in awk

As the title suggests, this problem has enough layers of complexity to make simple escaping characters difficult for me. I have a bash script which, for a large part, has an embedded awk script that ...
2
votes
7answers
83 views

Printing specific section of a line when a trigger value is present

I'm parsing data in the following format: prop1=value1:prop2=value2:prop3=value3+prop1=value4:prop2=value5 parts of the string are delimited by + properties can appear in any order the desired ...
3
votes
1answer
49 views

grep upto matching brackets

I have a file like this:- pin(ABC) { a b c d e f { abc } } pin(XYZ) { g h i j k { j k { ...
14
votes
10answers
2k views

There must be a better way to replace single newlines only?

I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
1
vote
2answers
36 views

awk - read program AND input from the standard input?

Is it possible for awk to read the program and the input from the standard input? I would like to be able to pipe a file to the following function. process_data () { awk -f - <<EOF {print} ...
6
votes
6answers
9k views

Show sum of file sizes in directory listing

The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current ...
1
vote
2answers
101 views

Extracting text string from file, assigning them to variables then writing them to another file?

Basically, its a game server (minecraft) and I have a situation where I am attempting to automate new world generation. Basically, I have a file that gets generated happily which has this snippet in ...
0
votes
2answers
43 views

remove duplicate entry from file1 if entry present in file2

I want to delete duplicate entry e from file1 if e is also present in file2. Input file1: x1 y1 x2 y2 x3 y3 x4 y4 y1 x1 x5 y5 y3 x3 x6 y6 x5 y5 Input file2: y1 x1 y2 x2 y3 x3 y4 x4 ...
3
votes
3answers
45 views

Patterns and file processing

Let's say I have to perform these actions from an input file: extract nth field from a line starting with a given pattern (in the exemple: 2nd field of the line starting with pattern 'name') ...
0
votes
2answers
122 views

AWK command failing for large file

I have a .csv file of 6.5 GB. I need to sort the files with 1st, 2nd, 3rd, 4th, 5th and 13th column and need to get the unique records. Also I need to get the duplicate records and redirect it to some ...
-2
votes
0answers
22 views

remove duplicates from file1 if entry present in file2 [duplicate]

Input file1: x1 y1 x2 y2 x3 y3 x4 y4 y1 x1 x5 y5 y3 x3 x6 y6 x5 y5 Input file2: y1 x1 y2 x2 y3 x3 y4 x4 x1 y1 y5 x5 x3 y3 y6 x6 x5 y5 Desired output: x1 y1 x2 y2 x3 y3 x4 ...
1
vote
2answers
28 views

rename a batch of files after reading from a source file

I have a directory full of over a thousand files and the names are bit long (and no pattern found in the naming style). I would like to simplify the filenames. E.g.: ...
0
votes
2answers
61 views

How match different column in file 1 and file2 with awk?

I have the following two files. The first file is : 3184 2014-07-28 04:15 global.Remote-Access 10.111.8.25 81.245.6.25 tcp 3268 3035 2014-07-28 04:16 global.Remote-Access 10.111.8.12 81.245.6.25 ...
1
vote
2answers
40 views

How can I sort a .txt file by two columns?

Feb 7 domainserver dovecot[37495]: auth(default): od(tjones): lookup failed for user: tjones Above is a sample output of what I'd like to sort. I'd like to sort by date on the first column, but ...
0
votes
1answer
59 views

grepping of common lines between two file based on specific columns values in case of multi matching

I want to match column3 of file1 & column2 of file2 , after that i want to print all column of matching lines in a single line, i know how to do it but the problem is that if there are not unique ...
0
votes
3answers
32 views

Finding and replacing text inside awk block

I want to find and replace a value inside awk block. Here is a sample input file to my script P1 10,9:11/18013013582 ,10:1 ,167:0 ,487:5/E.164 11,9:15/310410532169026 ,10:60 ,167:0 ,487:4/IMSI ...
2
votes
4answers
1k views

Split long output into two columns

Is there a simple utility or script to columnate the output from one of my scripts? I have data in some form: A aldkhasdfljhaf B klajsdfhalsdfh C salkjsdjkladdag D lseuiorlhisnflkc E sdjklfhnslkdfhn ...
0
votes
2answers
75 views

how to remove all double quotes from csv except specific field

I want to remove all double quotes from my csv but not the fourth field (because the four fields represent PATH of file) Please advice how to implement this by sed or awk or perl one liner , etc ...
3
votes
1answer
25 views

awk + gsub + how to work with all kind of characters

The following awk command target is remove spaces and unnecessary quotes from CSV file and only from single words between separators awk 'gsub(/("[ ]+|[ ...
3
votes
2answers
736 views

Replace a pattern in a file with a long string

I'm writing a little shell script to generate a directory listing. To make the output easily customizable, the script just builds a HTML-table and than should replace a specific token in a template ...
2
votes
2answers
70 views

Replace column entry with sum of other column entries

Let's say I have a file that contains these lines: 02.03.14 14.50 14.50 0.00 Desc 02.03.14 17.00 0.00 17.00 Desc 01.03.14 2.82 1.68 calc Desc 02.03.14 1.04 0.00 ...
1
vote
1answer
45 views

Combining awk statements

I'd like to combine those awk statements: awk -F'\t' 'NF>1 && $NF!=""' #prints every row with more than two fields awk '$1 ~ /^[0-9]/' #prints every row starting with a number awk ...
4
votes
4answers
126 views

Printing selective fields following 1st column instructions

Suppose I have a file in which I'd like to simultaneously print different awk commands following the first column instuctions, without messing up the original file (as it would happen with two ...
1
vote
2answers
80 views

Search for a string in a line and append related lines

I have a file containing some ID (identifier) and DS (description) lines. I'd like to keep the rows containing the string value in their ID line, as well as all the following (related) DS lines before ...
4
votes
2answers
192 views

Print lines if given column starts with a capital letter

I have a file like this: ID A56 DS /A56 DS AGE 56 And I'd like to print the whole line only if the second column starts with a capital letter. Expected output: ID A56 DS AGE 56 What I've ...
0
votes
2answers
51 views

How to print a string at the end of line using awk?

I am very new to awk. I am trying to add a new calculated column. But awk insert the column to the next line rather to the end of the current line. Here is an example of my data: ...
1
vote
2answers
39 views

Format output to a specific line length

So I have some output that has very long lines, and I want to have it formatted to lines no more than 80 columns wide, but I don't want to split words because they are in the column limit. I tried ...
10
votes
4answers
12k views

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string 'bar' in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
2
votes
3answers
132 views

Aggregate and group text file in bash

I have a big text file(with 5m lines) in this format(4 columns, separated by ;): string1; string2; string3; userId The first 3 strings (SHA1s) form a single ID, called appId (so it can be ...
1
vote
2answers
54 views

How to determine the newly closed file within a continuous audit log rotation?

I am attempting to find the best way to determine when the second file (of a matching criteria) is created. The context is an audit log rotation. Given a directory where audit logs are created every ...
1
vote
2answers
67 views

Assignment in braces vs outside braces in awk

From this question, I answer with this solution: awk 'FNR==NR{a[FNR]=$2;next}{$NF=a[FNR]}1' file2 file1 It works as expected. But the first solution I thought is: $ awk ...
2
votes
2answers
53 views

Merging columns from two separate files

How to create a new file merging selective columns from two separate files using awk? Without messing up the elements orders of BOTH files. Exemple: File 3 may contain column 1,2,3 from File 1 and ...
3
votes
1answer
72 views

Is there a way to pass a variable from an awk statement to a bash function as an argument? [closed]

I am trying to look a for some specific values from (such as 1 or 4 and 2 or 3) a text file using an awk statement in a bash script. If this value is found in the file (within the awk statement), then ...
6
votes
4answers
2k views

How to extract logs between two time stamps

I want to extract all logs between two timestamps. Some lines may not have the timestamp, but I want those lines also. In short, I want every line that falls under two time stamps. My log structure ...
3
votes
5answers
354 views

Print last element of each row

How can one print the element after the last tab in a file? Exemple: File1 A 3 8 6 7 B 4 6 2 3 6 8 c 1 9 would return: 7 8 9
4
votes
3answers
265 views

awk with if statements

I am trying to print from a file using awk, but my output is empty. Here is my code so far accountNum=$1 while read -r LINE || [[ -n $LINE ]] ; do awk -F',' '{ if($1==accountNum) { print ...
5
votes
1answer
2k views

Removing leading zeros from DATE

I made an alias of the date command to display date in the following format: 2013.06.14.12.10.02 using this command: alias date = date +"%Y.%m.%d.%H.%M.%S" Everything works great, except I ...
0
votes
2answers
184 views

Awk printf number in width and round it up

I need to printf a number out, but with given width and rounded (with awk!) %10s I have this and somehow I need to connect the %d but everything I do, ends up with too much parametres for awk ...
6
votes
2answers
218 views

Number of comma-separated fields in a text file

I'm trying to build an awk statement to read this file: A 1,2,3 * A 4,5,6 ** B 1 B 4,5 * and build a file like this: A 1,2,3 * 3 1 0.333 A 4,5,6 ** 3 2 0.666 B ...
2
votes
1answer
55 views

Backslash with pipe in system command from awk

I want to invoke shell command from awk so I am using system() but when I am passing expression in I am getting error at backslash. Here is my expression echo "select * from abc limit 1;"| ...
5
votes
1answer
38 views

Count duplicate rows with constraint

I have the following file: A B 1990 A C 2001 A C 2013 A B 2001 B C 2013 A D 2001 A D 2014 I need to count duplicate pairs (in the 1st and 2nd column) and to each such pair assign the lowest value ...
0
votes
1answer
40 views

Check values in a text file

Is there a simple way to check if values in the second column of a file ( File 2) exist in the second column of another file (File 1), for example by printing a star next to the concerned line? If ...
3
votes
1answer
32 views

Calculate intervals

How to explicit the intervals between numerical values in a file? (better if an awk one-liner) Input: A 1-3 B 5-7 Output A 1,2,3 B 5,6,7 Code I've tried so far: sed -i 's/-/\t/g' ...
4
votes
2answers
107 views

Get specific line from command output in awk

I have used pipeline to read specific record from impala shell.Here is what I got [cloudera@localhost ~]$ echo "select * from abc where key > 'a-26052014015400' limit 1;" | impala-shell Starting ...
0
votes
1answer
23 views

Split values rewriting key and value in newline

File1.txt A 1 2 3 File2.txt A 1 A 2 A 3 File 2 can be converted into File 1 using the following code (the first column is the primary key): cat File2.txt| awk '{line="";for (i = 2; ...
1
vote
3answers
32 views

Delete whole line if after “:” less than 4 chars

I have a file with many lines, such as : blue:fish red:tomato black:moon I would like the whole line to be deleted if after ":", there is less than 4 chars. Thanks for your help!
0
votes
1answer
33 views

Sorting file by second column

How can I manipulate a two columns tab-separated text file by sorting by the first element of the second column (only if the first column element is the same)? Exemple: Input File 1 A 1-2 A ...
1
vote
2answers
82 views

merge csv files, escape comma inside quotes

I have 3 csv files I want to join by first column (id column) Each file has the same 3 columns Row example : id | timestamp | Name 3792318, 2014-07-15 00:00:00, "A, B" When I join ...