awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

0
votes
2answers
10 views

rename a batch of files after reading from a source file

I have a directory full of over a thousand files and the names are bit long (and no pattern found in the naming style). I would like to simplify the filenames. E.g.: ...
1
vote
2answers
31 views

How can I sort a .txt file by two columns?

Feb 7 domainserver dovecot[37495]: auth(default): od(tjones): lookup failed for user: tjones Above is a sample output of what I'd like to sort. I'd like to sort by date on the first column, but ...
1
vote
1answer
28 views

grepping of common lines between two file based on specific columns values in case of multi matching

I want to match column3 of file1 & column2 of file2 , after that i want to print all column of matching lines in a single line, i know how to do it but the problem is that if there are not unique ...
0
votes
2answers
30 views

How match different column in file 1 and file2 with awk?

I have the following two files. The first file is : 3184 2014-07-28 04:15 global.Remote-Access 10.111.8.25 81.245.6.25 tcp 3268 3035 2014-07-28 04:16 global.Remote-Access 10.111.8.12 81.245.6.25 ...
2
votes
2answers
30 views

Patterns and file processing

Let's say I have to perform these actions from an input file: extract nth field from a line starting with a given pattern (in the exemple: 2nd field of the line starting with pattern 'name') ...
0
votes
2answers
42 views

how to remove all double quotes from csv except specific field

I want to remove all double quotes from my csv but not the fourth field (because the four fields represent PATH of file) Please advice how to implement this by sed or awk or perl one liner , etc ...
2
votes
1answer
13 views

awk + gsub + how to work with all kind of characters

The following awk command target is remove spaces and unnecessary quotes from CSV file and only from single words between separators awk 'gsub(/("[ ]+|[ ...
2
votes
2answers
64 views

Replace column entry with sum of other column entries

Let's say I have a file that contains these lines: 02.03.14 14.50 14.50 0.00 Desc 02.03.14 17.00 0.00 17.00 Desc 01.03.14 2.82 1.68 calc Desc 02.03.14 1.04 0.00 ...
1
vote
1answer
40 views

Combining awk statements

I'd like to combine those awk statements: awk -F'\t' 'NF>1 && $NF!=""' #prints every row with more than two fields awk '$1 ~ /^[0-9]/' #prints every row starting with a number awk ...
4
votes
4answers
110 views

Printing selective fields following 1st column instructions

Suppose I have a file in which I'd like to simultaneously print different awk commands following the first column instuctions, without messing up the original file (as it would happen with two ...
4
votes
2answers
163 views

Print lines if given column starts with a capital letter

I have a file like this: ID A56 DS /A56 DS AGE 56 And I'd like to print the whole line only if the second column starts with a capital letter. Expected output: ID A56 DS AGE 56 What I've ...
1
vote
2answers
74 views

Search for a string in a line and append related lines

I have a file containing some ID (identifier) and DS (description) lines. I'd like to keep the rows containing the string value in their ID line, as well as all the following (related) DS lines before ...
0
votes
2answers
35 views

How to print a string at the end of line using awk?

I am very new to awk. I am trying to add a new calculated column. But awk insert the column to the next line rather to the end of the current line. Here is an example of my data: ...
1
vote
2answers
33 views

Format output to a specific line length

So I have some output that has very long lines, and I want to have it formatted to lines no more than 80 columns wide, but I don't want to split words because they are in the column limit. I tried ...
2
votes
3answers
93 views

Aggregate and group text file in bash

I have a big text file(with 5m lines) in this format(4 columns, separated by ;): string1; string2; string3; userId The first 3 strings (SHA1s) form a single ID, called appId (so it can be ...
3
votes
1answer
62 views

Is there a way to pass a variable from an awk statement to a bash function as an argument? [closed]

I am trying to look a for some specific values from (such as 1 or 4 and 2 or 3) a text file using an awk statement in a bash script. If this value is found in the file (within the awk statement), then ...
1
vote
2answers
43 views

How to determine the newly closed file within a continuous audit log rotation?

I am attempting to find the best way to determine when the second file (of a matching criteria) is created. The context is an audit log rotation. Given a directory where audit logs are created every ...
3
votes
5answers
342 views

Print last element of each row

How can one print the element after the last tab in a file? Exemple: File1 A 3 8 6 7 B 4 6 2 3 6 8 c 1 9 would return: 7 8 9
1
vote
2answers
60 views

Assignment in braces vs outside braces in awk

From this question, I answer with this solution: awk 'FNR==NR{a[FNR]=$2;next}{$NF=a[FNR]}1' file2 file1 It works as expected. But the first solution I thought is: $ awk ...
2
votes
2answers
33 views

Merging columns from two separate files

How to create a new file merging selective columns from two separate files using awk? Without messing up the elements orders of BOTH files. Exemple: File 3 may contain column 1,2,3 from File 1 and ...
4
votes
3answers
245 views

awk with if statements

I am trying to print from a file using awk, but my output is empty. Here is my code so far accountNum=$1 while read -r LINE || [[ -n $LINE ]] ; do awk -F',' '{ if($1==accountNum) { print ...
6
votes
2answers
188 views

Number of comma-separated fields in a text file

I'm trying to build an awk statement to read this file: A 1,2,3 * A 4,5,6 ** B 1 B 4,5 * and build a file like this: A 1,2,3 * 3 1 0.333 A 4,5,6 ** 3 2 0.666 B ...
5
votes
1answer
37 views

Count duplicate rows with constraint

I have the following file: A B 1990 A C 2001 A C 2013 A B 2001 B C 2013 A D 2001 A D 2014 I need to count duplicate pairs (in the 1st and 2nd column) and to each such pair assign the lowest value ...
2
votes
1answer
43 views

Backslash with pipe in system command from awk

I want to invoke shell command from awk so I am using system() but when I am passing expression in I am getting error at backslash. Here is my expression echo "select * from abc limit 1;"| ...
3
votes
1answer
31 views

Calculate intervals

How to explicit the intervals between numerical values in a file? (better if an awk one-liner) Input: A 1-3 B 5-7 Output A 1,2,3 B 5,6,7 Code I've tried so far: sed -i 's/-/\t/g' ...
0
votes
1answer
19 views

Split values rewriting key and value in newline

File1.txt A 1 2 3 File2.txt A 1 A 2 A 3 File 2 can be converted into File 1 using the following code (the first column is the primary key): cat File2.txt| awk '{line="";for (i = 2; ...
0
votes
1answer
33 views

Check values in a text file

Is there a simple way to check if values in the second column of a file ( File 2) exist in the second column of another file (File 1), for example by printing a star next to the concerned line? If ...
1
vote
3answers
27 views

Delete whole line if after “:” less than 4 chars

I have a file with many lines, such as : blue:fish red:tomato black:moon I would like the whole line to be deleted if after ":", there is less than 4 chars. Thanks for your help!
0
votes
1answer
19 views

Sorting file by second column

How can I manipulate a two columns tab-separated text file by sorting by the first element of the second column (only if the first column element is the same)? Exemple: Input File 1 A 1-2 A ...
1
vote
2answers
60 views

merge csv files, escape comma inside quotes

I have 3 csv files I want to join by first column (id column) Each file has the same 3 columns Row example : id | timestamp | Name 3792318, 2014-07-15 00:00:00, "A, B" When I join ...
1
vote
4answers
100 views

How can I output the difference between 2 files?

My file consists of the the following; roughly: username:username:username:username:username The above line continues to about 600 characters. I use the awk command in order to use it as an ...
2
votes
1answer
59 views

Doesn't awk print all lines if it is passed empty arguments

I thought that awk without arguments is equivalent to awk {print} file. But when I do: awk '' file Nothing is printed. Why?
3
votes
2answers
88 views

Get specific line from command output in awk

I have used pipeline to read specific record from impala shell.Here is what I got [cloudera@localhost ~]$ echo "select * from abc where key > 'a-26052014015400' limit 1;" | impala-shell Starting ...
4
votes
2answers
66 views

Get the maximum sum for a file on 2 keys

I have a file as below. key1 key2 key3 a1 2 l1 a1 2.5 l2 a2 2 l2 a3 2.5 l3 a3 2.1 l4 a3 2.2 l5 I am trying to find the maximum possible sum of key2 that satisfies the ...
3
votes
2answers
92 views

Counting comma separated characters in a row

How can I count how many elements are separated with comma in a given column of a file (tab_delimited), using awk or sed? Exemple: AC 1,23,5 DC 2,8,89 Would return: AC 3 DC 3
1
vote
1answer
25 views

awk regex matches wrong?

Playing around with awk I noticed this behavior: [root@ror6ax3 ~]# grep open * | awk '$2 ~ /opens*/ {print $0}' install.log:Installing openldap-2.4.23-32.el6_4.1.x86_64 install.log:Installing ...
1
vote
2answers
48 views

Merge csv files with conditions

I have 2 CSV files for each date (csv_2014_4_15 and csv_2014_4_16) , with basic structure and couple of unique columns as below. id,name,created_at,updated_at,other columns 12, joe, 2013-1-1 18:30, ...
2
votes
3answers
39 views

Merging multiple files by row

Is it possible to copy the rows of File1 and File2 in a new File4 following the instruction given by File3 by using a simple bash script (sed? awk?). File1: /*two or more columns, tab delimited*/ ...
2
votes
1answer
65 views

merge csv files by first column

I have 3 csv files like this. csv 1: 1,aaaa,bbb,2014-04-01 2,qwe,rty,2014-04-03 3,zxc,cvb,2014-04-05 csv 2: 2,j,k,2014-04-01 3,a,s,2014-04-04 5,g,h,2014-04-08 csv 3: 2,a,s,d,f,g,2014-04-01 ...
1
vote
2answers
72 views

Replace columns using a shell

How could I replace column 1 in File1 with column 2 in File2, when a match is found? File1 id1 36987 id2 75685 id3 12345 id4 12896 File2 id1 ID1 id2 ID2 id3 ID3 id5 ID5 I'd like to write this on ...
2
votes
1answer
71 views

Compare the values of a column with all the values in another column

I have 2 input files. Each line of File1 should be compared to each and every line of File2. Logic is: If Column1 of File1 does not match Column1 (all the values under it) of File2, then print the ...
7
votes
2answers
421 views

Conditional block vs conditional statement (if)

Say I have a file: PRO 1 GLN 5.55112e-17 ILE -6.245e-17 THR 5.55112e-17 I want every line that has a number unequal to 1 in the second column to change it to 0 and keep the rest. If I use if (i.e ...
5
votes
5answers
494 views

Why does awk not sum to 0

I have this file and I want to sum all the number in first column. Easy: awk '{s+=$1;print $1,s}' file -0.4157 -0.4157 0.0143 -0.4014 0.5973 0.1959 -0.5679 -0.372 -0.2041 -0.5761 0.7130 0.1369 ...
6
votes
4answers
143 views

How to insert lines with numbers from X to Y into another file after line Z?

I created two files: echo -e "1\n2\n3\n4\n5" > 123.txt echo -e "a\nb\nc\nd\ne" > abc.txt I want to get the file 123.txt with the following contents: 1 2 3 b c d 4 5 So in other words ...
5
votes
2answers
115 views

Is there a way to make this one-liner faster?

Context I have a directory of thousands of zip files that are dated in the form YYYYMMDD_hhmmss.zip and each about 300K. Within each zip file is about 400 xml files each about 3K. The problem I ...
5
votes
2answers
134 views

Sum in block with AWK (restart the sum when the pattern changes)

I have a file like this: A 100 A 200 A 300 #sum=600 B 400 B 500 #sum=900 A 600 A 700 A 800 #sum=2100 I would like the output to be: A 600 B 900 A 2100 C sum_of_C D sum_of_D I can do that with ...
0
votes
1answer
37 views

change date format and store in variable using awk

I have date in format YYYY-MM-DD in variable 'a'. I want to convert it in format YYYYMMDD and store in variable 'b'. The variable is in an awk variable. How can I do the same ?
1
vote
3answers
34 views

Sed + how to set parameters in sed command in order to display lines from text file

I find the following way to print line from file In this example we want to print the first line from the hosts file sed -n '1,1p' /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 ...
6
votes
5answers
227 views

Strategy to extract movies's name from this ununiform dataset?

I am working on a movie database problem to improve regular expressions, this is the problem I'm running into. My dataset looks like this: Movie Name (variable space and tabs) year Movie1(can ...
2
votes
2answers
46 views

How can I compare two files based on the value of the first column?

I want to compare two files based on the value of their first column: file1 aaa 23 3 bbb 56 6 ccc 45 2 file2 bbb m a aaa a m ccc m m The output file should be: aaa a m bbb m a ccc m m ...