awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

0
votes
3answers
71 views

Fixing header and print

I have header starting with '>' and I want fix the header by keeping first word and removing other shown as in output.txt and print it input.txt >AGAJ01065549.1 ...
0
votes
3answers
847 views

Split and print

I have tab-demlimted text file with 50,000lines. I want to split the first column and print as shown in output.txt Input.txt rt|371443144|mb|MN556661.1| 2200443 A rt|371443344|mb|MN556645.1| ...
5
votes
2answers
2k views

Replacing missing value blank space with zero

I have input.txt tab-delimited text file around 30K lines, I would like to check each row (s1..s30K lines) for missing value (i.e blank space) and fill the missing value with zero value.See out.txt ...
2
votes
1answer
202 views

Count occurrences where two fields are equal

Is it possible to write an awk one-liner which will compare two fields on the same line to see if they are equal, and output the number of lines where this is true? I tried: awk '$1==$2 { print($1 ...
6
votes
5answers
1k views

Find images by size: find / file / awk

I've been trying to find png image files a certain height (over 500px). I know that file will return image dimensions. Example: $ file TestImg1a.png TestImg1a.png: PNG image data, 764 x 200, ...
1
vote
2answers
326 views

Print range of files based on time range using awk?

I have done a few simple things with awk but I cannot find a way to do this! I have a text file containing entries named with year, julian date and time as such: blah_2012251_130500_blah ...
1
vote
3answers
2k views

Find common elements in a given column from two files and output the column values from each file

I have two files with tab-seperated values that look like this: file1: A 1 B 3 C 1 D 4 file2: E 1 B 3 C 2 A 9 I would like to find rows between files 1 and 2 where the ...
1
vote
4answers
633 views

Replace text between brackets

I'm using awk '{ gsub(/BAR|WIBBLE/, "FOO"); print }' to replace text in data like: SOMETHING [BAR, WIBBLE] SOMETHING [BAR] This gives the desired result of: SOMETHING [FOO, FOO] SOMETHING [FOO] ...
2
votes
3answers
508 views

Regex/pattern matching problem to extract data

What is the best way to remove distances in the following lines and output cdv format. Can this be done reasonably easily on the command line? I'd like to avoid having to write a full-blow script. ...
3
votes
3answers
132 views

Print only lines that are completely numeric

I'd like to filter through a text file and only print the lines where each column is a valid floating point number. For example: 3 6 2 -4.2 21.2 3 x 4.2 21.2 3 2 2.2.2 Only the first line would ...
1
vote
1answer
214 views

Combining files with 4 columns each, into 4 files for each column [closed]

So I have a bunch of text files which look like this: T1 r1 a b c d r2 d e f g ... T2 r1 h i j k r2 l m n o ... And I want to make FOUR different text files in the following ...
1
vote
2answers
132 views

Extracting a multiline regex without leading tabs

I've been trying to hack a little code extraction script together, but I can't get it to work. My goal is to examine all .txt files in a directory. If it contains a line which doesn't start with a ...
1
vote
5answers
214 views

frequency of occurence

I have a tab-delimited text file, that is around 3K lines long. I want to calculate frequency of data appearing in it, in 10,000 ranges. Input.txt Price 500 1500 10001 15000 17000 ...
2
votes
4answers
194 views

Output some sections of a text file [closed]

I have input like the following: >gi|481443199|gb|MH447771.1| 49 0 0 0 1 50 0 0 1 0 >gi|987078004|gb|RTYJ01067061.1| 412 0 0 1 ...
1
vote
2answers
322 views

List info about files/directories with spaces

I am trying to use ls to get information about files and directories. My current command fails to properly get the name whenever a file/directory has a space. I am using this to list all ...
2
votes
4answers
744 views

Change currency format to generic number using sed

I am working on a large csv file with all coma separated entries. The entire document is composed of plain numbers, except for a few columns reflecting currencies that are formatted as text. For ...
5
votes
2answers
477 views

Use a script parameter in awk

Here is my script (to find the files that contain a specified pattern) : find . -type f -exec awk -v vawk="$1" '/'"$vawk"'/ {c++} c>0 {print ARGV[1] ; exit 0 } END { if (! c) {exit 1}}' \{\} \; ...
1
vote
3answers
3k views

bash merge 2 lists removing duplicated

I have an embedded linux system using Busybox (OpenWRT) - so commands are limited. I have two files that look like: first file aaaaaa bbbbbb cccccc mmmmmm nnnnnn second file mmmmmm nnnnnn ...
3
votes
3answers
1k views

Average rows with same first column

Given a file with two columns: Id ht 510 69 510 67 510 65 5l0 62 510 59 601 29 601 26 601 21 601 20 I need a way to coalesce all rows with the same ID into one that has an average height. In this ...
4
votes
4answers
3k views

AWK - how to reduce the greed of a regular expression?

I have an example: echo "@article{gjn2010jucs, Author = {Grzegorz J. Nalepa}, " | awk '{ sub(/@.*,/,""); print }' Is it possible to write a regular expression that selects the shorter pattern? ...
1
vote
1answer
1k views

Variable Substitution in Awk Print Statement -v

I've spent over an hour poking at this. It can't be this hard... I want to print a column of data from one file to another file. The column I want to print is dependent on what is passed in and stuff ...
2
votes
1answer
111 views

Extracting some files from a folder and creating a list of these files

I am just a beginner with Unix and so my skills in Unix commands is limited. I am currently dealing with a folder containing about 1000 files and I have to extract some filenames from this folder and ...
2
votes
3answers
4k views

Extracting column from comma separated text

I have a long comma-separated delimited file with 20K lines. Here's a sample: "","id","number1","number2","number3","number4","number5","number6","number7" ...
4
votes
1answer
2k views

Awk to remove line if argument is encountered in a specific column

I need to cycle through an entire file of unknown size and remove any line in which a given word (passed in as argument 1) appears in a specified column. In addition, I need to keep track of how many ...
1
vote
4answers
249 views

How to process these DHCP logs to be human readable?

I have this input: Sep 23 13:43 192.168.6.200 Sep 23 13:44 192.168.6.166 Sep 23 13:45 192.168.6.200 Sep 23 13:46 192.168.6.166 Sep 23 13:47 192.168.6.200 Sep 23 13:48 192.168.6.166 Sep 23 13:49 ...
4
votes
1answer
2k views

How to find out common elements between two files?

For an example, I have 2 files having following info: File #1: 12 13 14 15 File #2: 12 1 13 2 14 2 15 6 16 7 17 8 Output File: 1 2 2 6 In the output file, I want only the second column ...
5
votes
2answers
120 views

Why does awk execute both actions?

When I run the command: awk '/from/ {print $7} /to/ { print $7}' erroMuitoDoido.txt The file is: May 19 04:44:43 server postfix/smtpd[32595]: CDAB515013: ...
3
votes
1answer
1k views

File size limit exceeded in bash [closed]

I have tried this shell script on a SUSE 10 server, kernel 2.6.16.60, ext3 filesystem. The script has a line like this: cat file | awk '{print $1" "$2" "$3}' | sort -n > result The file's size ...
8
votes
5answers
9k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
4
votes
6answers
351 views

What is the easiest way to check if column A and B values goes both ways with AWK? [closed]

What is the easiest way to check if column A and B values goes both ways? Output to check: Mike John John Mike Pamela Barbara Barbara Pamela Mike Paul Roger Paul Desired output Mike <-> ...
5
votes
1answer
2k views

Fast elimination of duplicate lines across multiple files

I have a huge amount of data in which each (data-)line should be unique. There are a lot of files in one folder in which this is already true. It is about 15GB splitted into roughly 170 files with ...
5
votes
4answers
3k views

Concatenate lines by first column (AWK or SED)

How can I use AWK in the following situation? I want to concatenate lines that start with the same column. Only the first column is kept after the join (in this case aaa, www, hhh). The file may be ...
0
votes
5answers
5k views

compare two files get identical list

file1.txt (50 lines) TERYUFV00000010753 TERYUFV00000009526 file2.txt (500 lines) TERYUFV00000009526 refids_739_known_8/10_target TERYUFV00000018907 refids_12023_known_21/22_target ...
3
votes
2answers
1k views

What is the best way to find a list of several strings within a large text file

The short, general question is: In Unix/Linux, what is the best way to find a list of several (about 150) strings within a large text file? I am asking this to all Unix/Linux experts as a general ...
0
votes
1answer
143 views

awk extended pattern matching (embedding pattern matching in actions for already matched strings) [closed]

I want handle strings of the form: PREFIX_TYPE_N,DATA So, does the *awk (gawk, mawk, nawk) support including pattern matching in the action for already matched string? Something like this (of ...
2
votes
3answers
116 views

Need to parse a double entry table with two pairs

I have this two-entry table resulting from a program output: Variable1 : Value1 Variable2 : Value2 Variable3 : Value3 Variable4 : Value4 Variable5 : Value5 Variable6 : Value6 ...
4
votes
2answers
446 views

AWK help for filling up rest of the columns with &nbsp in file

I'm trying to get AWK filling up following text file for "empty" columns". Basic idea of finding longest line with AWK NF and checking which line has no value on that column. Then adding x times N/A ...
16
votes
3answers
4k views

Piping from grep to awk not working

I am trying to grep the ongoing tail of file log and get the nth word from a line. Example file: $ cat > test.txt <<EOL Beam goes blah John goes hey Beam goes what? John goes forget it Beam ...
1
vote
3answers
1k views

how to sum output of awk or other expression with xargs

Suppose i have the following bash shell script: #!/bin/bash export count=0; for i in `ls ./mydoc` ;do pdfinfo ./mydoc/$i | egrep Pages |awk {'print $2'} |xargs -+ $count ; ...
0
votes
2answers
204 views

Getting data from matching list

list.txt GETID_17049_knownids_1/2_Confidence_0.625_Length_2532 GETID_9248_knownids_6/10_Confidence_0.439_Length_2474 GETID_11084_knownids_3/3_Confidence_0.600_Length_1451 ...
2
votes
2answers
381 views

AWK match string, not Regex

How do I match a string with AWK? Say I have the string [(*.+ it should only match strings containing the string [(*.+. The string is entered by the user, so if you know a cli utility that can ...
1
vote
4answers
4k views

sed/awk/perl - delete spaces between two strings/words + solaris 10

please advice how to delete the spaces between "=" to the value/string ( after the "=" character) , should be implemented by awk or sed or perl command and command must to fit for solaris OS , ( ...
8
votes
7answers
369 views

Swapping an unlimited number of columns

I have a file with columns. See below for an example: a b c ... z 1 2 3 ... 26 I'd like to swap all columns where the 1st becomes the last, the second becomes the one before last...etc.. z y x ...
3
votes
1answer
263 views

grep list of names and information from bigger file

I have two files: one with list of names (500 entries) and other having some more information for each entry in A.txt and extra entries too. File A.txt (each line is starting with > (fasta format) ...
4
votes
2answers
136 views

Column mismatch and substituting

input.txt (tab-delimted) TTTTOTTT00000000008 RTTTT899 5.00E-28 TTTTOTTT00000000046 RTTTWRR 3.00E-31 TTTTOTTT00000000051 2.00E-11 TTTTOTTT00000000051 7.00E-12 TTTTOTTT00000000054 ...
3
votes
3answers
932 views

reading entries from multiple files

I have few files like A.txt, B.txt and C.txt, each having a only single number in it i.e A.txt has 45 B.txt has 35 and C.txt has 100. How could I read these entries from each files and report with its ...
3
votes
2answers
780 views

merging two column

input_file (columns are tab delimited, only column 4 contains spaces) input A B C D;<>X;<>Y 0 C D E F;<>Z;<>X 1 output A B C D;<>X;<>Y;<>apple=0 C D ...
2
votes
3answers
123 views

Finding out how many programs I have not installed from a list with grep/awk in archlinux

I want to see how many packages from a list are not installed. I use archlinux, so I use pacman -Q to determine if a package is installed. If it is not installed, it returns a message like: "error: ...
2
votes
2answers
306 views

Get row data for non-matching column values

I want to read the file (1600 rows) and get rows only the columns have different values (sno1, sno2, sno3 & sno4-should be not be equal value) and it should be above 50%.The example of output ...
11
votes
1answer
3k views

Making a progressbar with “dialog” from rsync output

I'm looking for a way to filter/redirect rsync output in a manner where it can be fed to the "dialog --gauge" command, so I can get a nice looking progressbar during file sync. Currently I have only ...