awk - pattern-directed scanning and processing language
5
votes
2answers
107 views
Why does awk execute both actions?
When I run the command:
awk '/from/ {print $7} /to/ { print $7}' erroMuitoDoido.txt
The file is:
May 19 04:44:43 server postfix/smtpd[32595]: CDAB515013: ...
3
votes
1answer
498 views
File size limit exceeded in bash [closed]
I have tried this shell script on a SUSE 10 server, kernel 2.6.16.60, ext3 filesystem.
The script has a line like this:
cat file | awk '{print $1" "$2" "$3}' | sort -n > result
The file's size ...
7
votes
5answers
2k views
Remove comma between the quotes only in a comma delimited file
I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row
123,"ABC, DEV 23",345,534.202,NAME
I need to ...
4
votes
6answers
249 views
What is the easiest way to check if column A and B values goes both ways with AWK? [closed]
What is the easiest way to check if column A and B values goes both ways?
Output to check:
Mike John
John Mike
Pamela Barbara
Barbara Pamela
Mike Paul
Roger Paul
Desired output
Mike <-> ...
5
votes
1answer
778 views
Fast elimination of duplicate lines across multiple files
I have a huge amount of data in which each (data-)line should be unique.
There are a lot of files in one folder in which this is already true. It is about 15GB splitted into roughly 170 files with ...
4
votes
4answers
1k views
Concatenate lines by first column (AWK or SED)
How can I use AWK in the following situation?
I want to concatenate lines that start with the same column. Only the first column is kept after the join (in this case aaa, www, hhh).
The file may be ...
0
votes
5answers
1k views
compare two files get identical list
file1.txt (50 lines)
TERYUFV00000010753
TERYUFV00000009526
file2.txt (500 lines)
TERYUFV00000009526 refids_739_known_8/10_target
TERYUFV00000018907 refids_12023_known_21/22_target
...
3
votes
2answers
345 views
What is the best way to find a list of several strings within a large text file
The short, general question is: In Unix/Linux, what is the best way to find a list of several (about 150) strings within a large text file?
I am asking this to all Unix/Linux experts as a general ...
0
votes
1answer
104 views
awk extended pattern matching (embedding pattern matching in actions for already matched strings) [closed]
I want handle strings of the form:
PREFIX_TYPE_N,DATA
So, does the *awk (gawk, mawk, nawk) support including pattern matching in the action for already matched string? Something like this (of ...
2
votes
3answers
81 views
Need to parse a double entry table with two pairs
I have this two-entry table resulting from a program output:
Variable1 : Value1
Variable2 : Value2
Variable3 : Value3
Variable4 : Value4
Variable5 : Value5
Variable6 : Value6
...
4
votes
2answers
195 views
AWK help for filling up rest of the columns with   in file
I'm trying to get AWK filling up following text file for "empty" columns".
Basic idea of finding longest line with AWK NF and checking which line has no value on that column. Then adding x times N/A ...
14
votes
3answers
880 views
Piping from grep to awk not working
I am trying to grep the ongoing tail of file log and get the nth word from a line. Example file:
$ cat > test.txt <<EOL
Beam goes blah
John goes hey
Beam goes what?
John goes forget it
Beam ...
1
vote
3answers
555 views
how to sum output of awk or other expression with xargs
Suppose i have the following bash shell script:
#!/bin/bash
export count=0;
for i in `ls ./mydoc` ;do
pdfinfo ./mydoc/$i | egrep Pages |awk {'print $2'} |xargs -+ $count ;
...
0
votes
2answers
169 views
Getting data from matching list
list.txt
GETID_17049_knownids_1/2_Confidence_0.625_Length_2532
GETID_9248_knownids_6/10_Confidence_0.439_Length_2474
GETID_11084_knownids_3/3_Confidence_0.600_Length_1451
...
2
votes
2answers
216 views
AWK match string, not Regex
How do I match a string with AWK?
Say I have the string [(*.+ it should only match strings containing the string [(*.+.
The string is entered by the user, so if you know a cli utility that can ...
1
vote
3answers
1k views
sed/awk/perl - delete spaces between two strings/words + solaris 10
please advice how to delete the spaces between "=" to the value/string ( after the "=" character) , should be implemented by awk or sed or perl command and command must to fit for solaris OS , ( ...
5
votes
7answers
251 views
Swapping an unlimited number of columns
I have a file with columns. See below for an example:
a b c ... z
1 2 3 ... 26
I'd like to swap all columns where the 1st becomes the last, the second becomes the one before last...etc..
z y x ...
3
votes
1answer
186 views
grep list of names and information from bigger file
I have two files: one with list of names (500 entries) and other having some more information for each entry in A.txt and extra entries too.
File A.txt (each line is starting with > (fasta format) ...
3
votes
2answers
100 views
Column mismatch and substituting
input.txt (tab-delimted)
TTTTOTTT00000000008 RTTTT899 5.00E-28
TTTTOTTT00000000046 RTTTWRR 3.00E-31
TTTTOTTT00000000051 2.00E-11
TTTTOTTT00000000051 7.00E-12
TTTTOTTT00000000054 ...
2
votes
3answers
168 views
reading entries from multiple files
I have few files like A.txt, B.txt and C.txt, each having a only single number in it i.e A.txt has 45 B.txt has 35 and C.txt has 100. How could I read these entries from each files and report with its ...
3
votes
2answers
283 views
merging two column
input_file (columns are tab delimited, only column 4 contains spaces)
input
A B C D;<>X;<>Y 0
C D E F;<>Z;<>X 1
output
A B C D;<>X;<>Y;<>apple=0
C D ...
1
vote
3answers
96 views
Finding out how many programs I have not installed from a list with grep/awk in archlinux
I want to see how many packages from a list are not installed. I use archlinux, so I use pacman -Q to determine if a package is installed. If it is not installed, it returns a message like: "error: ...
2
votes
2answers
189 views
Get row data for non-matching column values
I want to read the file (1600 rows) and get rows only the columns have different values (sno1, sno2, sno3 & sno4-should be not be equal value) and it should be above
50%.The example of output ...
11
votes
1answer
1k views
Making a progressbar with “dialog” from rsync output
I'm looking for a way to filter/redirect rsync output in a manner where it can be fed to the "dialog --gauge" command, so I can get a nice looking progressbar during file sync.
Currently I have only ...
2
votes
4answers
2k views
awk-Printing column value without new line and adding comma
input.txt
EN1
EN2
EN3
EN4
EN5
output
EN1,EN2,EN3,EN4,EN5
I have tried awk.But it is not printing with comma
awk 'BEGIN { OFS = ","} { printf $1}' input.txt
I have GNU Awk 4.0.0 version
3
votes
2answers
171 views
Print row contain maximum and minimum value
I need to read the file (contains 16K rows)and print the entire row if any of columns and all columns contains max value (100) and all columns contain min value (0).The ouput example is given
...
3
votes
2answers
1k views
AWK-Find maximum value in a row & print with header
I need to read a file, find the maximum value in each row, and print the ID of the row, the column the maximum value came from (sno#), and the value from the associated lc# column. If the maximum ...
3
votes
5answers
482 views
extract fields from “tail -f” of a syslog stream
Example line from syslog file:
Aug 1 10:25:50 10.10.10.1 id=firewall sn=XXXX time="2012-08-01 14:35:18 UTC" fw=x.x.x.x pri=6 c=1024 m=537 msg="Connection Closed" f=11 n=195273698 ...
4
votes
3answers
208 views
Print a line only if the next line does NOT contain a particular match
I am trying to search a log file for logged activities that did not complete. For example, I log a "Starting activity for ID 1234..." and if successful, the next line will be "Activity 1234 ...
1
vote
2answers
143 views
Matching values within columns
I have tab-delimited text file containing about more than 1500 rows.
input.txt
id sno1
EN1 Nucus_2158_mri_1/2_Co_1.0_Le_3104
EN2 Nucus_2158_mri_4/2_Co_1.0_Le_3104
EN3 ...
3
votes
2answers
1k views
Working with columns - awk and sed
I am trying to parse a text file that is generated by an expec script that
pulls down some information from a switch.
Here is a sample output:
192 0000.0000.0000 1/g23 Dynamic ...
3
votes
2answers
2k views
Join two files with matching columns
File1.txt
id No
gi|371443199|gb|JH556661.1| 7907290
gi|371443198|gb|JH556662.1| 7573913
gi|371443197|gb|JH556663.1| 7384412
gi|371440577|gb|JH559283.1| ...
3
votes
3answers
892 views
Piping in awk scripts
I'm trying to write a ls wrapper that uses awk to parse the output of ls -lhF. Right now I've split the program into two files - my_ls.sh and my_ls.awk. my_ls.sh's only purpose is to pipe the output ...
0
votes
1answer
326 views
Extract value from different column and replace in another column
I am a biologist, I need one more help in text file editing. I would like to get the help of computer experts
##fsdfsd
##sdd-ver gen 5.5.7
Xm Gen CDS 1 148 . + . ...
2
votes
1answer
187 views
text processing-perl
I am a biologist. I would like help of computer experts to help in text file editing.
Here is what the input file looks like:
##dsfsd2
##sdf-sdf sasg 5.6.3
gi34_ex Gen CDS 161 317 . ...
4
votes
2answers
2k views
Compare two file columns
I have long text files with space-delimited fields:
cat file1.txt
Id leng sal mon
25671 34343 56565 5565
44888 56565 45554 6868
23343 23423 26226 6224
77765 88688 87464 6848
66776 23343 63463 ...
4
votes
2answers
206 views
How can I get awk to print both matches and non-matches?
Basically, all I want to do is alter outputs if the input matches a filter, but the part being altered isn't what is being filtered (or I would use sed). The problem so far is that my awk is only ...
3
votes
2answers
256 views
Parsing a text-file table and aggregating information
I have long text file with the following columns, space-delimited:
Id Pos Ref Var Cn SF:R1 SR He Ho NC
cm|371443199 22 G A R Pass:8 0 1 0 0
...
8
votes
3answers
739 views
How to sort the string which combined with string + numeric using bash script?
This is the data what I want to sort. But sort treats the numeric to string, the data it no sorted as I expected.
/home/files/profile1
/home/files/profile10
/home/files/profile11
...
3
votes
6answers
835 views
Text file look-up by column
I have a file in this format:
[#] OWNER_NAME NAME SIZE
[6] Robottinosino Software 200
[42] Robottinosino Ideas worth zero 188
[12] ...
4
votes
2answers
213 views
only output when first columns are the same
INPUT:
[USER@NOTEBOOK ~/proba] find . -type f | xargs -I "{}" md5sum "{}" | sort
18b17ef2e55f6fd2deb044943a8a769d ./yes
2b00042f7481c7b056c4b410d28f33cf ./tt.txt
2b00042f7481c7b056c4b410d28f33cf ...
2
votes
4answers
399 views
shorten long lines in a log file
I keep a logfile of a program's output. The trouble is sometimes errors occur dumping very long lines of 7bit ascii data(encoded binary) I don't care to keep. These lines can be 200KB+ long before ...
3
votes
1answer
445 views
Make awk use bash with the system() command
Is there a way to make awk use bash instead of sh when running system commands using the system() call?
I want to use some bash-specific features such as [[ ]] and < > string comparison operators:
...
0
votes
1answer
2k views
Replacing one test string with another in particular column
I would like to replace "CC" with "C" and "AA" with A" in a particular column of Tab delimited file (using awk probably).
15
votes
9answers
956 views
Remove duplicate $PATH entries with awk command
I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my path environment.
I was told that it is possible to achieve this with a one line ...
3
votes
2answers
161 views
Subtracting one text string from another leaving unique characters, perhaps using awk
Is there any way to subtract one column of data containing text from another column containing text and get third column containing unique charcters, for example using awk eg.
Input
ab a
cd d ...
1
vote
1answer
518 views
Extract marked strings from text file using Bash
I have files which are of the following style - these are parametrised configuration files; the values within the #characters are replaced with real values from a database depending on environment.
...
-6
votes
1answer
171 views
sed and awk challenge [closed]
Here is a great challenge for someone to attack!
Is there a way to achieve the following using sed and/or awk. Please see myfile.txt below:
$cat myfile.txt:
entry
"entry-header"
entry-top
name
...
2
votes
3answers
637 views
Grepping several lines from Nmap output
I apologize in advance as this is a simple question.
So I'm attempting to scan a Class C to identify all mail servers by IP. The vast majority of IPs aren't mail servers. I'm trying to filter out the ...
1
vote
1answer
319 views
Append timestamp to top command output
All,
I'm using
top -l 0 -s 1 -pid XXX -stats rsize | awk 'NR%13==0; fflush(stdout)' > output.txt
to store the rsize value of a process in a file. Currently the contents of output.txt look ...
