awk - pattern-directed scanning and processing language
3
votes
3answers
898 views
Piping in awk scripts
I'm trying to write a ls wrapper that uses awk to parse the output of ls -lhF. Right now I've split the program into two files - my_ls.sh and my_ls.awk. my_ls.sh's only purpose is to pipe the output ...
0
votes
1answer
326 views
Extract value from different column and replace in another column
I am a biologist, I need one more help in text file editing. I would like to get the help of computer experts
##fsdfsd
##sdd-ver gen 5.5.7
Xm Gen CDS 1 148 . + . ...
4
votes
2answers
3k views
Compare two file columns
I have long text files with space-delimited fields:
cat file1.txt
Id leng sal mon
25671 34343 56565 5565
44888 56565 45554 6868
23343 23423 26226 6224
77765 88688 87464 6848
66776 23343 63463 ...
3
votes
2answers
256 views
Parsing a text-file table and aggregating information
I have long text file with the following columns, space-delimited:
Id Pos Ref Var Cn SF:R1 SR He Ho NC
cm|371443199 22 G A R Pass:8 0 1 0 0
...
4
votes
2answers
207 views
How can I get awk to print both matches and non-matches?
Basically, all I want to do is alter outputs if the input matches a filter, but the part being altered isn't what is being filtered (or I would use sed). The problem so far is that my awk is only ...
8
votes
6answers
220 views
Grab certain contents of a file
So I know tools exist for this problem because I've heard about them, but I don't know what they are.
I want to do something like filter out all data but the usernames in /etc/passwd.
For example, I ...
8
votes
3answers
742 views
How to sort the string which combined with string + numeric using bash script?
This is the data what I want to sort. But sort treats the numeric to string, the data it no sorted as I expected.
/home/files/profile1
/home/files/profile10
/home/files/profile11
...
3
votes
6answers
835 views
Text file look-up by column
I have a file in this format:
[#] OWNER_NAME NAME SIZE
[6] Robottinosino Software 200
[42] Robottinosino Ideas worth zero 188
[12] ...
4
votes
2answers
213 views
only output when first columns are the same
INPUT:
[USER@NOTEBOOK ~/proba] find . -type f | xargs -I "{}" md5sum "{}" | sort
18b17ef2e55f6fd2deb044943a8a769d ./yes
2b00042f7481c7b056c4b410d28f33cf ./tt.txt
2b00042f7481c7b056c4b410d28f33cf ...
2
votes
4answers
399 views
shorten long lines in a log file
I keep a logfile of a program's output. The trouble is sometimes errors occur dumping very long lines of 7bit ascii data(encoded binary) I don't care to keep. These lines can be 200KB+ long before ...
3
votes
1answer
445 views
Make awk use bash with the system() command
Is there a way to make awk use bash instead of sh when running system commands using the system() call?
I want to use some bash-specific features such as [[ ]] and < > string comparison operators:
...
3
votes
3answers
433 views
Using sed to edit lines in a file with a variable
I need to edit lines in a file using sed. Now the problem is I am replacing a particular pattern with a combination of text and number. This number is a variable which keeps on incrementing for every ...
0
votes
1answer
2k views
Replacing one test string with another in particular column
I would like to replace "CC" with "C" and "AA" with A" in a particular column of Tab delimited file (using awk probably).
3
votes
2answers
161 views
Subtracting one text string from another leaving unique characters, perhaps using awk
Is there any way to subtract one column of data containing text from another column containing text and get third column containing unique charcters, for example using awk eg.
Input
ab a
cd d ...
1
vote
1answer
518 views
Extract marked strings from text file using Bash
I have files which are of the following style - these are parametrised configuration files; the values within the #characters are replaced with real values from a database depending on environment.
...
-6
votes
1answer
171 views
sed and awk challenge [closed]
Here is a great challenge for someone to attack!
Is there a way to achieve the following using sed and/or awk. Please see myfile.txt below:
$cat myfile.txt:
entry
"entry-header"
entry-top
name
...
2
votes
3answers
637 views
Grepping several lines from Nmap output
I apologize in advance as this is a simple question.
So I'm attempting to scan a Class C to identify all mail servers by IP. The vast majority of IPs aren't mail servers. I'm trying to filter out the ...
1
vote
1answer
320 views
Append timestamp to top command output
All,
I'm using
top -l 0 -s 1 -pid XXX -stats rsize | awk 'NR%13==0; fflush(stdout)' > output.txt
to store the rsize value of a process in a file. Currently the contents of output.txt look ...
7
votes
2answers
982 views
A shell script for joining two files
I want to write a shell script that get two files A and B, and get a result like this:
File A:
user_a tel_a addr_a
user_b tel_b addr_b
File B:
process_1 user_a
process_2 user_a
process_3 user_b
...
2
votes
2answers
153 views
How to make a list and count LaTeX keywords in a source file?
I am a beginner, trying to make lists (and eventually count and make statistics) the LaTeX keywords appearing in some documents.
Basically a LaTeX "valid" keyword can be identified by:
The fact ...
2
votes
2answers
154 views
How do I ad an if statement (regarding punctuation in a word) to this bash script
I have this bash script
#!/bin/bash
cat $@ | while read line
do
for word in $line
do
echo $word | circling-the-square
# here's where i need to add the if statement:
...
6
votes
6answers
2k views
How can you combine all lines that end with a backslash character?
Using a common command line tool like sed or awk, is it possible to join all lines that end with a given character, like a backslash?
For example, given the file:
foo bar \
bash \
baz
dude \
happy
...
2
votes
2answers
377 views
How can I turn ugly output into pretty, useful information?
How can I turn this ugly output into pretty, useful data?
The output:
/* ---------- TA#box#AbC_p ---------- */
insert_job: TA#box#AbC_p job_type: a
#owner: bob
permission: gx
date_conditions: 1
...
1
vote
2answers
98 views
How to parse fields out of a text file and write them to another file
I have executed a database command and have generated an output file of the results.
Each field name starts with dbt_xxxxxx
Each field value is the value after the equal sign.
how can I parse the file ...
4
votes
3answers
236 views
How to parse a file to extract 3 digits numbers kept in a “group number”
I am a beginner, and trying to write a shell script to parse a text file that is extracted from a standardization pdf file. I would like for each test group (identified by Group 0, Group 1... to get ...
1
vote
1answer
561 views
AWK - change the file name (special characters)
I have two files:
~/Pulpit/kot$ find . -name "*jpg" -printf "%f\n"
1.jpg
`[~!($%^_&*){.}\___"`]`1.jpg
I want to change their name to:
_home_pic_Pulpit_kot_1.jpg
...
3
votes
1answer
630 views
escaping a single dot with double backslash - awk
"effective awk programming" book has an example on Field-Splitting. here is the example:
If you want fields to be separated by a literal period
followed by any single character, use ‘FS = ...
5
votes
2answers
105 views
Sort fields inline
I'm trying to sort within a line of input over an unknown number of fields:
Input:
ab bc
bc ab
cd ef bc
bc cd ef
cd bc ab
ef ab bc cd gh
Output:
ab bc
ab bc
bc cd ef
bc cd ef
ab cb cd
ab bc cd ...
0
votes
2answers
705 views
Using awk to process ls output with spaces in filenames/paths
All,
I have a script in which I use find command to filter out all png files in a given folder and list them along with their size. I want the output in the following format:
someFile.png => 1.2K ...
2
votes
3answers
1k views
How to delete files filtered out by awk
I have the following files in a directory:
-rw-r--r-- 1 smsc sys 46 Apr 22 12:09 bills.50.1.3G.MO.X.20120422120453.Z
-rw-r--r-- 1 smsc sys 28 Apr 22 12:15 bills.50.1.3G.MO.X.20120422120953.Z
...
0
votes
2answers
589 views
nmap script needed to output text into csv format
ultimately I am looking to insert some info into a database. So I'm trying to output a file from something (probably nmap) that is formatted like this:
10.1.150.1,Up,1/3/2012,host.domain.lan
...
3
votes
3answers
261 views
swapping fields in columns by fixed pattern (yes, no, no, yes)
I want to produce a following list of pages (for imposition work); (number of pages is variable, but I make a sample of my needs using a list of 16 pages, logic is the same for longer lists)
1,16
8,9
...
3
votes
3answers
2k views
How to list all unique ip address currently connected to a specific port?
Say I want to know how many unique clients are connected to port 5222 on a server.
Can you find a way better/faster/stronger than this?
netstat -nt | grep ':5222.*ESTABLISHED' | awk '{ print $5 }' \
...
0
votes
1answer
271 views
Using Awk To Parse Individual Lines
Here is the output from qacct -o
Total : OWNER WALLCLOCK UTIME STIME CPU MEMORY IO IOW ...
1
vote
2answers
272 views
How can I delete lines that has an empty first column?
INPUT:
CC2352345|m,safnlasndfmnyxdcvyxcvyxcv |klasjdf |3|lasdjflasdf| |2345567356
CC3543353|asdfasdffghntz |klasjdf |3|lsajdfl | ...
11
votes
6answers
14k views
Is there a way to get the min, max, median, and average of a list of numbers in a single command?
I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script.
Although my immediate situation is for ...
4
votes
2answers
373 views
Why does this awk command not play as well with find as sed does?
I am very much a newbie at Unix/Linux command line stuff, and mostly get to where I'm going by copying, pasting, and modifying code I find on the internet. Just mentioning that in hopes people will ...
1
vote
3answers
468 views
Combining Awk with Curl
How would I set my curl statement so that it puts the value of each $3 in for the username, and the value of each $4 for the wallclock information?
Some further information
I am trying to take the ...
1
vote
1answer
672 views
Use AWK add new line if match exists
I have a file with the following format:
aaa@ bbb
bbb@ ccc& ddddd@ aaaf
ccc@ ddd& eee: abcd& dassda
The output should be:
aaaa@ bbb
bbb@ ccc&
ddddd@ aaaf
ccc@ ddd&
eee@ ...
2
votes
2answers
189 views
Turn a bash and php script into a distributable rpm
I'm semi unix knowledgeable and I want to be able to call down an rpm from my server, run it with something like rpm -i , and have access to the script anywhere.
The script is a combination awk and ...
3
votes
2answers
473 views
simple script rotating table
I'm trying to write a simple script, that just rotates csv table.
I mean I have some file:
head1;head2;head3
field11;field12;field13
and so on.
All I want, just to make my file
...
2
votes
2answers
301 views
How to divide a list of values by a number in command line?
I am trying to translate a simple program to the command line using unix utilities. For example, if I have a frequency list (after piping through uniq and sort)
5 x
4 y
1 z
I want to print out, ...
1
vote
2answers
162 views
Why does awk print on non-null strings and positive numbers
I've noticed some awk examples which use 1 instead of print to print $0 (eg. To conserve space, I normally use '1'. and on this site).
Is this a documented / safe practice, or is is it subject to ...
4
votes
2answers
976 views
Why does awk do full buffering when reading from a pipe
I'm reading from a serial port connected to a gps device sending nmea strings.
A simplified invocation to illustrate my point:
$ awk '{ print $0 }' /dev/ttyPSC9
...
2
votes
2answers
353 views
splitting the array and send the values to hashset in awk
I am getting var as argument from shell script to awk
awk -v var = "blah1|blah2|blah3" -f awk_script.awk
I need to put each blah's into a hashset, currently, I am doing it like this. I am getting ...
2
votes
3answers
1k views
Faster awk script to get the substring / string we wanted
ORDER EVENT .........[] [] ... so many other tags... [Account<25106>=ACCT1] [Destination...] .. so many other tags.
I am currently trying to get the account like this. I tried using match in ...
2
votes
1answer
310 views
remove trailing zeros in awk not working. syntax error
regex = "\\.*0+$";
subst = "";
system("echo "id "| awk '{sub(\\.*0+$," subst"); print}'");
It is giving the following error:
awk: cmd. line:1: {sub(\.*0+$,); print}
awk: cmd. line:1: ^ ...
2
votes
3answers
767 views
Sorting columns from two files with awk
I have two input files
28a39a289906c01159f999a68996091a tom@dong.com
274d1d2c7e931fb55ac0c91dd41f2be7 go@far.com
44d25d3b1b70b240d5058f1be1cef576 person@domain.com
2227a768f6d253b7bf81bb4ecb15b52d ...
0
votes
1answer
230 views
Shell script to check server state
I am writing a script that would ssh into the VMWare server and check if the VMs are up or not. There's some error in the server=awk "/$i/{print $2}" /home/dharmit/scripts/vmids line. I am getting ...
3
votes
5answers
308 views
changing pattern of a text file
A text file has contents something like
chair
table
pen
desk
Now i want it to be changed and stored in a variable say var as below
('chair','chair'),('table','table'),('pen','pen'),('desk','desk')
...
