awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
2answers
194 views

Splitting giant files into a specific number of lines

I have a CSV that is 40 000 lines long. What is the simplest tool to read from line x to line y? I am sure there are more professional ways to do it but I would just like a bash script that works in ...
1
vote
3answers
1k views

How to awk the “df -Pm” command to get FS names that have more then 90% the FS usage?

I'm trying to get the names of all disks with at least 90% usage, using this oneliner: df -Pm | awk '+$5 >= 90 {print}' But it doesn't show the "100% usage" case, I need to grep for "100%". Why ...
3
votes
3answers
2k views

Using sed to edit lines in a file with a variable

I need to edit lines in a file using sed. Now the problem is I am replacing a particular pattern with a combination of text and number. This number is a variable which keeps on incrementing for every ...
6
votes
1answer
566 views

Why are capital letters included in a range of lower-case letters in an awk regex?

$ echo ABC | awk '$0 ~ /^[a-b]/' ABC $ echo ABC | awk '$0 ~ /^[a-a]/' $ echo ABC | awk '$0 ~ /^a/' $ You see. /[a-b]/ captures A, but /[a-a]/ or /a/ doesn't. Why?
4
votes
2answers
736 views

How to test if a function is defined in awk?

I am writing an awk program and I want it to run it both with mawk and gawk. I need a sorting function. I have written my own, but I want to use the builtin asort() function in gawk if available. How ...
4
votes
4answers
2k views

How to echo an escaped string

How can I echo an escaped string that contains $ in Bourne Shell? user@server:~$ cat test.sh #!/bin/sh echo $1 user@server:~$ ./test.sh \$sad\$test $sad$test I want it to return an escaped ...
2
votes
1answer
259 views

Awk : Range of PIDs

I am learning awk from this tutorial. It's quite basic. I have a list of processes in a file which I got by doing ps aux > processes Now according to the tutorial, doing awk '$2 ~ 14022, $2 ~ ...
6
votes
7answers
14k views

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...
3
votes
3answers
1k views

In awk, how to print a range of fields with the original $0 delimiters intact?

How to extract a substring of $0, in awk, when that substring's position and length is based on the positions of fields in the original $0? It effectively equates to removing the first n.lead fields ...
3
votes
2answers
2k views

How would I extract the user agent strings from a log file?

Currently I am running a command like this, to get the most requested content: grep "17\/Jul\/2011" other_vhosts_access.log | awk '{print $8}' | sort | uniq -c | sort -nr I want to now see the user ...
2
votes
2answers
393 views

maximum of column with matching id

I have a file with the following fields NM001 rp1 210 NM003 rp1 220 NM005 rag 200 NM004 rag 100 I would like to have an output that for every change in column2, find maximum in column4 but also ...
1
vote
1answer
85 views

awk expand window to 500% for two numbers from column file

I've got a file with two columns corresponding to start-end ranges like this: awk '{print $2"-"$3}' file.txt 91082802-91082990 94018177-94018321 146945732-146945949 88701397-88701594 ...
4
votes
1answer
2k views

awk: forcing a return status?

This is a followup to my earlier question. I am validating the number of fields in /etc/passwd using this handy snippit. In the following example, the users 'fieldcount1' and 'fieldcount2' have the ...
-1
votes
1answer
2k views

Sum of grep results

I need to output the sum of a grep results for a certain time, for now i hard coded the time grep but any help will be very much appreciated, my problem is i cant output the results on the script. If ...
2
votes
1answer
2k views

Time/Date grep for 15 mins

I need to grep a specific type of line in a specific time of the log, do you have any ideas? Here's my working script: cat *.log |grep -E '2011-06-30 (1[0-1]:[0-1][0-5]|10:16)'| grep -ach '0110 ...
1
vote
1answer
629 views

Inconsistent field separator behaviour of space in awk

The following awk script shows how a leading space ' ' is ignored as a field separator, but other characters are treated as a separator with a preceding null $1 field. Is there some way to make ...
1
vote
1answer
262 views

Need help processing a text file with awk to conform to CSV flat file format

I have the following problem. I collected data on reaction time from over 100 participants for an experiment I am running. Unfortunately, the separators between fields were not consistent, but after a ...
3
votes
2answers
2k views

only display df lines that has more fs usage then 80%

root@SERVER ~$ df Filesystem 512-blocks Free %Used Iused %Iused Mounted on /dev/YXCV 655360 365632 45% 6322 13% / /dev/ASDF 3801088 670648 83% 41759 32% /usr /dev/ASR ...
2
votes
3answers
616 views

pad a number with zeroes

I need have put 0 on the results of my grep so my script format will be fine and i don't have any idea how to do it. here's my grep result : 261 : 261 = 0 | 1192 : 1184 = 8 | 283 : 283 = 0 | 666 : ...
9
votes
3answers
6k views

Delete the first n bytes of files

I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way. I want to delete the first 31 bytes of each file ...
25
votes
9answers
35k views

Is there a way to get the min, max, median, and average of a list of numbers in a single command?

I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script. Although my immediate situation is for ...
12
votes
6answers
6k views

How can you combine all lines that end with a backslash character?

Using a common command line tool like sed or awk, is it possible to join all lines that end with a given character, like a backslash? For example, given the file: foo bar \ bash \ baz dude \ happy ...
5
votes
4answers
1k views

How to append end of every line with the line itself

This is in file.txt: redcar bluecar greencar Im looking for ways to make it become: redcar redcar bluecar bluecar greencar greencar I've tried many ways using sed with no luck
3
votes
1answer
1k views

using system command in awk script

In an AWK script I am using a command system(date) to print the current date in a file but after this command is executed next line is also added implicitly. Is there any way in AWK to print ...
-4
votes
2answers
297 views

Manipulating a file with awk

I have a text file in the following format: Surname, Forename: Day.Month.Year: Degree Sellen, Jo: 03.07.1986: MSc CSE Parfitt, Harry: 20.03.1984: MSc IT How can I write a shell script that uses ...
6
votes
2answers
3k views

sed one-liner to delete any line that does not contain lowercase letters

So, basically THIS LINE WOULD BE DELETED and (THIS LINE WOULD ALSO BE DELETED) but Indeed, THIS LINE WOULD NOT
7
votes
6answers
2k views

Is there any alternative to grep's -A -B -C switches (to print few lines before and after )?

grep -A 2 -B 3 prints 2 lines after the grep string and prints 3 lines before. grep -C 3 prints 3 Lines before and 3 lines after Unfortunately, the grep I'm using does not support these ...
3
votes
2answers
3k views

My awk program to change spaces into tabs doesn't work

How to write a shell script which uses awk to read in the data file students.txt and output the data in the tabbed format as shown: Surname Forename MSc Stream Date of Birth Smith John IT 15.01.1986 ...
5
votes
5answers
19k views

Multiline pattern match using sed, awk or grep

Is it possible to do a multiline pattern match using sed, awk or grep? Take for example, I would like to get all the lines between { and } So it should be able to match 1. {} 2. {.....} 3. ...
3
votes
4answers
261 views

Remove empty configuration section

Files like ~/.config/vlc/vlcrc are 99% junk if you want to version control only the configuration options. I've got a script to remove the comments, but there's a ton of empty configuration sections ...
2
votes
3answers
6k views

Loop a list through awk

I'm sorry if this is extremely elementary, but I just can't figure out how to do this, and my research has failed me as well. I have two files: data.csv and list.txt. Here's an example of what they ...
3
votes
2answers
350 views

Truncate a file on a certain pattern

How would I go about truncating a binary file when a certain pattern is found? For instance, I want to truncate the file at the first occurrence of the pattern 0xFFFFFFFF. I think something like awk ...
2
votes
3answers
3k views

Extracting tokens from a line of text

Using bash scripting and grep/awk/sed, how can I split a line matching a known pattern with a single character delimiter into an array, e.g. convert token1;token2;token3;token4 into a[0] = token1 … ...
3
votes
2answers
659 views

Using grep/sed/awk to classify log file entries

I need to process a very large log file with many lines in different formats. My goal is to extract unique line entries who have the same starting pattern, e.g. '^2011-02-21.*MyKeyword.*Error', ...
2
votes
2answers
1k views

How to build a long command string?

I've a sequence of commands to be used along with lot of pipings, something like this: awk '{ if ($8 ~ "isad") {print $2, $5, "SE"} else {print $2, $5, "ANT"} }' ...
0
votes
2answers
1k views

How to run a shell script containing an awk command

How to run this script (called count.sh)? awk 'BEGIN{ x=0 ; while (x < 5) { x+=1 ; print x ; } }' I'm trying to run it with sh count.sh but it's giving me an error.
14
votes
10answers
2k views

There must be a better way to replace single newlines only?

I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
3
votes
1answer
7k views

Filtering on dates with grep and awk

I have created the alias below in my .bash_aliases file alias auth="grep \"$(date|awk '{print $2,$3}')\" /var/log/auth.log | grep -E '(BREAK-IN|Invalid user|Failed|refused|su|Illegal)'" ...
5
votes
3answers
795 views

extracting “tag” information from git with a shell script

The vc bundle is a neat little package that extracts information about a git repo for easy insertion into a LaTeX document. It doesn't currently extract information about whether the current commit is ...
6
votes
3answers
2k views

AWK: wrap lines to 72 characters

$ awk 'length > 72' {HOW TO PRINT THE LINEs IN PCS?} msg ie I want it to add \n after 72 chars and continue, so initially you may need to remove all single \ns and the add them. It may be easier ...
5
votes
4answers
225 views

How could I simplify this command to only use awk?

awk '/user/ {print $1 }' /etc/userdomains | sed 's/://' the format of /etc/userdomains is domain.tld: user otherdomain.tld: otheruser
54
votes
7answers
15k views

Is there a basic tutorial for grep, awk and sed? [closed]

I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...
16
votes
9answers
9k views

Tool in UNIX to subtract dates

Is there any tool in Solaris UNIX (so no GNU tool available) to subtract dates? I know that in Linux we have gawk that can subtract one date from another. But in Solaris the maximum we have is nawk ...
8
votes
6answers
238 views

Grab certain contents of a file

So I know tools exist for this problem because I've heard about them, but I don't know what they are. I want to do something like filter out all data but the usernames in /etc/passwd. For example, I ...
5
votes
1answer
2k views

Replacing multiple lines in sed or awk [on hold]

I'm trying to use sed or awk to replace 5 lines in a smb file but I just don't have any idea how to deal with the newlines.