awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

23
votes
7answers
58k views

How to read first and last line from cat output?

I have text file. Task - get first and last line from file after $ cat file | grep -E "1|2|3|4" | commandtoprint $ cat file 1 2 3 4 5 Need this without cat output (only 1 and 5). ~$ cat file | ...
0
votes
1answer
18 views

Is there a way to get all the NEWS.gz which is/was this year in /usr/share/doc/ of $PACKAGENAME

If people go into /usr/share/doc/$PACKAGENAME (you can use any valid package name which you have installed on your system) you will see that the directory has various files in it, at the very least ...
-5
votes
1answer
37 views

What happen if record separator and field separator are the same

In awk, what happen if record separator RS and field separator FS are the same? How can I tell fields from records?
1
vote
1answer
39 views

Need to filter the data on the basis of two column having many to one relationship using awk

I have a big file having 50s columns and 100K of rows delimited by |. Now $2(col 2) has multiple type of $1(col 1) value which means col 2 will be repeated. So I have sorted the file. I need now to ...
2
votes
1answer
50 views

on-line tally of unique lines

uniq -c is very useful for counting the number of times the same line appears consecutively $ seq 1 1000 | awk '{ if ($1 > 100 && $1 <= 200) { print "hi" } else {print "bye"} }' | uniq ...
3
votes
1answer
56 views

Rename files based on JSON content

I have two sets of files; file1.txt file1.json file2.txt file2.json file3.txt file3.json ... fileN.txt fileN.json json files contain the following format: {"ago": "59 sec ago", ...
4
votes
2answers
756 views

How can I round all numbers in a file?

How can I round all numbers in a file that countains several columns of numbers up to a certain precision? Can this be done with awk? A single line looks like this: text - 0.1655456615 - 0.158645 - ...
11
votes
3answers
680 views

How do you remove dot character from string without calling sed or awk again?

I have a file called hostlist.txt that contains text like this: host1.mydomain.com host2.mydomain.com anotherhost www.mydomain.com login.mydomain.com somehost host3.mydomain.com I have the ...
2
votes
4answers
64 views

Can Awk read escaped newlines? I.e. read past newlines

I have input looking like this: entry1line1 entry2line1\ entry2line2\ entry2line3 entry3line1 I would like to read such input into an array in AWK, to then process it together with the embedded ...
0
votes
1answer
29 views

Understanding sed expression

I am a beginner and would appreciate help on the interpretation of the sed+awk command below, sed -e '1,$s/ /:/g' -e '1,$s/;/:/g' |awk -F ":" '{print ...
3
votes
2answers
81 views

Organizing three dimensional data with awk/sed/perl

I have this file (a sparse matrix): PC.354 OTU1 6 PC.354 OTU2 1 PC.356 OTU0 4 PC.356 OTU2 7 PC.356 OTU3 3 I want an output like this (dense matrix -classic .biom table): OTU_ID PC.354 PC.355 ...
1
vote
3answers
51 views

Awk/sed: add line numbers to text from within a range

So I have a range, say 2-4. And I have three lines: first second third I need my output to be: 2 first 3 second 4 third I'm trying this on BSD (Mac) awk/sed, which seems to be making it harder.
0
votes
3answers
53 views

How to extract logs between the current time and the last 15 minutes

I want to extract the logs between the current time stamp and 15 minutes before and sent an email to the people configured. I developed the below script but it's not working properly; can someone help ...
1
vote
2answers
28 views

replace the content of one field based on the content in another field

I would like to divide the first column based on the delimiter, here -. Based on the last value, last column, here column 2, should be populated. If the value is 01 or 99 then replace with 2 or 1 ...
5
votes
3answers
18k views

Extracting part of lines with specific pattern using awk,sed

I have a question regarding the awk/sed operators. I have a big file which has following set of lines repeated Expression loweWallrhoPhi : sum=-6.97168e-09 Expression leftWallrhoPhi : ...
0
votes
2answers
69 views

Replicate lines in text file according to a column value

I have a text file with a column containing text/data followed by an abundance column, e.g. Car 3 Apple 2 In order to subsample these columns I would like to replicate each text column by ...
4
votes
2answers
318 views

How do I print the (numerical) ASCII values of each character in a file?

How can I print the numerical ASCII values of each character in a text file. Like cat, but showing the ASCII values only... (hex or decimal is fine). Example output for a file containing the word ...
3
votes
2answers
102 views

Question about concatenation instructions

I have a semicolon-delimited csv file and a Korn shell script that do some elaboration. I'm searching to understand what these two instructions do in Korn shell: num_rec=`cat test.csv |wc -l|tr -d " ...
-1
votes
2answers
40 views

Check if a column is float

I have a requirement that i want to check if a column is float or not. If its a float then do nothing else concatenate with .00. File: 2017 47.0000 0.783333 0.2500 1663 1920.0000 32 8.0000 ...
1
vote
1answer
31 views

Need help with sed/awk with deleting string

i have a txt file contenting the following information -Xmx4096m -Xms4096m -verbose:gc -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+ScavengeBeforeFullGC -Dsun.net.inetaddr.ttl=3600 ...
5
votes
4answers
131 views

Build table - Add column depending on filenames

I have a huge amount of files having the following naming style: WBM_MIROC_rcp8p5_mississippi.txt WBM_GFDL_rcp8p5_nosoc_mississippi.txt DBH_HADGEM_rcp4p5_co2_mississippi.txt ...
1
vote
1answer
24 views

Wrong output in comparing floats

I'm trying to compare two floats in bash and something is going wrong. Here is the code sample based on solution here num1=0.502E-01 num2=0.01 echo $num1'>'$num2 | bc -l echo $num2'>'$num1 | bc ...
-3
votes
0answers
32 views

How to compare an input name (first and last) with names in a file (revised) [closed]

How do you compare an input name (first and last) with names in a file and print out either name is good or name is bad? I can't seem to find this anywhere. Do I need to set up the file a certain way? ...
1
vote
4answers
296 views

How to split text file into records by first non-blank character using awk and print only records containing string

I have a text file with information separated in blocks. I would like to separate these blocks into records so that I can handle them separately. This is my text file (file.txt): Alarm Stats: ...
19
votes
4answers
35k views

Pass shell variable as a /pattern/ to awk

Having the following in one of my shell functions: function _process () { awk -v l="$line" ' BEGIN {p=0} /'"$1"'/ {p=1} END{ if(p) print l >> "outfile.txt" } ' } , so when called as ...
0
votes
1answer
34 views

have a log file that I would like to break up into several files based on content

I have a log file that I would like to break up into several files Table of Contents *** Configuration Database *** Logs *** Configuration Logs *** Query Logs . . and so on The issue ...
3
votes
1answer
44 views

Modify values in a column based on number of colons it contains

I would manipulate the contents of the second column based on the number of colons it contains. If the second field contains more than one colon then I would require the content before first colon ...
1
vote
2answers
74 views

Print text between multi-line delimiters

I am trying to extract data between 2 matching patterns, only if the content is not empty and I have issues doing it: here is an example: ============================== Queue Manager is : ...
4
votes
7answers
718 views

How to split a filename list in 5GB sets?

I have a pretty big list of files which I like to group in smaller list of 5GB each for further processing. The file list is in the following form: "size" (from du) "tab" "absolute path", like this ...
1
vote
1answer
59 views

AWK, Sum of category

I have tons of CSV files with similar content. The values are usually comma separated and they look like this. product_a, domestic, 500 product_a, abroad, 15 product_b, domestic, 313 ...
1
vote
2answers
48 views

How can I use Awk to convert a unix timestamp to human readable when doing a Tail -f logfile?

I'm currently doing a tail -f mysql.log and also have it color coded. But how would I use awk to convert the unix timestamps to human readable format? tail -f /var/lib/mysql/mysql-slow.log | awk ...
-1
votes
3answers
42 views

Insert array of years into a list of URLS

I have a list of URLS like the example below. /test/test I also have an array of years [1999, 2000, 2001]. I am trying to find a way to automate the task of printing out a list of URLS with ...
0
votes
1answer
35 views

How to Tail -f output with colors using only Awk and show the rest of the output?

I was able to do a tail -f output with colors using the Awk command. But I'm unsure how to output the rest of the log as it seems to be filtering only those 2 lines. I also want to see the other lines ...
2
votes
4answers
339 views

How can I split a line into two lines if the length is greater than 7 using awk? [closed]

For example I just wanted to print something like this in the command line. Let say I have a file called file.txt. What is life? how are you? hi whatup this is more than And I want to print it ...
0
votes
3answers
69 views

insert variable text after another variable text on a specific line

I am trying to insert variable text after another variable text on a specific line. Below is my attempt so far, as far as I can tell, if I can escape the quotations in awk so that the variable can be ...
7
votes
6answers
6k views

How to remove all comments from a file?

I have a file with a few lines commented out: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I just want to print all the uncommented code: foo bar stuff morestuff ...
1
vote
2answers
41 views

sed + add last word on each string only if word not appears

I have the following file: linuxA.mtt.corp linux3V linux4B linux2A linux5v.mtt.corp how to add the .mtt.corp on each string that not have this word?
0
votes
4answers
3k views

Use SED or AWK to move rows to columns

I have a report that is generated in rows and I need to move it to columns. The number of rows per record will always be the same. This will be in a BASH script on Linux, so I would like to use ...
6
votes
5answers
5k views

Print line after nth occurrence of a match

I am looking to display the line 4598 in the following file. Effectively I want to display the line AFTER the nth occurrence of a match. In this case, the line after the 3rd occurrence of <Car>. ...
2
votes
2answers
50 views

find duplicate in first, second columns and sum numbers in third column

I have this input: Fred apples 20 Susy oranges 5 Mark watermellons 12 Robert pears 4 Robert oranges 17 Terry oranges 9 Lisa peaches 7 Susy oranges 12 Mark grapes 39 Anne mangoes 7 Greg pineapples 3 ...
1
vote
2answers
38 views

awk + how to append list in file

I have the following list in file more file.ex efifc1a hygg4a hdy5d nhdw4s wesf3a fjfhyr jfhg gsfar how to append any second list after the first list and any third list after the second list ...
1
vote
1answer
34 views

extract time range from log file using AWK

I am trying to extract time range from a logfile by passing shell variables in awk. This is how my log file is. 2014-11-24 12:58:59.290 1.1.1.1 etc.. 2014-11-24 13:58:59.290 2.2.2.2 etc.. ...
4
votes
4answers
422 views

Print only the Nth line before each line that matches a pattern

I'm trying to print only the <N>th line before a search pattern. grep -B<N> prints all the <N> lines before the search pattern. I saw the awk code here that can print only the ...
6
votes
2answers
9k views

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
-1
votes
0answers
36 views

how to pass multiple column value in a command

I have a file oldnewmap.txt. This file has hundreds of records like the following, where column one represents oldnumber and column two represents newnumber. 21382,45678 34567,45663 44245,42455 ...
4
votes
4answers
110 views

Replace strings in a file based on a list of strings and a list of corresponding replacements

I am trying to replace strings in a file A: Hello Peter, how is your dad? where is mom? where the strings to be replaced are in file B: Peter dad mom and their corresponding replacements are in ...
0
votes
1answer
28 views

subtract previous line based on another column

I'm trying to figure out how to subtract previous line based on another column using AWK. Here is the input: site year count 040130013 1997 34 040130013 1998 55 040130019 2006 79 040130019 2007 135 ...
1
vote
4answers
82 views

Impose character between repeated characters

Given a string of characters ,,a,,,b,,,, I want to impose the character - at the beginning, if no character is preceded comma, and between every comma that has no character. Meaning the result is ...
1
vote
2answers
59 views

Process several files and print filename without extension and the value of a specific key

I have ASCII file and I want to output the file name and a specific field from this file beside each other. For example: The contents of the ASCII file "data1.txt" is as follow: Name: John Age: 90 ...
-3
votes
2answers
45 views

linux + convert file format columns to one column

I have the following file (Linux.txt) linux-test01 linuxER LinuxCV LinuxWE LinuxPL LinuxDF LinuxXC LinuxSD LinuxAQ how to print using awk or perl one-liner to produce the following ...