awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

11
votes
3answers
754 views

How do you remove dot character from string without calling sed or awk again?

I have a file called hostlist.txt that contains text like this: host1.mydomain.com host2.mydomain.com anotherhost www.mydomain.com login.mydomain.com somehost host3.mydomain.com I have the ...
2
votes
4answers
65 views

Can Awk read escaped newlines? I.e. read past newlines

I have input looking like this: entry1line1 entry2line1\ entry2line2\ entry2line3 entry3line1 I would like to read such input into an array in AWK, to then process it together with the embedded ...
0
votes
1answer
32 views

Understanding sed expression

I am a beginner and would appreciate help on the interpretation of the sed+awk command below, sed -e '1,$s/ /:/g' -e '1,$s/;/:/g' |awk -F ":" '{print $1,$2,$3,$4,$12,$14,$18,$20,$22,$24,$26,$28,$30,$...
3
votes
2answers
85 views

Organizing three dimensional data with awk/sed/perl

I have this file (a sparse matrix): PC.354 OTU1 6 PC.354 OTU2 1 PC.356 OTU0 4 PC.356 OTU2 7 PC.356 OTU3 3 I want an output like this (dense matrix -classic .biom table): OTU_ID PC.354 PC.355 PC....
1
vote
3answers
57 views

Awk/sed: add line numbers to text from within a range

So I have a range, say 2-4. And I have three lines: first second third I need my output to be: 2 first 3 second 4 third I'm trying this on BSD (Mac) awk/sed, which seems to be making it harder.
0
votes
3answers
60 views

How to extract logs between the current time and the last 15 minutes

I want to extract the logs between the current time stamp and 15 minutes before and sent an email to the people configured. I developed the below script but it's not working properly; can someone help ...
1
vote
2answers
30 views

replace the content of one field based on the content in another field

I would like to divide the first column based on the delimiter, here -. Based on the last value, last column, here column 2, should be populated. If the value is 01 or 99 then replace with 2 or 1 ...
5
votes
3answers
20k views

Extracting part of lines with specific pattern using awk,sed

I have a question regarding the awk/sed operators. I have a big file which has following set of lines repeated Expression loweWallrhoPhi : sum=-6.97168e-09 Expression leftWallrhoPhi : sum=6.97168e-...
0
votes
2answers
71 views

Replicate lines in text file according to a column value

I have a text file with a column containing text/data followed by an abundance column, e.g. Car 3 Apple 2 In order to subsample these columns I would like to replicate each text column by ...
4
votes
2answers
321 views

How do I print the (numerical) ASCII values of each character in a file?

How can I print the numerical ASCII values of each character in a text file. Like cat, but showing the ASCII values only... (hex or decimal is fine). Example output for a file containing the word ...
3
votes
2answers
105 views

Question about concatenation instructions

I have a semicolon-delimited csv file and a Korn shell script that do some elaboration. I'm searching to understand what these two instructions do in Korn shell: num_rec=`cat test.csv |wc -l|tr -d " "...
-1
votes
2answers
49 views

Check if a column is float

I have a requirement that i want to check if a column is float or not. If its a float then do nothing else concatenate with .00. File: 2017 47.0000 0.783333 0.2500 1663 1920.0000 32 8.0000 ...
1
vote
1answer
32 views

Need help with sed/awk with deleting string

i have a txt file contenting the following information -Xmx4096m -Xms4096m -verbose:gc -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+ScavengeBeforeFullGC -Dsun.net.inetaddr.ttl=3600 -XX:...
5
votes
4answers
132 views

Build table - Add column depending on filenames

I have a huge amount of files having the following naming style: WBM_MIROC_rcp8p5_mississippi.txt WBM_GFDL_rcp8p5_nosoc_mississippi.txt DBH_HADGEM_rcp4p5_co2_mississippi.txt ...
1
vote
1answer
24 views

Wrong output in comparing floats

I'm trying to compare two floats in bash and something is going wrong. Here is the code sample based on solution here num1=0.502E-01 num2=0.01 echo $num1'>'$num2 | bc -l echo $num2'>'$num1 | bc ...
1
vote
4answers
303 views

How to split text file into records by first non-blank character using awk and print only records containing string

I have a text file with information separated in blocks. I would like to separate these blocks into records so that I can handle them separately. This is my text file (file.txt): Alarm Stats: com....
20
votes
4answers
36k views

Pass shell variable as a /pattern/ to awk

Having the following in one of my shell functions: function _process () { awk -v l="$line" ' BEGIN {p=0} /'"$1"'/ {p=1} END{ if(p) print l >> "outfile.txt" } ' } , so when called as ...
0
votes
1answer
35 views

have a log file that I would like to break up into several files based on content

I have a log file that I would like to break up into several files Table of Contents *** Configuration Database *** Logs *** Configuration Logs *** Query Logs . . and so on The issue ...
3
votes
1answer
45 views

Modify values in a column based on number of colons it contains

I would manipulate the contents of the second column based on the number of colons it contains. If the second field contains more than one colon then I would require the content before first colon ...
1
vote
2answers
77 views

Print text between multi-line delimiters

I am trying to extract data between 2 matching patterns, only if the content is not empty and I have issues doing it: here is an example: ============================== Queue Manager is : ...
4
votes
7answers
728 views

How to split a filename list in 5GB sets?

I have a pretty big list of files which I like to group in smaller list of 5GB each for further processing. The file list is in the following form: "size" (from du) "tab" "absolute path", like this (...
1
vote
1answer
63 views

AWK, Sum of category

I have tons of CSV files with similar content. The values are usually comma separated and they look like this. product_a, domestic, 500 product_a, abroad, 15 product_b, domestic, 313 ...
1
vote
2answers
62 views

How can I use Awk to convert a unix timestamp to human readable when doing a Tail -f logfile?

I'm currently doing a tail -f mysql.log and also have it color coded. But how would I use awk to convert the unix timestamps to human readable format? tail -f /var/lib/mysql/mysql-slow.log | awk '/...
-1
votes
3answers
44 views

Insert array of years into a list of URLS

I have a list of URLS like the example below. /test/test I also have an array of years [1999, 2000, 2001]. I am trying to find a way to automate the task of printing out a list of URLS with ...
0
votes
1answer
42 views

How to Tail -f output with colors using only Awk and show the rest of the output?

I was able to do a tail -f output with colors using the Awk command. But I'm unsure how to output the rest of the log as it seems to be filtering only those 2 lines. I also want to see the other lines ...
2
votes
4answers
360 views

How can I split a line into two lines if the length is greater than 7 using awk? [closed]

For example I just wanted to print something like this in the command line. Let say I have a file called file.txt. What is life? how are you? hi whatup this is more than And I want to print it ...
0
votes
3answers
78 views

insert variable text after another variable text on a specific line

I am trying to insert variable text after another variable text on a specific line. Below is my attempt so far, as far as I can tell, if I can escape the quotations in awk so that the variable can be ...
7
votes
6answers
7k views

How to remove all comments from a file?

I have a file with a few lines commented out: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I just want to print all the uncommented code: foo bar stuff morestuff ...
1
vote
2answers
43 views

sed + add last word on each string only if word not appears

I have the following file: linuxA.mtt.corp linux3V linux4B linux2A linux5v.mtt.corp how to add the .mtt.corp on each string that not have this word?
0
votes
4answers
3k views

Use SED or AWK to move rows to columns

I have a report that is generated in rows and I need to move it to columns. The number of rows per record will always be the same. This will be in a BASH script on Linux, so I would like to use ...
6
votes
5answers
5k views

Print line after nth occurrence of a match

I am looking to display the line 4598 in the following file. Effectively I want to display the line AFTER the nth occurrence of a match. In this case, the line after the 3rd occurrence of <Car>. ...
2
votes
2answers
60 views

find duplicate in first, second columns and sum numbers in third column

I have this input: Fred apples 20 Susy oranges 5 Mark watermellons 12 Robert pears 4 Robert oranges 17 Terry oranges 9 Lisa peaches 7 Susy oranges 12 Mark grapes 39 Anne mangoes 7 Greg pineapples 3 ...
1
vote
2answers
39 views

awk + how to append list in file

I have the following list in file more file.ex efifc1a hygg4a hdy5d nhdw4s wesf3a fjfhyr jfhg gsfar how to append any second list after the first list and any third list after the second list ...
1
vote
1answer
42 views

extract time range from log file using AWK

I am trying to extract time range from a logfile by passing shell variables in awk. This is how my log file is. 2014-11-24 12:58:59.290 1.1.1.1 etc.. 2014-11-24 13:58:59.290 2.2.2.2 etc.. 2014-11-...
4
votes
4answers
442 views

Print only the Nth line before each line that matches a pattern

I'm trying to print only the <N>th line before a search pattern. grep -B<N> prints all the <N> lines before the search pattern. I saw the awk code here that can print only the <N&...
6
votes
2answers
10k views

Remove lines based on duplicates within one column without sort

I have large 3-column files (~10,000 lines) and I would like to remove lines when the contents of the third column of that line appear in the third column of another line. The files' sizes make sort a ...
4
votes
4answers
124 views

Replace strings in a file based on a list of strings and a list of corresponding replacements

I am trying to replace strings in a file A: Hello Peter, how is your dad? where is mom? where the strings to be replaced are in file B: Peter dad mom and their corresponding replacements are in ...
0
votes
1answer
35 views

subtract previous line based on another column

I'm trying to figure out how to subtract previous line based on another column using AWK. Here is the input: site year count 040130013 1997 34 040130013 1998 55 040130019 2006 79 040130019 2007 135 ...
1
vote
4answers
84 views

Impose character between repeated characters

Given a string of characters ,,a,,,b,,,, I want to impose the character - at the beginning, if no character is preceded comma, and between every comma that has no character. Meaning the result is -,-,...
1
vote
2answers
59 views

Process several files and print filename without extension and the value of a specific key

I have ASCII file and I want to output the file name and a specific field from this file beside each other. For example: The contents of the ASCII file "data1.txt" is as follow: Name: John Age: 90 ...
-3
votes
2answers
47 views

linux + convert file format columns to one column

I have the following file (Linux.txt) linux-test01 linuxER LinuxCV LinuxWE LinuxPL LinuxDF LinuxXC LinuxSD LinuxAQ how to print using awk or perl one-liner to produce the following ...
0
votes
1answer
55 views

Compare strings with specialized charcters

If I have a lookup table cat tmp1 [//rtwttwtr*fgg]\\\erw``~ 1 ^774574574565665f[[[//]\] 2 ()42543^[[D^[[D^[[D^[[D^[[D353535345****@3242- 3 and a number of strings to be looked up from the first ...
21
votes
9answers
27k views

How to print certain columns by name?

I have the following file: id name age 1 ed 50 2 joe 70 I want to print just the id and age columns. Right now I just use awk: cat file.tsv | awk '{ print $1, $3 }' However, this ...
3
votes
1answer
32 views

Awk Search - Combined Character Classes in Same String

Good Evening, I am attempting to utilize multiple character classes at the same time without success. Given a column that contains the URI of HTTP traffic, I want the REGEX to identify lines that ...
0
votes
2answers
57 views

How to determine what IP addresses are available in a hosts table?

I'm working on a script that will read from the /etc/hosts file and determine what IPs are available in the hosts file. The way the hosts file will look is as followed: 172.17.164.2 System1 172.17....
3
votes
1answer
38 views

Splitting a large file by column with values in the header as file names

I have a large tab-delimited text file which contains a header and data. I want to split the file into smaller files by column, with the header excluded, and the files named as the column name. pos ...
0
votes
2answers
45 views

Awk: Print last N columns, where N is passed through a variable

My input variable $dirPath contains directory path. The action is to be able to retrieve the last N values in the directory path, where N is passed through a variable $depth. For a fixed N value (say ...
0
votes
1answer
57 views

Redirect output of: yum repolist all

If you run the command: yum repolist all you will see there are basically four columns of data. repo id, repo name, status, install count. If however your run something like: yum repolist all > ...
3
votes
2answers
52 views

Simple awk wrapper function or alias

I had seen someone use a bash alias or function based on awk, which I liked for its simplicity, but can't remember how to implement it (in bash). Lets say you have a large formatted file, from which ...
0
votes
3answers
69 views

Right syntax for awk usage in combination with other command inside xargs sh –c

How to make this command work: ls * | xargs -I {} sh -c 'echo {}; awk '{print $1}' {} | uniq' It should do the simple thing: print for each file in the folder its name and uniq values in the first ...