awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

0
votes
1answer
41 views

How to get the data between specified Starttime and Endtime in a log file?

This is my LogFile(Test.Log): May 10 01:06:05 A May 10 01:06:06 B May 10 01:06:07 C May 10 01:07:01 D May 10 01:07:02 E May 10 01:07:09 F May 10 01:07:11 G May 10 01:07:11 H May 10 01:07:12 I May 10 ...
2
votes
1answer
50 views

How to choose the highest number element in many files with many arrays

I have many files which each one contain many arrays(like input 1 and 2, below). Each array has two element numbers in the second row. I need to use a logical criterion to choose the highest number ...
16
votes
7answers
26k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
3
votes
3answers
53 views

Grep/awk/sed for lines composed of only two letters, and lines that start with a letter and meet a certain length

Not sure how many people are familiar with DNA sequencing data, but if this is part of my file (lines starting with ">" are IDs, and lines starting with a letter are DNA sequences): >NB501013:9:...
0
votes
2answers
45 views

Sum column numbers elements in many arrays

I need to sum the the numbers located in the same column in an array. The input is: 1: 6.1703 44 3.5 0.3 46 7.3 2.3 2: 6.1932 44 5.29379 2.9 46 1.72261 7.2261 3: 6.5664 45 4.756 5.6 46 1.6 2.5 4:...
0
votes
0answers
33 views

awk or sed Commands to Print Columns at desired locations [duplicate]

I have a source file which contain 15 columns and so may rows: 2973695.500 633952.188 2440.000 255 0 255 1 1 OP UNK opt TIME SKZ meter ms 2973695.500 633952.188 2440.000 0 255 255 1 1 ...
0
votes
0answers
37 views

get user input in terms of size (GB or MB)

I created script to calculate shmmax shmall shmmini Total_Memory_M=$(dmidecode -t 17 | grep Size | awk '{print $2}' | grep -o '[0-9]\+' | paste -sd+ | bc) Total_Memory_G=$(echo $Total_Memory_M /1024 |...
2
votes
1answer
36 views

How join two files by one common column (1ª) between two file (File1 and file2)?

I am trying join two files by one common column (1ª) between two file (File1 and file2), resulting in file 3, but the resultant file is not working. Can someone help me? I used this command: awk 'NR=...
3
votes
1answer
906 views

Awk compare 2 files, print match together with non match lines filled with 0

I have two files. I want to match the 1st column of first file to 1st column of second file. If there is a match I'd like to print the 2st column of second file, if not 0. I have tried awk (awk 'FNR=...
1
vote
4answers
70 views

Extract X number of minutes from the current time from a logfile

I have a logfile with the following timestamp format 2016-06-02 13:59:58.069 The log I am working with spans about an hour. 2016-06-02 13:05:06.144 2016-06-02 13:05:06.144 2016-06-02 13:05:06.160 ...
5
votes
2answers
412 views

Looking for lines which is in one file but not in other using Unix and Awk [duplicate]

I have 2 files with 7 fields and I want to print the lines which are present in file1 but not in file2 based on field1 and field2. Logic: I want to print all the lines, where there is a particular ...
1
vote
4answers
66 views

How to use Unindented line as the Record Separate in awk cli

I have a log file that look like: 2016-05-31 09:54:36 (16667) heritage_w? From: ip68-8-49-100.sd.sd.cox.net User: user1wizard (wizard) Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/...
1
vote
1answer
49 views

How to sum specific rows of a matrix

I need to sum the matrix elements of specific rows for each column. An example of the matrix is given below(input). So the matrix has 4 columns and 6 rows of numbers. I need to sum the numbers of ...
1
vote
2answers
44 views

How to append some string combined with the pattern at the end of line with the pattern

I want to append .com at the end of line for every line matching the pattern in /etc/hosts file. Sample file content: 127.0.0.1 localhost 1.2.3.4 hostname1 hostname1.xyz hostname1.xyz. 1.2.3....
3
votes
1answer
52 views

awk + how to perform OR in awk

by the following command I capture available size from df echo "$DF" | awk '$NF == "/var" { print $(NF-2) }' how to set the syntax if I want to set NF == "/var" or NF == "/" because I ...
0
votes
1answer
41 views

How to use TWO “for statements” in a shell script?

Writing script to create user accounts. Script is written to echo the useradd command (so I can make sure it will work as expected). I have a file newemploy.txt with Employee Names (first and last, ...
2
votes
3answers
95 views

Mysql output format in Bash Script

I'm writing a bash script where I'm using mysql query in that script where the format which I'm getting is quite different. Query: root@debian:~# mysql -u root -ptoor super_market -h 0 -e "select * ...
15
votes
4answers
75k views

How to run grep on a single column?

I want to grep the output of my ls -l command: -rw-r--r-- 1 root root 1866 Feb 14 07:47 rahmu.file -rw-r--r-- 1 rahmu user 95653 Feb 14 07:47 foo.file -rw-r--r-- 1 rahmu user ...
3
votes
6answers
1k views

How do I remove a words ending in the letter S if duplicates without an S on the end are on the same list?

I have a large list of words. Many of the words are only different because they have the letter s on the end. If a word on the list is the exact same as another word on the list, except one of the ...
1
vote
1answer
47 views

completely ignore lines that start with a specific pattern

I have a script which analyzes the output of a computation software. sometimes the output comes with some extra data that are irrelevant to my script. This data can be huge and makes running my simple ...
0
votes
1answer
27 views

How to move lines between patterns to the end of a file

I want to move several lines to the end of the file. Example: 1 2 3 4 5 6 blkn XYZ indf 7 8 9 10 11 lnkn ABCYDF knnfd 12 13 14 15 I want to move lines 6 to 11 to the end of the file. The patterns ...
24
votes
7answers
67k views

How to read first and last line from cat output?

I have text file. Task - get first and last line from file after $ cat file | grep -E "1|2|3|4" | commandtoprint $ cat file 1 2 3 4 5 Need this without cat output (only 1 and 5). ~$ cat file | ...
0
votes
1answer
18 views

Is there a way to get all the NEWS.gz which is/was this year in /usr/share/doc/ of $PACKAGENAME

If people go into /usr/share/doc/$PACKAGENAME (you can use any valid package name which you have installed on your system) you will see that the directory has various files in it, at the very least ...
-5
votes
1answer
42 views

What happen if record separator and field separator are the same

In awk, what happen if record separator RS and field separator FS are the same? How can I tell fields from records?
1
vote
1answer
45 views

Need to filter the data on the basis of two column having many to one relationship using awk

I have a big file having 50s columns and 100K of rows delimited by |. Now $2(col 2) has multiple type of $1(col 1) value which means col 2 will be repeated. So I have sorted the file. I need now to ...
2
votes
1answer
61 views

on-line tally of unique lines

uniq -c is very useful for counting the number of times the same line appears consecutively $ seq 1 1000 | awk '{ if ($1 > 100 && $1 <= 200) { print "hi" } else {print "bye"} }' | uniq -...
3
votes
1answer
70 views

Rename files based on JSON content

I have two sets of files; file1.txt file1.json file2.txt file2.json file3.txt file3.json ... fileN.txt fileN.json json files contain the following format: {"ago": "59 sec ago", "...
4
votes
2answers
790 views

How can I round all numbers in a file?

How can I round all numbers in a file that countains several columns of numbers up to a certain precision? Can this be done with awk? A single line looks like this: text - 0.1655456615 - 0.158645 - ...
11
votes
3answers
774 views

How do you remove dot character from string without calling sed or awk again?

I have a file called hostlist.txt that contains text like this: host1.mydomain.com host2.mydomain.com anotherhost www.mydomain.com login.mydomain.com somehost host3.mydomain.com I have the ...
2
votes
4answers
65 views

Can Awk read escaped newlines? I.e. read past newlines

I have input looking like this: entry1line1 entry2line1\ entry2line2\ entry2line3 entry3line1 I would like to read such input into an array in AWK, to then process it together with the embedded ...
0
votes
1answer
33 views

Understanding sed expression

I am a beginner and would appreciate help on the interpretation of the sed+awk command below, sed -e '1,$s/ /:/g' -e '1,$s/;/:/g' |awk -F ":" '{print $1,$2,$3,$4,$12,$14,$18,$20,$22,$24,$26,$28,$30,$...
3
votes
2answers
86 views

Organizing three dimensional data with awk/sed/perl

I have this file (a sparse matrix): PC.354 OTU1 6 PC.354 OTU2 1 PC.356 OTU0 4 PC.356 OTU2 7 PC.356 OTU3 3 I want an output like this (dense matrix -classic .biom table): OTU_ID PC.354 PC.355 PC....
1
vote
3answers
60 views

Awk/sed: add line numbers to text from within a range

So I have a range, say 2-4. And I have three lines: first second third I need my output to be: 2 first 3 second 4 third I'm trying this on BSD (Mac) awk/sed, which seems to be making it harder.
0
votes
3answers
66 views

How to extract logs between the current time and the last 15 minutes

I want to extract the logs between the current time stamp and 15 minutes before and sent an email to the people configured. I developed the below script but it's not working properly; can someone help ...
1
vote
2answers
35 views

replace the content of one field based on the content in another field

I would like to divide the first column based on the delimiter, here -. Based on the last value, last column, here column 2, should be populated. If the value is 01 or 99 then replace with 2 or 1 ...
5
votes
3answers
22k views

Extracting part of lines with specific pattern using awk,sed

I have a question regarding the awk/sed operators. I have a big file which has following set of lines repeated Expression loweWallrhoPhi : sum=-6.97168e-09 Expression leftWallrhoPhi : sum=6.97168e-...
0
votes
2answers
73 views

Replicate lines in text file according to a column value

I have a text file with a column containing text/data followed by an abundance column, e.g. Car 3 Apple 2 In order to subsample these columns I would like to replicate each text column by ...
4
votes
2answers
333 views

How do I print the (numerical) ASCII values of each character in a file?

How can I print the numerical ASCII values of each character in a text file. Like cat, but showing the ASCII values only... (hex or decimal is fine). Example output for a file containing the word ...
3
votes
2answers
107 views

Question about concatenation instructions

I have a semicolon-delimited csv file and a Korn shell script that do some elaboration. I'm searching to understand what these two instructions do in Korn shell: num_rec=`cat test.csv |wc -l|tr -d " "...
-1
votes
2answers
59 views

Check if a column is float

I have a requirement that i want to check if a column is float or not. If its a float then do nothing else concatenate with .00. File: 2017 47.0000 0.783333 0.2500 1663 1920.0000 32 8.0000 ...
1
vote
1answer
33 views

Need help with sed/awk with deleting string

i have a txt file contenting the following information -Xmx4096m -Xms4096m -verbose:gc -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+ScavengeBeforeFullGC -Dsun.net.inetaddr.ttl=3600 -XX:...
5
votes
4answers
134 views

Build table - Add column depending on filenames

I have a huge amount of files having the following naming style: WBM_MIROC_rcp8p5_mississippi.txt WBM_GFDL_rcp8p5_nosoc_mississippi.txt DBH_HADGEM_rcp4p5_co2_mississippi.txt ...
1
vote
1answer
31 views

Wrong output in comparing floats

I'm trying to compare two floats in bash and something is going wrong. Here is the code sample based on solution here num1=0.502E-01 num2=0.01 echo $num1'>'$num2 | bc -l echo $num2'>'$num1 | bc ...
1
vote
4answers
309 views

How to split text file into records by first non-blank character using awk and print only records containing string

I have a text file with information separated in blocks. I would like to separate these blocks into records so that I can handle them separately. This is my text file (file.txt): Alarm Stats: com....
0
votes
1answer
36 views

have a log file that I would like to break up into several files based on content

I have a log file that I would like to break up into several files Table of Contents *** Configuration Database *** Logs *** Configuration Logs *** Query Logs . . and so on The issue ...
3
votes
1answer
45 views

Modify values in a column based on number of colons it contains

I would manipulate the contents of the second column based on the number of colons it contains. If the second field contains more than one colon then I would require the content before first colon ...
1
vote
2answers
83 views

Print text between multi-line delimiters

I am trying to extract data between 2 matching patterns, only if the content is not empty and I have issues doing it: here is an example: ============================== Queue Manager is : ...
4
votes
7answers
734 views

How to split a filename list in 5GB sets?

I have a pretty big list of files which I like to group in smaller list of 5GB each for further processing. The file list is in the following form: "size" (from du) "tab" "absolute path", like this (...
1
vote
1answer
63 views

AWK, Sum of category

I have tons of CSV files with similar content. The values are usually comma separated and they look like this. product_a, domestic, 500 product_a, abroad, 15 product_b, domestic, 313 ...
1
vote
2answers
66 views

How can I use Awk to convert a unix timestamp to human readable when doing a Tail -f logfile?

I'm currently doing a tail -f mysql.log and also have it color coded. But how would I use awk to convert the unix timestamps to human readable format? tail -f /var/lib/mysql/mysql-slow.log | awk '/...