awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
3answers
74 views

Creating a new file inside awk but in different directory based on input field value

I am trying to work with awk datetime=`date +\%Y\%m\%d` logdatetime=`date +\%Y\%m\%d` foldermonth=`date +\%B_\%Y` folderday=`date +\%d` inputdir="~/sboper/Standalone/fsplit/GLMS" ...
3
votes
3answers
90 views

How to do a loop to process two input files

I am using awk to do match two files and then multiply the elements of file2 and file1. awk 'NR == FNR{a[$1]=$2; b[$1]=$3; next} /:/ || !NF{print; next} {print $1, $2*a[$1], $2*b[$1]}' ...
2
votes
6answers
118 views

Comparing 3 lines and 2 lines in BASH

I have a text file which looks like this. 2014-11-24 12:59:42.169 101.0.0.0 source 2014-11-24 12:59:40.375 104.156.80.0 destination 2014-11-24 12:59:36.729 104.219.48.0 destination 2014-11-24 12:59:...
0
votes
2answers
58 views

Comparing two lines in BASH using awk

I have text file like this. 2015-11-24 12:59:37.112 128.206.6.136 source 2014-11-24 12:59:36.920 8.8.8.8 source 2014-11-24 14:59:38.112 23.234.22.106 destination 2014-11-24 13:59:37.113 23.234.22.106 ...
1
vote
2answers
80 views

After small code change 'awk' prints header and blank lines, (not header repeated)

data-file content: k = 0.1667 0.0962 0.0000 ( 4922 PWs) bands (ev): -7.1689 -7.1043 -6.1682 -6.0658 1.1309 1.5196 3.0894 3.1087 3.3846 3.7588 4.2143 4.2220 4.3882 ...
3
votes
4answers
646 views

how to change one column and add other columns in a row by awk

I have a file MDD.TXT with following records. file : MDD.TXT 0|ABC|055309|20120929053309|11965150 0|ABC|061434|20120929063434|11155537 0|ABC|070228|20120929073228|11965150 0|ABC|082213|...
0
votes
3answers
68 views

Awk command to get the next line from the matched pattern and crop the string in between two patterns

I need to get the next line after matched pattern and need cut or crop the value in between two patterns from that line. Sample Source file <h2>Commodity Information</h2> <dl> ...
-1
votes
2answers
163 views

How to sum many numbers inside 2D array using awk

I need to sum numbers located in a file like this one: column1 column2 column3 row1 a(1,1) a(1,2) a(1,3) row2 a(2,1) a(2,2) a(2,3) row3 a(3,1) a(3,2) a(3,3) row4 a(4,1) ...
1
vote
1answer
40 views

Awk hangs during math operation on Solaris

I'm working on a SunOS 5.11/Solaris 11.3 machine. I have a bash script that calculates and exports CPU frequency because I use it frequently in some test scripts. Here are the two lines of interest: ...
1
vote
4answers
853 views

How to extract column name (header) from a CSV file which contains the max value in a row?

I am trying to extract the column name with the maximum value in each row using bash script i.e., the column header value or the value from the same column in the first row. I am using the following ...
0
votes
3answers
30 views

Extract a part of one line and sort

I have output similar to the following: [test output] my name is bob this is a random string [test output] my name is jim this is a randong string2 [test output] my name is bob this is a randong ...
1
vote
2answers
54 views

How to check if some recurring text is missing in a text file?

I have a text file that looks like this: { "mimeType": "web", "body": "adsfdf", "data_source_name": "abc", "format": "web", "url": "http://google.com/", "urls": "...
-2
votes
2answers
37 views

Data Manupulation required your help

I have a data as mentioned below. host_name Server1.domain.com contacts DL - Desktop contact_groups ravi, raj, rahim host_name Server2.domain.com contact_groups DL-Server host_name Server3....
1
vote
1answer
65 views

Convert vertical array data to horizontal array data

I have a file with 25 matrices in a file, and each matrix is has 128 rows and 5 columns. Each matrix are putted one after the other, in a vertical way. Like this File input(file1): Array1 Array2 ...
5
votes
4answers
12k views

Using bash shell function inside AWK

Is it possible to use bash function inside AWK somehow? Example file (string, int, int, int) Mike 247808 247809 247810 Trying to convert values from decimal to hexadecimal. Function defined ...
2
votes
0answers
54 views

how to capture the latest kernel version from grub.conf

how to capture the latest kernel version from grub.conf ( in case default=0) for example the latest kernel from the file is 2.6.18-409.el5 # all kernel and initrd paths are relative to /...
1
vote
2answers
62 views

Merging Two Files Columns

I want to add 6 field of File2 in last field of File1, i'm using following command but unable to get output. awk 'FNR==NR{a[$1]=$6; next} {print $0,a[$1]}' File2 File1 File1: zehriscollection.co....
0
votes
1answer
40 views

How to get the data between specified Starttime and Endtime in a log file?

This is my LogFile(Test.Log): May 10 01:06:05 A May 10 01:06:06 B May 10 01:06:07 C May 10 01:07:01 D May 10 01:07:02 E May 10 01:07:09 F May 10 01:07:11 G May 10 01:07:11 H May 10 01:07:12 I May 10 ...
3
votes
1answer
59 views

awk or sed command to start each column of file from desired locations or Byte Position

I have a source file format like below containing 15 columns and so many rows: 23.5 31.5 5.00 255 0 255 2 1 sal unknown sa time sps meter ms 25.5 32.5 6.00 0 255 0 2 2 sal unknown sa time sps ...
2
votes
1answer
50 views

How to choose the highest number element in many files with many arrays

I have many files which each one contain many arrays(like input 1 and 2, below). Each array has two element numbers in the second row. I need to use a logical criterion to choose the highest number ...
16
votes
7answers
25k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
3
votes
3answers
52 views

Grep/awk/sed for lines composed of only two letters, and lines that start with a letter and meet a certain length

Not sure how many people are familiar with DNA sequencing data, but if this is part of my file (lines starting with ">" are IDs, and lines starting with a letter are DNA sequences): >NB501013:9:...
0
votes
2answers
40 views

Sum column numbers elements in many arrays

I need to sum the the numbers located in the same column in an array. The input is: 1: 6.1703 44 3.5 0.3 46 7.3 2.3 2: 6.1932 44 5.29379 2.9 46 1.72261 7.2261 3: 6.5664 45 4.756 5.6 46 1.6 2.5 4:...
0
votes
0answers
33 views

awk or sed Commands to Print Columns at desired locations [duplicate]

I have a source file which contain 15 columns and so may rows: 2973695.500 633952.188 2440.000 255 0 255 1 1 OP UNK opt TIME SKZ meter ms 2973695.500 633952.188 2440.000 0 255 255 1 1 ...
-1
votes
0answers
22 views

How to copy data from a text file to a particular Excel row (or) column using shell script

I am facing the problem in copying text data into a particular Excel column using a shell script. I have used the following command awk -v cols=6 '{printf "%s%s", $1, (NR % cols == 0 ? "\n" : "\t")}...
0
votes
0answers
27 views

get user input in terms of size (GB or MB)

I created script to calculate shmmax shmall shmmini Total_Memory_M=$(dmidecode -t 17 | grep Size | awk '{print $2}' | grep -o '[0-9]\+' | paste -sd+ | bc) Total_Memory_G=$(echo $Total_Memory_M /1024 |...
2
votes
1answer
35 views

How join two files by one common column (1ª) between two file (File1 and file2)?

I am trying join two files by one common column (1ª) between two file (File1 and file2), resulting in file 3, but the resultant file is not working. Can someone help me? I used this command: awk 'NR=...
3
votes
1answer
889 views

Awk compare 2 files, print match together with non match lines filled with 0

I have two files. I want to match the 1st column of first file to 1st column of second file. If there is a match I'd like to print the 2st column of second file, if not 0. I have tried awk (awk 'FNR=...
1
vote
4answers
64 views

Extract X number of minutes from the current time from a logfile

I have a logfile with the following timestamp format 2016-06-02 13:59:58.069 The log I am working with spans about an hour. 2016-06-02 13:05:06.144 2016-06-02 13:05:06.144 2016-06-02 13:05:06.160 ...
5
votes
2answers
407 views

Looking for lines which is in one file but not in other using Unix and Awk [duplicate]

I have 2 files with 7 fields and I want to print the lines which are present in file1 but not in file2 based on field1 and field2. Logic: I want to print all the lines, where there is a particular ...
5
votes
4answers
10k views

How to remove duplicate lines with awk whilst keeping empty lines?

Below awk command removes all duplicate lines as explained here: awk '!seen[$0]++' If the text contains empty lines, all but one empty line will be deleted. How can I keep all empty lines whilst ...
1
vote
4answers
64 views

How to use Unindented line as the Record Separate in awk cli

I have a log file that look like: 2016-05-31 09:54:36 (16667) heritage_w? From: ip68-8-49-100.sd.sd.cox.net User: user1wizard (wizard) Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/...
1
vote
1answer
46 views

How to sum specific rows of a matrix

I need to sum the matrix elements of specific rows for each column. An example of the matrix is given below(input). So the matrix has 4 columns and 6 rows of numbers. I need to sum the numbers of ...
1
vote
2answers
41 views

How to append some string combined with the pattern at the end of line with the pattern

I want to append .com at the end of line for every line matching the pattern in /etc/hosts file. Sample file content: 127.0.0.1 localhost 1.2.3.4 hostname1 hostname1.xyz hostname1.xyz. 1.2.3....
3
votes
1answer
51 views

awk + how to perform OR in awk

by the following command I capture available size from df echo "$DF" | awk '$NF == "/var" { print $(NF-2) }' how to set the syntax if I want to set NF == "/var" or NF == "/" because I ...
0
votes
1answer
41 views

How to use TWO “for statements” in a shell script?

Writing script to create user accounts. Script is written to echo the useradd command (so I can make sure it will work as expected). I have a file newemploy.txt with Employee Names (first and last, ...
2
votes
3answers
63 views

Mysql output format in Bash Script

I'm writing a bash script where I'm using mysql query in that script where the format which I'm getting is quite different. Query: root@debian:~# mysql -u root -ptoor super_market -h 0 -e "select * ...
15
votes
4answers
73k views

How to run grep on a single column?

I want to grep the output of my ls -l command: -rw-r--r-- 1 root root 1866 Feb 14 07:47 rahmu.file -rw-r--r-- 1 rahmu user 95653 Feb 14 07:47 foo.file -rw-r--r-- 1 rahmu user ...
3
votes
6answers
740 views

How do I remove a words ending in the letter S if duplicates without an S on the end are on the same list?

I have a large list of words. Many of the words are only different because they have the letter s on the end. If a word on the list is the exact same as another word on the list, except one of the ...
1
vote
1answer
46 views

completely ignore lines that start with a specific pattern

I have a script which analyzes the output of a computation software. sometimes the output comes with some extra data that are irrelevant to my script. This data can be huge and makes running my simple ...
0
votes
1answer
26 views

How to move lines between patterns to the end of a file

I want to move several lines to the end of the file. Example: 1 2 3 4 5 6 blkn XYZ indf 7 8 9 10 11 lnkn ABCYDF knnfd 12 13 14 15 I want to move lines 6 to 11 to the end of the file. The patterns ...
23
votes
7answers
63k views

How to read first and last line from cat output?

I have text file. Task - get first and last line from file after $ cat file | grep -E "1|2|3|4" | commandtoprint $ cat file 1 2 3 4 5 Need this without cat output (only 1 and 5). ~$ cat file | ...
0
votes
1answer
18 views

Is there a way to get all the NEWS.gz which is/was this year in /usr/share/doc/ of $PACKAGENAME

If people go into /usr/share/doc/$PACKAGENAME (you can use any valid package name which you have installed on your system) you will see that the directory has various files in it, at the very least ...
-5
votes
1answer
42 views

What happen if record separator and field separator are the same

In awk, what happen if record separator RS and field separator FS are the same? How can I tell fields from records?
1
vote
1answer
45 views

Need to filter the data on the basis of two column having many to one relationship using awk

I have a big file having 50s columns and 100K of rows delimited by |. Now $2(col 2) has multiple type of $1(col 1) value which means col 2 will be repeated. So I have sorted the file. I need now to ...
2
votes
1answer
58 views

on-line tally of unique lines

uniq -c is very useful for counting the number of times the same line appears consecutively $ seq 1 1000 | awk '{ if ($1 > 100 && $1 <= 200) { print "hi" } else {print "bye"} }' | uniq -...
3
votes
1answer
62 views

Rename files based on JSON content

I have two sets of files; file1.txt file1.json file2.txt file2.json file3.txt file3.json ... fileN.txt fileN.json json files contain the following format: {"ago": "59 sec ago", "...
4
votes
2answers
776 views

How can I round all numbers in a file?

How can I round all numbers in a file that countains several columns of numbers up to a certain precision? Can this be done with awk? A single line looks like this: text - 0.1655456615 - 0.158645 - ...
11
votes
3answers
753 views

How do you remove dot character from string without calling sed or awk again?

I have a file called hostlist.txt that contains text like this: host1.mydomain.com host2.mydomain.com anotherhost www.mydomain.com login.mydomain.com somehost host3.mydomain.com I have the ...
2
votes
4answers
65 views

Can Awk read escaped newlines? I.e. read past newlines

I have input looking like this: entry1line1 entry2line1\ entry2line2\ entry2line3 entry3line1 I would like to read such input into an array in AWK, to then process it together with the embedded ...