awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

0
votes
1answer
99 views

Single read filter csv based on 1st column values

I have a gigantic csv file that is around a GB in size. The csv's 1st column contains around 300 unique integer values. I want to process the csv once and save the output to different files based on ...
1
vote
2answers
51 views

Extract keyword from line

I need to extract similar patterns from data below. The pattern starts with '.' or '' then to the keyword I want, then followed by ''. We can assume the format is LSUFE-numbers_ PCRF-numbers_ ...
1
vote
2answers
58 views

Stack data from multiple delimited files into one, with variable columns

I'm new to scripting. Got struck with a file merge issue in Unix. Was looking for some direction and stumbled upon this site. I saw many great posts and replies but couldn't find a solution to my ...
4
votes
1answer
90 views

How to compare float values in a shell script?

I need to alert if my load average is more than 10. I could do it like this if my awk command returned an integer: declare -i MAX_LOAD=10 declare -i L1=`cat /proc/loadavg | awk {'print $1'}` if [ $...
1
vote
1answer
21 views

Paragraph based searching using before and after conditions UNIX KSH

I have a file like: <Set of long paragraphs here> <Set of long paragraphs here> <Set of long paragraphs here> After Address1 Address2 Address3 Before <Set of long paragraphs here&...
3
votes
3answers
91 views

How can I remove lines under a sub heading in a file using linux commands

I have a file as below . In which I want to remove lines under a particular sub heading alone. I tried using some sed and awk commands but I couldn't get it. Can any one help to crack this with some ...
1
vote
2answers
632 views

How can I print the second to last line of many files into one file? [duplicate]

I have many CSV files in one directory which have various lengths. I'd like to put the second to last line of each file into one file. I tried something like tail -2 * | head -1 > file.txt, then ...
3
votes
2answers
156 views

Bypass a nawk snippet if the input file is empty

I have a script which generates several output files and use these output files during runtime. Following are the some of the files generated by the script: apple.txt, section_fruit_out.csv, ...
1
vote
2answers
115 views

Efficient way to search array in text file by AWK

I have one array SPLNO with approx 10k numbers.Now i want to search the subscriber number from MDN.TXT file (containing approx 1.5 lac record)from the array.if subscriber number found in array it will ...
3
votes
2answers
86 views

different results from awk and grep

This gives result 124: awk 'BEGIN {FS = ","; count = 0}; { if ($7 ~ /Nature Life/) { count++ }} END {print count}' file.csv This gives result 123: grep -cE '^([^,]*,){6}[^,]*Nature Life' file....
4
votes
3answers
3k views

How to grep lines between start and end pattern? [duplicate]

I have a file which is having following content: zdk aaa b12 cdn dke kdn Input1: aaa and cdn Output 1: aaa b12 cdn Input 2: zdk and dke Output 2: zdk aaa b12 cdn dke I could use below ...
1
vote
1answer
62 views

Find multiple matches in a tabular file and print second column?

I have a tabular file like this which is my index: a X001 a X002 a X003 b X002 c X006 z X007 z X008 z X001 I want to search the following single column file against that index and ...
0
votes
2answers
144 views

Prepend lines with file prefix using sed or awk

How do I change a sample input file so as to get something like the output file below? Input file: /tmp/logs 10/13/15 00:19:13 templogs1 10/13/15 00:19:13 dummylogs 10/13/15 01:19:12 test 10/13/15 ...
7
votes
5answers
681 views

How to programmatically detect awk flavor (e.g. gawk vs nawk)

I'm using a command-line application which is essentially a collection of bash shell scripts. The application was written to run on BSD/OSX and also on Linux. One of the scripts relies on awk. It ...
0
votes
4answers
222 views

Check if a field of a file exists in another file and update the the first file's specific column

I have 2 files as below: a.txt (tab delimited and it has 3 columns and too many lines): city plate region istanbul 36 marmara trabzon 61 karadeniz b.txt (comma delimited and it has 4 ...
2
votes
1answer
221 views

Why the value of $1 changed in `{$total=$total+$1}`?

I wrote an awk command try to count the total value of a particular column, but something unexpected happened: # THE CONTENT OF TEST FILE 1 2 3 4 # MY AWK COMMAND awk 'BEGIN {total=0} {print $1; $...
0
votes
0answers
14 views

AWK searching for numeric patterns in a string [duplicate]

I'm trying to remove any occurrence of six-digit numbers occurring in my data. for instance: "This is test data 765648.txt" would become "this is test data.txt" and "This example 457623 has embedded....
2
votes
3answers
79 views

Bash script to echo the first positional of each line into file with name of the second

I have a text file that is organized as follows: Group 1 asdsdsdsdf.html jeff xcvxcvxcvx.html bob vrgeiuvhif.html sue Group 2 iwdowijdoi.html mary pokpompojm.html doug ndkjfsjfbs.html lisa I ...
2
votes
1answer
295 views

Extract hostnames from non-hashed ssh known_hosts

I would like to extract hostnames/ips from .ssh/known_hosts file in all (known) format conditions: Hostname only hostname pubkey Hostname with ip: hostname,1.1.1.1 pubkey Hostname with ip that is ...
0
votes
2answers
1k views

Shell Script to remove double quotes within the column value

I have a input text file with 10 columns, while processing this file, in one of the middle column, I am getting this type of data.I required column value to be as below: Input Column value: "This is ...
0
votes
1answer
62 views

How do I sort a file and output the latest record in case duplicates exist?

I save all the requests, received via API, in a file. I wish to process only the latest request if duplicate requests arrive. Sort command is not helping, I am using : sort -r -k 1.69,1.88 ./...
2
votes
4answers
154 views

Only leave longest line when first column is duplicate

INPUT: user@server:~/bar/foobar$ SAT=$(date -dsaturday +%Y-%m-%d); SUN=$(date -dsunday +%Y-%m-%d) user@server:~/bar/foobar$ awk 'BEGIN {FS="'^'"} {print $1"\t"$3"\t"$6}' STs.csv STt.csv | egrep -w "$...
0
votes
2answers
79 views

awk command with multiple commands

When i am trying below code getting error like - + awk '{if ($1 > 1) { print "Memory utilisation is high \n Please find history of the memory utilisation below" sar -r|awk {print' ',,,}| column -...
0
votes
2answers
42 views

need to replace end of column with a character if it does not already exist

I am trying to use awk but can use anything else that will work lets say i have a file with these lines ewhueh wnlknlweknlnw/ qwejmdpdhhd 47578 enhkine enoekjohnkjiof elkjehbjuwekjuk 88989 ehnjkine ...
3
votes
2answers
68 views

Join (large) files on alphanumeric pattern

I have: File 1 like: sting_of_printable_characters*sting_of_printable_characters*sting_of_printable_characters*ALPHANUMERIC_PATTERN File 2 like: sting_of_printable_characters*ALPHANUMERIC_PATTERN ...
1
vote
1answer
192 views

BASH AWK Script add days to fixed date

My task is to export data from a database with the UNLOAD TO statement in dbaccess and convert it into a file, that can be inserted into a similar database. Now the problem: The old database (let's ...
0
votes
2answers
35 views

filter data by column's value

I want to search only lines where a column start with 03: (hours), but while using below command, showing all the rows where value contain 03. Its an example the data is pretty much huge in a file. ...
0
votes
2answers
42 views

Find the position of repeated character “&”

How can determine the position of “&” in the column number 5 to substring the value 2 or substring value 3 by awk functions column1|column2|column3|column4|variable1:value1 & variable2:...
0
votes
2answers
112 views

awk output has extra carriage returns and is buffered

My ultimate goal is to get a zenity progress dialog to work while encoding with fdkaac. I first started with some code that is working when I encode "aa.wav" to "aa.mp3" with lame. This results in a ...
1
vote
1answer
64 views

Find matches from an index file without exact matching and print the last field

I have a large file (~6mill rows) with 2 columns which looks like this: 1111.aaaaabbb.b.cccc.c ValueA 2222.dddddeee.e.ffff.f ValueB 3333.gggghhhh.h.iiii.i ValueC I want to use that as my index ...
0
votes
1answer
31 views

Merging and appending tables to create a master table of multiple tables

How to we can merge and append the multiple tables in the same format having same as well as different data point and merge all of the together by inputting zero to the null data points. Desired input ...
5
votes
4answers
133 views

Build table - Add column depending on filenames

I have a huge amount of files having the following naming style: WBM_MIROC_rcp8p5_mississippi.txt WBM_GFDL_rcp8p5_nosoc_mississippi.txt DBH_HADGEM_rcp4p5_co2_mississippi.txt ...
1
vote
2answers
25 views

Summation for a field related to another one.

How can I find the summation for a certain field that is related to another field by awk, consider below sample: File: boo,foo,VO,1000 boo,foo,VO,1000 bar,var,DT,1000 var,bar,VO,2000 in this ...
4
votes
4answers
213 views

print only lines where the first column is unique

I am looking for a way to sort a list and print all lines, whose first column appears only once - i.e., match only on the first column. For example, I have a file where the first column is a path and ...
6
votes
2answers
446 views

Delete multiple lines in a file

Usually I find my answers in the previously answered questions, but not this time (or I don't understand the answers and can't modify them to do what I want them to do), so I'm asking my first ...
2
votes
1answer
148 views

Adding two columns together in CSV and outputting to new CSV file

I have a problem I have been trying to figure out: We have a stock file CSV, which contains the stock in multiple locations. The csv looks like this: stock_no,primary,secondary,tertiary,cstock,...
4
votes
6answers
172 views

Make longer strings from substrings keeping new line between each string

I have a file that contains strings as: GTACGACGGAGTGTTATAAGATGGGAAATCGGATACCAGATGAAATTGTGGATCGGTGCAAAA GTCGGCAGATATCGTTGAAGTCATAGGTGATTATGTTCAATTAAAGAAGCAAGGCCGAAACTAC ...
0
votes
1answer
63 views

Finding Matches between two columns of different files and printing it out a different column

I was trying to figure out how to see if there were any matches between column 2 of file1 and column 6 of file2. If so, I wanted that match printed along with the respective string in column 14 of ...
3
votes
3answers
114 views

Passing variables to the sed command,

We have a file containing 6 columns. FILEDESCRIPTOR . DESCR00001 FILEDESCRIPTOR . DESCRIPTIVENAME FILEDESCRIPTOR . LSTUPDNAM FILEDESCRIPTOR . LASTUPDATENAME FILEDESCRIPTOR . LOCAT00001 ...
4
votes
3answers
87 views

awk ward problem

I'm trying to use awk to pull some different fields from a string. Example: selected="a2.flac|a3.flac|a4.flac" first_file=$(echo "$selected" | awk -F'|' '{print $1}') The above code returns a2....
0
votes
1answer
45 views

Search for a pattern in a file and move it to the new files one by one

I am trying to search for a particular pattern in a file and move it to a new files one by one. I am a noob and I tried sed -n 's/CREATE PROCEDURE/,/END ;/p' File It is again printing the same file. ...
0
votes
2answers
33 views

Passing timestamp variable from shell to gawk [duplicate]

I want to take time difference of two time stamp but getting an error while passing timestamp variable from shell toawk. Shell code: FTIMESTAMP="2015-07-01 12:30:50" gawk -v FTIMESTAMP=$FTIMESTAMP -...
1
vote
2answers
171 views

Printing all lines of a file with multiple columns in a shell script

I have a file with 4 columns with the below format: name;user_id;dob;date_of_joining I want to print all the variables in below output via shell script The employee "$name" is having "$user_id" ...
1
vote
4answers
108 views

Add a space or tab before the last character in each line

I have a file with a list of values that end with a letter. Is there an easy way to add a space or tab before the last character of each line to separate the number from the letter? 44A 354T ...
1
vote
1answer
128 views

How can I write a bash script that sends a graphical notification to all users' X displays?

This is a request for improvement and update (NOT a duplicate!) on these 2 questions & answers: Can I launch a graphical program on another user's desktop as root? Show a notification across ...
1
vote
2answers
122 views

How can I use the environment variables inside a regular expression in awk?

#!/bin/sh ALTER="1153" NAME="aio" for i in $(ps -eo pid,etime,comm | \ awk -v alt="$ALTER" -v name="$NAME" '$2~/[^0-alt]-/ && $3~/name/{print$1} '); do echo $i done
12
votes
6answers
6k views

How to delete the last column of a file in Linux

I want to delete the last column of a txt file, while I do not know what the column number is. How could I do this? Example: Input: 1223 1234 1323 ... 2222 123 1233 1234 1233 ... 3444 125 0000 5553 ...
-1
votes
2answers
54 views

how to run a command in many files by running the command only one time?

I would like to run following command on 31 separated file name chr1.txt ... chr31.txt , and get 31 outputs only by one run. this is command : cat input.txt | sed 's/.\{5\}/& /g' > output....
2
votes
1answer
48 views

How to limit printed output based on number of occurences (AWK)

So I am trying to track down individuals on our network surfing porn via https. The pattern is pretty easy to see, 15+ hits in a row for images without a referrer in the HTTP header on our Bro logs. ...
3
votes
2answers
107 views

Using find -exec how to print name of file in front of each line?

Similar to, but beyond: find . -iname "somepatter*txt" -print -exec cat {} \; How might the file name from each file be printed in front of each line with the results from find?