awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

2
votes
3answers
84 views

Replace a column and preserve spacing

This is a follow up to unix: replace one entire column in one file with a single value from another file I am trying to replace one column of a file (file1) with one specific value from another file ...
3
votes
3answers
217 views

Awk script to extract a portion of input file

How do I write a awk script that scans the input file for "start" and finds the line containing "next" and displays the following line? Something like this: [user]$ cat test.txt start ...
0
votes
2answers
25 views

Search for a non exact match and return column 1 of that file

I have a tab seperated file which looks like this: Adam Group1,Group2,Group8,Group9,Group10 James Group1,Group11,Group12,Group7 Kelly Group18 Alfred Group1,Group8,Group10 Manuel ...
1
vote
1answer
85 views

How to merge multiple files based on a timestamp

I have 2 files File 1: 01:12:00,001 Some text 01:14:00,003 Some text 02:12:01,394 Some text File 2: 01:12:00,001 Some text 01:12:01,029 Some text 01:13:21,123 Some text I need output as ...
0
votes
1answer
46 views

Quick shell (bash+awk) script to extract a column using the output of another command

Have to repeat below 100+ times. So first command takes paramater $1 and get me string A001C7F28AB20. Then second command is using that string. # Command1 ->snmpget -v1 -c elephantblue $1 ...
0
votes
2answers
43 views

Combine rows from different files, if pattern is matched

Input: two tab-delimited files, File1 and File2. File2 ($2) has to be parsed by patterns found in File1 ($1). Expected output: tab-delimited file, File3. File3 has to contain the same rows as File2, ...
2
votes
3answers
255 views

unix: replace one entire column in one file with a single value from another file

I want to replace a column in one file with a single value from another file using unix. File 1 is a pdb file, structured like this: HETATM 14 H4B FAD B 600 95.544 50.240 71.308 1.00 ...
2
votes
5answers
502 views

Replace characters in user part of email address

I have a list that looks like this: user-name@domain.example user.name@domain.example user_name@domain.example ... I would like to replace any occurence of a dot in the username part. So that: ...
2
votes
3answers
98 views

How to handle ^M in csv files by using sed & awk?

I have a file with delimiter ~, and was not able to load into database: Input file: 24~1~2~2~unsubscribed~wap~7~2015-10-23 20:03:00~2015-10-24 20:04:00~2015-10-25 02:53:26~Testing duplicate msisdn ...
2
votes
2answers
30 views

bash: how to add assigned value to file

I have bash script looping through several files in a directory. The file identifier has been saved as a variable $num. What I would like to do is add $num as another column in my data like in the ...
0
votes
1answer
47 views

Remove trailing spaces from first field only

I have a multi-lined, tab-delimited file like this: 99999bbbbbt999999bbt99t99t99t99 (where b=spaces, t=tabs) Trailing spaces in the 2nd field are ok, but the trailing spaces in the 1st field need ...
0
votes
2answers
41 views

Merge two lines of a file and adding number in the end of lines

I have a large text file abc.txt with the following format: BALT -6.110 130.600 4.06874414 -0.03254425 BALT -6.620 154.460 3.92286595 -0.17842244 BARM 32.740 140.680 3.94326190 -0.15802649 I need ...
3
votes
3answers
97 views

awk when both delimiter and quotes are used for a field

I have a file in the following format: field1|field2|field3 field1|"field2|field2"|field3 Notice the second row contains double quotes. The string within the double quotes belongs to field 2. How ...
0
votes
1answer
54 views

Single read filter csv based on 1st column values

I have a gigantic csv file that is around a GB in size. The csv's 1st column contains around 300 unique integer values. I want to process the csv once and save the output to different files based on ...
1
vote
2answers
42 views

Extract keyword from line

I need to extract similar patterns from data below. The pattern starts with '.' or '' then to the keyword I want, then followed by ''. We can assume the format is LSUFE-numbers_ PCRF-numbers_ ...
1
vote
2answers
48 views

Stack data from multiple delimited files into one, with variable columns

I'm new to scripting. Got struck with a file merge issue in Unix. Was looking for some direction and stumbled upon this site. I saw many great posts and replies but couldn't find a solution to my ...
4
votes
1answer
69 views

How to compare float values in a shell script?

I need to alert if my load average is more than 10. I could do it like this if my awk command returned an integer: declare -i MAX_LOAD=10 declare -i L1=`cat /proc/loadavg | awk {'print $1'}` if [ ...
1
vote
1answer
21 views

Paragraph based searching using before and after conditions UNIX KSH

I have a file like: <Set of long paragraphs here> <Set of long paragraphs here> <Set of long paragraphs here> After Address1 Address2 Address3 Before <Set of long paragraphs ...
3
votes
3answers
79 views

How can I remove lines under a sub heading in a file using linux commands

I have a file as below . In which I want to remove lines under a particular sub heading alone. I tried using some sed and awk commands but I couldn't get it. Can any one help to crack this with some ...
1
vote
2answers
241 views

How can I print the second to last line of many files into one file? [duplicate]

I have many CSV files in one directory which have various lengths. I'd like to put the second to last line of each file into one file. I tried something like tail -2 * | head -1 > file.txt, then ...
3
votes
2answers
142 views

Bypass a nawk snippet if the input file is empty

I have a script which generates several output files and use these output files during runtime. Following are the some of the files generated by the script: apple.txt, section_fruit_out.csv, ...
1
vote
2answers
93 views

Efficient way to search array in text file by AWK

I have one array SPLNO with approx 10k numbers.Now i want to search the subscriber number from MDN.TXT file (containing approx 1.5 lac record)from the array.if subscriber number found in array it will ...
3
votes
2answers
81 views

different results from awk and grep

This gives result 124: awk 'BEGIN {FS = ","; count = 0}; { if ($7 ~ /Nature Life/) { count++ }} END {print count}' file.csv This gives result 123: grep -cE '^([^,]*,){6}[^,]*Nature Life' ...
4
votes
3answers
937 views

How to grep lines between start and end pattern? [duplicate]

I have a file which is having following content: zdk aaa b12 cdn dke kdn Input1: aaa and cdn Output 1: aaa b12 cdn Input 2: zdk and dke Output 2: zdk aaa b12 cdn dke I could use below ...
1
vote
1answer
53 views

Find multiple matches in a tabular file and print second column?

I have a tabular file like this which is my index: a X001 a X002 a X003 b X002 c X006 z X007 z X008 z X001 I want to search the following single column file against that index and ...
0
votes
2answers
102 views

Prepend lines with file prefix using sed or awk

How do I change a sample input file so as to get something like the output file below? Input file: /tmp/logs 10/13/15 00:19:13 templogs1 10/13/15 00:19:13 dummylogs 10/13/15 01:19:12 test 10/13/15 ...
6
votes
5answers
510 views

How to programmatically detect awk flavor (e.g. gawk vs nawk)

I'm using a command-line application which is essentially a collection of bash shell scripts. The application was written to run on BSD/OSX and also on Linux. One of the scripts relies on awk. It ...
0
votes
4answers
91 views

Check if a field of a file exists in another file and update the the first file's specific column

I have 2 files as below: a.txt (tab delimited and it has 3 columns and too many lines): city plate region istanbul 36 marmara trabzon 61 karadeniz b.txt (comma delimited and it has 4 ...
2
votes
1answer
215 views

Why the value of $1 changed in `{$total=$total+$1}`?

I wrote an awk command try to count the total value of a particular column, but something unexpected happened: # THE CONTENT OF TEST FILE 1 2 3 4 # MY AWK COMMAND awk 'BEGIN {total=0} {print $1; ...
0
votes
0answers
14 views

AWK searching for numeric patterns in a string [duplicate]

I'm trying to remove any occurrence of six-digit numbers occurring in my data. for instance: "This is test data 765648.txt" would become "this is test data.txt" and "This example 457623 has ...
2
votes
3answers
57 views

Bash script to echo the first positional of each line into file with name of the second

I have a text file that is organized as follows: Group 1 asdsdsdsdf.html jeff xcvxcvxcvx.html bob vrgeiuvhif.html sue Group 2 iwdowijdoi.html mary pokpompojm.html doug ndkjfsjfbs.html lisa I ...
2
votes
1answer
162 views

Extract hostnames from non-hashed ssh known_hosts

I would like to extract hostnames/ips from .ssh/known_hosts file in all (known) format conditions: Hostname only hostname pubkey Hostname with ip: hostname,1.1.1.1 pubkey Hostname with ip that is ...
0
votes
2answers
361 views

Shell Script to remove double quotes within the column value

I have a input text file with 10 columns, while processing this file, in one of the middle column, I am getting this type of data.I required column value to be as below: Input Column value: "This is ...
1
vote
2answers
47 views

awk regex for apache rewriterule

I have these variant of urls RewriteRule ^/trendy/the-reason-for-example/? http://www.example.com/trendy/the-reason [NC, L, R=301] RewriteRule ^/lol/2015/10..._for-example ...
0
votes
1answer
52 views

How do I sort a file and output the latest record in case duplicates exist?

I save all the requests, received via API, in a file. I wish to process only the latest request if duplicate requests arrive. Sort command is not helping, I am using : sort -r -k 1.69,1.88 ...
2
votes
4answers
150 views

Only leave longest line when first column is duplicate

INPUT: user@server:~/bar/foobar$ SAT=$(date -dsaturday +%Y-%m-%d); SUN=$(date -dsunday +%Y-%m-%d) user@server:~/bar/foobar$ awk 'BEGIN {FS="'^'"} {print $1"\t"$3"\t"$6}' STs.csv STt.csv | egrep -w ...
0
votes
2answers
60 views

awk command with multiple commands

When i am trying below code getting error like - + awk '{if ($1 > 1) { print "Memory utilisation is high \n Please find history of the memory utilisation below" sar -r|awk {print' ',,,}| column ...
0
votes
2answers
33 views

need to replace end of column with a character if it does not already exist

I am trying to use awk but can use anything else that will work lets say i have a file with these lines ewhueh wnlknlweknlnw/ qwejmdpdhhd 47578 enhkine enoekjohnkjiof elkjehbjuwekjuk 88989 ehnjkine ...
3
votes
2answers
58 views

Join (large) files on alphanumeric pattern

I have: File 1 like: sting_of_printable_characters*sting_of_printable_characters*sting_of_printable_characters*ALPHANUMERIC_PATTERN File 2 like: sting_of_printable_characters*ALPHANUMERIC_PATTERN ...
1
vote
1answer
117 views

BASH AWK Script add days to fixed date

My task is to export data from a database with the UNLOAD TO statement in dbaccess and convert it into a file, that can be inserted into a similar database. Now the problem: The old database (let's ...
0
votes
2answers
28 views

filter data by column's value

I want to search only lines where a column start with 03: (hours), but while using below command, showing all the rows where value contain 03. Its an example the data is pretty much huge in a file. ...
0
votes
2answers
38 views

Find the position of repeated character “&”

How can determine the position of “&” in the column number 5 to substring the value 2 or substring value 3 by awk functions column1|column2|column3|column4|variable1:value1 & ...
0
votes
2answers
81 views

awk output has extra carriage returns and is buffered

My ultimate goal is to get a zenity progress dialog to work while encoding with fdkaac. I first started with some code that is working when I encode "aa.wav" to "aa.mp3" with lame. This results in a ...
1
vote
1answer
45 views

Find matches from an index file without exact matching and print the last field

I have a large file (~6mill rows) with 2 columns which looks like this: 1111.aaaaabbb.b.cccc.c ValueA 2222.dddddeee.e.ffff.f ValueB 3333.gggghhhh.h.iiii.i ValueC I want to use that as my index ...
0
votes
1answer
29 views

Merging and appending tables to create a master table of multiple tables

How to we can merge and append the multiple tables in the same format having same as well as different data point and merge all of the together by inputting zero to the null data points. Desired input ...
1
vote
2answers
23 views

Summation for a field related to another one.

How can I find the summation for a certain field that is related to another field by awk, consider below sample: File: boo,foo,VO,1000 boo,foo,VO,1000 bar,var,DT,1000 var,bar,VO,2000 in this ...
3
votes
4answers
109 views

print only lines where the first column is unique

I am looking for a way to sort a list and print all lines, whose first column appears only once - i.e., match only on the first column. For example, I have a file where the first column is a path and ...
6
votes
2answers
342 views

Delete multiple lines in a file

Usually I find my answers in the previously answered questions, but not this time (or I don't understand the answers and can't modify them to do what I want them to do), so I'm asking my first ...
2
votes
1answer
47 views

Adding two columns together in CSV and outputting to new CSV file

I have a problem I have been trying to figure out: We have a stock file CSV, which contains the stock in multiple locations. The csv looks like this: ...
4
votes
6answers
154 views

Make longer strings from substrings keeping new line between each string

I have a file that contains strings as: GTACGACGGAGTGTTATAAGATGGGAAATCGGATACCAGATGAAATTGTGGATCGGTGCAAAA GTCGGCAGATATCGTTGAAGTCATAGGTGATTATGTTCAATTAAAGAAGCAAGGCCGAAACTAC ...