awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
1answer
34 views

bash ignorecase match pattern in file and color that line, print all to screen

I have almost zero idea about coding or bash scripting, so please bare with me and excuse my horrible formatting. What I want is to cat a log file and highlight certain lines with color based on a ...
0
votes
2answers
26 views

Replace fields in file inline

I have file (a.txt) with following contents: Redhat master 12345678 V1A 20.0/20.0.1 Windows slave 12345678 V1B 19.0/19.0.1 I want to grep line with word master and replace 4th and fifth field with ...
4
votes
4answers
2k views

Merge 2 files based on all values of the first column of the first file

I need to merge below 2 files: file1: TABLES REF-IO HEAD-IO DIFF-IO test 200 500 -300 exam 2 3 -1 final 2 1 1 mail 4 2 2 TOTAL 208 506 -298 ...
1
vote
2answers
18k views

How to select, copy and paste some columns from different files and create one file from them?

I want to create and organize data in a file from a number of files by selecting parts of the columns of the given files. I have more than 10 file to copy the second, third and forth columns of each ...
1
vote
1answer
68 views

How to select rows from a CSV file based on different column values?

I have a CSV file with values like the following: col1,col2,col3,col4,col5,col6,col7 1,0,0,0,0,BTS,Active 4,5,3,1,1,LocalMode,Offlne 1,2,5,6,3,PermFault,Offline 1,2,6,6,2,BSC,Active ...
0
votes
3answers
34 views

How to get index values for multiple matches in the same line in awk?

I know the following expression: echo "bob alice robert alice" | awk '{print index($0,"alice")}' 5 Will output the index of the first alice match, is there any way to get the index of all matches?, ...
2
votes
2answers
386 views

Merge csv files with conditions

I have 2 CSV files for each date (csv_2014_4_15 and csv_2014_4_16) , with basic structure and couple of unique columns as below. id,name,created_at,updated_at,other columns 12, joe, 2013-1-1 18:30, ...
4
votes
3answers
2k views

awk remove the rows based on another column value

I have a file like below which I have sorted based on the username field. UserID score UserName 1234 200 Jack 5678 150 Jill 8543 200 Jill 5678 100 John I am trying to remove ...
2
votes
4answers
79 views

Print alternating lines with conditional

Still struggling with messy logfiles. What I want is to find lines that first matching string X and then string Y. I then want to print them together. The problem is that, sometimes there is X, but no ...
7
votes
3answers
3k views

Add thousands separator in a number

In python re.sub(r"(?<=.)(?=(?:...)+$)", ",", stroke ) To split a number by triplets, e.g.: echo 123456789 | python -c 'import sys;import re; print re.sub(r"(?<=.)(?=(?:...)+$)", ",", ...
7
votes
7answers
260 views

How to efficiently split up a large text file wihout splitting multiline records?

I have a big text file (~50Gb when gz'ed). The file contains 4*N lines or N records; that is every record consists of 4 lines. I would like to split this file into 4 smaller files each sized roughly ...
4
votes
2answers
90 views

Usage of awk to search for string that contains characters

I am facing an issue of searching through lines of text in a text file. Currently, I am using this command check=`awk -F : -v "title=$title" 'tolower($1) ~ tolower(title)' test.txt` It works fine ...
1
vote
3answers
71 views

how to break a line after matching a certain character using shell command?

File1: John Daniel Hommes Cameron;Emily Terry Mussy Barbara Mimi Papu;David Swiss Jen Hans Peter Iril;Kelvin Lilly Gucci Kate Nik;Forum Bill June;Jill and Jack Output file: John Daniel Hommes ...
0
votes
2answers
52 views

How to count the number of words of each sentences in a file using shell command?

I have text files like this: Mr.P.K.Baneerjee has visited the site today at 4.30pm The permit tag has been set in the ds busbar Opearation has been performed in presence of control engineer Opeation ...
0
votes
0answers
21 views

Output a particular character if the same column consists of a few defined characters with AWK

Let say I have a text file as below: G W G A A T G A A T R A T A R A G - G A And I already know that if the same column consist of "A", "G" and "T", I would like to output a "D". For column 2, ...
7
votes
3answers
397 views

How to use shell to derive an IPv6 address from a MAC address?

We know that we can use the MAC address to create an interface identifier, e.g. for a link-local IPv6 address which should be unique in the Network. The image shows the way to do this: My ...
17
votes
11answers
3k views

There must be a better way to replace single newlines only?

I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
1
vote
2answers
68 views

How to extract 1st, 3rd, 5th of group of three lines by sed or awk?

Say I have a file: Line1 Line2 Line3 Line4 Line5 Line6 Line7 Line8 Line9 Line10 Line11 Line12 … I want to get: Line1 Line2 Line3 Line7 Line8 Line9 Line13 Line14 Line15 It means that I want ...
10
votes
5answers
1k views

Case matching pattern replacement with sed

I have a source code spread across several files. It has a pattern abcdef which I need to replace with pqrstuvxyz. The pattern could be Abcdef (Sentence Case) then it needs to be replaced with ...
0
votes
1answer
50 views

What does vRS=“\n20” mean on the gawk command line?

Can any one please explain the below command: gawk -vRS="\n20" '/Job-157625/' log.1 We are using this command to get the log for the job. I know gawk is used to find/scan a pattern but I want to ...
0
votes
1answer
32 views

Need to parse nginx access.log by minute, chunks for statistics - is there better way?

I need collect current data from it every minute to monitore amount of requests and errors. It's nginx's frontend log - with a lot of requests. It's formatted, rotated than archived every hour. So - ...
1
vote
2answers
111 views

Grep multiple patterns and print a different number of lines below each of the patterns?

I am trying to find a way to grep/awk multiple patterns and then print a certain number of lines below the first pattern and a certain number of lines below the second pattern. For example .... ...
1
vote
0answers
64 views

How to search for columns that contain specific values

I have a large tab-delimited text file (~40000 columns x ~20000 rows) that I am trying to search through. Each column in the file has a unique identifier in row 1. I have a separate list of a subset ...
4
votes
3answers
3k views

GREP / SED or AWK: Print entire paragraph in a file on pattern match [duplicate]

I have a file with hundreds of paragraphs of around 15 lines each. I need to search for a pattern, say Occurance: 1. If this pattern is found in the para, I need to print the entire paragraph. Note ...
2
votes
1answer
44 views

How to read and do a min/max/avg from a NMEA stream

I have a weather station attached to my device spitting NMEA 0183 on /tty/USB0 I have alreeady sorted out how to parse the data and echo what I want in a script. What I need to do now is write the ...
0
votes
0answers
37 views

print duplicate fields in different columns from two files using awk

I have two large files. I am attempting to identify and print any duplicate fields to a new file. I have a 19 columns and over 100k rows. I need to compare columns 5-19 for duplicate values. ...
1
vote
1answer
61 views

Pick columns from a variable length csv file

I have two CSV files, which share a column that is unique to each row in each file, like an ID. The files don't have a header. The file_2 has variable length columns e.g. 1,2,3 4,5,6,7,8 I joined ...
-1
votes
1answer
74 views

how to extract lines between 2 patterns, and save them to different files

hej all, Let' say that we have this file: rrows 35 cols 51 spin 2 %band5 efermi=0.012538 0.004352 0.004317 0.004213 0.004047 0.003828 0.003570 0.003291 0.003011 0.002754 0.002544 0.002408 ...
1
vote
3answers
47 views

compare multiple files(more than two) with two different columns

I have four different files: file1, file2, file3, file4. Each file has 2 different columns separated by tab. I want to match first column of file1 (as reference) with the first column of second file, ...
4
votes
2answers
3k views

Remove entire row in a file if first column is repeated

I have a file containing two columns and 10 million rows. The first column contains many repeated values, but there is a distinct value in column 2. I want to remove the repeated rows and want to keep ...
3
votes
2answers
1k views

Merging columns from two separate files

How to create a new file merging selective columns from two separate files using awk? Without messing up the elements orders of BOTH files. Exemple: File 3 may contain column 1,2,3 from File 1 and ...
1
vote
1answer
185 views

How to copy rows of different files and paste column-wise?

I want to create a new file by copying rows of each file and pasting columnwise according to their file name orders. Here is the sample input and required output files below. file1.txt 1234 5678 ...
2
votes
4answers
184 views

Replace values in column that contain numbers

I have a file that looks as so: chrom start stop strand isoform mu_codon mut_codon2 more_info chr22 43089055 43089055 - NM_017436 C 300 903delC chr22 43089715 43089717 ...
3
votes
4answers
140 views

How to compare two dates without using Stat?

I'm trying to compare the last date modified of a file to today's date. If the modification date is before today's date then my script will do one thing, if the date matches todays date i need it to ...
0
votes
1answer
44 views

Python output in XML format

I have a python script which compare IP list with IP Database and gives output of matching criteria. (I got the script from this forum). convert.py #!/usr/bin/python import socket,struct ...
4
votes
4answers
2k views

merge 2 rows based on the same column values

I have a file like below. 47196436 47723284 name1 1.77273 42672249 52856963 name2 1.06061 52856963 430695 name2 1.16667 55094959 380983 name3 1.55613 17926380 55584836 name4 1.02461 3213456 34211 ...
0
votes
3answers
58 views

How to sort/reformat text information in a file?

I have a file (called customers) with information that needs to be sorted; example: qweasd,asdqwd@asd.org,256.62394383 qweasdczasd,asdghjghqwd@assssd.org,256.62394383 ...
3
votes
5answers
93 views

parsing a huge file with awk and extracting a subset

I have a huge file which looks like this chr10 98072 1 chr10 98073 1 chr10 98074 1 chr10 98075 2 chr10 98076 2 chr10 98077 3 chr10 98078 5 chr10 98079 5 chr11 98080 ...
1
vote
3answers
77 views

Remove characters in a specific column before a specific character

I have a file that looks like so: chr22 43089055 43089055 - NM_017436 C 300 903delC chr22 43089715-43089717 43089715-43089717 - NM_017436 CTT 79 I want to remove all the ...
0
votes
3answers
64 views

Bash to match two variables for time difference

I have a bash command which gives me following result (Fetch user account details from the mysql using command select creationdate from users; Name Account Creation Date Mobile User1 ...
0
votes
1answer
69 views

kill processes in one line with kill, awk, ps, and grep

I have a few processes that spring up and I am able to print a line of the pgid's that I would like to enter into a kill command. here is what I have: sudo ps o pgid,args | grep mininet: | sudo awk ...
3
votes
2answers
333 views

Extract only chinese characters

I am working on a text written in Italian and Chinese and I need to extract only Chinese characters using AWK. How can I do this? I tried: [The range of Chinese Unicode chars is 4E00 thru 9FFF (344 ...
0
votes
1answer
23 views

Awk split big file conditionally in multi output file

I have to split a 90 GB file in multiple file conditionally (the file is a multiple table file to split by single table). I have create a solution in ksh and awk that cycle on the table rules and ...
1
vote
3answers
223 views

Put big data of heterogenous byte offset into arrays by AWK

Assume the data consists of byte offset which is not fixed i.e. the distance of two subsequent file headers varies. The point of this thread is to go through each size of events separately in arrays. ...
2
votes
1answer
76 views

Convert IP Address to Coordinates using database

I have a file which contains IP and I wanted to compare it to Coordinates using a local database (instead of API). I tried following a process and break down to pieces and created scripts referring ...
1
vote
1answer
60 views

Extracting trace information using gawk

I am new to awk programming and am using gawk to extract trace information from a mrt file in order to use it further for analysis. I have successfully been able to extract trace information from a ...
0
votes
1answer
52 views

How to extract particular fields from lines using shell commands?

File1: 27/07/2014 10.10.03 Breaker1 was closed by Mr. A.Basu Spectrum ui6dms 29/07/2014 04.50.53 Breaker1 was opened by Mr. D.Gupta Spectrum ui6dms and one note added 30/08/2014 03.20.01 EarthSwitch ...
2
votes
2answers
59 views

How to exclude lines that don't include a string from the output of an awk script?

Completely new to scripting in general, and have been working on a script to go through our logs which contain entries for a variety of different things. What I'm interested in, is the entries that ...
1
vote
1answer
54 views

How to count the frequency of a word >20

Using exclusively AWK, I need to count the occurrence of a word only when greater than 20 characters. I've already found the way to count the frequency of a word, but I have some problems in setting ...
5
votes
1answer
76 views

awk Command Loop

awk '{print $1 ,": " $3}' Src.txt | column -t awk '{for(x=2;$x;++x) print $1, $x "\n"}' Src.txt | column -t These commands are working but formatting is not proper either I can only print column ...