awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
1answer
79 views

Replace pattern in file with pattern in second file

I want to replace part of a file's data, with data from another file. Suppose file1 has data as is written below and file2 has some data I want to store and replace file1 data from file2 from ...
12
votes
8answers
2k views

How to perform a multi line grep

How would you perform a grep for text that appears on two lines? For example: pbsnodes is a command I use that returns the utilization of a linux cluster root$ pbsnodes node1 state = free ...
7
votes
3answers
472 views

Distributing a script: Should I use /bin/gawk or /usr/bin/gawk for shebang?

Is gawk in /bin or /usr/bin usually? I would go with #!/usr/bin/env gawk but then I can't use arguments. Right now I'm using #!/bin/gawk -f. The script is very long and contains a lot of single ...
1
vote
2answers
65 views

How to match a number in one file to a name in another file?

Okay, I hope this is the right place to ask. I have 2 files. The first file has 4 fields: class CRN #, Instructor ID #, Class Section #, Number of Students. Example: 0002 T00005 006 15 0004 ...
4
votes
2answers
215 views

Comma separated words to next line in new file

I have a input file like this (input.txt): Main1 one,two,three,four Main2 five,six,seven,eight I want the output like this in output file (output.txt): Main1 one Main1 two Main1 three Main1 four ...
4
votes
3answers
3k views

Diff, show modified line from right file only

I have file a and b and I would like to output lines of b that changed since it was cloned from a. Just the modified lines, no surrounding context, no diff offset marks. How can I do that using shell ...
2
votes
4answers
80 views

how to print a new line in awk

I am trying to print collected data from a file where each set gets a new line (\n) Code: awk '/mail:|fullName:|uid:/{s=s", "$0} END{print substr(s,3)}' dump2.txt Output now: mail: ...
1
vote
2answers
93 views

grepping for multiple items

I have about 6 items I need extracted from and LDAP base dump file fullName: uid: email: ... the colon is required for the search, mainly because uid appears in a number of places in the file but ...
-2
votes
1answer
56 views

Extract/print every 3rd occurence of pattern

I need help with extracting/printing lines until a stop pattern. This needs to happen after every 3rd occurrence of start pattern, i.e ERROR_FILE_NOT_FOUND and continue until the end of the file. ...
1
vote
3answers
261 views

sed + find value before word in line

How can I find the value before the word "free"? top -n1 | grep Mem Mem: 2054968k av, 2034120k used, 20848k free, 0k shrd, 186768k buff First I tried top -n1 | grep ...
0
votes
3answers
206 views

How does the following shell script work in finding all files larger than 20MB?

I came across a post advising that the following will find all files greater than 20Mb in size. find / -type f -size +20000k -exec ls -lh {} \; 2> /dev/null | awk '{ print $NF ": " $5 }' | ...
-2
votes
1answer
4k views

convert CSV to XLS file on linux [closed]

The following Perl script can convert CSV file to XLS file the problem is that I need to install on customer Linux machine many Perl modules in order to run this Perl script , and actually I cant ...
1
vote
2answers
75 views

Extracting file name and string from multiple files

I have a lot of files (>100,000) from which I would like to extract a specific string and the filename itself. Files are organized like this: Filename: sequence.php?ID=gi|90022703|ref|YP_528530.1| ...
0
votes
5answers
313 views

df -k + how to match the root volume line

please advise how to match the "/" volume from df -k command? I tried the following: [root@machine /]# df -k | grep "/" /dev/vx/dsk/bootdg/rootvol 130926685 71378710 58238709 56% / /devices ...
0
votes
1answer
71 views

Finding words with one vowel group

I am trying to use a command to find all words in a list of words with one group vowels. Like four, for and far, but also words like ore and or. What I got so far: grep -i ...
1
vote
3answers
103 views

Find a string in one section of a file with mutiple sections

How to sed/awk for a string contained in another string. TESTVAR=' { icon : "icons/773_l.png", id : 80, initialState : true, isPng : false, label : "Imagery", opacity : 1, requestType : ...
4
votes
3answers
7k views

How can I use SED or AWK to replace placeholders in a template file with variable content that contains special characters?

CentOS 6.3 I'm trying to get a small script to send an email containing a copy of email headers in the body (for the purpose of internal reporting). The template file contains the following: ...
1
vote
4answers
490 views

splitting a column using awk

I have a file like shown below. The 9th column has values joined by **. chrXV 234346 234546 snR81 + SNR81 chrXV 234357 0.0003015891774815342**0.131826816475 + chrXV 234346 ...
2
votes
1answer
139 views

Getting matched fasta file

list.txt 58759__len__2903 58759__len__2903 673957__len__1655 673957__len__1655 3566454__len__1744 seq.fasta >58759__len__2903 TTTTCCGTAGAGGAGATCCCTATTTTTAGGTTTGTAAGAGATCATTTT ...
1
vote
2answers
142 views

Extract text from one iteration of top

I want to execute the top command from my bash script the problem is that usually when I type the top command on the console , I exit by Ctrl+C. but how to exit from top when I run the top command ...
2
votes
0answers
217 views

extracting data from a log file and plotting it using Gnuplot

I have a logfile with the following data in it: Sat Sep 20 11:25:01 BST 2014 -- temp=53.0'C -- message from script /usr/src/scripts/wifi_test_2.sh Sat Sep 20 11:26:01 BST 2014 -- temp=53.0'C -- ...
-1
votes
1answer
235 views

search for a character in one column and replace value in another column of matched records

I have a requirement to find value 2 in column 4 and wherever it matches replace 6th column with '-'sign in the beginning suppose if i have csv file - file1 like this abc,0,def,1234,ijk ...
2
votes
3answers
2k views

Using bash shell function inside AWK

Is it possible to use bash function inside AWK somehow? Example file (string, int, int, int) Mike 247808 247809 247810 Trying to convert values from decimal to hexadecimal. Function defined ...
-1
votes
1answer
85 views

Match and combine records in Bash [duplicate]

Consider two csv files: $ cat given.csv 123,John 246,Paul 369,George 987,Ringo $ cat family.csv 246,McCartney 123,Lennon 987,Starr 369,Harrison Note that the lines are not ordered! How might I use ...
2
votes
1answer
84 views

Substitute one field with other if its entry starts with specific letter

If I have a file something like: 223 898 teeth 223 899 mouth 223 900 r9 445 800 round My question is how can I use awk command to substitute column 2 with column 3, only for the row in which column ...
0
votes
2answers
66 views

Convert two rows to two columns

I would like to convert every two rows to two columns using awk. Any help appreciated. input.txt: # Query: gi|11465907|ref|NC_001872.1| Chlamydomonas eugametos genome, complete genome # 0 hits found ...
2
votes
2answers
56 views

Gather job usage information on cluster

I am running simulations on a linux cluster shared by hundreds of other users. There is a job scheduler installed on the cluster that determines who runs where, and one of the scheduler commands ...
8
votes
2answers
306 views

Reindexing a large CSV file

I went through the answers in this helpful thread, but my problem seems to be different enough that I can't think of good answer (at least with sed). I have a large CSV file (200+ GB) with rows that ...
4
votes
6answers
586 views

Sed — Replace first k instances of a word in the file

I want to replace only the first k instances of a word. How can I do this? Eg. Say file foo.txt contains 100 instances occurrences of word 'linux' . I need to replace first 50 occurrences only.
2
votes
3answers
58 views

Merging two tables including multiple ocurrence of column identifiers and unique lines

I asked a similar question yesterday (Merging two tables including multiple ocurrence of column identifiers) but ran into a problem with unique lines. I would like to merge two tables based on column ...
55
votes
7answers
17k views

Is there a basic tutorial for grep, awk and sed? [closed]

I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...
0
votes
4answers
271 views

Use SED or AWK to move rows to columns

I have a report that is generated in rows and I need to move it to columns. The number of rows per record will always be the same. This will be in a BASH script on Linux, so I would like to use ...
2
votes
2answers
122 views

Merging two tables including multiple ocurrence of column identifiers

I would like to merge two tables: File 1: 1 today 2 tomorrow 3 red File 2: 1 a lot 1 sometimes 2 at work 2 at home 2 sometimes 3 new Desired output (file 3): 1 today ...
1
vote
1answer
56 views

Take the subset of the file related to the time stamp

I have some json file in dictionary format. The lines looks like here: {"a":"1", "b":"2", "c":"3", "time":1334572551435} {"a":"1", "b":"2", "c":"4", "time":1334575352456} {"a":"2", "b":"2", "c":"7", ...
1
vote
1answer
114 views

Sed / awk script for a non-standard apache access log ( Oracle Weblogic )

I have hit a wall with my limited sed scripting abilities, and i wonder if any of you guys could help me out. I have a non-standard apache access log with the following format: #Version: 1.0 ...
3
votes
2answers
261 views

awk: Extracting a fixed number of rows where the last row number may vary

This question may look like a duplicate, but only at first glance. Of course, I would no longer need help in how to code a one-liner that extracts a fixed number of continuous lines (e. g. 5 in this ...
1
vote
4answers
169 views

How to split text file into records by first non-blank charecter using awk and print only records containing string

I have a text file with information separated in blocks. I would like to separate these blocks into records so that I can handle them separately. This is my text file (file.txt): Alarm Stats: ...
1
vote
4answers
80 views

Parsing complex text file using Unix commands

I have the following text structure which I would like to parse: >Cluster 423 0 56aa, >HWI-ST1448:257:C3V2HACXX:1:1106:19087:2550.1... at 92.86% 1 64aa, ...
1
vote
2answers
109 views

Use awk to find first occurrence

I have a log file that gets updated when my script runs. The script will insert "Start of script" text when it starts and "End of script" text when it finishes running. I am trying to capture the ...
1
vote
2answers
420 views

Replace columns using a shell

How could I replace column 1 in File1 with column 2 in File2, when a match is found? File1 id1 36987 id2 75685 id3 12345 id4 12896 File2 id1 ID1 id2 ID2 id3 ID3 id5 ID5 I'd like to write this on ...
5
votes
5answers
260 views

Replace data at specific positions in txt file using data from another file

I have a text file in the below format: $data This is the experimental data good data This is good file datafile 1 4324 3673 6.2e+11 7687 67576 2 3565 8768 8760 5780 8778 "This is line ...
0
votes
4answers
168 views

Adding a character to every other text line

I have a huge (ca. 20G) text file which contains millions of passages (a.k.a. paragraphs) with headers. Headers and paragraphs are always one line each, e.g., Sunshine This is a sunny day. ...
5
votes
3answers
305 views

How to use awk through multiple files?

I want a solution for the general case (N folder) I'm using awk to process a file and extract its content and put it in a variable then echo it: This is the file: H1 H2 H3 H4 H5 H6 H7 H8 H9 not ...
0
votes
3answers
51 views

How to Print certain Fields if a condition is met

I have a big Input file as shown below. If "PRIMER_LEFT_NUM_RETURNED" is not equal to 0 then print the corresponding SEQUENCE_ID, SEQUENCE_TEMPLATE,PRIMER_LEFT_NUM_RETURNED, PRIMER_RIGHT_NUM_RETURNED, ...
3
votes
4answers
112 views

Need to remove - (Minus) sign at the end in number from many columns and insert it on columns removed

I have a huge csv file which has - (Minus) sign at the end of numbers. Now I need to move these - minus signs to begin of the numbers, i.e I need to remove the last character if it has '-' sign and ...
0
votes
1answer
121 views

Haskell, C++, AWK, Bash, Lisp … Which one is faster? [closed]

I was wondering, which of these programming languages are faster at handling big data files of only strings: Haskell, C++(Boost), AWK, Bash, Lisp... So far I have found the paper Haskell vs. Ada ...
0
votes
3answers
35 views

AWK remove one line?

I need some help with awk. This is my uptime uptime=`uptime|grep days|awk '{print$3,$4}'` But it displays now 23 days, I want to remove , How can I accomplish that? cheers and thanks
2
votes
7answers
90 views

How to replace the third character in number to other character

IPS parameter include diff IP address as: IPS="192.9.200.12-14 172.17.200.12-89 12.21.1.10-25 127.0.0.1-127" I want to replace the third “.” In the IP address number to “,” How to achieve this by ...
3
votes
2answers
177 views

Compare 2 tab delimited files and output differences with column header

I would like to compare 2 similar files on a common column. The files will have identical headers. file1.txt mem_id Date Time Building aa bb cc dd ee ff ...
4
votes
4answers
281 views

extract lines according to a particular column

I have a CSV file like CK,ck XYZ,xyz,xyzs ABC,abc,abcs PQR,pqr,pqrs LMN,lmn, IJK,ijk, I have to check if something is written in column 3 except whitespace, then display the whole line.the output ...