awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

0
votes
1answer
16 views

splitting a column using awk

I have a file like shown below. The 9th column has values joined by **. chrXV 234346 234546 snR81 + SNR81 chrXV 234357 0.0003015891774815342**0.131826816475 + chrXV 234346 ...
0
votes
2answers
48 views

linux - top command + use top command in bash script

I want to execute the top command from my bash script the problem is that usually when I type the top command on the console , I exit by Ctrl+C. but how to exit from top when I run the top command ...
-1
votes
1answer
13 views

search for a character in one column and replace value in another column of matched records

I have a requirement to find value 2 in column 4 and wherever it matches replace 6th column with '-'sign in the beginning suppose if i have csv file - file1 like this abc,0,def,1234,ijk ...
1
vote
1answer
21 views

Getting matched fasta file

list.txt 58759__len__2903 58759__len__2903 673957__len__1655 673957__len__1655 3566454__len__1744 seq.fasta >58759__len__2903 TTTTCCGTAGAGGAGATCCCTATTTTTAGGTTTGTAAGAGATCATTTT ...
2
votes
3answers
1k views

Using bash shell function inside AWK

Is it possible to use bash function inside AWK somehow? Example file (string, int, int, int) Mike 247808 247809 247810 Trying to convert values from decimal to hexadecimal. Function defined ...
3
votes
2answers
547 views

Split log file by time range

I have a log file and would like to cut it into copies of 15min. In the log file the lines start with: dd mmm yyyy hh:mm:ss,xxx e.g. 12 Feb 2013 16:05:02,xxx log text... and the file rotates ...
-1
votes
1answer
36 views

Match and combine records in Bash [duplicate]

Consider two csv files: $ cat given.csv 123,John 246,Paul 369,George 987,Ringo $ cat family.csv 246,McCartney 123,Lennon 987,Starr 369,Harrison Note that the lines are not ordered! How might I use ...
2
votes
1answer
30 views

Substitute one field with other if its entry starts with specific letter

If I have a file something like: 223 898 teeth 223 899 mouth 223 900 r9 445 800 round My question is how can I use awk command to substitute column 2 with column 3, only for the row in which column ...
0
votes
2answers
37 views

Convert two rows to two columns

I would like to convert every two rows to two columns using awk. Any help appreciated. input.txt: # Query: gi|11465907|ref|NC_001872.1| Chlamydomonas eugametos genome, complete genome # 0 hits found ...
2
votes
2answers
25 views

Gather job usage information on cluster

I am running simulations on a linux cluster shared by hundreds of other users. There is a job scheduler installed on the cluster that determines who runs where, and one of the scheduler commands ...
8
votes
2answers
225 views

Reindexing a large CSV file

I went through the answers in this helpful thread, but my problem seems to be different enough that I can't think of good answer (at least with sed). I have a large CSV file (200+ GB) with rows that ...
0
votes
0answers
48 views

what is hexa dump and how to handle in unix? [on hold]

I am trying to extract some portion of text from the file by using awk command, The file is separated by comma. awk -F, '{print $1 FS $3 FS $6}' sample.csv But always I am getting wrong output , ...
4
votes
6answers
133 views

Sed — Replace first k instances of a word in the file

I want to replace only the first k instances of a word. How can I do this? Eg. Say file foo.txt contains 100 instances occurrences of word 'linux' . I need to replace first 50 occurrences only.
2
votes
3answers
82 views

Use AWK to pad date and time formats

I would like to use AWK to pad the date and time fields in my report. Here is my current .csv that is actually tab delimited. AA 9/7/2014 2:30:38 PM AA0000011111 08 Y A Jane, Doe AA ...
-3
votes
0answers
27 views

How to extract strings from a given input file and store them in two different files in a column wise manner

Given below is the input file pattern: *********************************************************** Search URL: ...
0
votes
0answers
24 views

how to insert seperate lines ( elemnts ) in parameter [closed]

I have the following csv file more file.csv 1,yes,yes,customer1 2,no,yes,customer5 3,yes,yes,customer11 4,no,no,customer14 5,yes,no,customer15 6,yes,yes,customer21 7,no,yes,customer34 ...
2
votes
3answers
43 views

Merging two tables including multiple ocurrence of column identifiers and unique lines

I asked a similar question yesterday (Merging two tables including multiple ocurrence of column identifiers) but ran into a problem with unique lines. I would like to merge two tables based on column ...
54
votes
7answers
14k views

Is there a basic tutorial for grep, awk and sed? [closed]

I've been a Linux user for a while, and I've a pretty decent understanding of most the common command line utilities. However, ones that come up and up again in relation to programming are grep, awk, ...
0
votes
4answers
64 views

Use SED or AWK to move rows to columns

I have a report that is generated in rows and I need to move it to columns. The number of rows per record will always be the same. This will be in a BASH script on Linux, so I would like to use ...
2
votes
2answers
94 views

Merging two tables including multiple ocurrence of column identifiers

I would like to merge two tables: File 1: 1 today 2 tomorrow 3 red File 2: 1 a lot 1 sometimes 2 at work 2 at home 2 sometimes 3 new Desired output (file 3): 1 today ...
1
vote
1answer
37 views

Take the subset of the file related to the time stamp

I have some json file in dictionary format. The lines looks like here: {"a":"1", "b":"2", "c":"3", "time":1334572551435} {"a":"1", "b":"2", "c":"4", "time":1334575352456} {"a":"2", "b":"2", "c":"7", ...
1
vote
1answer
17 views

Sed / awk script for a non-standard apache access log ( Oracle Weblogic )

I have hit a wall with my limited sed scripting abilities, and i wonder if any of you guys could help me out. I have a non-standard apache access log with the following format: #Version: 1.0 ...
3
votes
2answers
177 views

awk: Extracting a fixed number of rows where the last row number may vary

This question may look like a duplicate, but only at first glance. Of course, I would no longer need help in how to code a one-liner that extracts a fixed number of continuous lines (e. g. 5 in this ...
24
votes
9answers
33k views

Is there a way to get the min, max, median, and average of a list of numbers in a single command?

I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script. Although my immediate situation is for ...
1
vote
4answers
95 views

How to split text file into records by first non-blank charecter using awk and print only records containing string

I have a text file with information separated in blocks. I would like to separate these blocks into records so that I can handle them separately. This is my text file (file.txt): Alarm Stats: ...
1
vote
4answers
47 views

Parsing complex text file using Unix commands

I have the following text structure which I would like to parse: >Cluster 423 0 56aa, >HWI-ST1448:257:C3V2HACXX:1:1106:19087:2550.1... at 92.86% 1 64aa, ...
2
votes
2answers
34 views

Use awk to find first occurrence

I have a log file that gets updated when my script runs. The script will insert "Start of script" text when it starts and "End of script" text when it finishes running. I am trying to capture the ...
1
vote
2answers
142 views

Replace columns using a shell

How could I replace column 1 in File1 with column 2 in File2, when a match is found? File1 id1 36987 id2 75685 id3 12345 id4 12896 File2 id1 ID1 id2 ID2 id3 ID3 id5 ID5 I'd like to write this on ...
5
votes
5answers
83 views

Replace data at specific positions in txt file using data from another file

I have a text file in the below format: $data This is the experimental data good data This is good file datafile 1 4324 3673 6.2e+11 7687 67576 2 3565 8768 8760 5780 8778 "This is line ...
0
votes
4answers
46 views

Adding a character to every other text line

I have a huge (ca. 20G) text file which contains millions of passages (a.k.a. paragraphs) with headers. Headers and paragraphs are always one line each, e.g., Sunshine This is a sunny day. ...
5
votes
3answers
74 views

How to use awk through multiple files?

I want a solution for the general case (N folder) I'm using awk to process a file and extract its content and put it in a variable then echo it: This is the file: H1 H2 H3 H4 H5 H6 H7 H8 H9 not ...
0
votes
3answers
36 views

How to Print certain Fields if a condition is met

I have a big Input file as shown below. If "PRIMER_LEFT_NUM_RETURNED" is not equal to 0 then print the corresponding SEQUENCE_ID, SEQUENCE_TEMPLATE,PRIMER_LEFT_NUM_RETURNED, PRIMER_RIGHT_NUM_RETURNED, ...
3
votes
4answers
42 views

Need to remove - (Minus) sign at the end in number from many columns and insert it on columns removed

I have a huge csv file which has - (Minus) sign at the end of numbers. Now I need to move these - minus signs to begin of the numbers, i.e I need to remove the last character if it has '-' sign and ...
0
votes
1answer
67 views

Haskell, C++, AWK, Bash, Lisp … Which one is faster? [closed]

I was wondering, which of these programming languages are faster at handling big data files of only strings: Haskell, C++(Boost), AWK, Bash, Lisp... So far I have found the paper Haskell vs. Ada ...
0
votes
3answers
29 views

AWK remove one line?

I need some help with awk. This is my uptime uptime=`uptime|grep days|awk '{print$3,$4}'` But it displays now 23 days, I want to remove , How can I accomplish that? cheers and thanks
2
votes
7answers
75 views

How to replace the third character in number to other character

IPS parameter include diff IP address as: IPS="192.9.200.12-14 172.17.200.12-89 12.21.1.10-25 127.0.0.1-127" I want to replace the third “.” In the IP address number to “,” How to achieve this by ...
3
votes
2answers
53 views

Compare 2 tab delimited files and output differences with column header

I would like to compare 2 similar files on a common column. The files will have identical headers. file1.txt mem_id Date Time Building aa bb cc dd ee ff ...
7
votes
3answers
278 views

Distributing a script: Should I use /bin/gawk or /usr/bin/gawk for shebang?

Is gawk in /bin or /usr/bin usually? I would go with #!/usr/bin/env gawk but then I can't use arguments. Right now I'm using #!/bin/gawk -f. The script is very long and contains a lot of single ...
4
votes
4answers
141 views

extract lines according to a particular column

I have a CSV file like CK,ck XYZ,xyz,xyzs ABC,abc,abcs PQR,pqr,pqrs LMN,lmn, IJK,ijk, I have to check if something is written in column 3 except whitespace, then display the whole line.the output ...
1
vote
2answers
38 views

Error: ambiguous redirect while transferring output to command

Output to redirect in script: 511@ubuntu:~/Unix/test$ ls -ltr |awk '{print $9}' default.txt dfah.txt fruit.txt fruit_prices.txt dfh.txt header.txt testfile.txt topoutput.txt Script written on shell ...
30
votes
4answers
2k views

Reverse grepping

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
12
votes
1answer
3k views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
1
vote
0answers
27 views

Running multiples query on Hbase shell without calling hbase shell again [migrated]

To invoke shell again takes time , i want to execute multiple commands by invoking hbase shell once.Below code run only single query. cmd="echo \"put 'test', 'row1', 'cf:a', 'value1'\"| hbase shell" ...
1
vote
4answers
28 views

Loop through array

I have records in list.txt (INPUT) as List of animals SET 1=Dog 2=Crow 3=Snake Pet,bird,reptile List of Countries SET 1=France 2=Singapore 3=Columbia Europe,Asia,SouthAmerica List of Parts SET ...
1
vote
1answer
51 views

Why is the output of this paste command split on separate lines with indents?

I was trying to merge files using either of these commands: paste data_{1..2}.txt > 1_2.txt paste <(awk '{print $1}' data_2.txt ) <(awk '{print $1}' data_2.txt) > 1_2.txt But the ...
4
votes
3answers
598 views

Replace “o” with “0” in large wordlist and save the original word?

I have a large text file with words. Every word is on a single line (typical wordlist). I want to replace all characters "o" with the number "0" in every word, and the new formed word to be pasted on ...
2
votes
4answers
244 views

Only cat from specific line X (with a pattern) to other specific line Y (with a pattern)

A little extended problem from "cat line x to line y on a huge file": I have a huge file (2-3 GB). I'd like to cat/print only from the line having "foo:" to the line having "goo:". Assume that "foo:" ...
-1
votes
2answers
35 views

Searching out a pattern from Logs

The format of the logs is in this manner: Junk Junk Junk : Junk Junk Junk:IP/Port Junk Junk I want to search using the IP and the output should be only the IP/Port part of each line in that log ...
0
votes
0answers
44 views

wget: Retrieving a list of URLs when modifying input data file on the fly

This issue is currently driving me up the wall. It just does not work as it should. I have a file inp with audio samples to download where I preserved the internal ID number by parsing some other ...
1
vote
2answers
54 views

awk: for-loop with break option

I am trying to use awk to manipulate data.I have a data file with two columns and I would like to find the row that contains an specific value. But there is more than one row that contains this value ...