awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
4answers
174 views

Sed/awk/perl: reversing order of comma separated values, preserving other text

I have this text: LABEL1 .BYTE 01, 02, 03, 04, 05 .BYTE 01, 02, 03 I need to reverse only comma separated values order: LABEL1 .BYTE 05, 04, 03, 02, 01 .BYTE 03, 02, 01 I need ...
3
votes
2answers
111 views

How can I partition a set of words given pairs that must end up together?

I have a space or comma separated table with two columns, each row representing equivalence of the two words. A B B C B D C E F G What I want is a table with each row listing all mutually ...
2
votes
2answers
90 views

Environment variable not expanded inside the command line argument

I have a file user-pid.out2 which has "usernumber" and "process id" as two columns. based on usernumber I want to find corresponding process id. the first two lines below does not show the output ...
0
votes
2answers
19 views

Replicate lines in text fine according to a column value

I have a text file with a column containing text/data followed by an abundance column, e.g. Car 3 Apple 2 In order to subsample these columns I would like to replicate each text column by ...
1
vote
1answer
56 views

count matches and mismatches group by

Please help with a shell script of the following. I need count number of consistent variables in each lane (col1) across samples (col2). For example, since all the values (col4) of lane1 variable 1 ...
2
votes
2answers
58 views

Sed/awk/perl: modify text preserving parts and align to a column

I have a text like this: A1JOURNEY0TO1 .BYTE 00, 00, 00 A2JOURNEY0TO2 .BYTE 00, 01, 00 A3JOURNEY1TO0 .BYTE 00, 01, 01 I need to have: JOURNEY_01 ; 00 TO ...
1
vote
1answer
63 views

redirect output of awk gsub command to file

I found an answer on how to remove commas between quotes of a comma delimited csv file over here on stackexchange (cannot comment there as I don't have the needed rep). awk -F'"' -v OFS='' '{ for ...
1
vote
2answers
31 views

When interpret awk as a command or a programming language?

I know that awk is a script programming language, but sometimes I get confused about when I interpret as command or as a program. Eg. 1 - Here I interpret as a command: awk '{print $2}' file.txt ...
0
votes
1answer
110 views

Extract values from filename and add it into a text file

I have a gridded model data (separated by different depth) saved in text files. The structure is like this: Format of every column in a row in the text file: x_coordinate y_coordinate density ...
-1
votes
3answers
57 views

Filtering a .CSV file based on consecutive numbers in a line

I have a CSV file as follows: 1st 2nd 3rd 4th ID ... Res Res Res Res (other columns) ... RZ_AUTO_1, 1cx0, C118, B, C119, B, A165, ...
2
votes
2answers
107 views

Compare 2 delimited files and output differences

I have asked similar questions here a couple times in the past with great success, but now my needs have slightly changed and I am struggling to get the exact output I am looking for. I would like to ...
6
votes
5answers
185 views

Using awk to identify the number identical columns

I am new to all this, and this is probably a rather simple question, but I am stuck: I have a large number of individual files that contain six columns each (number of rows can vary). As a simple ...
1
vote
1answer
88 views

Summing by common strings in different files

I have a file file1 with the amount of times a user shows up in the files, something like this: 4 userC 2 userA 1 userB and I have another file file2 with users and other info like: userC, degree2 ...
0
votes
1answer
82 views

Cut a specific part of a column from a csv file [closed]

I have to process a csv containing about 15 columns one of which contains a particular timestamp that includes milliseconds. When I go to insert this csv into a DB, I get an error because of this ...
1
vote
1answer
123 views

splitting up a MySQL dump using awk

I have a large-ish (250G) MySQL dump file from which I only want / need one database. I was hoping to use awk to split the file up so I did some searching and found this expression: awk ...
2
votes
4answers
95 views

replace several comma out side one or more sets of curly braces and exception that in one or more sets of curly braces

In the text file I have multi records. Every record has multi column with comma separated, some column have one sets of curly brace and other have more than one curly brace. I need: If a comma is ...
4
votes
2answers
962 views

Remove entire row in a file if first column is repeated

I have a file containing two columns and 10 million rows. The first column contains many repeated values, but there is a distinct value in column 2. I want to remove the repeated rows and want to keep ...
1
vote
2answers
172 views

Merging 2 files with based on field match

I want to create a file that contains columns from two input files. File1 is like: aa 32 bb 15 cc 78 File2 is: fa 19 bc 23 cc 50 de 28 aa 45 bb 31 The task is, read through File1, if the 1st field ...
2
votes
1answer
146 views

Sed to Modfiy Hosts File

I have a text file, modifyhostslist.txt, which contains entries that correspond to entries found in my hosts file. Not every entry in my hosts file needs to be modified, only entries also found in ...
1
vote
1answer
96 views

Sum values in the other columns for the 1st column with the same ID

The input txt file looks like (there are more columns in the actual txt file): target_id length eff_length tot_counts uniq_counts est_counts eff_counts mthl7 61 0 0 0 0 0 loqs 72 ...
1
vote
3answers
233 views

keep only certain part of a string in a certain column

I have a file like this: id target_id length eff_length 1 intron_FBgn0000721:20_FBgn0000721:18 1136 243.944268 1 intron_FBgn0000721:19_FBgn0000721:18 1122 ...
1
vote
1answer
79 views

Replace pattern in file with pattern in second file

I want to replace part of a file's data, with data from another file. Suppose file1 has data as is written below and file2 has some data I want to store and replace file1 data from file2 from ...
12
votes
2answers
681 views

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++' <f q w e r $ The original code I have found on the internets ...
4
votes
2answers
214 views

Comma separated words to next line in new file

I have a input file like this (input.txt): Main1 one,two,three,four Main2 five,six,seven,eight I want the output like this in output file (output.txt): Main1 one Main1 two Main1 three Main1 four ...
1
vote
2answers
65 views

How to match a number in one file to a name in another file?

Okay, I hope this is the right place to ask. I have 2 files. The first file has 4 fields: class CRN #, Instructor ID #, Class Section #, Number of Students. Example: 0002 T00005 006 15 0004 ...
2
votes
4answers
80 views

how to print a new line in awk

I am trying to print collected data from a file where each set gets a new line (\n) Code: awk '/mail:|fullName:|uid:/{s=s", "$0} END{print substr(s,3)}' dump2.txt Output now: mail: ...
1
vote
2answers
91 views

grepping for multiple items

I have about 6 items I need extracted from and LDAP base dump file fullName: uid: email: ... the colon is required for the search, mainly because uid appears in a number of places in the file but ...
1
vote
3answers
260 views

sed + find value before word in line

How can I find the value before the word "free"? top -n1 | grep Mem Mem: 2054968k av, 2034120k used, 20848k free, 0k shrd, 186768k buff First I tried top -n1 | grep ...
-2
votes
1answer
56 views

Extract/print every 3rd occurence of pattern

I need help with extracting/printing lines until a stop pattern. This needs to happen after every 3rd occurrence of start pattern, i.e ERROR_FILE_NOT_FOUND and continue until the end of the file. ...
-2
votes
1answer
4k views

convert CSV to XLS file on linux [closed]

The following Perl script can convert CSV file to XLS file the problem is that I need to install on customer Linux machine many Perl modules in order to run this Perl script , and actually I cant ...
0
votes
3answers
205 views

How does the following shell script work in finding all files larger than 20MB?

I came across a post advising that the following will find all files greater than 20Mb in size. find / -type f -size +20000k -exec ls -lh {} \; 2> /dev/null | awk '{ print $NF ": " $5 }' | ...
0
votes
5answers
296 views

df -k + how to match the root volume line

please advise how to match the "/" volume from df -k command? I tried the following: [root@machine /]# df -k | grep "/" /dev/vx/dsk/bootdg/rootvol 130926685 71378710 58238709 56% / /devices ...
1
vote
2answers
74 views

Extracting file name and string from multiple files

I have a lot of files (>100,000) from which I would like to extract a specific string and the filename itself. Files are organized like this: Filename: sequence.php?ID=gi|90022703|ref|YP_528530.1| ...
0
votes
1answer
70 views

Finding words with one vowel group

I am trying to use a command to find all words in a list of words with one group vowels. Like four, for and far, but also words like ore and or. What I got so far: grep -i ...
1
vote
3answers
100 views

Find a string in one section of a file with mutiple sections

How to sed/awk for a string contained in another string. TESTVAR=' { icon : "icons/773_l.png", id : 80, initialState : true, isPng : false, label : "Imagery", opacity : 1, requestType : ...
4
votes
4answers
233 views

How to remove all comments from a file?

I have a file with a few lines commented out: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I just want to print all the uncommented code: foo bar stuff morestuff ...
2
votes
0answers
216 views

extracting data from a log file and plotting it using Gnuplot

I have a logfile with the following data in it: Sat Sep 20 11:25:01 BST 2014 -- temp=53.0'C -- message from script /usr/src/scripts/wifi_test_2.sh Sat Sep 20 11:26:01 BST 2014 -- temp=53.0'C -- ...
1
vote
4answers
461 views

splitting a column using awk

I have a file like shown below. The 9th column has values joined by **. chrXV 234346 234546 snR81 + SNR81 chrXV 234357 0.0003015891774815342**0.131826816475 + chrXV 234346 ...
1
vote
2answers
140 views

Extract text from one iteration of top

I want to execute the top command from my bash script the problem is that usually when I type the top command on the console , I exit by Ctrl+C. but how to exit from top when I run the top command ...
-1
votes
1answer
226 views

search for a character in one column and replace value in another column of matched records

I have a requirement to find value 2 in column 4 and wherever it matches replace 6th column with '-'sign in the beginning suppose if i have csv file - file1 like this abc,0,def,1234,ijk ...
2
votes
1answer
135 views

Getting matched fasta file

list.txt 58759__len__2903 58759__len__2903 673957__len__1655 673957__len__1655 3566454__len__1744 seq.fasta >58759__len__2903 TTTTCCGTAGAGGAGATCCCTATTTTTAGGTTTGTAAGAGATCATTTT ...
-1
votes
1answer
85 views

Match and combine records in Bash [duplicate]

Consider two csv files: $ cat given.csv 123,John 246,Paul 369,George 987,Ringo $ cat family.csv 246,McCartney 123,Lennon 987,Starr 369,Harrison Note that the lines are not ordered! How might I use ...
2
votes
1answer
82 views

Substitute one field with other if its entry starts with specific letter

If I have a file something like: 223 898 teeth 223 899 mouth 223 900 r9 445 800 round My question is how can I use awk command to substitute column 2 with column 3, only for the row in which column ...
0
votes
2answers
64 views

Convert two rows to two columns

I would like to convert every two rows to two columns using awk. Any help appreciated. input.txt: # Query: gi|11465907|ref|NC_001872.1| Chlamydomonas eugametos genome, complete genome # 0 hits found ...
2
votes
2answers
56 views

Gather job usage information on cluster

I am running simulations on a linux cluster shared by hundreds of other users. There is a job scheduler installed on the cluster that determines who runs where, and one of the scheduler commands ...
8
votes
2answers
305 views

Reindexing a large CSV file

I went through the answers in this helpful thread, but my problem seems to be different enough that I can't think of good answer (at least with sed). I have a large CSV file (200+ GB) with rows that ...
2
votes
3answers
158 views

Use AWK to pad date and time formats

I would like to use AWK to pad the date and time fields in my report. Here is my current .csv that is actually tab delimited. AA 9/7/2014 2:30:38 PM AA0000011111 08 Y A Jane, Doe AA ...
2
votes
3answers
57 views

Merging two tables including multiple ocurrence of column identifiers and unique lines

I asked a similar question yesterday (Merging two tables including multiple ocurrence of column identifiers) but ran into a problem with unique lines. I would like to merge two tables based on column ...
4
votes
6answers
550 views

Sed — Replace first k instances of a word in the file

I want to replace only the first k instances of a word. How can I do this? Eg. Say file foo.txt contains 100 instances occurrences of word 'linux' . I need to replace first 50 occurrences only.
2
votes
2answers
121 views

Merging two tables including multiple ocurrence of column identifiers

I would like to merge two tables: File 1: 1 today 2 tomorrow 3 red File 2: 1 a lot 1 sometimes 2 at work 2 at home 2 sometimes 3 new Desired output (file 3): 1 today ...