0
votes
2answers
36 views

how to remove all double quotes from csv except specific field

I want to remove all double quotes from my csv but not the fourth field (because the four fields represent PATH of file) Please advice how to implement this by sed or awk or perl one liner , etc ...
5
votes
1answer
36 views

Count duplicate rows with constraint

I have the following file: A B 1990 A C 2001 A C 2013 A B 2001 B C 2013 A D 2001 A D 2014 I need to count duplicate pairs (in the 1st and 2nd column) and to each such pair assign the lowest value ...
4
votes
2answers
66 views

Get the maximum sum for a file on 2 keys

I have a file as below. key1 key2 key3 a1 2 l1 a1 2.5 l2 a2 2 l2 a3 2.5 l3 a3 2.1 l4 a3 2.2 l5 I am trying to find the maximum possible sum of key2 that satisfies the ...
1
vote
1answer
38 views

Retrieving fasta sequences using bed file information from locally installed file

I have a .bed file containing around 30000 rows for which I have the sequences retrieved using fetch-sequences module of the rsat tool (http://rsat.ulb.ac.be/rsat/help.fetch-sequences.html#usage). ...
3
votes
3answers
204 views

calculate sum of squares using shell script in perl/awk

I have 2 files as below. file1 0.34 0.27 0.32 file2 0.15 0.21 0.15 Now, I would like to calculate the sum of squares between each column. For example, [(0.34 - 0.15)^2 + (0.27 - 0.21)^2 + ...
2
votes
2answers
54 views

How to print top five highest numbers from a column?

I have a text file with four columns. I need to read till end of file and print the highest number from column3 (top 5 values) along with column 1. input.txt xm|340034177|ref|RT_235820.1| 139697 ...
2
votes
3answers
140 views

replace column of a csv file with column from another file

I have two sample files like this: $ cat file1 abc,sachin cat,kumar $ cat file2 xyz pressure $ cat file3 xyz,sachin pressure, kumar I want the first column of file1 to be replaced with file2. ...
0
votes
1answer
83 views

Duplicate specific text in text file, using perl or grep

I need to duplicate part of a plain text file as follows: The text file consists of a sequence of "pages", of the form: <page [lots of lines of text] /page> <page [lots of ...
0
votes
4answers
63 views

Print text between tags (inclusive) if certain text is found

I've got a task to extract data from several Apache servers. The task is to print out: <Directory ...> ... </Directory> where +ExecCGI is located within. Let me give an example to ...
0
votes
2answers
56 views

Method to paste specific column when in terminal

I'd like to paste only the third column of data from within the terminal. I originally have something like: 4 SF0086 SF0086 12 SF0087 SF0087 17 TSF141 TSF141 29 TSF070 TSF070 30 TSF028 TSF028 31 ...
6
votes
3answers
510 views

Is there a perl or awk solution for this problem?

I have an input file (input.txt) like below. id1 id2 name weight 53723848 12651711 timburnes 1.36667 53530214 12651711 timburnes 1.51191 53723848 53530214 timburnes 1.94 764157 ...
4
votes
7answers
319 views

How can I groups numbers in a file

I have a file with numbers in float format. I can review them via sort -rn numbers.txt | less I would like though to be able to "group" them. I.e. easily see how many are in the same range. To give an ...
3
votes
4answers
249 views

merge 2 rows based on the same column values

I have a file like below. 47196436 47723284 name1 1.77273 42672249 52856963 name2 1.06061 52856963 430695 name2 1.16667 55094959 380983 name3 1.55613 17926380 55584836 name4 1.02461 3213456 34211 ...
2
votes
3answers
102 views

awk - to add a column after matching the remaining columns

I have a file (file1.txt) with the contents as below. 8145216 3377090 1.5 1405541 53595498 1.53637 I need to generate all the possible combinations for the first 2 columns in the above file. I use ...
3
votes
2answers
73 views

introduce line break at the same position based on another file

I have 3 input files all having the same number of lines. I have come up with a movie example to make the question look interesting. file1.txt -- Essentially contains the actor names. ...
1
vote
3answers
345 views

Extract part of string using grep

I have 3 files in a directory MYO144064T MYO144064TA MYO144064TX Where the digits and 11th character will change. In my csh script, I want to extract the letters 'A' and 'X' probably using grep ...
1
vote
3answers
241 views

Print the last-but-one field

How to print the word before the last word in line (with ksh or awk or sed or perl one liner) Example 1: echo one two three will print "two" Example 2: echo 1 2 3 4 5 6 will print "5"
2
votes
1answer
54 views

awk to split the input file into 2 files(one for duplicated values and one for unique values)

I have an input file like below. file.txt 14 2.692022 author1 0 14 2.57826 author2 1 229 2 author3 2 323 2 author4 3 514 2.0625 author5 4 I am trying to split the file with the duplicated values ...
0
votes
2answers
69 views

search for first pattern and copy the string inside the ( ) into the 3rd field in lines where the second pattern found

Here are my spec: first search for "#firstpattern(" copy the content inside the ( ) to the 3rd field inside the ( ) of all lines where the "secondpattern" found. remove the entire line with the ...
1
vote
1answer
118 views

replace a pattern found with a number plus 1 on the same line

I have a file where each line contains a keyword (HELLO) and one or more numerical fields designated A1, A2, etc. The value in the numerical field is enclosed in brackets after the field name and is ...
2
votes
5answers
239 views

Can the string “abcedf” be matched to string “bafcde” in a single line command?

I am planning to implement an indexing structure in my program. For example, if I have 100 rows in the table, I will number these rows from 1 to 100 in another column by appending an _ to the end of ...
1
vote
4answers
233 views

Pattern Matching and Delete the whole line

I want to delete all the Lines of File 1, if Column1 of File1 matches exactly with Column 1 File2. File 1: r001:21:10 21 AAAAAATTTGC * = XM:21 r002:21:10 21 YAAAATTTGC * ...
4
votes
2answers
338 views

Replace string with sequential index

Can someone suggest an elegant way to accomplish this? Input: test instant () test instant () ... test instant () //total 1000 lines output should be: test instant1 () test ...
1
vote
1answer
83 views

pcregrep fails due to segmentation fault

I am extracting the HTML source of a web page using the below command. curl http://www.linkedin.com/pub/ramesh-venkataraman/63/621/b42 -s | w3m -dump -T text/html > foo.txt The foo.txt has the ...
1
vote
1answer
121 views

Is Perl ever a better tool than awk for text processing [closed]

This question was inspired by a discussion on stackoverflow - see in particular the comment thread under Ed Morton's answer. I figured this was the site where I could get a definitive answer: In ...
3
votes
1answer
85 views

Reorder Multiple Line Blocks with Sed

I am trying to reorder generated output as follows; given a tex fragment as follows, I want to move the /^Constant/ down below /Dummies/, with the line before and after /^Constant/. Further, I would ...
-2
votes
3answers
84 views

Deleting Lines, which match a particular Identifier from another file

I have 2 files. File 1, has an identifier (eg. D7MHBF:11:1449:1988) and every new entry starts with @. It has few more fields, which are not important in our analysis. File 2 consists of a column of ...
6
votes
1answer
857 views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
1
vote
1answer
188 views

how to subtract a constant value from a data file?

I want to subtract a constant number from my data to get new data file. For example, I want to subtract 10 from my data. data.txt Id a b c d e 1 1 2 3 4 5 2 2 3 4 5 6 ...
1
vote
3answers
176 views

How to calculate the sum of the data that have the same ID in the first column?

I want to sum the values of the rows of my data that have the same ID in the first column. My data looks like data.txt Id a b c d e 1 1 2 3 4 5 1 2 3 4 5 ...
0
votes
2answers
81 views

Merging two different files with conditions

I have two different files as shown below. Content of a.txt: HDR|1|||||||||| DTL|@||||||||||| TLR||||||||||||| HDR|1|||||||||||| ...
0
votes
2answers
86 views

How to Lookup the values of an index values from a big data file?

I have a data file (data_array.txt in the example) and index file, index.txt, at which I want to extract the data from the data file that have the same ID in the index file and store in to a new file, ...
0
votes
1answer
36 views

rearranging from colum to row in a csv [duplicate]

I have a CSV file that presently has an output like this 1 2 3 4 5 what I need to do is reconfigure the output as follows 1 2 3 4 5 all the entry data on one line, any help would be greatly ...
4
votes
5answers
4k views

extract the 4th column from a csv file using unix command

I have a csv file formatted as below. "col1","col2","col3","col4" "1","text1","<p>big html text</p> ","4th column" "2","text2","<p>big2 html2 text2</p> ...
3
votes
1answer
31 views

How to calculate (weighted) majority over columns?

I have two files: the first one contains lines where the first column is an ID, and the remaining columns are votes, e.g. 3242 -1 1 -1 1 1 1 In a first step I want to calculate the majority for ...
1
vote
3answers
2k views

How to convert from one date format to other format?

I have one problem with my date format. I want to change from one format to the other and vice versa. My date formats are Format 1 YYYY-MM-DD Format 2 MM/DD/YYYY I want to change format 1 to ...
4
votes
6answers
98 views

Only show the lines that are in all textfiles at least once

a.txt cat a.txt a b x c b.txt cat b.txt d e a f the q: SOMEMAGICK *.txt a Q: how can I show only the lines that are in all *.txt files?
1
vote
1answer
60 views

how to clean free row/column spaces after the row of your data

I have a problem in cleaning my file where there are unnecessary free spaces below my last data row. This is created when I am copying and pasting my data from excel to text. I want my text file to ...
3
votes
3answers
3k views

importing data from a text file to a bash script

I need to import a text file like this below: AMBI CBNK CCXI CERE CLACW CNYD DAEG DLIA DLLR DNDN DSGX HAST HIBB, HPOL IRDMZ MARK NEPT NFEC NINE NRCIB OMER PLCC PLPM PSUN UNTK There are 25 entries, ...
1
vote
3answers
395 views

How to limit MAX an MIN length of a string?

"nothing" means no OUTPUT. Q: how can I limit the MIN and MAX length of a piped STRING? (MIN: x>=3 chars, MAX: x<=32 chars) INPUT: echo 12 | MAGIC OUTPUT: nothing INPUT: echo 123 | ...
1
vote
1answer
51 views

How to make URL's out of this text with least resources?

INPUT: nyeeeeeeeeeeeeeee.txt:sadfsadf sadjflsajfd:jasdlfjalsfd foo/asdfsafd.txt:laksjdlfajsfd OUTPUT: <a href="nyeeeeeeeeeeeeeee.txt">nyeeeeeeeeeeeeeee.txt</a>:sadfsadf <a ...
1
vote
6answers
2k views

Repeat each line multiple times

Would like to have each line in a file repeated a fixed number of times. e.g. have each line repeated four times: a b c becomes: a a a a b b b b c c c c I've done some searching, and there are ...
2
votes
3answers
745 views

I want to find duplicates in a file and add a character to the end of the line on the 1st match

I am trying to find duplicates in a file and once a match is found mark the 1st match with a character or word on the end of the line. eg my file (test.html) contains the following entries host= ...
2
votes
3answers
1k views

Easy way to parse syslog date format

I'd like to write a script that analyses the last week's syslog. But my syslog logs in a stupid date format, e.g. "May 22". If it logged as 2013-05-22, this task would be trivial. Is there a way ...
3
votes
3answers
153 views

sed + how to remove character/s that start or ended on each number

How do I remove the . character(s) that start in the beginning of each number or end on each number? Remark – perl one liner also good alternative for sed. Example input: .23.12.44.5. .233.3.3.3 ...
6
votes
4answers
2k views

How to remove multiple blank lines from a file?

I have some text-files I use to take notes in - just plain text, usually just using cat >> file. Occasionally I use a blank line or two (just return - the new-line character) to specify a new ...
1
vote
2answers
56 views

Trimming one part if equal to another part?

I use an e-mail quote attribution string like this: On dd mmm yyyy hh:mm +hhmm, from info@example.org (Example Dot Org): where info@example.org is the sender's e-mail address and Example Dot Org is ...
0
votes
2answers
205 views

Reordering strings in linux

How can I change lines around using the command line? e.g. I have these lines: Acct-Status-Type = Start User-Name = "37XXXXXXX" Event-Timestamp = "Apr 12 2013 15:56:55 AMT" I need to ...
0
votes
3answers
371 views

Sorting IP address according to the second field in file

in my ksh script I need to add the following task ( OS - linux/solaris ) I have the following file more test.txt /etc/backup/app 172.1.120 172.110.120 /etc/backup/app 172.1.120.12 ...
-4
votes
1answer
140 views

How to transform two delimited ASCII files

These are 2 common delimited, text file transformation problems I'd like to find scripts to do (I dont know if awk or perl is best for these tasks). csh is the default shell on my Solaris10 unix ...