0
votes
0answers
18 views

Row merger due to redunt splits of a single row in Unix

Thanks for your explanation and time Avinash. The code is working as expected but my depiction of problem truned much simpler than the actual problem. Problem: I have a file which has ~5Million of ...
2
votes
2answers
36 views

Sed/awk/perl: modify text preserving parts and align to a column

I have a text like this: A1JOURNEY0TO1 .BYTE 00, 00, 00 A2JOURNEY0TO2 .BYTE 00, 01, 00 A3JOURNEY1TO0 .BYTE 00, 01, 01 I need to have: JOURNEY_01 ; 00 TO ...
0
votes
1answer
56 views

Cut a specific part of a column from a csv file [closed]

I have to process a csv containing about 15 columns one of which contains a particular timestamp that includes milliseconds. When I go to insert this csv into a DB, I get an error because of this ...
1
vote
1answer
41 views

How to write this replacement with condition

I want to replace now within my working directories the word -> with $\to$ if the word is not within verbatim environment. Pseudocode Replace all entriens of -> with $\to$ everywhere else but not ...
3
votes
2answers
88 views

How to write this pseudocode where condition

This question is extension of this question about listing questions of directories. I get a problem that I need to understand the original section of the text. Current output where you see the ...
1
vote
1answer
29 views

Perl one-liner for replacing values greater than a threshhold

I have a text file which contains 7 columns of floating point values (). I would like to use perl one-liner and find the values which are greater than a threshold in the 7th column. Then, I want to ...
0
votes
3answers
72 views

Combinatorics of letters and words - preferably bash, will accept perl or other [closed]

I am trying to write a shell script to generate all possible words in the English language less than 20 characters. I doubt there is any truly efficient way to do this other than to brute force some ...
3
votes
1answer
50 views

Handling dynamically changing column positions and splitting file

I have a unique problem. I have a file with following columns PersonId, AB1, AB2, AB3, DG1_B1,DG1_B2,DG2 1,0,1,0,3,5,7 2,0,3,1,3,7,4 I want to split this and create multiple files based on the ...
2
votes
3answers
34 views

Delete until given character starting from given string

root@server:~/foo$ cat INPUT.txt limit. Current u'^'ZO limit. Current usage'^'FR limit. Current us'^'AS limit. Current usage 95 %.'^'FR limit. Cur'^'AEW root@server:~/foo$ sed ...
2
votes
2answers
138 views

Format the content of the file using Perl

I am new to Perl programming and I am stuck with this problem for some time. Can you please help me in solving it? I have a file with the following format: >Country1 Aus trali a >Country5 ...
2
votes
1answer
475 views

Convert xml subtitle file to srt format

I have a xml subtitle that looks like this: <?xml version="1.0" encoding="utf-8" ?><transcript> <Item from="1.16" duration="4.68"><![CDATA[(Dong-hyuk is coming to see you now.) ...
1
vote
1answer
51 views

Retrieving fasta sequences using bed file information from locally installed file

I have a .bed file containing around 30000 rows for which I have the sequences retrieved using fetch-sequences module of the rsat tool (http://rsat.ulb.ac.be/rsat/help.fetch-sequences.html#usage). ...
1
vote
1answer
66 views

Extract time stamps and find the difference

I have a large set of logs as shown below: June 5 15:51:56.606 [3997]: TR#153. State machine - state = CC_STATE_IDLE, event = EVT_ISDN_SETUP June 5 15:51:56.607 [3997]: TR#153): SC ...
3
votes
5answers
133 views

Get values block by block in same file

I have a file say "SAMPLE.txt" with following content, P1 10,9:6/123456 P2 blah blah P1 10,9:5/98765 P2 blah blah P1 blah blah P2 I want a output file say "RESULT.txt" as, Value1:123456 ...
1
vote
3answers
2k views

compare two columns of different files and print if it matches

I am using solaris 10 and Grep options involving -f doesnt work . I have two pipe separated files: file1: abc|123|BNY|apple| cab|234|cyx|orange| def|kumar|pki|bird| file 2 : abc|123| ...
0
votes
3answers
2k views

compare two files based on a column and print it

I have two big files of 400,000 lines. I want to compare the column 1 of the second file with column 1 of first file recursively. If they match I would like to print the whole line. It is a sorted ...
4
votes
2answers
67 views

Using perl to modify a LaTeX document

I am trying to use perl in order to modify a LaTeX document. In fact, I am trying to change the citation format from for example: . \cite{poelma20113d} to \cite{poelma20113d}. In other words, I ...
0
votes
4answers
78 views

Print text between tags (inclusive) if certain text is found

I've got a task to extract data from several Apache servers. The task is to print out: <Directory ...> ... </Directory> where +ExecCGI is located within. Let me give an example to ...
3
votes
7answers
541 views

Finding the maximum of the values in a file

I have a file which contains several numbers, as follows. 21 12 33 35 21 12 33 44 52 63 14 12 23 34 11 12 13 53 1 12 43 33 44 How can I find the maximum of these values, which is 63? I wanted to ...
2
votes
4answers
120 views

Take a sum based upon particular columns

I am having a huge pile of data around 30 to 40 Million data. We need to process these files and send to another interface team. Below is my file format which we recevie c1 c2 c3 c4 c5 c6 A B ...
4
votes
7answers
331 views

How can I groups numbers in a file

I have a file with numbers in float format. I can review them via sort -rn numbers.txt | less I would like though to be able to "group" them. I.e. easily see how many are in the same range. To give an ...
1
vote
1answer
126 views

replace a pattern found with a number plus 1 on the same line

I have a file where each line contains a keyword (HELLO) and one or more numerical fields designated A1, A2, etc. The value in the numerical field is enclosed in brackets after the field name and is ...
1
vote
4answers
549 views

Pattern Matching and Delete the whole line

I want to delete all the Lines of File 1, if Column1 of File1 matches exactly with Column 1 File2. File 1: r001:21:10 21 AAAAAATTTGC * = XM:21 r002:21:10 21 YAAAATTTGC * ...
4
votes
2answers
656 views

Replace string with sequential index

Can someone suggest an elegant way to accomplish this? Input: test instant () test instant () ... test instant () //total 1000 lines output should be: test instant1 () test ...
-2
votes
3answers
95 views

Deleting Lines, which match a particular Identifier from another file

I have 2 files. File 1, has an identifier (eg. D7MHBF:11:1449:1988) and every new entry starts with @. It has few more fields, which are not important in our analysis. File 2 consists of a column of ...
15
votes
1answer
8k views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
3
votes
1answer
63 views

Count and merge consecutive patterns

I'm searching for a short snippet to find, count and merge consecutive duplicates using standard tools or a common scripting language. Say our input is: 1 2 2 2 7 22 a b b c c c c c d dd 2 2 c c ...
3
votes
2answers
3k views

How to convert a .txt subtitle file to .srt format?

I have a subtitle file, it looks like this: 00:00:44:" Myślę, więc jestem".|Kartezjusz, 1596-1650 00:01:01:Trzynaste Pietro 00:01:06:Podobno niewiedza uszczęśliwia. 00:01:10:Po raz pierwszy w ...
1
vote
1answer
234 views

how to subtract a constant value from a data file?

I want to subtract a constant number from my data to get new data file. For example, I want to subtract 10 from my data. data.txt Id a b c d e 1 1 2 3 4 5 2 2 3 4 5 6 ...
1
vote
3answers
250 views

How to calculate the sum of the data that have the same ID in the first column?

I want to sum the values of the rows of my data that have the same ID in the first column. My data looks like data.txt Id a b c d e 1 1 2 3 4 5 1 2 3 4 5 ...
0
votes
2answers
110 views

How to Lookup the values of an index values from a big data file?

I have a data file (data_array.txt in the example) and index file, index.txt, at which I want to extract the data from the data file that have the same ID in the index file and store in to a new file, ...
4
votes
5answers
6k views

extract the 4th column from a csv file using unix command

I have a csv file formatted as below. "col1","col2","col3","col4" "1","text1","<p>big html text</p> ","4th column" "2","text2","<p>big2 html2 text2</p> ...
4
votes
4answers
202 views

Howto find and move a text line inside a file?

I need to change a text line position inside a file; from one line position to another one, underneath a defined text. For example, I have the following text block (on /etc/pam.d/system-auth file) ...
1
vote
1answer
122 views

Parsing the headers of sequence file

I have a multiple sequence file as >abc|d017961 sequence1...... >cdf|rhtdm9 sequence2...... >ijm|smthr12 sequence3...... >abc|d011wejr sequence4...... >stg|eethwe77 sequence5...... ...
1
vote
3answers
3k views

How to convert from one date format to other format?

I have one problem with my date format. I want to change from one format to the other and vice versa. My date formats are Format 1 YYYY-MM-DD Format 2 MM/DD/YYYY I want to change format 1 to ...
1
vote
1answer
64 views

how to clean free row/column spaces after the row of your data

I have a problem in cleaning my file where there are unnecessary free spaces below my last data row. This is created when I am copying and pasting my data from excel to text. I want my text file to ...
1
vote
1answer
105 views

How do I modify this Perl solution so that it will substitute embedded double quotes with single quotes?

Previously in this question: "string having doublequotes in between apart from the enclosing quotes" @BernieReiter asked a follow-up question where he wanted to take CSV entries such as the following: ...
2
votes
6answers
3k views

Repeat each line multiple times

Would like to have each line in a file repeated a fixed number of times. e.g. have each line repeated four times: a b c becomes: a a a a b b b b c c c c I've done some searching, and there are ...
2
votes
3answers
1k views

I want to find duplicates in a file and add a character to the end of the line on the 1st match

I am trying to find duplicates in a file and once a match is found mark the 1st match with a character or word on the end of the line. eg my file (test.html) contains the following entries host= ...
3
votes
3answers
165 views

sed + how to remove character/s that start or ended on each number

How do I remove the . character(s) that start in the beginning of each number or end on each number? Remark – perl one liner also good alternative for sed. Example input: .23.12.44.5. .233.3.3.3 ...
0
votes
1answer
106 views

read line by line and take a string in line and assign that string to the subsequent line using perl [duplicate]

Data file is fixed length file, and I want to read file line by line and take a string from the current line, and then append that string to the subsequent line and write into another file. For ...
-4
votes
1answer
151 views

How to transform two delimited ASCII files

These are 2 common delimited, text file transformation problems I'd like to find scripts to do (I dont know if awk or perl is best for these tasks). csh is the default shell on my Solaris10 unix ...
2
votes
5answers
3k views

How can I split a text file into multiple text files using Perl?

I have a file ABC_TabDelim.txt that contains the following: 00:00:00:00 00:00:05:00 01SC_001.jpg 00:00:14:29 00:00:19:29 01SC_002.jpg 00:01:07:20 00:01:12:20 01SC_003.jpg 00:00:00:00 00:00:03:25 ...
5
votes
2answers
2k views

Replacing missing value blank space with zero

I have input.txt tab-delimited text file around 30K lines, I would like to check each row (s1..s30K lines) for missing value (i.e blank space) and fill the missing value with zero value.See out.txt ...
2
votes
3answers
4k views

Extracting column from comma separated text

I have a long comma-separated delimited file with 20K lines. Here's a sample: "","id","number1","number2","number3","number4","number5","number6","number7" ...
0
votes
1answer
94 views

How to get rows between lines?

I have a perl oneliner: echo -e 'sla\ndfjAAAAk\nsaljdfla\nsjdfBBBBsadf\njasdf' | perl -E 'my $_ = do { local $/; <> }; say join qq!\n!, m!\QAAAA\E\K(.*?)(?=\QBBBB\E)!gs' k saljdfla sjdf it ...
4
votes
6answers
370 views

What is the easiest way to check if column A and B values goes both ways with AWK? [closed]

What is the easiest way to check if column A and B values goes both ways? Output to check: Mike John John Mike Pamela Barbara Barbara Pamela Mike Paul Roger Paul Desired output Mike <-> ...
4
votes
2answers
143 views

Column mismatch and substituting

input.txt (tab-delimted) TTTTOTTT00000000008 RTTTT899 5.00E-28 TTTTOTTT00000000046 RTTTWRR 3.00E-31 TTTTOTTT00000000051 2.00E-11 TTTTOTTT00000000051 7.00E-12 TTTTOTTT00000000054 ...
2
votes
1answer
228 views

text processing-perl

I am a biologist. I would like help of computer experts to help in text file editing. Here is what the input file looks like: ##dsfsd2 ##sdf-sdf sasg 5.6.3 gi34_ex Gen CDS 161 317 . ...
2
votes
2answers
2k views

How to do a regex search in a UTF-16LE file while in a UTF-8 locale?

EDIT: Due to a comment Warren Young made, it made me realize that I was not clear on one quite relevant point. My search string is already in UTF-16LE order (not in Unicode Codepoint order, which is ...