Manipulation of text by programs, scripts, etc.

learn more… | top users | synonyms

0
votes
0answers
16 views

problem in extracting fasta sequence based on header name

I'm facing a crazy problem during extracting fasta sequence based on their header name. I have a fasta sequence file and a text file containing their header name. I tried to extract sequence of ...
-1
votes
1answer
24 views

How to add a word at the first of every lines? [duplicate]

File1: Data will be inserted on Thursday Messages has been sent at 6pm Office hours is from 9am to 6pm All the employees are present Output what I want: DMS Data will be inserted on ...
2
votes
1answer
61 views

Show nth occurrence of a record

Using awk, how to show the nth occurrence of a record and delete the other occurrence, for example" Input File: 79933,2 79933,4 79933,5 55251,1 49494,4 49494,2 49494,2 49494,1 in this example, ...
2
votes
5answers
128 views

How to sort file by character occurrences per line?

I'm quite new to Linux, and I've found quite a bit of useful information on how to do character counts in a file, but is there a way in Linux/terminal to sort a text file by the number of times a ...
1
vote
5answers
311 views

Check number of words in a file that contain a specific letter

Bash command to check number of words in a file that contain letter “a”
1
vote
2answers
29 views

Append mth and nth columns of a file with the columns of another file

I have 2 files having different number of columns and with huge number of rows: bash-3.00$ cat fileA a1,a2,a3,a4,a5,a6,a7,a8,a9 q,w,,,,y,u,, a,z,,,,q,n,, ......................... z,p,,,,w,e,, ...
2
votes
3answers
137 views

Remove the ^L (aka \f/FF/form-feed/page-break) character

How do I remove the ^L character and the blank lines that come after it on a unix file? I have tried the below and have been able to remove the VT and spaces but am failing to remove the ^L character ...
3
votes
3answers
122 views

If statement string comparison

I'm reading a csv file with 14 values per line. I want to convert the HEX value from 10th [f10] value to a DEC value. In order to do this if the 7th [f7] value must hold EnHr or EnSt or SpJb or Chem ...
1
vote
3answers
44 views

Adding a pattern to each field of the following line

I need some help here please. I have the following input: MX04A; DMX04A; DMX04A; LMX04A; LMX04A; -17.2; -15.3; -14.3; -13.6; -16.8; -15.4; -16.0; -15.3; LH36A; DLH36A; DLH36A; -11; -117.2; -11; ...
2
votes
5answers
70 views

bash expanding to same value instead of line by line value in sed command

I have a file containing numbers in it say numbers.txt as follows. 9374541632553 51243747879841 32030098896914 84654557358238 11000656847765 I want to convert it to the following format. ...
0
votes
0answers
32 views

Drawing a histogram from a bash command output

I have the following output from a miRNA Predictor: Read Sequence:hsa-miR-576-3p MIMAT0004796 (22 nt) Read Sequence:gi|182396|gb|M74088.1|HUMFAPAPC Human APC gene mRNA, complete cds (8972 nt) ...
3
votes
3answers
204 views

awk comparison using arrays

I have the following file: 6180,6180,0,1,,1,0,1,1,0,0,0,0,0,0,0,0,4326,4326,,0.440000, 6553,6553,0,1,,1,0,1,1,0,0,0,0,1,0,1,0,4326,4326,,9.000000, ...
2
votes
2answers
18 views

How to replace value for a given condition in specific column of file

I have file separetd by pipe | and I want that when column 6 contain letter I print 0758000 in column 7 and when the column 6 contain letter A print 0800000 in column 7, I can not find how to do it!!! ...
3
votes
6answers
360 views

Grep the text which matches with a pattern

I was looking for a command which will print the selected portion of a line which matches the conditions. Eg, my text line is like this: p1=X||p2=Y||p3=X||p4=X||p5=X||p6=Y||p7=X I want to print ...
1
vote
2answers
34 views

Escaping double quotes in string

I have a string message which contains data in double quotes. I need to put \ in front of every double quote how can u do it in shell ? I used Perl and it works fine, but I'm looking for sh command: ...
2
votes
2answers
29 views

Shell script - text processing help

I want to create a ldif modify file to update the users on LDAP. I have an input file(alluserdns.ldif) with list of all Dn entries in ldap: dn: cn=testuser2@abc.com, ou=persons,ou=pp,dc=abc,dc=com ...
1
vote
0answers
44 views

Convert / import flat file with multiple records, one variable per line

I am searching for an efficient way using (UNIX) command line tools to read a flat file into a(ny) database, perhaps after first converting to an intermediate structured format, for example XML or ...
2
votes
3answers
131 views

How to print the before lines of search keyword with out using grep in a gz file?

How to print the lines before a search keyword without using grep in a .gz file? ID:342N000390AAAAAAAA 07/14/15 10:26 (MV90 ) * Register Data Imported * Warning - No ...
0
votes
1answer
27 views

How to replace the content of specific column with awk? Tab Delimited File

Here's a similar question but I ran into a problem that was not addressed in the question. How to replace the content of specific column with awk? I want to do something similar, however my file is ...
1
vote
1answer
19 views

AWK: where is my error?

I have a file with :-delimited fields. If the 3rd field of a line is greater than 22, that line should be printed to file44. The file looks like this: 1:2:3:4 1:2:3:411 1:2:33:42 1:2:3:44444 ...
5
votes
4answers
134 views

Remove lines from file up to a pattern, unless the pattern doesn't exist

I am attempting to write a filter using something like sed or awk to do the following: If a given pattern does not exist in the input, copy the entire input to the output If the pattern exists in ...
1
vote
1answer
26 views

Loop over lines of file, partition file and classify each partition

In the following I've put up my code for iterating over lines of a file in order to collect data within a segment from the file in following fashion: Illustration for understanding procedure: segL ...
1
vote
4answers
134 views

UNIX to print a content from a .gz file

I have a .gz file with contents like the following: ID:123455 23-07-2015 mvni warning: RTM post warning : no profile data Using the string no profile data, I want to print the ID. I tried with ...
2
votes
3answers
60 views

Read a column from a file and append a specific column of another file

I have two files as such: fileA: Name,Site Name,Product UPEI,Product Name,NMS Flag,Product Model, SW FileB: NE_Name SW ABC 4.4 ASD 4.3 ... ... Note that ...
3
votes
4answers
98 views

Efficient data extraction from multiple files to a single CSV file

I have a large collection of XML files with the same exact structure: $ cat file_<ID>.xml ... ... ... <double>1.2342</double> <double>2.3456</double> ...
2
votes
5answers
164 views

Perl script to compare two consecutive lines for the first character and ignore the second line if both first characters are same

Can anyone help how to compare two consecutive lines from a text file for the first character and if both the first characters are same ignore/delete the second line and print into new file only the ...
0
votes
0answers
16 views

Adapt text lines to fixed length [duplicate]

With the shell command fmt -w 30 myfile it is possible to truncate the myfile lines so that their length is at most 30 characters. The result is however not a justified file: some lines will be ...
-1
votes
2answers
51 views

Substitute a string containing && with sed [closed]

I've wrote some script for automated technical acceptance tests. Since we're not allowed to write password in clear text, the script I used is associated with many variables, one of which is the ...
2
votes
4answers
191 views

Need to print the matching lines in different file

Eg: I have data like this 1111111 test1 test2 test3 1111111 block 1 block 2 block 3 1111111 page1 page2 page3 I would like to print the content as different files as bellow. Test1 - File 1111111 ...
0
votes
3answers
70 views

Search for string(s) in files and report four lines of context (match and three following)

I have a string (or a list of strings) that is/are to be searched in all the files of a directory.  For example, if I am searching for ABC, and file1.txt contains ABC on line 6, then I want lines 6, ...
1
vote
2answers
103 views

Remove lines from CSV file if the second column is in a word list

I have found a long list of free email providers that I want to remove from my email lists - https://gist.github.com/tbrianjones/5992856 Below are two commands I currently use that do the same job ...
1
vote
2answers
73 views

How to remove characters of text in one line before specific character

How to remove a characters of text in one line before specific character I will remove characters to the left of the first letter "F" Example: null###F01|54646466|00K4234001|IFD|1990101 ...
1
vote
1answer
33 views

Helping with sed - remove specific lines with certain string

These are the contents of the text file: cat dog elephant cat lion cat tiger penguin penguin emperor zebra I want to remove the line contain just cat and retain cat lion and cat tiger lines. There ...
0
votes
1answer
40 views

Replace pattern with line from file using awk/sed [duplicate]

I have a file first.html which contains the following code: <tr> <td class="headerValue">One</td> </tr> <tr> <td class="headerValue">Two</td> </tr> ...
2
votes
4answers
266 views

Replace regex match with string containing match

I have a file with text as Afghanistan=+93 Albania=+355 Algeria=+213 American Samoa=+1 Andorra=+376 Angola=+244 It has all the country list and its dialing code. I want to replace: ...
1
vote
1answer
41 views

Split column into separate columns

I want to split the column1 at dot . into two columns and rest of the columns remain same. Input file: chr21.9411551 7 13 53.8461538461538 7 29 24.1379310344828 Expected output file: chr21 ...
2
votes
1answer
55 views

combine two files to single file with combined columns

I am new to linux/unix shell scripting and struggling to combine two files into a single file with all columns from both files. I am providing my example files. File 1 chr loc T1 C1 chr1 100 2 ...
2
votes
4answers
56 views

CSV - Only keep certain entries

I have a bunch of data inside "" and separated by , - Example below of two lines "stampthisandthat.com","GANDI ...
5
votes
3answers
136 views

How to print a line if next line does not contain a particular string using nawk or sed

Input.txt: 8B0C remove 8B0D remove 8B0E remove 8B0F 8B10 remove 8B14 remove 8B15 remove 8B16 remove 8B17 remove 8AC0 ...
0
votes
3answers
35 views

How to print empty spaces in first column using awk or sed

Here is my sample input file. ip.txt: john math science paul math science rosy math jill science rob math science hary math Desired output: john paul ...
2
votes
5answers
286 views

Duplicate lines and make them unique?

I have the following structure of text lines: 3923 001 L05 LV,L05 RM 3923 002 L12 RA,L12 LA 3923 003 I06 ALL 3923 004 G04 RV,Z09 ALL but i would need this: 3923 001 L05 LV 3923 001 ...
1
vote
3answers
66 views

Random sampling and outputting the largest value

I have a fairly large data set ~500 million rows. The data set looks like below. Col 1 is float number, col 2 is mac id(device id) 1616.93,ac:22:0b:a6:22:c3 2872.32,c0:bd:d1:36:bb:49 ...
1
vote
2answers
51 views

Matching “keyword value” pairs from semi-structured input

On occasion I get output from a command that includes "key value" pairs, with possibly more than one pair per line. As a repeatable example, consider the command ip addr show dev eth0: ip addr show ...
0
votes
2answers
26 views

Replace fields in file inline

I have file (a.txt) with following contents: Redhat master 12345678 V1A 20.0/20.0.1 Windows slave 12345678 V1B 19.0/19.0.1 I want to grep line with word master and replace 4th and fifth field with ...
1
vote
1answer
32 views

Group by Functionality on Mac Address

I'm looking for a solution that should do grouping of identical mac address(device ids). I tried using sort -k 2, but I'm not confident of the result. Data Set looks like: 1616.93,ac:22:0b:a6:22:c3 ...
0
votes
2answers
52 views

Merging large data set in fastest possible way

I have a large data set ~100 GBs having smaller csv files ~ 100,000. I wish to merge all .csv files together in the fastest possible way. Files are located in two different folders. Someone ...
1
vote
1answer
36 views

bash ignorecase match pattern in file and color that line, print all to screen

I have almost zero idea about coding or bash scripting, so please bare with me and excuse my horrible formatting. What I want is to cat a log file and highlight certain lines with color based on a ...
11
votes
9answers
2k views

How can I delete the 5th word of every line in a file?

I want to delete the 5th word of each line in a file. The current content of the file: File is not updated or and will be removed System will shut down f within 10 seconds Please save your work ...
3
votes
1answer
37 views

Store specific output to a shell variable

I am looking to store the output of the svn info command, more precisely the revision number into a variable for further processing. I just need the 4 stored. Can anyone help me please? $ svn info ...
0
votes
3answers
44 views

Shell append string after last match

I would like to insert a string after the last match, example: String to insert after last instance of banana: Orange Input Apple Banana Apple Banana Output Apple Banana Apple Banana Orange I ...