Questions tagged [bioinformatics]
Use this tag for questions relating to common bioinformatics tasks performed on a *nix system. Things like manipulating/converting between standard biological text formats, extracting data of interest from such formats etc.
315
questions
46
votes
3
answers
184k
views
How to merge two files based on the matching of two columns?
I have file1 likes:
0 AFFX-SNP-000541 NA
0 AFFX-SNP-002255 NA
1 rs12103 0.6401
1 rs12103_1247494 0.696
1 rs12142199 0.7672
And a file2:
0 AFFX-SNP-000541 1
0 AFFX-...
20
votes
6
answers
8k
views
Running a command on many files
I've got a folder with many files (xyz1, xyz2, all the way up to xyz5025) and I need to run a script on every one of them, getting xyz1.faa, xyz2.faa, and so on as outputs.
The command for a single ...
20
votes
1
answer
51k
views
Best Linux Distribution for Scientific Computing? [closed]
I recently bought a new laptop to be used for scientific research when I'm at home. The machine has Intel i7 processor with 8 cores, 4 GB of RAM, and a graphic card from Nvidia (2 GB, Ivy Bridge). The ...
9
votes
8
answers
5k
views
How to count the number of characters in a line, except a specific character?
This is part file
N W N N N N N N N N N
N C N N N N N N N N N
N A N N N N N N N N N
N N N N N N N N N N N
N G N N N N N N N N N
N C N N N C N N N N N
N C C N N N N N N N N
In each line I want to ...
8
votes
5
answers
3k
views
Removing rows containing NA in every column
I have a tab delimited file which looks like this:
gene v1 v2 v3 v4
g1 NA NA NA NA
g2 NA NA 2 3
g3 NA NA NA NA
g4 1 2 3 2
The number of fields in every line is fixed and ...
8
votes
5
answers
1k
views
bash loop to replace middle of string after a certain character
I have 120 files (genomes.faa) that all have headers between each sequence
>GENOME1_00001 HYPOTHETICAL PROTEIN A
NQFTIAQSQVGLEDALLDL
>GENOME1_00002 HYPOTHETICAL PROTEIN B
NQFTIAQSQVGLEDALLDL
&...
8
votes
3
answers
14k
views
extract lines that match a list of words in another file
I have file 1 which have those lines:
ATM 1434.972183
BMPR2 10762.78192
BMPR2 10762.78192
BMPR2 1469.14535
BMPR2 1469.14535
BMPR2 1738.479639
BMS1 4907.841667
BMS1 4907.841667
BMS1 880.4532628
BMS1 ...
7
votes
2
answers
530
views
Find any line in VI that has something other than ATCG
I have a file of genomic data that is approximately 5 million lines long and should have only the characters A, T, C, and G in it. The problem is, I know how large the file should be, but it's ...
7
votes
4
answers
6k
views
How to merge two files with different number of rows in shell?
I have file1 like this:
CHR SNP TEST A1 A2 GENO O(HET) E(HET) P
0 AFFX-SNP-000541 ALL 0 0 0/0/0 ...
6
votes
5
answers
268
views
Remove all sub-fields in column-organized datafile that contain "_XX"
I have this type of file
#CHROM POS ID REF ALT QUAL FILTER INFO
chr1 69511 rs2691305 A G . PASS AC=70482;AN=83312;AF=0.846001;popmax=eas;...
6
votes
4
answers
27k
views
Remove string from a particular field using awk/sed
I have a file (>80,000 lines) that looks likes this:
chr1 GTF2GFF chromosome 1 249213345 . . . ID=chr1;Name=chr1
chr1 GTF2GFF gene 11874 14408 . + . ID=DDX11L1;Note=...
6
votes
9
answers
1k
views
Extract subsequence corresponding to n:th pattern from a file
I have below data blocks (multiple)
chr1.trna4 (17188416-17188486) Length: 71 bp
Type: Gly Anticodon: CCC at 33-35 (17188448-17188450) Score: 78.3
HMM Sc=56.60 Sec struct Sc=21.70
* ...
6
votes
5
answers
459
views
Extract the start and end position of a common identifier
I have a file that looks like this:
Id Chr Start End
Prom_1 chr1 3978952 3978953
Prom_1 chr1 3979165 3979166
Prom_1 chr1 3979192 3979193
Prom_2 chr1 4379047 ...
6
votes
3
answers
639
views
Unix fold command behaving strangely
So I have this fasta (biology) file that looks like this:
>m64093_191209_130050/133911/ccs_64
TTCAGGCTGTGTTCCATTTGATTTAAAATCAAATAATTTCATTCGCGTCAGAACACCTGGTTTCACGACC
...
6
votes
1
answer
496
views
awk split a file into multiple files with specific names in another index file
I have a cluster fasta file (called file) which looks like:
>1AB2
>1AB2 AA
NWWIEUNJRNIBGOWNGIOWGRBIGBRGRIOWGI
NCIDHFR8EHGBVPIWOBGIGRI
>1AB3 AA
WNIOREHUEBRGOUERGHBERGIORBGREUGEGO
...
6
votes
2
answers
880
views
Extract number of length n from field and return string
I have a tab delimited file with a combination of text and numbers. I want to keep each line as is, but I want to keep only the six digit numbers in the 5th column if present. For example:
gene1 ...
5
votes
5
answers
2k
views
AWK to replace character for lines not starting with ">"
I'm working with sequence data and I stupidly cannot find the correct way to replace "." by "X" in lines not starting with ">" using awk. I really need to use awk and not sed.
I got this far, but ...
5
votes
5
answers
830
views
How to convert all text into 1 except "0"s and first two fields in a csv file?
I have several large.csv files that I'd like to convert to binary (1 and 0) format. In which all cells containing text would become 1 and 0 would remain 0, except first two fields.
head Test.csv
Iss1,...
5
votes
6
answers
539
views
Count the longest stretch of consecutive patterns
I have a sequence file:
$ cat file
CACCGTTGCCAAACAATG
TTAGAAGCCTGTCAGCCT
CATTGCTCTCAGACCCAC
GATGTACGTCACATTAGA
ACACGGAATCTGCTTTTT
CAGAATTCCCAAAGATGG
I want to calculate the longest stretch of C+T. I ...
5
votes
2
answers
10k
views
Replace pattern between two characters
Staphylococcus_sp_HMSC14C01-KV792037.1:0.00371647154267842634,Staphylococcus_hominis_VCU122-AHLD01000058.1:0.00124439639436691308)69:0.00227646100249620856,(Staphylococcus_sp_HMSC072E01-KV814990.1:0....
5
votes
6
answers
218
views
subset columns from the 1st file using column names in 2nd file
I have two text files:
1st file is a Tab delimited file which looks like this:
chrom pos ref alt a1 a2 a3 a4
10 12345 C T aa bb cc dd
10 12345 C T aa bb cc dd
10 12345 C ...
5
votes
2
answers
7k
views
counting multiple patterns in a single pass with grep?
I've written a grep loop to iteratively count DNA trinucleotides within a gzipped DNA fasta file containing DNA sequences e.g.
declare -a tri=(AAA AAC AAG AAT CAA .. etc)
for i in ${tri[@]}
do
...
4
votes
2
answers
857
views
Merge fields in a file
I have a file with 7 columns, a GFF file having chromosomal regions.I want to collapse the rows where REGION ="exon" to only one row in the file.The row has to be collapsed on the basis of regions ...
4
votes
2
answers
15k
views
Extract lines containing a pattern
I want to extract all the lines in a file containting these patterns: "#1:" and "tree length for".
Input:
#1: nexus0002_Pseudomonas_10M
branch t N S dN/dS dN ...
4
votes
1
answer
160
views
Splitting file content into other files based on parsed parameter value in file [duplicate]
A regular file contains data organized in columns, separated by space. Based on the value of the second field, I want to split that initial data file into several other files.
The data file has 54,...
4
votes
2
answers
9k
views
Nested 'awk' in a 'while' loop, parse two files line by line and compare column values
I need some help with a combination of awk & while loop.
I have two simple files with columns (normal ones are very large), one representing simple intervals for an ID=10(of coding regions(exons),...
4
votes
3
answers
9k
views
merging two files according to a common column
I have 2 files. File1 is:
chr19 4124051 4124250 1
chrX 154458151 154458200 2
chr22 37019451 37019600 3
chr15 74995401 74995550 4
chr12 128823901 128824100 5
and File2 ...
3
votes
3
answers
498
views
Identify strings between patterns and print entire region between pattern if string is found. Perferably using awk
There are similar questions but none of them quite address the issue I have.
Simply put, I need to print every block that contains any string I'm looking for. Each block start line contains: <entry ...
3
votes
3
answers
15k
views
Extracting subset from fasta file
I have a fasta file which looks like this:
>chr1
ACGGTGTAGTCG
>chr2
ACGTGTATAGCT
>chrUn
ACGTGGATATTT
>chr21
ACGTTGATGAAA
>chrX
GTACGGGGGTGG
>chrUn5
TGATAGCTGTTG
I just want to ...
3
votes
2
answers
652
views
How to display the difference between two DNA Sequences via command line tools
I have the following problem:
My data sheet looks like this
AAAGGGTTT AAAGGG
AAAGGGCCC GGGCCC
and I want to display the difference between the two sequences in the third row like this
AAAGGGTTT ...
3
votes
2
answers
736
views
Counting a specific consecutive character with its occurrence position and length
I have a sequence file and want to count consecutive character "N" with its position of occurrence and the length
Say I have a file named mySequence.fasta like this:
>sequence-1
...
3
votes
4
answers
785
views
First two fields to be separated by _ and rest of the fields as they are
#CHROM POS REF ALT ../S101_sorted.bam ../S102_sorted.bam ../S105_sorted.bam ../S107_sorted.bam ../S113_sorted.bam ../S114_sorted.bam ../S115_sorted.bam ...
3
votes
4
answers
4k
views
Wrapping a loop around a 'sed'-command processing many files in a single directory
I have text-files containing many lines, of which some starts with ">" (it's a so-called *.fasta file, and the ">"s marks the beginning of a new information container):
>header_name1
sequence_info
...
3
votes
2
answers
153
views
Series of numbers at a certain position across multiple lines
I've got a dataset that looks like this:
>TRINITY_DN37_c0_g1_i1 len=333 path=[361:0-43 362:44-332] [-1, 361, 362, -2]
GCCGCCATCATGGATGCCAGCCGTGTGCAGCCCATCAAGCTTGCCAGAGTCACCAAGGTT
>...
3
votes
4
answers
5k
views
Delete everything after second underscore
I want to delete all the text after the second underscore (including the underscore itself), but not on every line. Every of the target lines begin with a pattern (>gi_).
EXAMPLE.
Input
>...
3
votes
3
answers
240
views
Repairing the headers of phylip bioinformatics files to accurately reflect the updated number of samples in the file(s)
I have a dataset that I am working with made up of phylip files that I have been editing. Phylip format is a bioinformatics format that contains as a header the number of samples and the sequence ...
3
votes
2
answers
121
views
Organizing three dimensional data with awk/sed/perl
I have this file (a sparse matrix):
PC.354 OTU1 6
PC.354 OTU2 1
PC.356 OTU0 4
PC.356 OTU2 7
PC.356 OTU3 3
I want an output like this (dense matrix -classic .biom table):
OTU_ID PC.354 PC.355 PC....
3
votes
2
answers
2k
views
Understanding an awk formula that unwraps fasta files
I have just found a formula which can be used to unwrap fasta files. Before I give the formula, I need to explain what unwrapping a fasta file is.
In short, the fasta format is like this:
>...
3
votes
1
answer
356
views
How to create a new file based on results from multiple files and keep the filenames as first column?
I wanted to extract specific lines from multiple bam (a binary file format) files. I can select the lines from a single bam file using this command:
samtools view -c TCGA-BH-A0BW-11A.sorted.bam "...
3
votes
3
answers
2k
views
How can I replace strings in a given column of a table using awk and regex?
I am studying bioinformatics, but I don't have long experience with awk and now I'm stuck.
I have a table with 13 columns.
In column 9 I have a lot of variations of strings like ELL1-XXXXXXXXX (e.g. ...
3
votes
1
answer
54
views
Update the values of a file using information from another file
File 1:
P SNP:
0.266234 1:11008
0.266234 1:11012
0.340042 1:13110
0.86724 rs201725126
0.86724 rs200579949
0.127269 1:1327
File 2:
snp chr:pos
rs201725126 1:...
3
votes
1
answer
120
views
How I run these commands at the same time of different files
You please imagine I have 16 separate folders named A1 to A16 inside each of them I have a file named aligned.sorted.bam
By these commands I want to convert aligned.sorted.bam to aligned.sam
module ...
3
votes
1
answer
292
views
merging files in a one dataframe
I have almost 160 txt files, each file has 7 columns and now i want to put that in one file. Each files 1st and 6th column data (Geneid, Length) is same so i just want that to be the 1st and 2nd ...
3
votes
2
answers
122
views
keep only those columns in the first file which their column numbers match to numbers in the second file
I have a genotype.file with 317 columns (1, 2, 3, ..., 317).
In the example below I am just showing the first few columns!
Input genotype.file:
Chr00c0002 56240 N N N A N A N N N ...
3
votes
5
answers
568
views
Combine file data, sort on chromosome, position next to each other - column vise
I have multiple files in format: chr position value.
I want to combine them in format "chr", "position", "samp1", "samp2", "samp3", "samp4",........
For example:
Samp1:
chr position value
1 ...
3
votes
3
answers
774
views
Comparing two files linewise and if pattern of file 1 is not found (fully or partially) in file 2 then print line of file 1
I have two files:
file1 (search):
1
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGTTTTTTTTT
2
GACGGAGGATGCAAGTGTTATCCGGAAT
3
GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGCGTCC
4
...
3
votes
2
answers
2k
views
How to sort data by the numbers in third column? [duplicate]
If I have a file consisting of data that looks as follows, how would I sort the data based on the numbers in the 3rd column ? The space between the first two columns are NOT tab delimited but some ...
3
votes
3
answers
212
views
regex from two files
I have two files that built of blocks of 4 lines, starting with a @. They are matching and in order. The files are too big (>10 GB) so I am putting here the first 3 blocks of each (12 lines).
1st: ...
2
votes
5
answers
892
views
Replace new lines with spaces using awk
I have a text file that I generated of all files in a directory. I'd like to use this file as input into a script that I have, but I need the text file to be formatted in a particular way to be parsed ...
2
votes
5
answers
420
views
How to split a given column's string values in a text file
I have a text file on a Linux machine with two columns:
Column 1 = id_no (most are 5, with some 6 digits long);
Column 2 = genetic_markers (all are 50674 digits long);
12345 0102010205
54322 ...