Questions tagged [bioinformatics]

Use this tag for questions relating to common bioinformatics tasks performed on a *nix system. Things like manipulating/converting between standard biological text formats, extracting data of interest from such formats etc.

Filter by
Sorted by
Tagged with
46 votes
3 answers
184k views

How to merge two files based on the matching of two columns?

I have file1 likes: 0 AFFX-SNP-000541 NA 0 AFFX-SNP-002255 NA 1 rs12103 0.6401 1 rs12103_1247494 0.696 1 rs12142199 0.7672 And a file2: 0 AFFX-SNP-000541 1 0 AFFX-...
Dadong Zhang's user avatar
20 votes
6 answers
8k views

Running a command on many files

I've got a folder with many files (xyz1, xyz2, all the way up to xyz5025) and I need to run a script on every one of them, getting xyz1.faa, xyz2.faa, and so on as outputs. The command for a single ...
Manuel's user avatar
  • 453
20 votes
1 answer
51k views

Best Linux Distribution for Scientific Computing? [closed]

I recently bought a new laptop to be used for scientific research when I'm at home. The machine has Intel i7 processor with 8 cores, 4 GB of RAM, and a graphic card from Nvidia (2 GB, Ivy Bridge). The ...
linux-user-to-be's user avatar
9 votes
8 answers
5k views

How to count the number of characters in a line, except a specific character?

This is part file N W N N N N N N N N N N C N N N N N N N N N N A N N N N N N N N N N N N N N N N N N N N N G N N N N N N N N N N C N N N C N N N N N N C C N N N N N N N N In each line I want to ...
Anna1364's user avatar
  • 1,016
8 votes
5 answers
3k views

Removing rows containing NA in every column

I have a tab delimited file which looks like this: gene v1 v2 v3 v4 g1 NA NA NA NA g2 NA NA 2 3 g3 NA NA NA NA g4 1 2 3 2 The number of fields in every line is fixed and ...
user3138373's user avatar
  • 2,559
8 votes
5 answers
1k views

bash loop to replace middle of string after a certain character

I have 120 files (genomes.faa) that all have headers between each sequence >GENOME1_00001 HYPOTHETICAL PROTEIN A NQFTIAQSQVGLEDALLDL >GENOME1_00002 HYPOTHETICAL PROTEIN B NQFTIAQSQVGLEDALLDL &...
Goodolgab's user avatar
8 votes
3 answers
14k views

extract lines that match a list of words in another file

I have file 1 which have those lines: ATM 1434.972183 BMPR2 10762.78192 BMPR2 10762.78192 BMPR2 1469.14535 BMPR2 1469.14535 BMPR2 1738.479639 BMS1 4907.841667 BMS1 4907.841667 BMS1 880.4532628 BMS1 ...
LamaMo's user avatar
  • 223
7 votes
2 answers
530 views

Find any line in VI that has something other than ATCG

I have a file of genomic data that is approximately 5 million lines long and should have only the characters A, T, C, and G in it. The problem is, I know how large the file should be, but it's ...
mah's user avatar
  • 429
7 votes
4 answers
6k views

How to merge two files with different number of rows in shell?

I have file1 like this: CHR SNP TEST A1 A2 GENO O(HET) E(HET) P 0 AFFX-SNP-000541 ALL 0 0 0/0/0 ...
Dadong Zhang's user avatar
6 votes
5 answers
268 views

Remove all sub-fields in column-organized datafile that contain "_XX"

I have this type of file #CHROM POS ID REF ALT QUAL FILTER INFO chr1 69511 rs2691305 A G . PASS AC=70482;AN=83312;AF=0.846001;popmax=eas;...
newuser2021's user avatar
6 votes
4 answers
27k views

Remove string from a particular field using awk/sed

I have a file (>80,000 lines) that looks likes this: chr1 GTF2GFF chromosome 1 249213345 . . . ID=chr1;Name=chr1 chr1 GTF2GFF gene 11874 14408 . + . ID=DDX11L1;Note=...
fridaymeetssunday's user avatar
6 votes
9 answers
1k views

Extract subsequence corresponding to n:th pattern from a file

I have below data blocks (multiple) chr1.trna4 (17188416-17188486) Length: 71 bp Type: Gly Anticodon: CCC at 33-35 (17188448-17188450) Score: 78.3 HMM Sc=56.60 Sec struct Sc=21.70 * ...
MO12's user avatar
  • 391
6 votes
5 answers
459 views

Extract the start and end position of a common identifier

I have a file that looks like this: Id Chr Start End Prom_1 chr1 3978952 3978953 Prom_1 chr1 3979165 3979166 Prom_1 chr1 3979192 3979193 Prom_2 chr1 4379047 ...
fusion.slope's user avatar
6 votes
3 answers
639 views

Unix fold command behaving strangely

So I have this fasta (biology) file that looks like this: >m64093_191209_130050/133911/ccs_64 TTCAGGCTGTGTTCCATTTGATTTAAAATCAAATAATTTCATTCGCGTCAGAACACCTGGTTTCACGACC ...
Ricardo Guerreiro's user avatar
6 votes
1 answer
496 views

awk split a file into multiple files with specific names in another index file

I have a cluster fasta file (called file) which looks like: >1AB2 >1AB2 AA NWWIEUNJRNIBGOWNGIOWGRBIGBRGRIOWGI NCIDHFR8EHGBVPIWOBGIGRI >1AB3 AA WNIOREHUEBRGOUERGHBERGIORBGREUGEGO ...
Jiao's user avatar
  • 375
6 votes
2 answers
880 views

Extract number of length n from field and return string

I have a tab delimited file with a combination of text and numbers. I want to keep each line as is, but I want to keep only the six digit numbers in the 5th column if present. For example: gene1 ...
DemosW's user avatar
  • 63
5 votes
5 answers
2k views

AWK to replace character for lines not starting with ">"

I'm working with sequence data and I stupidly cannot find the correct way to replace "." by "X" in lines not starting with ">" using awk. I really need to use awk and not sed. I got this far, but ...
TUnix's user avatar
  • 355
5 votes
5 answers
830 views

How to convert all text into 1 except "0"s and first two fields in a csv file?

I have several large.csv files that I'd like to convert to binary (1 and 0) format. In which all cells containing text would become 1 and 0 would remain 0, except first two fields. head Test.csv Iss1,...
Debajyoti Kabiraj's user avatar
5 votes
6 answers
539 views

Count the longest stretch of consecutive patterns

I have a sequence file: $ cat file CACCGTTGCCAAACAATG TTAGAAGCCTGTCAGCCT CATTGCTCTCAGACCCAC GATGTACGTCACATTAGA ACACGGAATCTGCTTTTT CAGAATTCCCAAAGATGG I want to calculate the longest stretch of C+T. I ...
CN_229133's user avatar
  • 115
5 votes
2 answers
10k views

Replace pattern between two characters

Staphylococcus_sp_HMSC14C01-KV792037.1:0.00371647154267842634,Staphylococcus_hominis_VCU122-AHLD01000058.1:0.00124439639436691308)69:0.00227646100249620856,(Staphylococcus_sp_HMSC072E01-KV814990.1:0....
Ahmed Magdy Moustafa's user avatar
5 votes
6 answers
218 views

subset columns from the 1st file using column names in 2nd file

I have two text files: 1st file is a Tab delimited file which looks like this: chrom pos ref alt a1 a2 a3 a4 10 12345 C T aa bb cc dd 10 12345 C T aa bb cc dd 10 12345 C ...
user3138373's user avatar
  • 2,559
5 votes
2 answers
7k views

counting multiple patterns in a single pass with grep?

I've written a grep loop to iteratively count DNA trinucleotides within a gzipped DNA fasta file containing DNA sequences e.g. declare -a tri=(AAA AAC AAG AAT CAA .. etc) for i in ${tri[@]} do ...
Stephen Henderson's user avatar
4 votes
2 answers
857 views

Merge fields in a file

I have a file with 7 columns, a GFF file having chromosomal regions.I want to collapse the rows where REGION ="exon" to only one row in the file.The row has to be collapsed on the basis of regions ...
Ron's user avatar
  • 1,057
4 votes
2 answers
15k views

Extract lines containing a pattern

I want to extract all the lines in a file containting these patterns: "#1:" and "tree length for". Input: #1: nexus0002_Pseudomonas_10M branch t N S dN/dS dN ...
Manuel's user avatar
  • 453
4 votes
1 answer
160 views

Splitting file content into other files based on parsed parameter value in file [duplicate]

A regular file contains data organized in columns, separated by space. Based on the value of the second field, I want to split that initial data file into several other files. The data file has 54,...
Amanda Botelho Alvarenga's user avatar
4 votes
2 answers
9k views

Nested 'awk' in a 'while' loop, parse two files line by line and compare column values

I need some help with a combination of awk & while loop. I have two simple files with columns (normal ones are very large), one representing simple intervals for an ID=10(of coding regions(exons),...
BlueBird's user avatar
4 votes
3 answers
9k views

merging two files according to a common column

I have 2 files. File1 is: chr19 4124051 4124250 1 chrX 154458151 154458200 2 chr22 37019451 37019600 3 chr15 74995401 74995550 4 chr12 128823901 128824100 5 and File2 ...
Chiara's user avatar
  • 43
3 votes
3 answers
498 views

Identify strings between patterns and print entire region between pattern if string is found. Perferably using awk

There are similar questions but none of them quite address the issue I have. Simply put, I need to print every block that contains any string I'm looking for. Each block start line contains: <entry ...
Giles's user avatar
  • 897
3 votes
3 answers
15k views

Extracting subset from fasta file

I have a fasta file which looks like this: >chr1 ACGGTGTAGTCG >chr2 ACGTGTATAGCT >chrUn ACGTGGATATTT >chr21 ACGTTGATGAAA >chrX GTACGGGGGTGG >chrUn5 TGATAGCTGTTG I just want to ...
user3138373's user avatar
  • 2,559
3 votes
2 answers
652 views

How to display the difference between two DNA Sequences via command line tools

I have the following problem: My data sheet looks like this AAAGGGTTT AAAGGG AAAGGGCCC GGGCCC and I want to display the difference between the two sequences in the third row like this AAAGGGTTT ...
Daniel R's user avatar
3 votes
2 answers
736 views

Counting a specific consecutive character with its occurrence position and length

I have a sequence file and want to count consecutive character "N" with its position of occurrence and the length Say I have a file named mySequence.fasta like this: >sequence-1 ...
Budding-bioinformatician's user avatar
3 votes
4 answers
785 views

First two fields to be separated by _ and rest of the fields as they are

#CHROM POS REF ALT ../S101_sorted.bam ../S102_sorted.bam ../S105_sorted.bam ../S107_sorted.bam ../S113_sorted.bam ../S114_sorted.bam ../S115_sorted.bam ...
gaurav agarwal's user avatar
3 votes
4 answers
4k views

Wrapping a loop around a 'sed'-command processing many files in a single directory

I have text-files containing many lines, of which some starts with ">" (it's a so-called *.fasta file, and the ">"s marks the beginning of a new information container): >header_name1 sequence_info ...
nouse's user avatar
  • 133
3 votes
2 answers
153 views

Series of numbers at a certain position across multiple lines

I've got a dataset that looks like this: >TRINITY_DN37_c0_g1_i1 len=333 path=[361:0-43 362:44-332] [-1, 361, 362, -2] GCCGCCATCATGGATGCCAGCCGTGTGCAGCCCATCAAGCTTGCCAGAGTCACCAAGGTT >...
Luca 's user avatar
  • 41
3 votes
4 answers
5k views

Delete everything after second underscore

I want to delete all the text after the second underscore (including the underscore itself), but not on every line. Every of the target lines begin with a pattern (>gi_). EXAMPLE. Input >...
Annick Raymond's user avatar
3 votes
3 answers
240 views

Repairing the headers of phylip bioinformatics files to accurately reflect the updated number of samples in the file(s)

I have a dataset that I am working with made up of phylip files that I have been editing. Phylip format is a bioinformatics format that contains as a header the number of samples and the sequence ...
erikusrex's user avatar
3 votes
2 answers
121 views

Organizing three dimensional data with awk/sed/perl

I have this file (a sparse matrix): PC.354 OTU1 6 PC.354 OTU2 1 PC.356 OTU0 4 PC.356 OTU2 7 PC.356 OTU3 3 I want an output like this (dense matrix -classic .biom table): OTU_ID PC.354 PC.355 PC....
Lika 's user avatar
  • 33
3 votes
2 answers
2k views

Understanding an awk formula that unwraps fasta files

I have just found a formula which can be used to unwrap fasta files. Before I give the formula, I need to explain what unwrapping a fasta file is. In short, the fasta format is like this: >...
Agathe's user avatar
  • 133
3 votes
1 answer
356 views

How to create a new file based on results from multiple files and keep the filenames as first column?

I wanted to extract specific lines from multiple bam (a binary file format) files. I can select the lines from a single bam file using this command: samtools view -c TCGA-BH-A0BW-11A.sorted.bam "...
beginner's user avatar
  • 277
3 votes
3 answers
2k views

How can I replace strings in a given column of a table using awk and regex?

I am studying bioinformatics, but I don't have long experience with awk and now I'm stuck. I have a table with 13 columns. In column 9 I have a lot of variations of strings like ELL1-XXXXXXXXX (e.g. ...
Orsi's user avatar
  • 33
3 votes
1 answer
54 views

Update the values of a file using information from another file

File 1: P SNP: 0.266234 1:11008 0.266234 1:11012 0.340042 1:13110 0.86724 rs201725126 0.86724 rs200579949 0.127269 1:1327 File 2: snp chr:pos rs201725126 1:...
VasoGene's user avatar
3 votes
1 answer
120 views

How I run these commands at the same time of different files

You please imagine I have 16 separate folders named A1 to A16 inside each of them I have a file named aligned.sorted.bam By these commands I want to convert aligned.sorted.bam to aligned.sam module ...
Zizogolu's user avatar
  • 171
3 votes
1 answer
292 views

merging files in a one dataframe

I have almost 160 txt files, each file has 7 columns and now i want to put that in one file. Each files 1st and 6th column data (Geneid, Length) is same so i just want that to be the 1st and 2nd ...
H.K's user avatar
  • 79
3 votes
2 answers
122 views

keep only those columns in the first file which their column numbers match to numbers in the second file

I have a genotype.file with 317 columns (1, 2, 3, ..., 317). In the example below I am just showing the first few columns! Input genotype.file: Chr00c0002 56240 N N N A N A N N N ...
Anna1364's user avatar
  • 1,016
3 votes
5 answers
568 views

Combine file data, sort on chromosome, position next to each other - column vise

I have multiple files in format: chr position value. I want to combine them in format "chr", "position", "samp1", "samp2", "samp3", "samp4",........ For example: Samp1: chr position value 1 ...
user381133's user avatar
3 votes
3 answers
774 views

Comparing two files linewise and if pattern of file 1 is not found (fully or partially) in file 2 then print line of file 1

I have two files: file1 (search): 1 GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGTTTTTTTTT 2 GACGGAGGATGCAAGTGTTATCCGGAAT 3 GACGGAGGATGCAAGTGTTATCCGGAATCACTGGGCGTAAAGCGTCC 4 ...
MSt's user avatar
  • 31
3 votes
2 answers
2k views

How to sort data by the numbers in third column? [duplicate]

If I have a file consisting of data that looks as follows, how would I sort the data based on the numbers in the 3rd column ? The space between the first two columns are NOT tab delimited but some ...
Jenna Maiz's user avatar
3 votes
3 answers
212 views

regex from two files

I have two files that built of blocks of 4 lines, starting with a @. They are matching and in order. The files are too big (>10 GB) so I am putting here the first 3 blocks of each (12 lines). 1st: ...
Hassan Massalha's user avatar
2 votes
5 answers
892 views

Replace new lines with spaces using awk

I have a text file that I generated of all files in a directory. I'd like to use this file as input into a script that I have, but I need the text file to be formatted in a particular way to be parsed ...
lovelyrubbish's user avatar
2 votes
5 answers
420 views

How to split a given column's string values in a text file

I have a text file on a Linux machine with two columns: Column 1 = id_no (most are 5, with some 6 digits long); Column 2 = genetic_markers (all are 50674 digits long); 12345 0102010205 54322 ...
Michiel Van Niekerk's user avatar

1
2 3 4 5
7