2

I have a text file that I generated of all files in a directory. I'd like to use this file as input into a script that I have, but I need the text file to be formatted in a particular way to be parsed correctly.

Currently the text file, which is a list of file names, is formatted like so:

A1_R1.fastq.gz
A1_R2.fastq.gz
A2_R1.fastq.gz
A2_R2.fastq.gz
A3_R1.fastq.gz
A3_R2.fastq.gz

I need the paired reads (files with the same name, but different RN values) for each sample to be on the same line, separated by a tab:

A1_R1.fastq.gz A1_R2.fastq.gz
A2_R1.fastq.gz A2_R2.fastq.gz
A3_R1.fastq.gz A3_R2.fastq.gz

Since I have >1000 entries, I am hoping for a method using awk or something similar to modify the file, but I don't have much experience with awk.

5
  • 2
    Hello, do you mean you have a R1 and R2 file for every prefix (A1, A2, etc.)? Thx. Feb 26 at 1:47
  • 4
    Do you need to explicitly match the prefixes, or can you simply take pairs of lines and place them side by side? If the latter, something like paste - - <yourfile would suffice. Feb 26 at 1:49
  • 1
    Yes, @jubilatious1 sorry just edited the OG post, I have a R1 and R2 file for every prefix I need to match prefixes and put the two reads side by side Feb 26 at 1:52
  • 5
    Does it matter whether A1_R1 comes before or after A1_R2 on the output line for that pair?
    – Ed Morton
    Feb 26 at 10:39
  • 1
    Does each of your pairs of associated _R1 and _R2 files come together in the listing, one immediately following the other? Feb 28 at 15:45

5 Answers 5

12

Assuming that the lines are ordered in the manner that you show in the question, then the paste command can do that:

$ paste - - < input_file
A1_R1.fastq.gz  A1_R2.fastq.gz
A2_R1.fastq.gz  A2_R2.fastq.gz
A3_R1.fastq.gz  A3_R2.fastq.gz

If you want the columns to be separated by anything other than the default TAB then use the -d option of paste. For a simple space:

$ paste -d ' ' - -  <input_file
1
  • See also paste -sd '\t\n' input_file which avoids a trailing tab if the input has an odd number of lines (though note it produces one empty line on empty input). Feb 26 at 13:54
9

For the input you show where all the paired lines are next to each other all you need with any awk is:

$ awk '{ORS=(NR%2 ? "\t" : RS)} 1' file
A1_R1.fastq.gz  A1_R2.fastq.gz
A2_R1.fastq.gz  A2_R2.fastq.gz
A3_R1.fastq.gz  A3_R2.fastq.gz

or if they aren't paired already:

$ shuf file > file1
$ cat file1
A3_R2.fastq.gz
A2_R2.fastq.gz
A1_R1.fastq.gz
A3_R1.fastq.gz
A1_R2.fastq.gz
A2_R1.fastq.gz

and so need to be paired then if you don't mind adding a call to sort:

$ awk '{ORS=(NR%2 ? "\t" : RS)} 1' <(sort file1)
A1_R1.fastq.gz  A1_R2.fastq.gz
A2_R1.fastq.gz  A2_R2.fastq.gz
A3_R1.fastq.gz  A3_R2.fastq.gz

or to pair them within awk:

$ awk -F'_' -v OFS='\t' '$1 in a{print a[$1], $0; next} {a[$1]=$0}' file1
A3_R2.fastq.gz  A3_R1.fastq.gz
A1_R1.fastq.gz  A1_R2.fastq.gz
A2_R2.fastq.gz  A2_R1.fastq.gz

Note in that last script that the R2 field is output before it's R1 partner in some cases. If that's an issue then you can order them when printing:

$ awk -F'_' -v OFS='\t' '
    $1 in a { print (a[$1] < $0 ? a[$1] OFS $0 : $0 OFS a[$1]); next }
    { a[$1] = $0 }
' file1
A3_R1.fastq.gz  A3_R2.fastq.gz
A1_R1.fastq.gz  A1_R2.fastq.gz
A2_R1.fastq.gz  A2_R2.fastq.gz

If your input file is actually many millions of lines long then adding delete a[$1]; before the next would speed up execution time in most cases, probably not worthwhile if it's just a few thousand (trading off the overhead of calling delete a[$1] for every pair vs the overhead of having a large hash table a[]).

If you need the output lines also sorted then you'd need to use GNU awk for PROCINFO["sorted_in"]:

$ awk '{a[$0]} END{PROCINFO["sorted_in"]="@ind_str_asc"; for (i in a) printf "%s%s", i, (++n % 2 ? "\t" : RS) }' file1
A1_R1.fastq.gz  A1_R2.fastq.gz
A2_R1.fastq.gz  A2_R2.fastq.gz
A3_R1.fastq.gz  A3_R2.fastq.gz

but, just like the solution that uses sort, that wouldn't produce the presumably expected order when the numbers in the input can be multiple digits because, for example, A11 would sort alphabetically before A2 - you'd need to split each string up into separate alphabetic and numeric parts and sort each separately or normalize them to always have the same number oc alphabetic and numeric characters in each position, e.g. map A1_R1 into 000A0001_000R0001 or similar before sorting.

4

Using Raku (formerly known as Perl_6)


If it's always paired reads, and they're always in the correct order:

~$ raku -e '.put for lines.rotor(2);'  file 

If it's always paired reads, but filenames are out-of-order:

~$ raku -e '.put for lines.sort.rotor(2);'  file 

Raku is a programming language in the Perl family. Similar to the excellent awk answer by @EdMorton, if you have to sort the filenames the resultant order will be alphabetic.

https://docs.raku.org/routine/lines
https://docs.raku.org/routine/rotor
https://raku.org


If sometimes files are missing, regardless of filename order:

~$ raku -ne 'BEGIN my %hash;     \  
             %hash.append: .match(/^ (<-[_]>+) _ /).[0]  =>  $_;   \
             END for %hash.sort { .values.put };'   file

#OR

~$ raku -ne 'BEGIN my %hash;     \ 
             %hash.append: m/^ (<-[_]>+) _ /.[0]  =>  $_;    \
             END for %hash.sort { .values.put };'   file 

This second approach is if the input data is less-than-pristine (missing filenames, etc.). Like Perl itself, Raku has an awk-like command-line mode invoked with the -ne ("non-autoprinting linewise") flags. Above, we BEGIN by declaring a %hash. In the body of this "one-liner" each line is .matched on (here .match is short for $_.match meaning to call the function on $_, the topic variable which holds the line text).

The .match (or m/ … /) routine/operator looks for ^ beginning-of-line followed by <-[_]>+ one-or-more of any-character-but _ underscore (i.e. a custom negative character class), followed by _ underscore itself. (FYI, a custom positive character class looks like so: <+[ … ]>). Parentheses capture the leading non-underscore text into $0 or more simply .[0].

Each line is thus deciphered into .[0] as key with $_ (the whole line) as value. The => is used to create a key/value pair. Hash data structures maintain unique keys, so every time the same key is encountered, the new value simply gets appended. At the END of reading lines, we sort the keys, and output the .values per key.

https://course.raku.org/essentials/associatives/hashes/
https://docs.raku.org/language/hashmap
https://raku.org


Sample Input:

A1_R1.fastq.gz
A1_R2.fastq.gz
A2_R1.fastq.gz
A2_R2.fastq.gz
A3_R1.fastq.gz
A3_R2.fastq.gz

Sample Output (either approach):

A1_R1.fastq.gz A1_R2.fastq.gz
A2_R1.fastq.gz A2_R2.fastq.gz
A3_R1.fastq.gz A3_R2.fastq.gz
0
2

I would probably start out with the simplest approach:

$ fmt -w30 < files.csv | tr ' ' '\t'
A1_R1.fastq.gz  A1_R2.fastq.gz
A2_R1.fastq.gz  A2_R2.fastq.gz
A3_R1.fastq.gz  A3_R2.fastq.gz

But if your heart is set on using awk, that's straightforward:

$ sort  < files.csv |
  awk 'NR % 2 == 0 {print prev"\t"$0}   {prev = $0}'
A1_R1.fastq.gz  A1_R2.fastq.gz
A2_R1.fastq.gz  A2_R2.fastq.gz
A3_R1.fastq.gz  A3_R2.fastq.gz

We're looking at low bit of NumberofRecord to trigger on every other line. You could always substitute a pattern match if you feel that's clearer.

6
  • 2
    What does fmt -w29 do?
    – terdon
    Feb 26 at 13:44
  • 1
    OP's filenames had length 14. What you see is copy-n-paste output from a run of BSD / MacOS fmt -- OP did not specify an execution platform. I refer you to the fmt man page: "Lines are joined or split only at white space". When porting this solution to fmt (GNU coreutils) 8.32, I see they apparently introduced an OBOB or made an undocumented change in the meaning of "maximum line width", as we find that at least fmt -w30 is needed. (I will update my Answer to use 30.)
    – J_H
    Feb 26 at 16:38
  • 1
    Not sure I understand. Let's say A1, A2, etc. are primer-set names, and R1/R2 are the paired reads. The sequencing software outputs files with these two strings joined via _ underscore. But what happens if some primer-set names are A1, A2, etc., while others are GTAAAACGACGGCCAG (i.e. the string length varies from primer-set to primer-set)? Feb 26 at 21:53
  • 1
    I'm just trying to understand your fmt approach, because it seems powerful. However AIUI to use fmt an end-user has to make the assumption that input linewise character lengths ('filenames', in this example), are non-varying. The fmt approach could work just fine for some identifiers (e.g. Ensembl_IDs (such as ENSRNOG00000000007, ENSRNOG00000071217, etc.), which seemingly maintain consistent length. Thank you. Feb 26 at 22:16
  • 2
    @jubilatious1 Depending on the naming scheme of the files, they definitely do not need to be of uniform length (it is common to use species names as a part of the filename, and the length of these varies greatly). I would advise against using this approach (with fmt) as a generic solution.
    – Kusalananda
    Feb 27 at 9:51
0

I'm a fan of sed, and it can join adjacent pairs of lines pretty easily:

sed 'N; s/\n/'$'\t''/' < input > output

The $'\t' in the middle of that is a bash-ism conveying a single tab character, not directly related to sed. In practice, I would probably use a literal tab character instead, but that does not come across clearly in this medium: sed 'N; s/\n/ /' < input > output

Explanation:

  • sed starts by reading the first line
  • the N instructs sed to read the next line of input and append it to the current one, separated by a newline.
  • the s command replaces the newline with a tab.
  • there being no more commands, the result of the previous steps is written, then sed reads the next line and cycles back
  • sed terminates when it exhausts its input.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .