I have a text file that's over 100MB, each line of which has the same number of columns:

Column No.: 0 1 2 3 4 5 6
            d x c c s b c

I want to add , at given locations. For example, location = 2, 3, 5

The desired output would be:

Column No.:  0 1 2  3  4 5  6
             d x c, c, s b, c

The location file would be a text or csv file


The text file has to have delimiters.


Sample data

enter image description here

  • Note: the line breaks are where I want to put delimiters
  • Column No. is the byte offset from the beginning of the line
  • It's unclear if the input file "Column No." refers to bytes, (the 6th column c would be the 7th byte of the file), or spaced delimited columns of strings, (the 6th column c would the the 13th byte). – agc Oct 22 '16 at 2:25
  • @agc I attached a sample data – John Hass Oct 22 '16 at 2:27
  • The question doesn't specify whether every line in the 100MB input file has the same number of columns. – agc Oct 22 '16 at 2:29
  • @agc I confirm evert line has same number of columns. – John Hass Oct 22 '16 at 2:30
  • @agc Any other conditions that I need to explain? – John Hass Oct 22 '16 at 2:32

Use Sed.

Note that you won't be using 0 indexed positions, but starting from 1. So I've incremented the numbers you gave.

Also, you have to go back to front since once you change the first one, the column positions change. So use Sed like so:

sed 's/./&,/6;s/./&,/4;s/./&,/3'


$ echo dxccsbc
$ echo dxccsbc | sed 's/./&,/6;s/./&,/4;s/./&,/3'

  • The s command in Sed is for substitution.
  • The pattern . matches any single character.
  • The & in the replacement text means "the text that was matched," and the comma is a literal comma.
  • The numerical flag after the last / means to only perform the substitution on the "nth" match on the line.

If you want to be really fancy, use Bash brace expansion to create the Sed commands:

$ echo dxccsbc | sed '-es/./&,/'{6,4,3}

But that's just icing on the cake and probably confusing unless you understand both Sed and Bash quite well. :)

If you want to pull the list of positions from a separate file (as you actually show in your question), you can do this like so:

sed -f <(sort -rn positionsfile | sed -n 's:^[1-9][0-9]*$:s/./\&,/&:p') file

Note this is Bash-specific as it uses process substitution syntax that is not POSIX. Also note I've made this quite robust as anything but actual numbers in the positionsfile (that don't start with 0) will be discarded.

Test results:

$ cat file 
$ cat positionsfile 
not a number
$ sed -f <(sort -rn positionsfile | sed -n 's:^[1-9][0-9]*$:s/./\&,/&:p') file
  • Good answer. I can never remember the syntax for these complex substitutions. Could you elaborate on the brace expansion method? Also, do you know if the syntax is the same for perl? – voices Oct 22 '16 at 0:45
  • @tjt263, you can read up on Brace Expansion either on this site or in the Bash man page at LESS='+/^ *brace expansion' man bash I don't know about Perl but I doubt the syntax is the same. – Wildcard Oct 22 '16 at 1:41
  • I understand typical bash style expansion. Something about it in this context seems unusual. – voices Oct 22 '16 at 2:33
  • 2
    @tjt263, since the brace expansion results will each be separate arguments, it's necessary to put the -e flag for Sed at the beginning so that it doesn't interpret them as files named s/./&,/4 and s/./&,/3. But no need for a space between -e and the Sed command following it, so it works out nicely. I'll grant that it is indeed an unusual use of brace expansion. :) (Also you should understand that Sed never sees the single quotes surrounding any command passed to it; those are purely for the shell.) – Wildcard Oct 22 '16 at 2:52
  • Inelegant addition to the really fancy code above, which parses the location input file also: eval "echo dxccsbc | sed '-es/./&,/'"$(tac location | numprocess /+1/ | tr '\n' ',' | sed 's/,$/}/;s/.*/{&/'); or assuming there's a data file for input: eval "sed '-es/./&,/'"$(tac location | numprocess /+1/ | tr '\n' ',' | sed 's/,$/}/;s/.*/{&/') input.txt – agc Oct 22 '16 at 3:29

With perl:

#!/usr/bin/env perl

my @pos;

while (<>)
    { push @pos, 1 + int; }
    { last if eof; }

@pos = sort { $b cmp $a } @pos;

while (<>) {
    for my $k (@pos)
        { s/^.{$k}\K/,/; }

Run it like this:

script.pl positions.txt file.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.