I have a very long series of urls with no separating character, in the same format as below:


I want each URL to be on a new line. I tried to do this by replacing all instances of "http://" with "\nhttp://" using sed

sed 's_http://_\nhttp://_g' urls.txt

but a segmentation fault occurs (memory violation). I can only surmise that the sheer size of the file (it's over 100GB) is causing sed to exceed some limit.

I could split the file into several smaller files for processing, but all instances of "http://" would need to be kept intact.

Is there a better way to do this?

  • I think sed doesn't like the 100GB without line endings as it tries to read a single line in its buffer. – jippie Apr 15 '12 at 8:52
  • splitting (irrespective of "where" the cut happens), processing, then reassembling should give however the correct result. – enzotib Apr 15 '12 at 9:09
  • 3
    If you truly have a 100GB text file containing a single long line, then you are better off writing a quick C program to do the work. – fpmurphy Apr 15 '12 at 16:28

With awk you can avoid reading huge amount of text at once:

awk -vRS='http://' -vORS='\nhttp://' 1 urls.txt > urlsperline.txt

The success may depend on the used awk implementation. For example gawk works fine, but mawk crashes.


This will do the job:

perl -pe 'BEGIN { $/ = "//" } s!(?=http://\z)!\n!' urls.txt

By setting $/, I've changed the definition of a line so it ends with // instead of a newline. This makes Perl read one URL at a time. It's unlikely that a URL contains // except after the scheme, but it's OK if one does, the regex will keep it from adding spurious newlines.

If you want to avoid adding a blank line before the first URL:

perl -pe 'BEGIN { $/ = "//"; print scalar <> } s!(?=http://\z)!\n!' urls.txt

You might try benchmarking to see whether s!http://\z!\nhttp://! is faster. They're equivalent. Note that the /g flag is not necessary on the substitution, because there can only be one match per "line".

  • Is the perl regexp engine okay with multi-gigabyte-long lines? – Alexios Apr 15 '12 at 11:24
  • 2
    @Alexios, probably not, but it doesn't need to be. Since I changed $/, it'll only be dealing with one URL at a time. – cjm Apr 15 '12 at 16:16
  • Ah, I see what you did there. It's been a while since the 90s, and I had to man perlvar, but it makes sense that way. – Alexios Apr 15 '12 at 17:26
  • Linux allows urls to have embedded multiple slashes in paths, so this code may fail if you have any of those. Testing for the whole string, http and all, won't have this problem. – Joe Apr 20 '12 at 22:24
  • @Joe, I am testing for the http: part in the regex. It will examine every //, but it won't add a newline unless it finds http://. – cjm Apr 20 '12 at 23:02
  1. Change all occurrences of a : with a newline, to chop up the file.
  2. Replace
    • http at the end of the line with
    • a newline followed by http: and append the next line to it
  3. Repeat once, so even and odd lines are updated

These steps look like:

tr ':' '\n' | sed -e '/http$/{N;s/http\n/\nhttp:/}' | sed -e '/http$/{N;s/http\n/\nhttp:/}'
  1. Check if there are lines that do not start with http://, print the line numbers. This would only occur if a : is somewhere in the URL other than after the http.

    grep -nv '^http://'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.