1

I have a file which is an HTML document, containing a <table> I want to extract data from and output into a csv.

This file has 544609657 characters, is about 545 megabytes, all in a single line.

I managed to extract the data into a csv by using sed and making many string replacements, but I wanted to speed things up by using GNU parallel. Is this possible, considering it's a single line file?

My attempts below have not increased processing speed nor improved memory usage:

parallel -a table.html --pipepart 'sed -e [...etc.]' > table.csv

Or

cat table.html | parallel --pipe 'sed -e [...etc.]' > table.csv

I'm guessing the problem is because the file has a single line. If so, what strategies could I used to process the file more efficiently?

Improve this question
3
  • 545 megabytes for an HTML file is highly unusual. Are you sure it isn't actually XML ? Have you tried an XML parser of some sort ? You may already have one available on your system.
    – Kate
    Mar 31, 2021 at 19:43
  • Unfortunately, it's really an HTML. It's bad practice but it's how the data is. Not sure if an XML parser would help speed up the processing of the file... I guess my question might be badly formulated but what I really wanted to know is how to parallel process a file like this one (when it's in a single line)
    – Bernardo Loureiro
    Mar 31, 2021 at 20:32
  • I can't see how that would be the solution. On multiple files perhaps. Even if you could do parallel processing disk access should be the bottleneck. Parsing such a big file is never going to be quick, that's why I would look at a native parser, rather than tools like sed. I think an XML parser could work as long as the file is properly structured. If you have xmllint you could try something.
    – Kate
    Mar 31, 2021 at 20:40

1 Answer 1

Reset to default
1

You have exactly the correct thoughts.

You just need to learn --recstart:

parallel --pipepart --recstart '<tr>' -a big --block -10 'sed ...' > table.csv

Here we assume each row of your HTML table starts with <tr>.

Improve this answer
3
  • This is great, thank you! --recstart allows specifying the start of records, which is defaulted to newlines. Using <tr> makes parallel treat every table row as a different record, allowing it to actually process the file in parallel.
    – Bernardo Loureiro
    Apr 7, 2021 at 18:13
  • @BernardoLoureiro Almost. --recend defaults to \n. But when --recstart is set --recend is set to "".
    – Ole Tange
    Apr 7, 2021 at 21:33
  • Makes sense, thanks for your clarification, Ole!
    – Bernardo Loureiro
    Apr 8, 2021 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.