1

I have a file which is an HTML document, containing a <table> I want to extract data from and output into a csv.

This file has 544609657 characters, is about 545 megabytes, all in a single line.

I managed to extract the data into a csv by using sed and making many string replacements, but I wanted to speed things up by using GNU parallel. Is this possible, considering it's a single line file?

My attempts below have not increased processing speed nor improved memory usage:

parallel -a table.html --pipepart 'sed -e [...etc.]' > table.csv

Or

cat table.html | parallel --pipe 'sed -e [...etc.]' > table.csv

I'm guessing the problem is because the file has a single line. If so, what strategies could I used to process the file more efficiently?

3
  • 545 megabytes for an HTML file is highly unusual. Are you sure it isn't actually XML ? Have you tried an XML parser of some sort ? You may already have one available on your system.
    – Kate
    Commented Mar 31, 2021 at 19:43
  • Unfortunately, it's really an HTML. It's bad practice but it's how the data is. Not sure if an XML parser would help speed up the processing of the file... I guess my question might be badly formulated but what I really wanted to know is how to parallel process a file like this one (when it's in a single line) Commented Mar 31, 2021 at 20:32
  • I can't see how that would be the solution. On multiple files perhaps. Even if you could do parallel processing disk access should be the bottleneck. Parsing such a big file is never going to be quick, that's why I would look at a native parser, rather than tools like sed. I think an XML parser could work as long as the file is properly structured. If you have xmllint you could try something.
    – Kate
    Commented Mar 31, 2021 at 20:40

1 Answer 1

1

You have exactly the correct thoughts.

You just need to learn --recstart:

parallel --pipepart --recstart '<tr>' -a big --block -10 'sed ...' > table.csv

Here we assume each row of your HTML table starts with <tr>.

3
  • This is great, thank you! --recstart allows specifying the start of records, which is defaulted to newlines. Using <tr> makes parallel treat every table row as a different record, allowing it to actually process the file in parallel. Commented Apr 7, 2021 at 18:13
  • @BernardoLoureiro Almost. --recend defaults to \n. But when --recstart is set --recend is set to "".
    – Ole Tange
    Commented Apr 7, 2021 at 21:33
  • Makes sense, thanks for your clarification, Ole! Commented Apr 8, 2021 at 22:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .