Reputation
6,320
Top tag
Next privilege 10,000 Rep.
Access moderator tools
Badges
17 65 141
Impact
~699k people reached

19h
awarded  Notable Question
Jul
24
revised Efficient data extraction from multiple files to a single CSV file
added 41 characters in body
Jul
24
comment Efficient data extraction from multiple files to a single CSV file
This is great. Thanks!! -- The explanations are very helpful. I measured the latest solutions and they don't have the "argument list too long" errors anymore. They run on ~20 secs against 10K XML files, making them on par with @mikeserv's slowest solution, but slower than mikeserv's faster solution. This is definitely a GREAT and very clear solution to the problem!
Jul
24
revised Efficient data extraction from multiple files to a single CSV file
added 162 characters in body
Jul
24
comment Efficient data extraction from multiple files to a single CSV file
The second solution still runs in ~ 15s
Jul
24
comment Efficient data extraction from multiple files to a single CSV file
Thanks @mikeserv I prepended Sol (1) with export LC_ALL=C; and the timing for it was still ~ 20.28 seconds (i.e. ~unchanged).
Jul
24
comment Efficient data extraction from multiple files to a single CSV file
Thanks @mikeserv! Both solutions now work great. This a huge speed up from the original solution in the OP (and 100X faster than a Python solution that properly loads xml's one by one to build a CSV dataframe in Pandas). Current times in my machine: Sol (1): 19 seconds. Sol (2): 14 seconds, both transforming ~10K XML files into a single CSV file.
Jul
24
awarded  Notable Question
Jul
23
comment Efficient data extraction from multiple files to a single CSV file
Awesome. By the way, I saw your posterior edits to the answer, which is great. Would you mind elaborating perhaps a bit on how the second solution works? It may help others adapt this answer to future multiple-file to-single-CSV transformation questions :).
Jul
23
revised Efficient data extraction from multiple files to a single CSV file
edited body
Jul
23
revised Efficient data extraction from multiple files to a single CSV file
deleted 921 characters in body
Jul
23
accepted Efficient data extraction from multiple files to a single CSV file
Jul
23
revised Efficient data extraction from multiple files to a single CSV file
deleted 1 character in body
Jul
23
comment Efficient data extraction from multiple files to a single CSV file
That worked! @mikeserve. I only had to fix type -f to have it be type f (editing your query if you don't mind). Impressive speed! I hope this is useful for others -- extracting structured data from many files into a csv file. I renamed the OP to signal that.
Jul
23
revised Efficient data extraction from multiple files to a single CSV file
added 310 characters in body
Jul
23
comment Efficient data extraction from multiple files to a single CSV file
And by the way, your assumption is correct (one dot per file and 168 numbers per file)
Jul
23
comment Efficient data extraction from multiple files to a single CSV file
Thanks -- This is really strange. I have updated my OP with your latest solution.
Jul
23
revised Efficient data extraction from multiple files to a single CSV file
added 631 characters in body
Jul
23
comment Efficient data extraction from multiple files to a single CSV file
Thanks -- This is interesting -- It doesn't seem to work yet, I basically end up with csv with two XML filenames per line, i.e. first line looks like file1.xml file2.xml , second line like file3.xml file4.xml, etc.
Jul
23
comment Efficient data extraction from multiple files to a single CSV file
Thanks -- For some reason this is very slow (slower than the solution in the OP) on my machine. I'm really surprised to see this, since your solution looks definitely cleaner and involves less piping etc. from what I can see. Hmm, I'm still investigating.