2

Rather straightforward question that I'm stumped on. I have a command line utility called twarc that I am executing. This can build rather large files. I would like to leave this running and generating files of 5GB size for easier downstream processing. I am trying to run the following command:

twarc "wordlist" > outputfile.jsonl | split -b 5G

However, the file ended up at 10.4GB overnight. I can run | split -b 5G manually, but I would prefer not to.

I am using split (GNU coreutils 8.25) and the Windows Linux Subsystem.

2
  • because previous command's stdout written to file not to pipe Commented Oct 3, 2020 at 14:29
  • Sorry, not sure I understand. Could you elaborate? Do I need to specify | split outputfile.jsonl -b 5G?
    – bashity
    Commented Oct 3, 2020 at 14:30

1 Answer 1

3

Following up from the comment, the answer was in front of me the whole time!

The problem is that I was writing the command's stdout to a file, which effectively ended the command. In order to pipe it into split I needed to not write to a file. For example:

twarc "wordslist" | split -b 1G

Hope that helps someone down the line!

3
  • 1
    Thank you for following up with your own answer. Tomorrow (?) you can accept it too, to show that this was the answer that helped you most Commented Oct 3, 2020 at 14:44
  • 1
    The split is going to call your files xaa, xab, xac ... xzz. See man split for options to change both the base name and the extension length and type. Also note the -b option will split at byte count without regard for lines. If your output is text, the -C or -l options may be preferable as they preserve whole lines. Commented Oct 3, 2020 at 21:23
  • @Paul_Pedant yep, exactly! I have been playing around with those options to ensure that I can create legible output files. split is such a great tool!
    – bashity
    Commented Oct 4, 2020 at 19:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .