5

Long ago, in Seventh Edition UNIX (a/k/a V7), there was a program called prep. Its primary use was to take files of text, and break them up into one word per line, for further processing by other tools in a pipeline. It could do a little bit of other manipulation too, like telling you the location of each individual word within a file, ignoring specific words in an ignore list, or only paying attention to words specifically mentioned in an include list. It's sort of difficult to explain exactly what it does, but here is a man page from 2.9BSD for it. It had an assortment of interesting uses - for example, building dictionaries, spell-checkers, and the like.

This program was rather short lived. It only existed in V7 and a couple of offshoots (and 2.9BSD was basically an offshoot of V7). It didn't previously exist in V6. It was removed from V8. It never even made it into 4.2BSD. It doesn't exist (at least not in this form) in any Linux distribution that I'm aware of, nor in FreeBSD and friends. There was another program that also (as far as I am aware) first appeared on V7, called deroff, that was primarily for a completely different purpose - but it had a "-w" option that told it to do the "split up files into one word per line" thing, similar to prep, but didn't do any of the other functions (like word numbering, include lists, and ignore lists). I assume for purposes like dictionary building, deroff -w subsumed the function of prep. That was comparatively much longer lived - but these days, there doesn't even seem to be a version of deroff packaged for any major Linux distribution, I know it's not in any recent version of RHEL, it's not in Fedora 32, and it's not in Debian 10 (but I'm pretty sure it actually was in Debian until not that long ago).

Why did prep go away? Was it really because deroff -w duplicated most of its function? I presume that deroff has disappeared in current Linux distributions because people generally don't deal with [nt]roff-formatted documents anymore, except maybe for man pages. But with both of these tools gone, what can one use to do the "split up a text file into one word per line" function? Is there anything packaged for any modern Linux distro that would perform this function? (If you're going to respond with, "you can probably do this yourself with a simple script", I concede that is probably correct - but that is not the answer I'm looking for right now, I'm looking for a way to do this with some existing tool that already exists in modern Linux distributions...) Ideally, I'd like to find something that implements all the features listed in the man page I linked (plus the "implied" behaviors that aren't explicitly specified in the man page, like not considering punctuation to be part of a word, and how numbers that appear as part of a "word" are handled). :-) Practically, I don't think the include and exclude lists are particularly crucial, and while I'd like to have the word numbering (it can sometimes be handy to know the location of a word in a file), it's not that important. Handling of hyphenated words at the end of a line would be desirable.

10
  • You miss out an entire decade leaping from the 1970s to Linux. Do not skip over XENIX and Ultrix. (-: – JdeBP Sep 23 '20 at 5:36
  • Granted, but I can only look for what I'm looking for in places that I have ready access to in an easily digestible form. ;-) FWIW, "prep" does exist in Ultrix-11, which is essentially a straight repackaging of DEC's V7M, which itself was a repackaging of V7 by DEC with some additional tools and facilities. As far as Xenix, I've tried to put it out of my head - I only ever used it on some proprietary Intel hardware that was normally intended to run iRMX, and to the best of my recollection of Xenix on this hardware, it was pretty unpleasant. ;-) – patbarron Sep 23 '20 at 5:51
  • 3
    The deroff removal bug confirms your suspicions. The “one word per line” feature of prep can be obtained with xargs -n 1. – Stephen Kitt Sep 23 '20 at 5:59
  • ... although as pointed out in comments to my deleted answer, there are a number of issues with this use of xargs. – Stephen Kitt Sep 23 '20 at 6:17
  • 1
    @roaima Punctuation needs to be removed, as punctuation adjacent to a word is not part of the actual word. – patbarron Sep 23 '20 at 8:40
0

It seems like tr -s " " "\n" < file ought to work for splitting a file to one word per line.

6
  • 1
    You probably also want the -s (squeeze) option to eliminate consecutive newlines (i.e. a lot of blank lines). You can also deal with punctuation by using complement with the characters you want to keep: tr -cs [:alnum:] '\n'. – Paul_Pedant Sep 23 '20 at 7:58
  • Yep, we don't want to pass any blank lines along to the output, and we don't want to count punctuation as part of a "word". We also don't want to count strings of consecutive digits as a "word", while something like "abc123" should get passed as a "word". There's another corner case that prep handles, which is that it understands what to do with hyphenated words at the end of a line, though in modern text you hardly ever see that. To do this right, it seems like you really do need some sort of lexical analyzer (which is one reason there was an entirely separate program to do this in V7). – patbarron Sep 23 '20 at 8:33
  • I have half a mind to just port the V7 prep program to modern systems, but the source is technically still encumbered so at minimum I'm not sure I could distribute it or use it for anything other than historical exploration. – patbarron Sep 23 '20 at 8:38
  • And now, looking more closely at the man page, it says "A word is a string of alphabetic characters and imbedded apostrophes, delimited by space or punctuation." So I'm not actually sure now what it would do with "abc123". I may need to just fire up an emulator running V7 and try it to see what happens. – patbarron Sep 23 '20 at 9:20
  • This was bugging me so much I actually did set up an emulator running V7 to just try it and see... The behavior of prep is not completely specified in the man page, and there's at least one bug... Everything is changed to lower-case in the output. Anything other than letters and the single-quote are treated as whitespace, so "abc123xyz" is split into "abc" and "xyz" as individual words. There is a bug handling contractions, where things like "you're" are passed through correctly, but "don't" is passed through as "don'". Handling of hyphenated words at the end of a line works. – patbarron Sep 24 '20 at 5:45
0

Using Raku (formerly known as Perl6)

~$ raku -ne '.words.join("\n").put;'  < file

HTH.

https://raku.org/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.