6

The short, general question is: In Unix/Linux, what is the best way to find a list of several (about 150) strings within a large text file?

I am asking this to all Unix/Linux experts as a general question, in the hopes that I can find a solution that pertains to my particular case: I have a feeling this is going to take a little tinkering.

I have a large text file (actually, an MVS dataset) on an IBM Unix System Services (USS) machine-- I believe it is somewhere around 6GB.

I also have a list of about 150 5-character identifiers in the format AAAAA that I need to find within this file-- that is, I'd like to extract rows from the file that contain any one of the 150 the specific identifers that I am looking for.

The format of each line in the large file is:

00000000000A00000000000000000AAAAA\n

where 0 represents a digit, and A represents an alphanumeric character. The string that I'm searching for is always at the end of the row.

Working with datasets seems to be a little awkward in USS, and I am not able to copy it over into the Unix environment because it is too large. The standard Unix utilities don't all operate on datasets (dd for example); however sed, awk, and grep seem to work to some degree (although the command line switches seem to be a bit different).

I can grep the dataset as follows:

cat  "//'MVS.DATASET'" | grep -e"LOOKFOR1" -e"LOOKFOR2" -e"LOOKFOR3" > output_to_file.txt

However, it won't allow me to grep for all 150 items on one line; I could split it up and run it several times, but I feel like there should a better way.

I tried using a sed script as follows, but I don't know sed at all, and I got an error that said "garbage after command". I saved the following in a file sed-script.txt:

s/AAA01/&/p
s/AAA30/&/p
s/AAA10/&/p
... etc ...

and then ran sed -f sed-script.txt "//'MVS.DATASET'"

Again, this failed with "sed: FSUM7294 garbage after command".

So, 1. How would one normally tackle this problem in the "average" Unix environment, and 2. Do you have any specific insights to this particular case?

Improve this question

2 Answers 2

Reset to default
11

grep supports getting patterns from a file -f, and becomes more efficient if you also specify fixed strings (-F):

grep -F -f patterns.txt "//'MVS.DATASET'"
Improve this answer
3
  • When I ran it exactly your way on USS, I got: FSUM6261 read error on file "//'MVS.DATASET'": EDC5114I Bad file descriptor. However, when I rewrote it as cat "//'MVS.DATASET'" | grep -F -f patterns.txt > output.txt, it worked. +1-- thank you! Also thank you for the -F-- I did not know that either! I wonder whether piping the output lessens the efficiency at all.
    – transistor1
    Sep 7, 2012 at 18:12
  • 1
    It shouldn't have a significant effect. If you can make redirection work you can cut the cat. Either < filename grep ... or grep ... < filename.
    – Thor
    Sep 7, 2012 at 18:26
  • Meh. When I use grep -F -f patterns.txt < "//'MVS.DATASET'", it says: FSUM7343 cannot open "//'MVS.DATASET'" for input: EDC5129I No such file or directory. I think I read somewhere that USS datasets can't be opened with open(), so utilities that use it can't "see" datasets? Or something like that. Not even sure if that makes sense.
    – transistor1
    Sep 7, 2012 at 18:56
0

You can store your list of 150 5 char strings in a file first and then ran a loop for grepping patterns -

for i in `cat file`
do
cat mvsfile | grep -i $i >> matched_lines
done
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .