My situation is that I have to search my entire cluster for files with .xyz extension and the sed/grep/awk/? off only records that contain two distinct values, "CONFNAME" and "INPUTSQL", respectively.

1. Structuring the proper pipeline that will return this information.
2. Cutting the 50 lines of junk out between these two values and formatting as csv with two columns.

example file:

1  !prototype|P|||${AIDQ_MP}/dA_lookIP.mp
2  CONFNAME|$|||ITEST_New_Deal
50 INPUTSQL||||Select \n 'ITOD|L1235'

The two pieces of data needed from these .zyz files are ITEST_New_Deal and ITOD|L1235 (comma separated).

Some of what I tried for problem 1

cd To_Correct_Search_Path


find . -type f -iname "*.xyz" -exec egrep -i CONFNAME --color=auto --with-filename {} \; 

//only finds one value


find . -type f \( -iname "*.xyz" -o -name "CONFNAME" -o -name "INPUTSQL" \) -exec sh -c 'echo "$0"' {} \; >

//only finds the last value if "INPUT_SQL"


find . -type f -iname "*.pset" -exec egrep -i CONFNAME egrep -i INPUTSQL --color=auto --with-filename {} \; 

//only finds the last value

for problem 2

To remove the garbage between the to values I want to key off of these two distinct values but include those values in information returned by the search. I have some ideas here but I need problem 1 to be fixed first. Definitely open to suggestions!

  • Can you post a sample of what the expected output is? And take a peek at the editing help and consider marking the data and commands as "preformatted text", it makes them much easier to tell apart from the rest, and also makes the whole question easier to read. – ilkkachu May 18 '18 at 14:11
  • Hey ikkachu, thanks for the response! The end goal is to take the value immediately after CONFNAME (which is easy b/c its consistent) and INPUTSQL (which is not consistent) and pipe just those values in a csv file with 2 columns CONFNAME,INPUTSQL. THANKS! – SSDdude May 18 '18 at 20:07

Regarding your first problem:

You can combine find and grep. And you can also combine multiple grep. The first grep doesn't print anything (-q) and the last only prints the file if the other has passed.

This will find only files containing both strings:

find . -type f -iname "*.xyz" \
  -exec sh -c 'grep -qi CONFNAME "{}" && grep -li INPUTSQL "{}"' \;

The second problem is probably worth a new question when you get stuck. Probably good for awk.

  • Don't embed {} in the child shell. If a file is called $(sudo reboot).xyz, it may reboot the system. – Kusalananda May 18 '18 at 14:48
  • Hey RoVo, thanks so much for the prompt reply but this does not work either... it simply returns ~300 lines of .xyz files and no return values of "CONFNAME" or "INPUTSQL" _____ FOR EXAMPLE___cat -n OUTPUT.txt | less 1 ./../../.xyz – SSDdude May 18 '18 at 14:50
  • Sure. This is intended. It returns only the files that contain BOTH stringsm You can then go on and use awk to find the specific lines and parse the output. – pLumo May 18 '18 at 15:21
  • Hey Rovo, thanks again.. so this gives me all the .xyz file locations and their names, I got that. But now I am not sure how to scan these line items and extract the CONNAME and INPUTSQL values. I am real new to bash. Thanks! – SSDdude May 18 '18 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.