0

I have a directory with multiple subfolders that all contain several text files which are formatted in the following way.

data01:data02

I need to extract just the data02 after the : and export this to a single file in the root directory. I got the extraction in place, but how can I run this on multiple directories and files?

  • Hi Gromit, this looks like a perfect task for command line tools "find" and "awk". Can you please explain, why you'd require python for this task? – Marvin Mar 15 at 11:20
  • If you have a suggestion resolve this just by using cmd, it's also a option I'am just not familiar with using awk etc. – Gromit Mar 15 at 11:24
1

You can customize the following command line with "find" and "awk"

find FOLDERLIST -type f -iname "PATTERN" \
     -exec awk -F":" 'NF>1 {print $2}' "{}" \; > /PATH/TO/RESULTFILE

where

  • FOLDERLIST is a space separated list of top folders you want to recursively search, whereby the "current folder" would be a dot: find . -type f ...
  • "-type f" for searching files only
  • PATTERN is the common pattern of the files you are interested in, e.g. an asterix "*" will find all files, "*.csv" will find CSV files, ...
  • /PATH/TO/RESULTFILE is the name of your result file in root directory
  • the "awk" part splits all found files at the ":" and skips empty results

EDIT: adjusted empty results check to NF>1 as suggested by steeldriver.

  • NF>1 {print $2} might be safer (in case $2 evaluates numerically as 0) – steeldriver Mar 15 at 12:00
  • You are right steeldriver, thank you :) – Marvin Mar 15 at 12:19
  • No problem - upvoted ;) – steeldriver Mar 15 at 12:20
0

You have not provided the format of the output file, so I am writing assuming that you want second field in separate line. You can use(assuming all files are in the format a:b as you said in your question):

find directory -type f \
-exec awk -F: '{print $2}' "{}" >> /output.txt \;

It will find files in directory and in it's sub-directory, and will execute command awk -F: '{print $2}' "{}" >> /output.txt, which will write value after :, to a file /output.txt.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.