extracting string from files and merge (python)

Question

I have a directory with multiple subfolders that all contain several text files which are formatted in the following way.

data01:data02

I need to extract just the data02 after the : and export this to a single file in the root directory. I got the extraction in place, but how can I run this on multiple directories and files?

Answer 1

You can customize the following command line with "find" and "awk"

find FOLDERLIST -type f -iname "PATTERN" \
     -exec awk -F":" 'NF>1 {print $2}' "{}" \; > /PATH/TO/RESULTFILE

where

• FOLDERLIST is a space separated list of top folders you want to recursively search, whereby the "current folder" would be a dot: find . -type f ...
• "-type f" for searching files only
• PATTERN is the common pattern of the files you are interested in, e.g. an asterix "*" will find all files, "*.csv" will find CSV files, ...
• /PATH/TO/RESULTFILE is the name of your result file in root directory
• the "awk" part splits all found files at the ":" and skips empty results

EDIT: adjusted empty results check to NF>1 as suggested by steeldriver.

Answer 2

You have not provided the format of the output file, so I am writing assuming that you want second field in separate line. You can use(assuming all files are in the format a:b as you said in your question):

find directory -type f \
-exec awk -F: '{print $2}' "{}" >> /output.txt \;

It will find files in directory and in it's sub-directory, and will execute command awk -F: '{print $2}' "{}" >> /output.txt, which will write value after :, to a file /output.txt.