1

I have lot of folders and files, this is the structure I have

26-09-2016/CHANGELOG_20160926.TXT
26-09-2016/FILE_CHANGELOG_20160926.TXT
27-09-2016/CHANGELOG_20160927.TXT
27-09-2016/FILE_CHANGELOG_20160927.TXT

I need the output as follows. All files with name like CHANGELOG_*.TXT shall be merged and add separate new line into one file like CHANGELOG_20160926-20160930.TXT, and all file with name FILE_CHANGELOG_*.TXT shall be merged and add separate new line into one file too like FILE_CHANGELOG_20160926-20160930.TXT.

How can I do that?

  • It is not quite clear where you want the newlines. Could you give a small example of what you want, please? – countermode Oct 1 '16 at 0:14
1

Since you did not specify any language requirement, here's a possibility using Python 3.

#/usr/bin/env python3

from glob import glob
from os.path import basename
import re

for prefix in ('CHANGELOG', 'FILE_CHANGELOG'):
    files = dict((int(re.split('[_.]', basename(f))[-2]), f)
                 for f in glob('*-*-*/%s_*.TXT' % prefix))
    out_file = '%s_%d-%d.TXT' % (prefix, min(files.keys()), max(files.keys()))

    with open(out_file, 'w') as f_out:
        for date in sorted(files.keys()):
            with open(files[date]) as f_in:
                for line in f_in:
                    f_out.write(line)
            f_out.write("\n")

It basically uses glob and basename to list and parse file names, sorting them by date. The min/max values are used to build the output filename, and all the files are written there in order. Don't forget to adjust the patterns to your actual directory structure, should it be necessary. Then, just chmod and run:

$ chmod +x script.py
$ ./script.py
  • Wow, this is very cool. Thanks for helping. Sorry late give you a comment. – MH Rohman Masyhar Oct 8 '16 at 0:14
0

Solution in TXR:

First, let's treat this as a text processing task assuming we have the example list of path names a an input file called paths. We transform paths into shell commands which cat the groups of files together and produce the required output files:

@(do
   (defstruct file-info nil
     full-name
     root-name
     date-key
     (:method equal (self) self.date-key)))
@(collect :vars (files))
@  (all)
@dd-@mm-@yyyy/@*{name}_@yyyy@mm@dd.TXT
@  (and)
@path
@  (end)
@  (bind files @(new file-info full-name path root-name name
                     date-key ^(,yyyy ,mm ,dd)))
@(end)
@(do
   (let ((h (group-by (usl root-name) files :equal-based)))
     [hash-update h sort]
     (dohash (name flist h)
       (let ((start (find-min flist))
             (end (find-max flist))
             (paths (mapcar (usl full-name) flist)))
         (put-line `cat @{paths " "} >\ \
                    @{start.root-name}_@{start.date-key ""}- \
                    @{end.date-key ""}.TXT`)))))

Run:

$ txr catfiles.txr paths
cat 26-09-2016/CHANGELOG_20160926.TXT 27-09-2016/CHANGELOG_20160927.TXT > CHANGELOG_20160926-20160927.TXT
cat 26-09-2016/FILE_CHANGELOG_20160926.TXT 27-09-2016/FILE_CHANGELOG_20160927.TXT > FILE_CHANGELOG_20160926-20160927.TXT

To work on real paths and do the cat commands requires simple modifications:

@(do
   (defstruct file-info nil
     full-name
     root-name
     date-key
     (:method equal (self) self.date-key)))
@(next :list (glob "*/*.TXT"))
@(collect :vars (files))
@  (all)
@dd-@mm-@yyyy/@*{name}_@yyyy@mm@dd.TXT
@  (and)
@path
@  (end)
@  (bind files @(new file-info full-name path root-name name
                     date-key ^(,yyyy ,mm ,dd)))
@(end)
@(do
   (let ((h (group-by (usl root-name) files :equal-based)))
     [hash-update h sort]
     (dohash (name flist h)
       (let ((start (find-min flist))
             (end (find-max flist))
             (paths (mapcar (usl full-name) flist)))
         (sh `cat @{paths " "} >\ \
              @{start.root-name}_@{start.date-key ""}- \
              @{end.date-key ""}.TXT`)))))

The only changes are the addition of a @(next :list (glob "*/*.TXT")) to redirect the input scanning over a list of paths globbed from the filesystem, and a switch from put-string to sh to have the cat commands executed.

If the lists of files can be very large, we will run into OS command/argv passing limits: we cannot cat them in a single command.

A possible fix for that is to change the last part of the code to:

@(do
   (let ((h (group-by (usl root-name) files :equal-based)))
     (hash-update h (op sort))
     (dohash (name flist h)
       (let* ((start (find-min flist))
              (end (find-max flist))
              (paths (mapcar (usl full-name) flist))
              (target `@{start.root-name}_@{start.date-key ""}- \
                       @{end.date-key ""}.TXT`))
         (sh `> @target`)
         (each ((group-of-ten (tuples 10 paths)))
           (sh `cat @{group-of-ten " "} >> @target`))))))

I.e. for each file, use > file to ensure it exists and is truncated to zero. Then use cat ... >> file to append the logs to it, in groups of ten.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.