0

I have daily incoming files in a local folder(/home/gas_uk). I want to put/copy the latest files from local for today to the HDFS directory (/apps/hive/warehouse/gas_uk/). I have tried :

find /home/gas_uk -type f -print0 | xargs -0 stat --format '%Y :%y %n' | sort -nr | cut -d ' ' -f5 | cut -d '/' -f7 | head

to find latest file but due to spaces in the name of file (eg:20191010_voice file cover.csv) I am unable to find the list.

7
  • I think the cut -d ' ' -f5 is causing the issue here - it's cutting off anything after the first space in a filename. Need some way to trim the date fields without chopping up the filename, or a way to sort on date without adding the date info
    – Smock
    Commented Oct 10, 2019 at 10:35
  • So it worked better when I removed the -f5 section. but now I am getting 10 files when I do a head but they all do not belong to today! Any suggestion about how we can pick latest files as per system date and not the date on files please ?
    – SweetyP
    Commented Oct 10, 2019 at 12:17
  • So you just want to list files for the current (todays) date? You could use the -mtime -1 option for find
    – Smock
    Commented Oct 10, 2019 at 12:34
  • And it worked !! find /home/gas_uk -mtime -1 -print0 | xargs -0 stat --format '%Y :%y %n' | sort -nr | cut -d '/' -f7 | head Thank you @Smock.
    – SweetyP
    Commented Oct 10, 2019 at 12:52
  • Do you have any clue on how we can send this list to hdfs ? As files have space in names.
    – SweetyP
    Commented Oct 10, 2019 at 12:57

1 Answer 1

1

Use find's -exec directive to process the moves: The filenames will not be split by spaces.

find "$srcDir" -type f -mtime 1 -exec echo mv -t "$destDir" {} +
# ....... remove echo if it looks OK: ^^^^
1
  • 1
    For this kind of thing, I find that printf '%q\n' is a better choice than echo, as it helps visualise the distinct arguments. Commented Oct 10, 2019 at 16:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .