0

I have daily incoming files in a local folder(/home/gas_uk). I want to put/copy the latest files from local for today to the HDFS directory (/apps/hive/warehouse/gas_uk/). I have tried :

find /home/gas_uk -type f -print0 | xargs -0 stat --format '%Y :%y %n' | sort -nr | cut -d ' ' -f5 | cut -d '/' -f7 | head

to find latest file but due to spaces in the name of file (eg:20191010_voice file cover.csv) I am unable to find the list.

  • I think the cut -d ' ' -f5 is causing the issue here - it's cutting off anything after the first space in a filename. Need some way to trim the date fields without chopping up the filename, or a way to sort on date without adding the date info – Smock Oct 10 at 10:35
  • So it worked better when I removed the -f5 section. but now I am getting 10 files when I do a head but they all do not belong to today! Any suggestion about how we can pick latest files as per system date and not the date on files please ? – SweetyP Oct 10 at 12:17
  • So you just want to list files for the current (todays) date? You could use the -mtime -1 option for find – Smock Oct 10 at 12:34
  • And it worked !! find /home/gas_uk -mtime -1 -print0 | xargs -0 stat --format '%Y :%y %n' | sort -nr | cut -d '/' -f7 | head Thank you @Smock. – SweetyP Oct 10 at 12:52
  • Do you have any clue on how we can send this list to hdfs ? As files have space in names. – SweetyP Oct 10 at 12:57
1

Use find's -exec directive to process the moves: The filenames will not be split by spaces.

find "$srcDir" -type f -mtime 1 -exec echo mv -t "$destDir" {} +
# ....... remove echo if it looks OK: ^^^^
  • 1
    For this kind of thing, I find that printf '%q\n' is a better choice than echo, as it helps visualise the distinct arguments. – Toby Speight Oct 10 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.