1

I have 3 massive folders containing lots of other folders that I need to give access to a third party for downloading via SFTP.

At the moment every folder in the main directory is set for download rights for SFTP so my idea is to make a list.txt containing the files that the user can not access and set the permissions to something? Or move these files to another folder?

The folder in question will have over 2000 folders containing million of files over 500GB and I need to remove access to half of them

Example folder list

(1) some test (2) more test

1. PLANT Madrid Two

2013 Folio ltd

2014-27

201-07-98

3M

3M 4M 5M

3M Comp LTD

5028 - Video

6398SRTTGDS

I was thinking something along the lines of a bash script that would even move the files to a new folder or change permissions. Any thoughts on what would be best with the amount of data, folders and user will be used SFTP to download the other folders?

while IFS= read -r dir; do
  mv -t path/to/Deny_folder -R -- "$dir"
done < list.txt

or

while IFS= read -r dir; do
  chown 700 "$dir"
done < list.txt
1

You can avoid the slow bash loop with something like this, which seems to works ok in my tests:

$ tr '\n' '\0' <file1 |xargs -0 -I{} mv -vt path/to/deny {}  #v for verbose.
#OR
$ cat file1 |xargs -d'\n' -I{} mv -vit path/to/deny {}  # set delimiter to new line

For a dry run you can make a test like this

cat file1 |xargs -d'\n' -I{} echo "mv -vt path/to/deny " {}

PS: My mv command in RHEL & Debian does not recognize the -R option in mv.

One pitfall of this solution is if the directory names in your file include newlines as part of their dirname. In all other cases (i.e dir names with spaces) both versions tested and work fine.

If you want to do it with a loop you could speed things up by avoiding calling mv for each line read by your file. You could "load" all the lines/directories in an array and call mv afterwards, like:

$ while IFS= read -r dir; do folders+=("$dir");done < list.txt
$ mv -t path/to/Deny_folder -- "${folders[@]}"  #-R is not available in Red Hat and Debian

Or even make a kind of mv grouping:

while IFS= read -r dir; do 
let "a++"
folders+=("$dir")
[ "$a" -gt 1000 ] && mv -vt path/to/Deny_folder -- "${folders[@]}" && a=1 && unset folders 
done < list.txt
  • @Grimlockz - See online test here: (tutorialspoint.com/…) – George Vasiliou Mar 14 '17 at 15:20
  • Thanks - Haven't seen that website either - Looks to work :) – Grimlockz Mar 14 '17 at 15:34
  • Could you preform a dry-run on this? – Grimlockz Mar 14 '17 at 15:51
  • @Grimlockz Unfortunatelly i did not found a way to make a dry run... I was also interested for this option. You can make a test like head -n10 |xargs .... to see how it goes with the first 10 lines of your file. – George Vasiliou Mar 14 '17 at 16:06
  • 1
    @Grimlockz Yes, we can do that - let me correct a little bit the command - need to be cat file1 |xargs -d'\n' -I{} echo -e "mv -vt deny " {} – George Vasiliou Mar 14 '17 at 16:20
0

Note: I would test the following on a small example folder before modifying your 500GB directory. Also, I would make a backup of the directory before changing it at all. Even though it's 500GB, having the backup is priceless.

tar -zcvf mybackup.tar.gz big_ol_directory

Then I would move the tar.gz, maybe just to your local machine, or to another machine on the network, any other machine. Two backups in the same place aren't quite as useful.

As far as the permissions, I like your permissions idea. Another idea is using group permissions to limit access.

# ensure that no one has access except the owner (root, your user, whatever)
chmod -R 600 big_ol_directory

# alternatively
# chown -R myuser:companygroup big_ol_directory
# chmod -R 660 big_ol_directory

# create a group and add a user:
# https://www.howtogeek.com/50787/add-a-user-to-a-group-or-second-group-on-linux/

# begin granting access to ftpusers 
chown -R root:ftpusers big_ol_directory

# use 770, 760, 740 as desired (g+rw is x6x)
chmod -R g+rw big_ol_directory/public

# files in the root of big_ol_directory, including directories
# other than public, will still be owner-editable only.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.