3

I'm moving files over from my old server to my new server. By the time I'm doing it for real (as opposed to just testing) I'll have all the "user" users (the non-system users, i.e. >= 1000) created on the new server. My only fear is that I have some file that I'm moving over in the home directory that will belong to one of these users (say apache for instance) that doesn't exist on the new server. I'm using

rsync -az -e ssh src dest

as user on the new server to do the copying. It does preserve the usernames (as opposed to ids) for the users that exist. However instead of complaining about non-existent users, it just falls back on the numeric ids if the user isn't found. The behavior seems to be as described in this paragraph from the man page:

If a user or group has no name on the source system or it has no
match  on  the  destination system, then the numeric ID from the
source system is used instead.  See also  the  comments  on  the
"use  chroot" setting in the rsyncd.conf manpage for information
on how the chroot setting affects rsync’s ability to look up the
names of the users and groups and what you can do about it.

While I haven't read the entire man page word for word, what I have read doesn't offer me any options to complain about non-existent users. What is the best way to make sure that whatever users exists as owners/groups of files under a directory (say /home) exist on the new machine. If it's not doable with rsync, what's the best way to get a list of all the users/groups that exist so I can manually check that they exist on the new machine, or fix them before copying them.

Summary:

How do I make sure that after I run rsync, none of the files have been copied using numeric ids instead of name ids?

4

The rsync command doesn't have a mechanism for handling this directly, so I would use a different approach. I would scan the source filesystem tree, collecting the usernames (and groups) of all files present there:

# List of usernames owning files under 'src'
find src -printf "%u\n" | sort -u | tee /tmp/src.users

# List of group memberships for files under 'src'
find src -printf "%g\n" | sort -u | tee /tmp/src.groups

# Copy files to target system
scp -p /tmp/src.{users,groups} dest:/tmp/

I would then ensure that all users existed on the target system ready for rsync to use. Run these commands on the target system:

# List "missing" users
getent passwd | cut -d: -f1 | sort -u | comm -13 - /tmp/src.users

# List "missing" groups
getent group | cut -d: -f1 | sort -u | comm -13 - /tmp/src.groups
  • Coincidentally I'm doing this just as you're editing it to improve it :P. Looks good, though I wrote a patch for rsync to report when a username doesn't exist on the destination. – CrazyCasta Jun 16 '15 at 21:50
  • @CrazyCasta simply replaced the awk with cut as it's a little cleaner – roaima Jun 16 '15 at 21:51
  • Thank you, luckily I only have a few users so it's easy to check manually. – CrazyCasta Jun 16 '15 at 21:51
  • @CrazyCasta the second part of my answer lists the missing users and groups for you. I hope I've now made that clearer in the answer – roaima Jun 16 '15 at 22:02
0

I've developed a patch for rsync that complains if the user doesn't exist on the destination. Here it is, I'm using it together with roaima's listing of users.

diff -ur rsync-3.1.1-orig/uidlist.c rsync-3.1.1/uidlist.c
--- rsync-3.1.1-orig/uidlist.c  2014-04-30 13:34:15.000000000 -0600
+++ rsync-3.1.1/uidlist.c       2015-06-15 10:03:50.282216140 -0600
@@ -232,10 +232,24 @@
        else if (*name && id) {
                if (idlist_ptr == &uidlist) {
                        uid_t uid;
-                       id2 = user_to_uid(name, &uid, False) ? uid : id;
+                       if (user_to_uid(name, &uid, False)) {
+                               id2 = uid;
+                       } else {
+                               rprintf(FINFO,
+                                       "User does not exist name %s uid %u\n",
+                                       name, (unsigned)id);
+                               id2 = id;
+                       }
                } else {
                        gid_t gid;
-                       id2 = group_to_gid(name, &gid, False) ? gid : id;
+                       if (group_to_gid(name, &gid, False)) {
+                               id2 = gid;
+                       } else {
+                               rprintf(FINFO,
+                                       "Group does not exist name %s uid %u\n",
+                                       name, (unsigned)id);
+                               id2 = id;
+                       }
                }
        } else
                id2 = id;

I'm not sure what limitations this might have, like how it'll work if the destination is remote. It does work for copying files from a remote location to a local destination over ssh, i.e.:

rsync -az -e ssh user@host:src_location dest_location

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.