chmod 777 /home is a problem—you shouldn't do that. Change it back. Stuff like that should never be needed when running as
root ignores permission bits.
Other than that, it's just telling you that it's not possible for it to verify the correctness of quota accounting if the filesystem is modified while its checking. Basically, the way it works is by scanning through all the files on the filesystem, totaling up their size (by user/group). So, given three files (A, B, and C), it might check A to get its size, user, and group, then B, then C. If midway through that (while it's adding in B, for example) someone adds 10MB to A,
quotacheck won't know that happened. So it'll have a sum that is 10MB too small—and thus complain about a quota accounting error when there isn't really one. It also might be possible to corrupt the quote accounting file, depending.
Normally you'd run
quotacheck on boot, before starting to use the filesystem (your distro's boot scripts probably already do this). Alternatively, you'd bring it down to single user so there aren't any users trying to modify their files.