0

So I have a Micro-SD card with an EXT4 file system on it. On there are a bunch of files that I used to save in this structure: /YYYY-MM/folders/files - The amount of files in the folders is usually just 1-10, but the amount of folders in YYYY-MM is a lot. I have now changed it to YYYY-MM/YYYY-MM-DD/.. to keep it nicer, but...

I need to copy/move all the files off of the µSD card. This worked for every folder except 2019-10 which I assume contains too many folders. Commands that failed (not all for moving):

  • rsync (over ssh): After ~2 days of "receiving incremental file list", it disconnected
  • mv: It creates the directory in DEST, then gets stuck
  • cp: Same thing, stuck at ./2019-10/ -> DEST/2019-10/
  • find .: Nothing for an hour, I CTRL+C'd after
  • ls: Same as above
  • du -chs 2019-10: I let this one run for around 3 days, nothing

The size of 2019-10 is around 15.5GB (determined by df -h after everything else was moved off). The amount of folders (I know because a database of the folders exists) in 2019-10 is 40472. Each of those has ~4 files on average inside.

Now my question is: How can I move all these folders out of there?

Btw: At first I thought the µSD-Card had issues and that's why I can't read the directory, but moving 2019-11 and 2019-12 worked just fine at normal speed. They both contained the same amount of files in total, but due to better structure they never had more than ~2000 folders in a single directory.

Side info: 32GB SanDisk A1 µSD-Card (no counterfeit), internal laptop SD Card reader (performs best), debian 10 OS (arch available), 8GB RAM and a 4x3.2GHz i7 CPU.


As I was writing this question, I realized a solution (happens so often it's crazy). I made a script that goes through the database and for every folder entry retrieves the name of the folder and then does a mv -v /sd-card/2019-10/${folderid}/ DEST to move it. Works perfectly fine at normal speed.

I still ask this question and want to know what to do in case I wouldn't have the database, this might be useful to others in the future.

  • Sounds like the SD card is dead. Unable to read blocks from the directory tree and just hanging. Do you have a backup? – Philip Couling Dec 22 '19 at 9:32
  • 1
    @PhilipCouling Please see the bottom part of my question on my solution for my specific case. The script finished with an avg. speed of 10MB/s which is definitely fine. The data is all there. I'm just interested if there is a solution in case someone else does not have a database with the foldernames to do what I did. Since my solution worked, I'm 99% sure the card is fine. – confetti Dec 22 '19 at 10:55
  • Actually... I just checked if I can use ls or find or anything on the SD card... The card is now pretty much empty (df -h checked) and the 2019-10 directory should be completely empty, however both ls and find . hang right at that folder still.... Weird. I can do a re-format with no problem since I saved all the data now, but an explanation would still be cool. Seems strange to me. The card itself should really be fine, it's just been in use for a few months and SanDisk's always been a very solid brand for me. – confetti Dec 22 '19 at 11:00
  • Okay, my bad. This was an ext4 filesystem, not FAT32. I have ran fsck.ext4 -f -y, after which the issue disappeared. A lot of Inode ref counts have been fixed and a bunch of filetypes have been set. – confetti Dec 22 '19 at 11:08
  • 1
    ah that makes more sense. ext4 is much less constained than fat32. I was going to suggest trying ls -U since ls can perform slowly with directories over 1000 files. But usually that doesn't make it "hang for an hour". And yes minor corruption on the file system can mess things up. Always try to eject SD cards properly. If you are manually mounting on the file system then always call sync between unmounting and removing the card. – Philip Couling Dec 22 '19 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.