0

OK so this is on an iPad Pro but I'm asking you guys because this is about the underlying Unix system behind iOS/OS X, not really iPad specific. (And yes I spent hours reading all relevant StackExchange's about "no space left on device" first.)

Problem #1: I can't create files larger than 2-8MB (it varies upon reboot). This renders the iPad virtually unusuable. Many apps won't launch, apps won't install, etc. It reports "no space left on device" when you try to create a file larger than the bizarre 2-8MB limit, despite having gigs of free space.

Problem #2: Disk space constantly keeps disappearing. I kept uninstalling apps (before this "no space left on device" issue began) and no matter how many I deleted, it would act full a few days later. At first it acted full at 1GB free. Then over several weeks it eventually became 2GB, then 3...4... 6...8... and eventually even with 9GB free, the device still acted like it was full! So I knew a HUGE amount of disk space was unaccounted for bc I had uninstalled dozens of gigs of apps.

Precipitating Incident: Something catastrophic happened a few months ago when I was legitimately very low on disk space and tried updating several apps at once. The iPad froze and several system databases were corrupted, and the iPad began asking me to setup certain passwords again, etc. Ever since then I've had various issues with it but was able to use it mostly. Until last week!

I ended up Jailbreaking the iPad because I'm at the end of my wits and going to have to erase the device if I can't solve it, and I was absolutely DYING to run a "du -h -d 1" to see just WHAT was consuming roughly 60GB of missing space!!

I ran a fsck_hfs on the drive (which was incredibly difficult to do!!) and SURE ENOUGH, it said something like 2 million blocks free - should be 16 million, and I did the math and it made perfect sense! The fsck completed and rebooted and BAM! Suddenly my missing space is back and I've got 71GB free!

But that's around the time the problem got so bad that I can't create any files bigger than 2-8MB. I literally ran:

dd if=/dev/zero of=testfile.bin bs=1M count=10

..and it will fail at a certain number that almost always is a perfect MiB power of 2 (like 2, 4, or 8MiB) with "No space left on device". BUT I CAN ALWAYS WRITE AS MANY MORE FILES OF THAT SIZE AS I WANT! Let's say the limit is 4.0MiB today. I can do that DD command with incremental filenames over and over. I've done it 7 times in a row creating 7 files and every time it worked perfectly. If I made it 4.1MiB, it fails. Even though I just created 7x4 (32MiB) of files!

And STILL, the disk space CONTINUES to shrink on its own, this morning its down to 39GB free. If I fsck_hfs it again, it will go back to the ~70GB free mark, and slowly begin dwindling once again.

I'm at a loss. Just HOW can the device give "No space left on device" errors when there's dozens of GB free? The iPad only has 1 disk, divided into a 4GB /System partition and the rest on /private/var. My System partition is only 75% full which is normal for any iOS device.

I even checked the inodes with df and there's something like 4 billion inodes free on the Data disk (/dev/disk0s1s2).

Here are some relevant printouts (from various days):

iPad:/private root# df
Filesystem     512-blocks      Used Available Capacity iused      ifree %iused  Mounted on
/dev/disk0s1s1    9316200   6795912   2427128    74%  125137 4294842142    0%   /
devfs                  99        99         0   100%     172          0  100%   /dev
/dev/disk0s1s2  486135960 476137152   9998808    98% 1217291 4293749988    0%   /private/var
iPad:/private root# df -h
Filesystem       Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk0s1s1  4.4Gi  3.2Gi  1.2Gi    74%  125137 4294842142    0%   /
devfs            50Ki   50Ki    0Bi   100%     172          0  100%   /dev
/dev/disk0s1s2  232Gi  227Gi  4.8Gi    98% 1217291 4293749988    0%   /private/var

iPad-Pro-256GB:/sbin root# mount
/dev/disk0s1s1 on / (hfs, local, journaled, noatime)
devfs on /dev (devfs, local, nobrowse)
/dev/disk0s1s2 on /private/var (hfs, local, nodev, nosuid, journaled, noatime, protect)

iPad-Pro-256GB:~ root# pwd
/var/root
iPad-Pro-256GB:~ root# dd if=/dev/zero of=test3.bin bs=1M count=20
dd: error writing 'test3.bin': No space left on device
9+0 records in
8+0 records out
8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.671137 s, 12.5 MB/s

Excerpt from one of the first fsck_hfs I ran when the device had about 9GB free but should've had 70GB free:

** Checking volume bitmap.
   Volume bitmap needs minor repair for orphaned blocks
   Volume bitmap needs repair for under-allocation
** Checking volume information.
   Invalid volume free block count
   (It should be 16884367 instead of 2063604)

A complete successful fsck_hfs:

iPad-Pro-256GB:/ root# umount -f /private/var && killall backboardd && fsck_hfs -f -y /dev/disk0s1s2
umount: /private/var: not currently mounted
iPad-Pro-256GB:/ root# fsck_hfs -f -y /dev/disk0s1s2
** /dev/rdisk0s1s2
   Executing fsck_hfs (version hfs-366.30.3).
** Checking Journaled HFS Plus volume.
** Detected a case-sensitive volume.
   The volume name is Data
** Checking extents overflow file.
** Checking catalog file.
   Incorrect size for file MediaLibrary.sqlitedb
   (It should be 1343488 instead of 1564672)
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking volume bitmap.
   Volume bitmap needs minor repair for orphaned blocks
** Checking volume information.
   Invalid volume free block count
   (It should be 16972349 instead of 14633343)
** Repairing volume.
   Limited repair mode, not all repairs available
** Rechecking volume.
** Checking Journaled HFS Plus volume.
** Detected a case-sensitive volume.
   The volume name is Data
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking volume bitmap.
** Checking volume information.
** Trimming unused blocks.
** The volume Data was repaired successfully.

Notes:

A. Nothing relevant on the syslog when large files fail to create.

B. Device: iPad Pro 9.7" 256GB iOS 10.2.1 HFS (not APFS which was introduced later in 10.3). Never jailbroken UNTIL long after this problem started.

closed as off-topic by Rui F Ribeiro, psusi, msp9011, Kiwy, Isaac Aug 29 '18 at 2:38

  • This question does not appear to be about Unix or Linux within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    FYI, there is an Ask Different Stack Exchange; your question may very well be on-topic for them; it's likely not here. – Jeff Schaller Aug 27 '18 at 12:20
  • 1
    @JeffSchaller Yes I'm a member of the Ask Different, but I'm pretty confident a question of this nature is more appropriate here than it is there, as this is low level hard core unix internal stuff. – Syclone0044 Aug 27 '18 at 12:22
  • 1
    @RuiFRibeiro If I had simply deleted the words iPad from my entire post, can you honestly say you'd still vote it off topic? – Syclone0044 Aug 27 '18 at 12:42
  • 2
    I honestly think that diagnosing faulty storage memory/RAM on your iPad is more on topic on Ask Different than here. This group is regularly (ab)used for having more knowledgable people hanging around for diagnosing faulty memory/sticks/disks, and it gets tiring after a while. – Rui F Ribeiro Aug 27 '18 at 13:19
  • 2
    This should probably be a Meta post, as I'm not sure whether IOS (being derived from OSX) is still on-topic. Seems neither U&L nor Ask Different get many HFS/IOS questions. I found only two Meta posts, from 2010 and 2011 that talk about IOS at all. It does appear from the IOS tag info that Renan and Mat think that "UNIX foundations of iOS" is on-topic. Note also that I did not VTC here. – Jeff Schaller Aug 30 '18 at 11:47
1

If a filesystem is very fragmented, it may be that there is lot of free space but there is no sufficcient space in larger blocks.

Your case looks as if this could be true on your filesystem.

Fragmentation typically happens if you copy lots of small files to the filesystem and then remove a random part of these small files. This frees fragments that cannot be recombined to larger blocks.

In former times, a usenet news filesystem did usually suffer from this problem in case that you copied a lot of articles to your private disk and use different keep times for the various newsgroups.

If there is no defragmentation tool for this filesystem, you could copy (not move) plenty of the small files to another place in the filesystem and then remove the old versions of the small files. If you get the right files while doing this, there is a big chance that the freed fragments fit together to create new larger free blocks.

  • Ooh, very interesting idea. This happens even on solid state flash storage like the iPad? And I did delete several thousand files for sure when I freed up that 70GB. Probably 10,000 or more. I'll think if there's a way to test your theory. – Syclone0044 Aug 27 '18 at 15:18
  • 2
    Fragmentation depends on the filesystem code and is independent from the background storage. If you delete more small files and then can write a larger file before you get ENOSPACE, my theory is verified. – schily Aug 27 '18 at 15:25
  • Results are inconclusive. At first, copying 2000 small files and deleting the originals, seemed to work as I was able to install a large app and create files close to 20MB. But very soon after, the original 8MB limit returned. I then did the same with 5 batches of ~1000 files per batch, and I could not exceed an 8MB file at any point. So I don't know if the first test was a fluke? With 70GB free, I can't imagine there's any possible level of fragmentation that would consistently prevent even a 10MB file from being created. Is there another way I can test? I'll try a fsck for bad blocks – Syclone0044 Sep 1 '18 at 5:39
  • If a filesystem is newary full (less than 10% of free space), then the management of fragments usually does not work as expected anymore. – schily Sep 1 '18 at 11:13
  • New results: I had forgotten the disk has a System partition. I'm able to run dd commands on there and create large files at 500MB/s speeds. Curiously, I was then able to copy one of those files 50MB in size to the User partition (the problematic one) but it transferred at an extremely slow rate, less than 1MB/sec. However I was quite surprised it did not simply fail with "No free space on device." I also found if I ran "dd if=testfile1.bin of=testfie2.bin bs=512KiB count=40" it DID create a 20MB file, but at an excruciatingly slow speed of something like 100KB/s. 1 of 2 CPUs sat at 100% also. – Syclone0044 Sep 2 '18 at 20:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.