Why does my thumb drive go into read-only mode after a while?

Question

My thumb drive auto-mounts as read-write, and I test it by writing a file.

Then 18 hours later, I run a cron job that does

cp -f -u $i "$TARG"/$i

and I get the message,

cp: cannot remove `/media/HP USB FD/./bin_backup.zip': Read-only file system

Why has it flipped to read-only? I am running Fedora 16, and this has worked before; I'm not sure what changed to stop it from working (although I do rotate thumb drives).

After a period of time, I get this in the dmesg log:

(tl;dr summary - halfway down it says, [3866698.685131] FAT-fs (sdb1): Filesystem has been set read-only)

[3807719.969704]  sdb: sdb1
[3807719.984477] sd 18:0:0:0: [sdb] Attached SCSI removable disk
[3865771.716500] swapper/0: page allocation failure: order:2, mode:0x4020
[3865771.716506] Pid: 0, comm: swapper/0 Not tainted 3.2.9-2.fc16.x86_64 #1
[3865771.716508] Call Trace:
[3865771.716510]  <IRQ>  [<ffffffff8111fd66>] warn_alloc_failed+0xf6/0x160
[3865771.716524]  [<ffffffff8112d241>] ? wakeup_kswapd+0x101/0x160
[3865771.716529]  [<ffffffff81123c59>] __alloc_pages_nodemask+0x6f9/0x8c0
[3865771.716536]  [<ffffffff815dc633>] kmalloc_large_node+0x57/0x85
[3865771.716541]  [<ffffffff81168e7c>] __kmalloc_node_track_caller+0x19c/0x1f0
[3865771.716546]  [<ffffffff814c813b>] ? __alloc_skb+0x4b/0x240
[3865771.716549]  [<ffffffff814c8955>] ? skb_copy+0x45/0xb0
[3865771.716553]  [<ffffffff814c8168>] __alloc_skb+0x78/0x240
[3865771.716556]  [<ffffffff814c8955>] skb_copy+0x45/0xb0
[3865771.716576]  [<ffffffffa017a9a4>] tg3_start_xmit+0xa34/0xc80 [tg3]
[3865771.716583]  [<ffffffff814d7282>] dev_hard_start_xmit+0x322/0x6e0
[3865771.716590]  [<ffffffff814f2daf>] sch_direct_xmit+0xef/0x1c0
[3865771.716595]  [<ffffffff814d77c4>] dev_queue_xmit+0x184/0x600
[3865771.716600]  [<ffffffff81500985>] ? nf_hook_slow+0x75/0x150
[3865771.716605]  [<ffffffff8150f8d0>] ? ip_fragment+0x810/0x810
[3865771.716609]  [<ffffffff8150fa3b>] ip_finish_output+0x16b/0x2f0
[3865771.716613]  [<ffffffff81510598>] ip_output+0x98/0xa0
[3865771.716617]  [<ffffffff8150fc64>] ? __ip_local_out+0xa4/0xb0
[3865771.716621]  [<ffffffff8150fc99>] ip_local_out+0x29/0x30
[3865771.716626]  [<ffffffff8150fdef>] ip_queue_xmit+0x14f/0x3f0
[3865771.716630]  [<ffffffff81527204>] tcp_transmit_skb+0x3e4/0x8d0
[3865771.716634]  [<ffffffff81527d57>] tcp_write_xmit+0xf7/0xa20
[3865771.716638]  [<ffffffff8152a9f0>] ? tcp_v4_md5_lookup+0x10/0x20
[3865771.716643]  [<ffffffff815286e6>] __tcp_push_pending_frames+0x26/0xd0
[3865771.716647]  [<ffffffff81524e0d>] tcp_rcv_established+0x10d/0x8b0
[3865771.716651]  [<ffffffff8152d09f>] tcp_v4_do_rcv+0x27f/0x430
[3865771.716656]  [<ffffffff8152ee70>] tcp_v4_rcv+0x610/0x8e0
[3865771.716660]  [<ffffffff8150acf0>] ? ip_rcv_finish+0x380/0x380
[3865771.716663]  [<ffffffff8150acf0>] ? ip_rcv_finish+0x380/0x380
[3865771.716668]  [<ffffffff8150adcd>] ip_local_deliver_finish+0xdd/0x280
[3865771.716672]  [<ffffffff8150b138>] ip_local_deliver+0x88/0x90
[3865771.716676]  [<ffffffff8150aaa1>] ip_rcv_finish+0x131/0x380
[3865771.716680]  [<ffffffff8150b375>] ip_rcv+0x235/0x300
[3865771.716685]  [<ffffffff814d4e23>] __netif_receive_skb+0x523/0x5c0
[3865771.716690]  [<ffffffff814d5a20>] netif_receive_skb+0x80/0x90
[3865771.716694]  [<ffffffff814d5d89>] ? dev_gro_receive+0x1b9/0x2c0
[3865771.716699]  [<ffffffff814d5a80>] napi_skb_finish+0x50/0x70
[3865771.716703]  [<ffffffff814d5f85>] napi_gro_receive+0xf5/0x140
[3865771.716711]  [<ffffffffa0171362>] tg3_poll_work+0x7a2/0xce0 [tg3]
[3865771.716717]  [<ffffffff81056d45>] ? check_preempt_curr+0x75/0xa0
[3865771.716725]  [<ffffffffa017868f>] tg3_poll+0x6f/0x3c0 [tg3]
[3865771.716730]  [<ffffffff814d617b>] net_rx_action+0x12b/0x270
[3865771.716735]  [<ffffffff81096ebd>] ? sched_clock_cpu+0xbd/0x110
[3865771.716740]  [<ffffffff81075d58>] __do_softirq+0xb8/0x230
[3865771.716745]  [<ffffffff8109acc6>] ? do_timer+0x2d6/0x500
[3865771.716749]  [<ffffffff815ee8ec>] call_softirq+0x1c/0x30
[3865771.716754]  [<ffffffff810162f5>] do_softirq+0x65/0xa0
[3865771.716758]  [<ffffffff8107616e>] irq_exit+0x9e/0xc0
[3865771.716762]  [<ffffffff815ef1a3>] do_IRQ+0x63/0xe0
[3865771.716766]  [<ffffffff815e4aee>] common_interrupt+0x6e/0x6e
[3865771.716768]  <EOI>  [<ffffffff81094189>] ? enqueue_hrtimer+0x39/0xc0
[3865771.716775]  [<ffffffff8131237d>] ? intel_idle+0xed/0x150
[3865771.716779]  [<ffffffff8131235f>] ? intel_idle+0xcf/0x150
[3865771.716784]  [<ffffffff81494fb1>] cpuidle_idle_call+0xc1/0x280
[3865771.716788]  [<ffffffff8101322a>] cpu_idle+0xca/0x120
[3865771.716792]  [<ffffffff815c199e>] rest_init+0x72/0x74
[3865771.716797]  [<ffffffff81aebbfe>] start_kernel+0x3ba/0x3c5
[3865771.716801]  [<ffffffff81aeb347>] x86_64_start_reservations+0x132/0x136
[3865771.716804]  [<ffffffff81aeb140>] ? early_idt_handlers+0x140/0x140
[3865771.716808]  [<ffffffff81aeb44d>] x86_64_start_kernel+0x102/0x111
[3866698.685128] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3866698.685131] FAT-fs (sdb1): Filesystem has been set read-only
[3866698.685145] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3866698.685148] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3866698.685149] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3866698.685151] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3866698.685153] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3866698.685154] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3866698.685156] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3866698.685158] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3866698.685161] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3953054.903337] fat_get_cluster: 88 callbacks suppressed
[3953054.903340] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3953054.903356] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3953054.903359] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3953054.903362] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3953054.903364] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3953054.903367] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3953054.903369] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3953054.903371] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3953054.903374] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)
[3953054.903378] FAT-fs (sdb1): error, fat_get_cluster: invalid cluster chain (i_pos 496175)

Answer 1

Is it just one drive, or any drive?

This might be a long shot, but if the drive has an ext filesystem on it and it was mounted with the errors=remount-ro option, it may be that the thumb drive is starting to throw errors?

You can check the errors behavior using mount (in case custom mount options were used) and tune2fs (for default behavior).

$ mount | grep /dev/sda
/dev/sda2 on / type ext4 (rw,noatime,errors=remount-ro,discard)

$ sudo tune2fs -l /dev/sda2 | grep Errors
Errors behavior:          Continue

If this is the case and the drive has started throwing errors, It is time to replace the drive.

If you want to continue using the drive in read/write mode (knowing that it may at any point stop working completely!), you can change the error mode to continue when errors are found. Doing this depends on how the error mode is set. First (with the drive plugged in), change the default behavior on the disk partition itself:

tune2fs -e continue /dev/sdb1

Then, if it is still mounting remount-ro, check /etc/fstab and remove the errors=remount-ro portion from the appropriate line. Though note, remounting in read-only mode is useful for notifying you when a device is starting to fail!

Answer 2

Your thumb drive goes read-only because the driver crashes. When the driver has crashed, any data it might feel tempted to write is possibly corrupt, so the filesystem is forced into read-only mode to avoid any (further) damage.

The driver crashes because it runs out of memory: the trace ends in warn_alloc_failed, and the lack of memory is shown by swapper/0: page allocation failure: order:2, mode:0x4020. On the face of it, it looks like you're running out of memory that's usable by the kernel. This isn't quite the same thing as running out of RAM, as the kernel can't use the whole RAM for itself. Running out of kernel memory is pretty rare, though. Are you running intensive applications that cause a lot of simultaneous disk usage or network traffic? Do you have a large amount of video memory?

If the answer is yes, you may need to switch to a 64-bit installation, or to reduce your kernel memory usage.

If the answer is no (or even if it's yes), this could be a kernel bug. Make sure you're runing the latest distribution-provided kernel. This could be a driver bug that's triggered by a broken filesystem or broken reader (the driver is supposed to resist to bad data but many drivers aren't tested a lot with misbehaving devices). It could also be failing RAM; do a memory test.