45

I have an external hard drive which is encrypted via LUKS. It contains an ext4 fs.

I just got an error from rsync for a file which is located on this drive:

rsync: readlink_stat("/home/some/dir/items.json") failed: Structure needs cleaning (117)

If I try to delete the file I get the same error:

rm /home/some/dir/items.json
rm: cannot remove ‘//home/some/dir/items.json’: Structure needs cleaning

Does anyone know what I can do to remove the file and fix related issues with the drive/fs (if there are any)?

Improve this question

8 Answers 8

Reset to default
43

That is strongly indicative of file-system corruption. You should unmount, make a sector-level backup of your disk, and then run e2fsck to see what is up. If there is major corruption, you may later be happy that you did a sector-level backup before letting e2fsck tamper with the data.

Improve this answer
2
  • 1
    Did what you suggested. e2fsck fixed some fs errors. Seems like I'm back to normal. Thx =)
    – Rotareti
    Dec 20, 2016 at 20:06
  • 5
    I strongly suggest people to use GUIs like e.g. gparted to do this. Gparted still wraps e2fsck (in the case of checking a partition), but the graphical interface makes it a lot easier to select the proper hard-disk, as well as ensuring that you run these tools with the correct arguments. With the wrong arguments you risk severe loss of data!
    – Qqwy
    Feb 7, 2019 at 22:50
10

If it helps anyone, I had a similar issue (rsync/rsnapshot backups, on an affected file). I posted my problem/solution, here:

https://ubuntuforums.org/showthread.php?t=2348768&p=13627299#post13627299

SUMMARY:

rsnapshot (rsync) backup error on arch linux x86_64 system; a corrupted, deeply-nested file was throwing that error, also shown when I tried to delete that file:

sudo rm -fR hourly.5/

rm: cannot remove 'hourly.5/snapshot_root/mnt/Vancouver/temp/temp - old/temp - 09 (Dec 07, 2014 - Sep 02, 2015)/a_OLD-gmail/victoria.a.stuart@gmail.com/[Gmail]/LINUX/rsync, rsnapshot; Other backups/19.bak': Structure needs cleaning

Here is the problem:

cd mnt/Vancouver/temp/temp\ -\ old/temp\ -\ 09\ \(Dec\ 07\,\ 2014\ -\ Sep\ 02\,\ 2015\)/a_OLD-gmail/victoria.a.stuart@gmail.com/\[Gmail\]/LINUX/rsync\,\ rsnapshot\;\ Other\ backups/

ls -l

ls: cannot access '19.bak': Structure needs cleaning
total 0
-????????? ? ? ? ?  ? 19.bak        ## << THAT IS THE PROBLEM!!

[ See also: https://www.reddit.com/r/linuxquestions/comments/4b47r2/has_anyone_ever_gotten_structure_needs_cleaning/ ]

My backup drive is /dev/sda1.

sudo umount /dev/sda1

sudo fsck.ext4 /dev/sda1  ## << accepted suggested fixes
  • Edit (2023-02): a precautionary note: you cannot run fsck on NAS - network associated (RAID) hard disk drives - commonly sold online (Amazon; ...), and which look like regular HDD; repairs to NAS drives require special software. If you apply fsck to a NAS HDD, you'll likely corrupt it.

Rebooted: all seems fine. Went into backups drive, deleted that problematic file:

/mnt/Backups/rsnapshot_backups/hourly.5/snapshot_root/mnt/Vancouver/temp/temp - old/temp - 09 (Dec 07, 2014 - Sep 02, 2015)/a_OLD-gmail/victoria.a.stuart@gmail.com/[Gmail]/LINUX/rsync, rsnapshot; Other backups/19.bak

Q.E.D.?!

[Update: yes; that worked: my backups are running normally, again! :-) ]

Improve this answer
1
  • This just got upvoted, prompting me to peek at this older answer. Note: don't run fsck on a NAS (Network Attached Storage) hard drive - commonly sold online (Amazon; ...) particularly one that is used as a regular (non-RAID) drive - you will corrupt it. If you have a non-RAID system, buy non-NAS HDD, and routinely back up to a second non-NAS HDD. When a NAS drive fails, it's probably best to just toss it, as (from what I understand) specialized software and procedures are needed to recover lost data, from the RAID (for which NAS drives are intended).
    – Victoria Stuart
    Jan 21 at 8:02
2

File systems sometimes are not in order and need cleaning. This can be done by fsck command. But remember, you must run fsck only to unmounted partitions to avoid risk of file corruption.

If your file system is ext4, try running this command :

fsck -AR -t ext4 -y
Improve this answer
2
  • 7
    a list of what options do is always welcome, as we should teach people to understand what they are running instead of just copy/paste some magic command. So -A is check all fstab devices, -R will exclude the root filesystem from -A, as it is mounted readwrite and usually can't be check at this time (can be check offline or if mounted read-only) , -t ext4 to limit the list to ext4 filesystems (ignoring the others that may exist) and -y to auto-answer yes to the repair questions
    – higuita
    May 25, 2020 at 14:02
  • Not that I'd run a command like that without looking up the CLOs first, but I do appreciate the heads up. That's a bad answer in that it's unlikely -All the drives/partitions attached to a system will have issues. As OP said, they're using an external HDD which means they'd want to scan that specific drive/partition, not all minus root.
    – s.co.tt
    Apr 17, 2021 at 5:20
1

I fixed this problem with this command in my proxmox shell:

  • stop the container 
    pct stop 100
  • fsck container, this repair the problem, on some issues files that are corrupt on container system 
    pct fsck 100
  • start the container 
    pct start 100

Now backup and all is good!

Improve this answer
4
  • 1
    So your answer is — stop any software that might be using the disk, run fsck, and resume using the disk.   How does this differ from DepressedDaniel's answer?
    – G-Man Says 'Reinstate Monica'
    Nov 25, 2018 at 1:27
  • 1
    Thjis is very different because it is a special case of proxmox containers. You can't fsck inside the lxc container nor have (a easy) direct control of the lxc devices to run the fsck, so this commands are the correct and easier answer for errors in a proxmox lxc container
    – higuita
    May 25, 2020 at 14:05
  • @higuita This question is about “an external hard drive”, not an LXC device.
    – Melebius
    Jun 29, 2020 at 9:47
  • 1
    @Melebius you are correct for the question but i was referring to this specific answer. While this answer do not solves the original question, it is a useful answer for the same problem in a proxmox container. It is useful for people that arrive here via search (like me)
    – higuita
    Jul 1, 2020 at 20:04
1

I got the same error message from rsync and same error message from rm when I tried to delete the file. Because the filesystem was the root filesystem there was no way to use fsck. But when I just rebooted the system the file was gone and the backup succeeded. I have no clue why this worked but at least that's an easy fix and it's worth to try a reboot first.

Improve this answer
1
  • It worked because your filesystem was flagged for fsck at reboot.
    – Auspex
    May 9, 2022 at 11:03
1

Just recently had this problem myself, so I'll add what I did here.

Ubuntu 20.04 3 disks (4tb each) - (sdd,sde and sdf) formatted with raid5 as md0. MD0/raid was formatted with LUKS. Luks contained LVM with 1 partition of ext4 mounted as /home.

looked like this:

sdd                     8:48   0   3.7T  0 disk  
└─md0                   9:0    0   7.3T  0 raid5 
  └─vgraid-lvraid     253:0    0   7.3T  0 lvm   
    └─luks-raid       253:4    0   7.3T  0 crypt /home
sde                     8:64   0   3.7T  0 disk  
└─md0                   9:0    0   7.3T  0 raid5 
  └─vgraid-lvraid     253:0    0   7.3T  0 lvm   
    └─luks-raid       253:4    0   7.3T  0 crypt /home
sdf                     8:80   0   3.7T  0 disk  
└─md0                   9:0    0   7.3T  0 raid5 
  └─vgraid-lvraid     253:0    0   7.3T  0 lvm   
    └─luks-raid       253:4    0   7.3T  0 crypt /home

No matter what I tried, couldn't un-mount /home.

I ended up rebooting into a Live CD (Ubuntu 20.04), and used the 'Try before installing' option.

Mounted/activated md0 raid using

$ sudo mdadm --assemble --scan

Then started up gnome-disks

$ sudo gnome-disks

Using Disks, I mounted the LUKS volume, and this gave access to the EXT4 partition where I used gnome-disks to perform filesystem check and repair on the unmounted ext4 partition.

Rebooted back to normal, and all was fixed.

Improve this answer
0

It's usual error when try remove .Trash-0 if you try remove windows in encoding CP1251 file system on Linux system. So fs is corrupt but this dosn't metter. Fs of Windows always corrupt as see from Linux. But is not so. You may try open this file from windows os All will be well. And then remove it in windows. And only after that clean trash.

Improve this answer
1
  • 1
    It's not a solution: it's LUKS drive encrypted with LUKS. Accessing (or at least trying access) it from Windows is in my opinion not the best idea.
    – mrc02_kr
    Oct 23, 2018 at 8:08
0

Contrary to others, I faced this issue, not because any of my disks were failing, but my lower RAM frequencies would cause weird issues including this one while under heavy load.
Basically, for me, it was my DDR4 RAM being set to 2133mhz instead of 3000mhz, coupled with 12700K, and a 20 thread full IO workload that resulted in me facing this error repeatedly. No amount of memtest,Aida64 stress test, etc would reveal this issue, until thanks to God, I finally meddled with ram frequency and it all got sorted out and errors were no more!

This seems like an issue especially for Intel's 12th gen cpus/motherboard that use DDR4 instead of DDR5, I'm not sure if this can happen in other cases, but if you are on the 12th gen, and use DDR4 and face such weird error, definitely check your RAM frequencies.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.