Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm having a little problem with my USB stick. There are files and folders on the USB stick I can see and manipulate on my Linux machine (Fedora 20) but some of those files and folders appear nowhere when use the USB stick on a Windows machine. Moreover, according to Windows the USB stick has got almost 2.9 GB of files stored on it, but the files I can use amount to just 500+ MB. I don't know why this happens, but it's not the first time. What can I do to solve this problem permanently?

share|improve this question
    
(1) What file system is the drive formatted with? (2) Can you show us a detailed listing of the drive from both Windows and Linux? –  Andrew Medico Apr 8 at 14:41
add comment

1 Answer 1

UPDATE

So, I really didn't think I would be researching NTFS this morning, but, thanks mostly to @AndrewMedico's comments below, I learned something.

The truth is file streams are weird, and they confuse me, but apparently it gets deeper. Behaving in a way very like NTFS file streams, Transactional NTFS commits file changes to some alternate cache until the change is guaranteed. When it is, the file is committed atomically and wholly in one piece. I confused the concept of the file stream - which I still believe must be a very fundamentally similar concept and likely involved in some way as there can be multiple named and only one unnamed streams for each file - with what is apparently happening with TNTFS. And I guess I'm not the only one:

Due to its complexity and various nuances which developers need to consider as part of application development, Microsoft is considering deprecating TxF APIs in a future version of Windows. Therefore, Microsoft strongly recommends developers investigate using the alternatives rather than adopting the Transactional NTFS API platform which may not be available in future versions of Windows.

When I was putting together that Windows image I mentioned below, I had a problem with disappearing, reappearing files - seemingly even after deleting the disk entirely - which, amounted mostly truncating it actually. My theory then was - and still is in part - that the Linux cache was not committing the changes to the file streams. I think now instead that it just wasn't handling atomic commits in the way that was expected:

Deleting a File

A file or directory that is deleted by calling the DeleteFileTransacted function remains visible to all outside readers.

*Note All* transacted handles to the file must be closed before the end of the transaction. If the handles are not properly closed, the delete does not occur. All open handles to the file must be closed before performing the commit for the delete operation to be considered part of the transaction. This is because the system does not actually delete a file until the last handle to it is closed, even when the operation is not transacted, as part of the Windows file I/O subsystem.

Deleting a Directory

A directory that is deleted by calling the RemoveDirectoryTransacted function remains visible to all outside readers.

Directory Locking Issues

If a file is modified in a transaction, all directory components of the path to the file are referred to as pinned against rename until the transaction ends. That is, the system prevents you from renaming them until the transaction is committed or rolled back. An attempt to rename a directory that is an ancestor to a file that has been modified in an ongoing transaction will fail with the error ERROR_CANT_BREAK_TRANSACTIONAL_DEPENDENCY.

And here is some more info from NTFS-3G:

Alternate Data Streams (ADS)

NTFS stores all data in streams. Every file has exactly one unnamed data stream and can have many named data streams. The size of a file is the size of its unnamed data stream. By default, ntfs-3g will only read the unnamed data stream. By using the options “streams_interface=windows” (not possible with lowntfs-3g), you will be able to read any named data streams, simply by specifying the stream’s name after a colon. For example:

cat some.mp3:artist

Named data streams act like normals files, so you can read from them, write to them and even delete them (using rm). You can list all the named data streams a file has by getting the “ntfs.streams.list” extended attribute.

And to set their use you use a module parameter:

streams_interface=value

This option controls how the user can access Alternate Data Streams (ADS) or in other words, named data streams. It can be set to, one of none, windows or xattr. If the option is set to none, the user will have no access to the named data streams. If it’s set to windows (not possible with lowntfs-3g), then the user can access them just like in Windows (eg. cat file:stream). If it’s set to xattr, then the named data streams are mapped to xattrs and user can be manipulated by using {get,set}fattr utilities. The default is xattr on Linux, none on other OSes.

We could get down and dirty with this I suppose. I think this could be related based on what I read above. Looking through their changelogs and I see:

ntfs-3g: fixed returned files types in readdir()

mkntfs: insert an $Info stream in $UpCase to comply with Windows 8

ntfs-3g: keep the name of a deleted file in place for easier undeletion

All of the above look at least somewhat related to some of the delete issues mentioned in the TNTFS docs above.

Here's what Tuxera NTFS-3G says:

Why doesn’t file deletion free disk space?

In most cases it does, except for the following scenarios:

In some desktop configurations files are not deleted for real but moved into a ‘Trash’ or ‘.Trash-username’ directory in the root of the partition. When these directories are emptied then the disk space is reclaimed.

By design, Linux and Unixes free the disk space of the deleted files permanently only if no software keeps them open anymore. NTFS is able to store small files and directories in fixed size (1 kB) MFT records (inodes). When such files are deleted then the MFT records are marked free for reuse or for undelete, and no space can be freed.

Status: Not NTFS-3G problem.

Why can’t I move files to trash?

Moving a file to trash is only possible if the trash directory is owned by the current user. This implies that ownership of files has been enabled, either by forcing ownership to the current user or by using a generic ownership and permissions mode.

WIPING GETS MY VOTE

This doesn't look good short of wiping the disk - which might not be a bad idea. If you can access the files in Linux, just back them up. Then get theexFAT driver and use it - it is much simpler, and, well, to be honest I'm starting to wish it was the partition thing...

Or maybe it's fixed?

Why do I get an “Operation not supported” message when creating a file?

The latest driver has solved this issue, please upgrade

OLD

This could be for a couple of reasons :

  1. In Linux your ntfs3g driver can - as I think yours does - show the file streams. This is a little known and little used feature of the NTFS file system that is mostly used for the shadow copies kept by Windows itself to version files. Anyway, the result is that sometimes the same file is two files. Or more even. This can get especially tedious if you delete files from an ntfs3g mount without properly handling Windows typed permissions on the disk. Effectively you will alter the files' streams. It's confusing, which is probably why it's not often used, but here you go: FILE STREAMS

  2. Your usb disk is partitioned. Windows does not handle well multi-partition disks marked with the removable flag - whereas that's all part and parcel to the Unix-way. There are options - Rufus and ImDisk are two I can think of off the top of my head.

I said more likely before for number two up there but looking closer at your question and I think number 1 is your problem.

DIGRESSION

I first learned about NTFS file streams when I built my first Windows installation image in Linux. Apparently much of what goes on during the Windows installation process is translating file streams in .wim archives into regular files during install. If you're at all interested in handling these in a Linux environment, I humbly recommend wimlib.

I guess there's a third possibility, and maybe this is it after all:

3. Missing, disappeared files or directories?

If the top directory is completely empty then it’s very probably that the NTFS volume is not mounted. If only some files are missing then please upgrade to at least NTFS-3G 2009.1.1 which has full, built-in Unicode UTF-8 conversion support.

If you use Mac OS X or FreeBSD and have at least one very long filename with national characters and the filename length translates into more than 255 UTF-8 bytes (higher chance with Korean and Greek languages) then Mac OS X and FreeBSD will not show any files in the directory.

If you are double-booting with Windows 8, you may have enabled its fast restarting feature. This may cause Windows 8 to ignore changes done by another OS on any internal partition. A safe way to avoid loss of data is to disable the fast restarting by issuing as a Windows Administrator the command :

powercfg /h off

If your computer has a SSD plugged in, it might be used by Windows as a cache controlled either by the hardware (“Intel Fast Response Technology”) or by software (“Expresscache”, “ReadyCache”, etc.). This feature is generally not compatible with both Windows and Linux and you may have to disable it.

3:

share|improve this answer
    
Point #1 is not accurate. Windows does not use NTFS streams (aka Alternate Data Streams) on files to implement volume shadow copies. "delete files from an ntfs3g mount without properly handling Windows typed permissions on the disk" does not make sense - if you delete items, they're gone and their permissions are gone too. –  Andrew Medico Apr 8 at 14:41
    
And point #3 does not seem relevant - it's about files from Windows not appearing when an NTFS volume is mounted by the NTFS-3G driver, but the OP is having the opposite problem. –  Andrew Medico Apr 8 at 14:44
    
@AndrewMedico Are you sure about this - NTFS streams are not used for shadow copies? And if you permissions to delete them, you change a version. I'm going to look up the shadow copy - I might have it confused with xfs or xss or whatever that giant filesystem is in the windows directory - in which case I will appreciate the correction. Though it would be nice if you said what it was they were used for when issuing it. –  mikeserv Apr 8 at 15:18
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.