3

I have a HP Microserver gen 8 with Centos 7 and only one hard disk in the bay area, 2 TB, mounted as NTFS.

I'm trying to copy files (about 100GB at once) over network from local.

I've tried with the following:

  • via Samba
  • via SCP
  • via Rsync

Common issues along the way for this methods are usually with this error messages:

  • System is read-only with I\O error (I've replaced the hard disk, because more likely it's causing this issue, not sure though)
  • Disk is full (this just saw, and I was using Samba), Disk is 92% empty.
  • mkstemp failed operation not permitted

So I'm thinking maybe there's a better way to copy big data; is there?

Also, i just tried to do this:

rsync -rvz -e "ssh" * [email protected]:/folder

and I got this error:

rsync: writefd_unbuffered failed to write 16385 bytes [sender]: Broken pipe (32)
rsync: write failed on "/folder/movie.mkv": Input/output error (5)
rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9]
rsync: connection unexpectedly closed (490 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/io.c(453) [sender=2.6.9]

Torrent answer couldn't be the only valid way to copy this much of data; I need a Linux based solution, without torrent.

--

Update:

After @zeppelin suggested to try one big file instead of many 4-5GB files, i've made one big archive using: tar -zcvf movies.tar.gz *

once i had the archive:

ls -alh movies.tar.gz 
-rw-r--r--  1 Abude  staff    77G Nov 11 13:54 movies.tar.gz

I did the rsync and got the following issue:

rsync -avz -e 'ssh' --progress movies.tar.gz [email protected]:/nas/media
/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
building file list ... 
1 file to consider
movies.tar.gz
 82671089587 100%   25.36MB/s    0:51:49 (xfer#1, to-check=0/1)
rsync: mkstemp "/nas/media/.movies.tar.gz.ezRUOM" failed: Operation not permitted (1)

sent 82698841857 bytes  received 42 bytes  26586993.06 bytes/sec
total size is 82671089587  speedup is 1.00
rsync error: some files could not be transferred (code 23) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/main.c(992) [sender=2.6.9]
9
  • 1
    You surely meant NTFS, no? Because FAT32 does not support bigger volumes than 32GB AFAIK. Nov 6, 2016 at 12:02
  • Actually, FAT32 only supports filesizes up to 4 GB.
    – pfnuesel
    Nov 6, 2016 at 12:04
  • @pfnuesel I meant volume size, not a single file size. Nov 6, 2016 at 12:06
  • @pfnuesel see update please
    – Abude
    Nov 7, 2016 at 9:05
  • 2
    I agree with this answer; rsync is the best solution.
    – Wildcard
    Nov 10, 2016 at 1:49

7 Answers 7

6

To copy big chunks of data, a private torrent may be subjectively speaking one of the best choices.

This is not a place to make a tutorial of torrenting though.

You may look here for example or look up something like:

How to create a private torrent

1
  • 1
    Despite your update, torrent sounds like a pretty valid solution. Having had to setup synchronization towards several Samba servers, I remember implementing some btsync puppet class, way easier to automate than any rsync-based cuisine.
    – SYN
    Nov 9, 2016 at 22:56
3

rsync is the best solution. I recommend you remove the -z so that compression is skipped. Compression could be taking up extra disk space for temporary compressed files. So you should be using:

rsync -av -e "ssh" * [email protected]:/folder

5
  • even if compression takes extra space, i have 92% free on the disk, so this is irrelevant, when i tried the rsync method i got the error from the question above.
    – Abude
    Nov 10, 2016 at 11:06
  • 1
    What happens when you try to rsync a single file of small size, say 10MB? Do you get the same erro?
    – GMaster
    Nov 10, 2016 at 11:08
  • @GMaster, shouldn't that just be rsync -av ./ [email protected]:/folder?
    – Wildcard
    Nov 10, 2016 at 11:10
  • * works fine as well. I tried it just now.
    – GMaster
    Nov 10, 2016 at 11:22
  • @GMaster any file under ~4GB has no errors. i even can push this number to about 8-10GB but not sure though.
    – Abude
    Nov 10, 2016 at 12:17
1
+50

Update

The permission error (below) boiled down to removing the '-a' (--archive) flag from the rsync command, to prevent it from trying to preserve ownership and permissions on the files copied.

After doing the local 'dd' test, a low-level IO error was detected, probably caused by a faulty disk, resulting in a filesystem corruption.


Operation not permitted (1)

The last error you see is a most probably simple permission issue, i.e. user 'abude' just does not have a write access to /nas/media/), try making this folder public writeable:

 chmod a+rwx /nas/media

and repeat your rsync command.

------- (disregard below, as OP is using a NTFS partition now) -----

Based on this:

I have a HP Microserver gen 8 with Centos 7 and only one hard disk in the >bay area, 2 TB, mounted as fat32.

...

rsync: write failed on "/folder/movie.mkv": Input/output error (5) rsync error: error in file IO (code 11) at receiver.c(322) >>[receiver=3.0.9]

and your later comment

@GMaster any file under ~4GB has no errors. i even can push this number to about 8-10GB but not sure though. – Abude 8 hours ago

I assume is that the problem you have it that you try to rsync (at least some) 4GB+ files to the FAT32 filesystem, which is not possible to do, as this is hard limit on the file size for FAT32.

Maximum file size 2^32 minus 1 bytes

(https://technet.microsoft.com/en-us/library/cc938937.aspx)

So rsync reports you a file IO error:

11     Error in file I/O

(https://linux.die.net/man/1/rsync)

The options you have are:

15
  • did you read my question? i've mentioned that i have NTFS filesystem.
    – Abude
    Nov 10, 2016 at 21:48
  • @Abude, ahh, I see, sorry :) Would you please strikeout (or otherwise remove) the original mounted as fat32 reference, then (as it is quite confusing).
    – zeppelin
    Nov 10, 2016 at 21:50
  • Also does you comment on files >4GB still apply ? What if you try to create a dummy 5GB file directly on the remote machine , like that: dd if=/dev/zero bs=1G count=5 > /mnt/mydisk/dummy.data ?
    – zeppelin
    Nov 10, 2016 at 21:53
  • Files under ~8GB most likely will successfully finish, but with size increasing it could fail at any time.
    – Abude
    Nov 10, 2016 at 22:05
  • Ok, but what if you try to create a really big file inplace with dd, say 32G in size? (to rule out a local issue)
    – zeppelin
    Nov 10, 2016 at 22:22
1

Assuming you are on a local or otherwise secure network, you can transmit the data directly with netcat or nc depending on your distro.

On the target machine:

nc -l 0.0.0.0 45999 | tar -xz

On the source machine:

tar -cz the_files_or_folder | nc <target IP> 45999

This will tar and compress all the desired files and directly pipe them (unencrypted hence the local or secure network) to the target machine over TCP, on the receiving side nc will be listening and pipe them to tar to decompress and unarchive. You can remove the z argument on both sending and receiving tar commands to omit compression and send the data as is.

Also on the target machine IP 0.0.0.0 will listen on all interfaces, this can be a bit more insecure if you have other interfaces, to avoid this just replace for the desired IP reachable by the sending machine.

1
  • One issue with this is if the connection gets broekn, you have to transfer everything again from the beginning.
    – GMaster
    Nov 9, 2016 at 23:49
0

error: error in file IO (code 11) at receiver.c(322) 1. is /folder writable for you?

rsync: connection unexpectedly closed (490 bytes received so far) [sender] 2. SSH dropped. can you ssh in? keyless or pw?

You said "I'm trying to copy files (about 100GB at once) over network from local." Hope I didnt' read it wrong. Target IP is non-routable IP (192.168.0.106)? May be you are on VPN.

rsync -avr * user@<remoteIP>:/folder

use --bwlimit to 'slow it down' if you get dropped from connection. use --partial to resume interrupted attempts

1
  • The commands in my question are working files for files under 8GB. not connectivity issue. thx
    – Abude
    Nov 10, 2016 at 17:24
0

Set up an FTP server on the machine with the data. You only have to run it as long as it takes to download the file, then you can shut it down. You could use a tool like wget to fetch it. That way if the download is interrupted it can be resumed without losing any time.

-2

Try to use dd over network.dd if=/dev/sda | ssh [email protected] "dd of=/dev/sdb" This makes a copy of SDA to sdb

5
  • hello, what do you mean by copy of sda to sdb? i don't want a perfect copy, files are just part of the partition on local machine.
    – Abude
    Nov 6, 2016 at 11:50
  • The dd command generate an exact copy of if. So if you try and if it works you can move the file and do want is to do
    – amarradi
    Nov 6, 2016 at 11:54
  • Sledgehammer, please meet nut. Nov 6, 2016 at 12:39
  • This is not what was asked (copy files, not copy a disk image). It won't work since a normal user can't write to /dev/sdb. Even if it did what was asked, it would be slow and brittle because of the use of dd. Nov 6, 2016 at 20:46
  • Oh sorry, in my way you can copy 100GB of Data (files or whatever). It is slow but it works
    – amarradi
    Nov 6, 2016 at 22:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .