Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I found a way in Windows to do such thing

echo "This is just a sample line appended  to create a big file. " > dummy.txt
for /L %i in (1,1,21) do type dummy.txt >> dummy.txt

http://www.windows-commandline.com/how-to-create-large-dummy-file/

Is there a way in UNIX to copy a file, append and then repeat the process? Something like for .. cat file1.txt > file1.txt?

share|improve this question
    
Why copy the file and append instead of just appending ? – 123 Mar 11 at 14:56
    
@123 append is good, but how to do the loop? – Thomas Lee Mar 11 at 14:59
4  
for i in {1..1000000};do echo "string" >> file;done in bash. – 123 Mar 11 at 15:00
9  
Does it have to be a text file? You can make any size of file from /dev/zero or /dev/urandom. – RealSkeptic Mar 11 at 15:06
2  
I'd expect type file >> file to run in an infinite loop (at least as soon as it's sufficiently large that it doesn't fit in a buffer). – Stéphane Chazelas Mar 11 at 15:14
up vote 27 down vote accepted
yes "Some text" | head -n 100000 > large-file

With csh/tcsh:

repeat 10000 echo some test > large-file

With zsh:

{repeat 10000 echo some test} > large-file

On GNU systems, see also:

seq 100000 > large-file

Or:

truncate -s 10T large-file

(creates a 10TiB sparse file (very large but doesn't take any space on disk)) and the other alternatives discussed at "Create a test file with lots of zero bytes".


Doing cat file >> file would be a bad idea.

First, it doesn't work with some cat implementations that refuse to read files that are the same as their output file. But even if you work around it by doing cat file | cat >> file, if file is larger than cat's internal buffer, that would cause cat to run in an infinite loop as it would end up reading the data that it has written earlier.

On file systems backed by a rotational hard drive, it would be pretty inefficient as well (after reaching a size greater than would possibly be cached in memory) as the drive would need to go back and forth between the location where to read the data, and that where to write it.

share|improve this answer
18  
Or dd if=/dev/zero of=large-file bs=1024 count=1024 for a 1MB file – Doug O'Neal Mar 11 at 15:13
6  
@DougO'Neal I find dd if=/dev/zero of=test bs=1M count=1 to be clearer. – 123 Mar 11 at 15:27
3  
@DougO'Neal, see Create a test file with lots of zero bytes – Stéphane Chazelas Mar 11 at 15:31
1  
I appreciate Stéphane's answers but I think of Doug's as the typical one. – roberto tomás Mar 12 at 1:50
2  
@robertotomás yes, everyone uses dd, but I have never understood why. In fact, I think I've only ever used it to read an MBR or similar fringe tasks. In my experience, other tools are faster, simpler and safer for the vast majority of cases where people use dd. I think this is one of those cases where common != optimal, like sudo su or cat file | grep foo. – terdon Mar 12 at 17:48

You can create a large file on Solaris using:

mkfile 10g /path/to/file

Another way which works on Solaris (and Linux):

truncate -s 10g /path/to file

It is also possible to use:

dd if=/dev/zero of=/path/to/file bs=1048576 count=10240
share|improve this answer
    
dd one is like a terabyte – 123 Mar 11 at 15:23
1  
Define "a large file" :-) But I edited since the other samples all state 10g... – Lambert Mar 11 at 15:27
    
truncate Can't execute 'truncate'. No such file or directory Truncate seems to be Linux only. – schily Mar 11 at 17:19
    
truncate does exist on Solaris 11.2+ – Lambert Mar 11 at 19:41

This will keep going until you CTRL-C:

yes This is stuff that I want to put into my file... >> dummy.txt

Be careful though, because you can get a hundreds of thousands of lines/second...

From man yes:

yes - output a string repeatedly until killed
share|improve this answer

If I understand you correctly, you are looking for something like:

echo "test line" > file;
for i in {1..21}; do echo "test line" >> file; done

That will create a file with 22 repetitions of "test line". If you want a specific file size, you can use something like this (on Linux). 1024 is one kilobyte:

while [ $(stat -c "%s" file) -le 1024 ]; do echo "test line" >> file; done

Personally, when I want to create a large file, I use two files and cat one into the other. You can repeat the process until you reach the desired size (1MB here):

echo "test line" > file;
while [ $(stat -c "%s" file) -le 1048576 ]; do 
    cat file >> newfile
    cat newfile >> file
done

Note that this solution will often exceed the desired size because if the file is under the limit, everything will be catted into it again.

Finally, if all you want is a file of the desired size and don't need it to actually contain anything, you ca use truncate:

truncate -s 1M file
share|improve this answer
1  
Does cating the file actually have any advantage to just appending though ? It would seem as though it would take longer as it has to fork two processes every loop and also move the entire contents multiple times. – 123 Mar 11 at 15:14
1  
@123 speed. The cat approach is much, much faster. It only makes sense for creating huge files but that created a 545M file in 10 seconds on my machine. The same while loop with echo "test line" >> file created a 96K file in the same amount of time. – terdon Mar 11 at 17:15

The fastest way possible to create a big file in a Linux system is fallocate:

sudo fallocate -l 2G bigfile

fallocate manipulates the files system, and does not actually writes to the data sectors by default, and as such is extremely fast. The downside it is that it has to be run as root.

Running it successively in a loop, you can fill the biggest of filesystems in a matter of seconds.

From man fallocate

fallocate is used to manipulate the allocated disk space for a file, either to deallocate or preallocate it.
For filesystems which support the fallocate system call, preallocation is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks. This is much faster than creating a file by filling it with zeros.
Supported for XFS (since Linux 2.6.38), ext4 (since Linux 3.0), Btrfs (since Linux 3.7) and tmpfs (since Linux 3.5).

share|improve this answer
    
This should be the accepted answer. Easy and fast. – ardochhigh Mar 13 at 20:54

By piping the contents of /dev/urandom to head you can redirect the output to a file, so :

 cat /dev/urandom | head --bytes=100 >> foo.bar

Will give you a file with 100 bytes of garbage.

share|improve this answer
echo "This is just a sample line appended  to create a big file. " > dummy.txt
i=1
while [ $i -le 21 ]
do
  cat dummy.txt >> bigfile
  cat bigfile > dummy.txt
  (( i++ ))
done

same effect of your windows script but in bash, you can not concatenate a file to itself, directly.

share|improve this answer
    
Apart from forgetting the .txt extension, you're leaving 2 big files at the end. – ott-- Mar 12 at 13:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.