Tag Info

New answers tagged

2

The RAM in a computer is useful for two things: to store the memory of programs, and as a cache of recently-used disk content. On a typical healthy desktop system, about half the memory goes into each. You can check your memory usage with the free command; the “used” column of the “-/+ buffers/cache” is the figure for memory used for program data, and the ...


1

http://en.wikipedia.org/wiki/Asynchronous_I/O, a program is firing up requests for I/O, but does not wait on them. However it can still accept and process the I/O responses once they come.


2

There looks to be a single thread writing a little bit more than 4KB about 111 times per second. This is sufficient to keep your disk 100% busy (111 iops * 9 ms service time = 1 second of service per second = 100%). As there are no other processes writing on that disk (that partition actually), the wait queue is empty, all requests are processed immediately. ...


14

Izkata's comment revealed the answer: locale-specific comparisons. The sort command uses the locale indicated by the environment, whereas Python defaults to a byte order comparison. Comparing UTF-8 strings is harder than comparing byte strings. $ time (LC_ALL=C sort <numbers.txt >s2.txt) real 0m5.485s user 0m14.028s sys 0m0.404s How about ...


7

This is more of an extra analysis than an actual answer but it does seem to vary depending on the data being sorted. First, a base reading: $ printf "%s\n" {1..1000000} > numbers.txt $ time python sort.py <numbers.txt >s1.txt real 0m0.521s user 0m0.216s sys 0m0.100s $ time sort <numbers.txt >s2.txt real 0m3.708s user ...


5

Both of the implementations are in C, so a level playing field there. Coreutils sort apparently uses the mergesort algorithm. Mergesort does a fixed number of comparisons which increases logarithmically to the input size, i.e. big O(n log n). Python's sort uses a unique hybrid merge/insertion sort, timsort, which will do a variable number of comparisons ...


3

In your output, the point where the kernel is actually loaded is this one: Init version 2.86 booting Which is after 23 seconds. After that, init, a userspace process, takes over and begins configuration of the userspace, although this inevitably provokes activation of various kernel drivers, possibly including loading appropriate modules. You haven't ...


2

The speed doesn't decrease. It slowly approaches the real speed that was there from the start. Such copies normally involve buffers at the OS level and these need to be pushed out for real, what the application doesn't notice. Before they fill up, they are measuring how fast things get copied in the output buffer, and once that is filled your network speed ...


0

You can compress data and based on nature of files, this can considerably reduce the disk usage so you can get more data with actual less reads. In your case, compressing binary files, would not reduce storage as much as in case of plain text files but still it help your data located consecutively in disk so reading those shared library files with less ...


3

using awk: ps aux --sort=-%cpu | awk 'NR==1{print $2,$3,$11}NR>1{if($3!=0.0) print $2,$3,$11}' > some_file.txt the above code will give all program with non-zero cpu usage. will give you pid,%cpu, command_name if you want cpu usage greater than equal to 60 replace $3!=0.0 to $3>=60 I have saved the output in file some_file.txt. you can cat the ...


3

Have you taken a look at ionice? You may want to run updatedb/find with posix fadvise to inform the OS that you don't want these programs to dirty your page cache. That way, your cached disk i/o stays in memory. You can also rework your cron jobs to have these background processes pushed to times when your computations are not running. Additionally, you ...


1

RAID10 does not accelerate any form of O_*SYNC, at least not for small files. You're writing 512b at a time, and after each write forcing it out to disk (plus the metadata required to read it back, e.g., file size). That requires RAID10 writes to at least 2 disks, probably 4 (i.e., all your disks). And all those writes need to be completed before it can ...


-1

You're not doing "100b" (byte? blocks? what was your intention?) chunks, you're doing 512 byte chunks. That is always slow because the RAID is using 512k chunks. That means that for every 512 bytes the RAID system has to read a 512k chunk, update 512 bytes in that chunk, compute its parity, and write the data + parity out to disk. It has to do the update, ...


0

Your dd command gives me 73.7 kB/s - on a SSD. So yes I guess it's normal. Or rather, dd just isn't a good benchmark. RAID certainly does not do any speedups for small files. Access times still remain the same, and for a small file that's what will take most of the effort for HDDs, getting the read head to the physical address of the file in the first place ...


0

Use lftp, its much faster than rsync and its best for mirroring websites (many small files). It can also transfer in parallel using multiple connections: lftp -u username,password sftp://ip-address -e 'mirror --only-newer --no-dereference --parallel=5 /remote/path/ /destination/;quit' If one connection breaks it will reconnect and continue. If you break ...


20

As you say, use rsync: rsync -azP /var/www/html/txt/ username@ip-address:/var/www/html/txt The options are: -a : enables archive mode, which preserves symbolic links and works recursively -z : compress the data transfer to minimise network usage -P : to display a progress bar and enables you to resume partial transfers As @aim says in his answer, make ...


6

Just use rsync over ssh! rsync -av username@ip:/var/www/html/txt /var/www/html/ From the man page: -a, --archive : This is equivalent to -rlptgoD. It is a quick way of saying you want recursion and want to preserve almost everything (with -H being a notable omission). The only exception to the above equivalence is when --files-from is ...


1

This really seems like the kind of question you should answer yourself: keep statistics while the machine is operating normally for a while (e.g. a week), and this will give you some idea of what the parameters of normal operation should be. Your snapshot shows you more or less maxing out one of two cores. Since your main process is multi-threaded, this ...


5

Compression and decompression take time, so it's a trade off. If you have a slow network connection, it's worth taking the time for compression. If you have a 1Gbps connection between two nodes, you're probably going to lose more time to compression than you'll gain from transferring less data.



Top 50 recent answers are included