Hot answers tagged

59

The common protocols HTTP, FTP and SFTP support range requests, so you can request part of a file. Note that this also requires server support, so it might or might not work in practice. You can use curl and the -r or --range option to specify the range and eventually just catting the files together. Example: curl -r 0-104857600 -o distro1.iso ...


48

You probably want to use the ServerAlive settings for this. They do not require any configuration on the server, and can be set on the command line if you wish. ssh -o ServerAliveInterval=5 -o ServerAliveCountMax=1 $HOST This will send a ssh keepalive message every 5 seconds, and if it comes time to send another keepalive, but a response to the last one ...


7

GNU Parallel does that and more (using ssh). It can even deal with mixed speed of machines, as it simply has a queue of jobs, that are started on the list of machines (e.g. one per CPU core). When one jobs finishes another one is started. So it does not divide the jobs into clusters before starting, but does it dynamically. Watch the intro videos to learn ...


7

Creating an ha environment has a lot of caveats and is complicated, and often times depends on the actual software (e.g. creating a master-slave environment for mysql is different than for postfix0 If you want to get started and only want to have two systems and don't have time to configure all your daemons accordingly you should have a look at drbd, raid-1 ...


6

You can't have multiple processes adding to the same tar archive (or any other usual archive format, compressed or not). Each file is stored contiguously, and there is no way to insert data in a file, only to append or overwrite, so continuing to write to a file that isn't the last one would overwrite subsequent files. If you know the file size in advance, ...


5

This tutorial, titled: Smooth Active-Active NFS failover with Heartbeat and Pacemaker shows how to setup a NFS active/active using NFS, Heartbeat, & Pacemaker. You'll also probably want to take a look at these other links to familiarize yourself with the various technologies and terminologies. Global File System (GFS) Configuration Example - NFS Over ...


5

An approach that works fairly well for me... Connect one of those obsolete monitors you have lying around "just in case" to each of the small computers (RaspberryPi, etc.). Run a tiny, fast, RAM-based O/S like Puppy Linux (see how it works) on every computer. Setup passwordless (pre-shared password distribution) SSH between all computers. Install KVM ...


5

There's the rocks linux distro which is made for clustering, and is based on CentOS/RHEL. The strong point of rocks is that it'll for the most part manage and do a lot of the minutia for you. It'll do automatic installation and reinstallation, and if your computers can boot via PXE, the initial install will consist of PXE booting your nodes. If you have a ...


5

Launch it with javaws (java webstart), e.g javaws ~/Downloads/ContestAppletProd.jnlp


5

It isn't normal for mandb to run continuously. It is typical to run mandb once a day in a cron job, to perform maintenance task such as updating an index of installed man pages and building or trimming a cache of formatted man pages. The daily job should run in a few seconds, perhaps a few minutes if you have a lot of man pages and a slow disk. If the job ...


4

It really depends on what you are trying to accomplish, and what you mean by "Distributed Computing Cluster." I did a similar thing once in Uni using old machines and PVM that's the "Cluster" in the sense of a bunch of machines acting as one single computer to do parallel processing - think Beowulf clusters. Of course, you will need code that is written to ...


4

Using a compute cluster? Since you're asking about qsub I'm going to assume you're using GridEngine or some computer clustering product. qsub When you run the command qsub script_name.sh you're telling the cluster's head node that you want to submit a script to run on that server. So that server would need to have access to the same filesystem's that ...


3

As already mentioned in the comments, since memory and CPU usage are negligible as you said, idle users won't hurt anyone. If you still want to get rid of them, you could automatically hunt down idle users and kick them off, either by e.g. a shell script you call in regular intervals via cron, or by setting ClientAliveInterval 1800 ClientAliveCountMax 0 ...


3

I suppose that when you say that myprogram writes in a fixed file, your mean that you cannot modify this program to make it write to another file (maybe you don't have the source code). I'm also pretty sure that what you want to do is not possible with NFS. Anyway the solution you mentioned using different mount points will work, but isn't very practical if ...


3

This is just one of the many ways in which NFS sucks. There is no way to do this just with NFS up to version 3. You are going to have to add on the functionality manually. This probably means: data replication or some shared storage IP takeover Some sort of heartbeat monitoring Cluster management There are packages out there to do all of this but you ...


3

Ok try with this command line: pdsh -w n032 sed -i.bak -e '"\$aGATEWAY=10.0.10.1"' test/test1; With ssh it works. If it works locally (on the nodes) the simple command, it should work this line too. When you try without quote ', the shell locally (on the current shell on entry computer) will expand and transform for the node the line in: sed -i.bak -e ...


3

I'm guessing you have added qsub to your $PATH in the .bashrc or .profile files of your user on the cluster. Those are not read when executing a command via ssh. It should work fine if you use the full path to qsub: ssh cluster '/usr/local/bin/qsub /home/user/casedir/run_script.sge' Obviously, you'll need to change /usr/local/bin/qsub to whatever the ...


3

Simple way to check the ssh service is up and running locally on the guest: ssh localhost alternatively: netstat -an | grep -w LISTEN | grep -w 22 If not installed, have a look to http://askubuntu.com/questions/51925/how-do-i-configure-a-new-ubuntu-installation-to-accept-ssh-connections


3

While @wnoise's answer is a the nicer solution, it might not be possible for you to implement it (i.e. do you administer the cluster?)... so, why not have a look at one of the 'cluster SSH' solutions @Chaleb mentioned here (pssh, pdsh, clusterssh, clusterit) or Fabric (also mentioned in this thread, by @Crankyadmin) to gather usage statistics. Add a ...


3

The rup command from the rstatclient package will poll all the machines on your subnet for information, including their load averages. The machines must be running rstatd to serve up that information, and I would tcpwrapper it to only respond to your admin desktops. You can also specify individual machines to collect data from. With rstatd running on the ...


3

I'm a little confused what exactly you want your cluster to do. It sounds a bit like all you want is starting programs and running them on different boxes. If that's the case, SSH/X-forwarding/NFS should do you ok. IMHO this doesn't really have anything to do with clustering, its simply remote controlling different machines. I really have to ask one thing ...


3

You can use Redhat Cluster Suit for the same. Let's understand little bit of clustering There is a different cluster for every problem. Generally speaking though, there are two main problems that clusters try to resolve. Performance and High Availability. and As per your requirement ( will continue to work even if a node in the cluster becomes inoperable ...


3

ZFS is not a clustered file system, so what you talk about wouldn't actually work as described. In order to increase the capacity of a server's storage, you'd need to add more disks to the host, either in the chassis, or, if that's full, in a storage array. A host running ZFS can have up to 2^64 zpools, and each zpool can contain up to 2^64 vdevs, so you are ...


3

Ceph and GlusterFS is where the clustered FS technology is currently going. Since I'm not familiar with GlusterFS I'll speak on the features of Ceph. Ceph scales horizontally; the more low-end nodes you add to it the better the performance. Unlike GlusterFS this is a primary benefit to Ceph as it shards the data. However, Ceph is in active development ...


2

It doesn't make any changes on the EMC storage. It only scans the fibre channel bus to see what world-wide names are visible through each local port. If it sees the same WWN through multiple ports, then it sets that storage device up for multipath I/O. Changes are made to emcp.conf in /kernel/drv, and those changes in turn effect how device paths are ...


2

The kernel recognizes certain file formats that it can execute natively. This includes at least one binary format. Additionally, files that begin with #! (shebang) are considered scripts; for example, if a file is located at /path/to/script and begins with #!/bin/bash then the kernel executes /bin/bash /path/to/script arg1 arg2 when you invoke ...


2

If SNMPD is running you can query the load-values of these machines with a simple snmpget. If you script that you can sort these with the load value and output the four lowest...


2

There are many "batch systems" that are designed to handle this sort of problem. One specifically tailored to handle "cycle stealing" from otherwise unoccupied systems is condor, a long running research project at the University of Wisconsin.


2

We have a small cluster that has openSUSE as its base distro, but I do not think it is too important. Ubuntu looks like a viable alternative and has quite a bit of documentation and community support. On top of linux, we run Sun Grid Engine (and our cluster even includes Mac OS machines pretty seamlessly), but slurm would probably work for a simple setup. ...


2

What you're talking about is called a Single System Image cluster, or sometimes a distributed shared memory system (in a more limiting context). There are some projects listed on the linked Wikipedia pages you should look into. I've used the SGI Altix cluster (the NUMAlink ones), and it can be quite powerful if you have a process that requires a huge ...



Only top voted, non community-wiki answers of a minimum length are eligible