Tag Info

Hot answers tagged

55

The common protocols HTTP, FTP and SFTP support range requests, so you can request part of a file. Note that this also requires server support, so it might or might not work in practice. You can use curl and the -r or --range option to specify the range and eventually just catting the files together. Example: curl -r 0-104857600 -o distro1.iso ...


30

You probably want to use the ServerAlive settings for this. They do not require any configuration on the server, and can be set on the command line if you wish. ssh -o ServerAliveInterval=5 -o ServerAliveCountMax=1 $HOST This will send a ssh keepalive message every 5 seconds, and if it comes time to send another keepalive, but a response to the last one ...


7

Creating an ha environment has a lot of caveats and is complicated, and often times depends on the actual software (e.g. creating a master-slave environment for mysql is different than for postfix0 If you want to get started and only want to have two systems and don't have time to configure all your daemons accordingly you should have a look at drbd, raid-1 ...


7

GNU Parallel does that and more (using ssh). It can even deal with mixed speed of machines, as it simply has a queue of jobs, that are started on the list of machines (e.g. one per CPU core). When one jobs finishes another one is started. So it does not divide the jobs into clusters before starting, but does it dynamically. Watch the intro videos to learn ...


6

You can't have multiple processes adding to the same tar archive (or any other usual archive format, compressed or not). Each file is stored contiguously, and there is no way to insert data in a file, only to append or overwrite, so continuing to write to a file that isn't the last one would overwrite subsequent files. If you know the file size in advance, ...


5

There's the rocks linux distro which is made for clustering, and is based on CentOS/RHEL. The strong point of rocks is that it'll for the most part manage and do a lot of the minutia for you. It'll do automatic installation and reinstallation, and if your computers can boot via PXE, the initial install will consist of PXE booting your nodes. If you have a ...


4

It isn't normal for mandb to run continuously. It is typical to run mandb once a day in a cron job, to perform maintenance task such as updating an index of installed man pages and building or trimming a cache of formatted man pages. The daily job should run in a few seconds, perhaps a few minutes if you have a lot of man pages and a slow disk. If the job ...


4

Using a compute cluster? Since you're asking about qsub I'm going to assume you're using GridEngine or some computer clustering product. qsub When you run the command qsub script_name.sh you're telling the cluster's head node that you want to submit a script to run on that server. So that server would need to have access to the same filesystem's that ...


3

You can use Redhat Cluster Suit for the same. Let's understand little bit of clustering There is a different cluster for every problem. Generally speaking though, there are two main problems that clusters try to resolve. Performance and High Availability. and As per your requirement ( will continue to work even if a node in the cluster becomes inoperable ...


3

I'm a little confused what exactly you want your cluster to do. It sounds a bit like all you want is starting programs and running them on different boxes. If that's the case, SSH/X-forwarding/NFS should do you ok. IMHO this doesn't really have anything to do with clustering, its simply remote controlling different machines. I really have to ask one thing ...


3

It really depends on what you are trying to accomplish, and what you mean by "Distributed Computing Cluster." I did a similar thing once in Uni using old machines and PVM that's the "Cluster" in the sense of a bunch of machines acting as one single computer to do parallel processing - think Beowulf clusters. Of course, you will need code that is written to ...


3

As already mentioned in the comments, since memory and CPU usage are negligible as you said, idle users won't hurt anyone. If you still want to get rid of them, you could automatically hunt down idle users and kick them off, either by e.g. a shell script you call in regular intervals via cron, or by setting ClientAliveInterval 1800 ClientAliveCountMax 0 ...


3

I suppose that when you say that myprogram writes in a fixed file, your mean that you cannot modify this program to make it write to another file (maybe you don't have the source code). I'm also pretty sure that what you want to do is not possible with NFS. Anyway the solution you mentioned using different mount points will work, but isn't very practical if ...


3

Ok try with this command line: pdsh -w n032 sed -i.bak -e '"\$aGATEWAY=10.0.10.1"' test/test1; With ssh it works. If it works locally (on the nodes) the simple command, it should work this line too. When you try without quote ', the shell locally (on the current shell on entry computer) will expand and transform for the node the line in: sed -i.bak -e ...


2

We have a small cluster that has openSUSE as its base distro, but I do not think it is too important. Ubuntu looks like a viable alternative and has quite a bit of documentation and community support. On top of linux, we run Sun Grid Engine (and our cluster even includes Mac OS machines pretty seamlessly), but slurm would probably work for a simple setup. ...


2

What you're talking about is called a Single System Image cluster, or sometimes a distributed shared memory system (in a more limiting context). There are some projects listed on the linked Wikipedia pages you should look into. I've used the SGI Altix cluster (the NUMAlink ones), and it can be quite powerful if you have a process that requires a huge ...


2

While @wnoise's answer is a the nicer solution, it might not be possible for you to implement it (i.e. do you administer the cluster?)... so, why not have a look at one of the 'cluster SSH' solutions @Chaleb mentioned here (pssh, pdsh, clusterssh, clusterit) or Fabric (also mentioned in this thread, by @Crankyadmin) to gather usage statistics. Add a ...


2

There are many "batch systems" that are designed to handle this sort of problem. One specifically tailored to handle "cycle stealing" from otherwise unoccupied systems is condor, a long running research project at the University of Wisconsin.


2

The rup command from the rstatclient package will poll all the machines on your subnet for information, including their load averages. The machines must be running rstatd to serve up that information, and I would tcpwrapper it to only respond to your admin desktops. You can also specify individual machines to collect data from. With rstatd running on the ...


2

Launch it with javaws (java webstart), e.g javaws ~/Downloads/ContestAppletProd.jnlp


2

It doesn't make any changes on the EMC storage. It only scans the fibre channel bus to see what world-wide names are visible through each local port. If it sees the same WWN through multiple ports, then it sets that storage device up for multipath I/O. Changes are made to emcp.conf in /kernel/drv, and those changes in turn effect how device paths are ...


2

ZFS is not a clustered file system, so what you talk about wouldn't actually work as described. In order to increase the capacity of a server's storage, you'd need to add more disks to the host, either in the chassis, or, if that's full, in a storage array. A host running ZFS can have up to 2^64 zpools, and each zpool can contain up to 2^64 vdevs, so you are ...


2

An approach that works fairly well for me... Connect one of those obsolete monitors you have lying around "just in case" to each of the small computers (RaspberryPi, etc.). Run a tiny, fast, RAM-based O/S like Puppy Linux (see how it works) on every computer. Setup passwordless (pre-shared password distribution) SSH between all computers. Install KVM ...


2

The kernel recognizes certain file formats that it can execute natively. This includes at least one binary format. Additionally, files that begin with #! (shebang) are considered scripts; for example, if a file is located at /path/to/script and begins with #!/bin/bash then the kernel executes /bin/bash /path/to/script arg1 arg2 when you invoke ...


2

You cannot have NFS as active-active cluster as far as I know. Now, for Active-Passive there are certain tools that help you achieve this. I'm more familiar with Solaris OS and Solaris Cluster that provide you the possibility to create a high-availability NFS share/shares depending on your objective. One thing that you will need is a shared storage that ...


2

This tutorial, titled: Smooth Active-Active NFS failover with Heartbeat and Pacemaker shows how to setup a NFS active/active using NFS, Heartbeat, & Pacemaker. You'll also probably want to take a look at these other links to familiarize yourself with the various technologies and terminologies. Global File System (GFS) Configuration Example - NFS Over ...


2

This is just one of the many ways in which NFS sucks. There is no way to do this just with NFS up to version 3. You are going to have to add on the functionality manually. This probably means: data replication or some shared storage IP takeover Some sort of heartbeat monitoring Cluster management There are packages out there to do all of this but you ...


2

I set up a redundant pair of NFS file servers in an "dual primary" active/active configuration using a load balancer, CentOS, NFS, DRBD (replication), GFS2 (clustered file system) and Pacemaker (cluster management). Here is a guide for this implementation at RackSpace, but it would be similar in any other environment. Howto setup Dual Primary DRBD on ...


2

I believe you can do this using VMware, at least it appear to be the case according to this article titled: Fence Device and Agent Information for Red Hat Enterprise Linux. As to setting it up I did find these 2 resources which are too long to include in an answer here so I'm only going to reference them. They come from a ProxMox tutorial (another ...


2

I'm assuming that you're using GridEngine as the clustering software when you submit this script to run. Something like this: $ qsub myscript.sh You can include environment variables to qsub that you want the resulting shells that get spawned on the HPC cluster nodes like so: $ qsub -v DISPLAY=$(hostname):0.0 myscript.sh This should "inject" the ...



Only top voted, non community-wiki answers of a minimum length are eligible