FAQ

How do I get an account on acm?

If you already have an SCS account access can be requested by going to https://tsg.scs.fsu.edu/scsum, logging in, selecting request cluster access, and select one of the PI's as a sponsor. If you do not already have an SCS account, you can request an SCS sponsored account by going to the mentioned page and clicking request an scs account, selecting sponsored account then selecting an SCS person who will act as the sponsor (the account then requires authorization from them).

How do I log into acm?

Access to acm can only be achieved through the head node, i.e. $ ssh acm. From there you can gain access to all other of acm's resources. Please note if you are not on SCS network you must first go through pamd to get past our firewall (i.e. $ssh pamd.scs.fsu.edu log in and from there $ssh acm).

How do I compile programs?

Compiling of programs should be done on the head node. The system will set the default path so that you have access to all the available compilers (at the time of writing this includes gnu, pgi, and NVIDIA). Of note for compilers that are licensed (pgi) only one compilation can happen at a time, which means the situation can occur that two people may be trying to compiler at the same time. When this happens just wait a little bit for one of the compilations to finish and then recompile.

How do I run programs?

Since the head node is provides many purpose programs should never be run directly on the head node. Instead all programs should be submitted via the queueing system (SGE). Batch style jobs should be run using qsub command, and if interactive login is required the qlogin command will allow for gaining access to particular nodes, while ssh between nodes in not currently disabled, please refrain from doing this as it does not inform the scheduler of the resource as being occupied. Technical tips of how to use SGE can be found here and here as well as usefull hints can be found at the offiicial documentation site http://gridengine.sunsource.net/documentation.html

How do I login to a specific node

There may be some testing that you need to do in an interactive manner such as compiling against a specific hardware. This is where the qlogin functionality comes into play. There currently are three different types of queues that are specific to the different types of hardware, acm_gts, acm_tesla, acm_tesla_double which correspond to the machines with the 8500 GTS cards, the single precision tesla, and the double precision tesla (when they become available). By typing

qlogin -q <queue-name>

you will be given a login on a node of that type. If you need to log into a specific node you can use the form

qlogin -q <queue-name>@<machine-name>

. For example to log into any node with a 8800GTS card you can type qlogin -q acm_gts or to get a log in on acm015 you could type qlogin -q acm_gts@acm015.scs.fsu.edu

MPI

What version on MPI are available?

As part of a standard OFED install mvapich, mvapich2 and openmpi are installed (mvapich and mvapich2 are infiniband optimized versions of mpich and mpich2 respectively). Each of these are compiled against all available compilers (gnu and pgi) which actually results in six different version of mpi installed.

What are the differences between the different MPI implementations?

For the most part all these mpi implementations are the same and provide the same functionality but in slightly different ways. OpenMPI? provides many utilities that make it more portable (can be taken from cluster to cluster) and dynamic (many options can be specified at run time as to not have to be recompiled) and is the suggested development platform. However occassionally some code will be locked into a particular implementation as well as some people are interested in performance enhancements provided by mvapich and mvapich2. These are available but be warned the biggest problem with these two implementation is that they are not very scheduler friendly and are known to leave processes running if a job dies unexpectedly.

What are the performance differences between different MPI's?

At the bottom of [[ClusterAcm][ this page] there is a series of graphs that describe the difference in performance. The large png images display two grids: the left one for latency and the right one for bandwidth from each machine to each machine in the cluster (hence the 22x22 grid). There is also a text document that plots typical bandwidth/latency vs message size curves for direct comparisons for the different mpi implementations.

How do I use the different MPI implementations?

Each MPI implementation names all of their programs the same things so it is disadvantages to have them defined simultaneuously. There are MPI environmental setup scripts located in /etc/mpivars which follow the naming convention of _-.(c)sh. So case in point if I use bash and would like to use openmpi 1.2.5 compiled for gcc, I would upon log in type $ source /etc/mpivars/openmpi_gcc-1.2.5.sh. It is always a good idea to unset your mpi environments by logging out and back in when swtiching between different implementations. Also if there is a particular implementation that you would like to use by default this can be set by the mpi-selector-menu command.

How do I run MPI jobs?

MPI jobs should be batch submitted to the queueing system by using the pe flag to select a parallel environment. To see a list of available parallel environments type

qconf -spl

. This currently includes

• make
• make_gts_fu
• make_gts_rr
• make_tesla_fu
• make_tesla_rr

The make environment is the default one provided by SGE, it includes all compute hosts and assigns slots in a round robin fashion (evenly distributes the jobs on all hosts). The gts and tesla queues only allow processes to be submitted to the nodes that have gts cards (acm004-acm020) or tesla cards (acm000-acm003) respectively. The fu and rr endings stand for fill_up and round_robin respectively, which decribe the way the processes are assigned to the nodes. Round robin evenly distributes all the jobs while fill up fills all available slots on a particular machine before placing jobs on other nodes.

The context is qsub -pe as an example if you wanted to run a job with 16 cores you would type

qsub -pe make 16 job.sh

Currently there is nothing in place to prevent you from using mpirun to run jobs, however this technique does no inform the scheduler about the resources occupied making the scheduler useless.

How fast is acm?

The standardized test that is used to compare computation clusters is called HPL (high performance linkpack), which essentially solves a dense matrix on many machines by breaking it up into smaller chunks. Unfortunately this hasn't been extended into GPU resources (a possible goal in the near future?) and there is no way to perform a direct comparison

GPU's

Unfortuneately there is currently no standardized test with which to demonstrate the high performance scalability for GPU's

CPU's

Using the standard HPL tests the acm cluster has achieved an R(peak) of about 300GFLOPS using just the CPU alone. The R(theoretical) of the this cluster using the cpu's alone is found by NUM CORES x CLOCK SPEED x NUM-FLOPS/cycle which is 84 * 2.4 Ghz * Num of FLOP/cycle 202 GFLOPS * Number of floating operations per cycle (I don't know this number off hand but it is between 2 and 4). This means the R(theoretical) is expected to be somewhere between 404 and 808 GFLOPS. In general is tough to even achieve 60%-70% of R(theoretical) so this number is expected. To give a sense of scale, as of October 2007 the HPC at FSU achieved 2.2 TFLOPS with the same tests.

Ganglia and Load monitoring

For those who are not on the SCS network and are interested in trying to view the ganglia graphs of system usage. You can go to http://watcher.scs.fsu.edu/ganglia and then click on the acm cluster to get an overview of it's usage