Matsim FAQ

---

General frequently asked questions about the Matsim cluster, this includes matsim and matsim-ib:

Table of Contents

How do I get an account on matsim?

Matsim is an owner-based resource. Accounts on matsim must be approved by Anter El-Azab

Why the two seperate clusters?

Because the hardware and backbone are so seperate we felt it would be easier for the end user to seperate the nodes into two clusters: one with ethernet only and one with infiniband.

What kind of performance should I expect?

Performance of high performance computing clusters is often measured by several benchmarks. The first statisic that people often qoute is the thoretical performance which is just a raw calculation that mutliples number of cores by number of operations. The next test is the HPL (High Performance Linpack) test. This test is not representative or real world problems and is completely dependent on the users ability to optimize the alogrythm and time invested into finding the perfect parameters to achieve a large performance. It is often a great achievement to get with 60-70% of the theoretical value. Note that this caluclations and performance marks do not include the cores on the headnode as these should general be reserved for special purpose. The break down is as follows

	matsim	matsim-ib
Theoretical Performance	230 GFLOPS	1.2 TFLOPS
Achieved Performance	70 GFLOPS	700 GFLOPS

To give a sense of scale as of Jan 30th, 2008 the HPC @ FSU cluster has achieved a mesured performnace of 2.1TFLOPS with HPL.

How do I gain access to the matsim cluster if I am not in DSL or on the SCS floor?

The SCS network is behind a firewall so to gain access to matsim you must first ssh to pamd.scs.fsu.edu and then ssh from pamd to matsim.

I had an account when matsim was in engineering, where are my files?

The files are located on matsim:/export/old-matsim, however because the uid's are different you need to contact ops@scs.fsu.edu and let them know what files are yours so that can give them the correct permissions to allow you to access them. As of initial install the these files are not automatically visible form matsim-ib and would need to be scp'd over.

What MPI implementations are available?

Because of the two different networking interfaces between the clusters, the same environments are not available in both. On matsim-ib you have the choice of OpenMPI?, MVAPICH, MVAPICH2 (mvapich and mvapich2 are infiniband optimized versions of MPICH and MPICH2). However on matsim only OpenMPI? is currently working (mvapich and mvapich2 are installed and can compile however the programs will not run on matsim). OpenMPI? is the preferred implementation because it is scheduler friendly and is easier to administer, and is transparent across the clusters. There are future plans to provide support for mpich and mpich2 but these are secondary projects right now.

What are the performance differences is MPI implentations?

The differences in MPI implementation only depends on the message passing protocols and as such will affect programs that reley heavily on MPI communications the most. The two numbers that are traditionally cited are the latency (time in overhead begin passing a message) and bandwidth. I have attached below graphs that layout the performances. The openmpi-tcp reflect the performances you will get on matsim, while everything is in regards to matsim-ib

How do I submit parallel jobs using SGE?

Each of the mpi implementations will often have its own special configurations which is why we create a seperate parallel environment for each. Hence if you wanted to submit two jobs that take 4 processors, one using mvapich and one using openmpi, you would type qsub -pe mvapich 4 myjob.sh, followed by qsub -pe openmpi 4

How do I use one MPI implementation above another?

Each of the mpi implementations have overlapping command names and hence can't be in your path at the same time. For this reason you should set your mpi implementation every time you would like to use it. There is a set of scripts that are located in /etc/mpivars that setup your particular implementation that are names by

<implementation>_<compiler>_<version>.c(sh)

(i.e. if I am using bash and I would like to use OpenMPI? 1.2.4 with the gcc compiler, I would type

source /etc/mpivars/openmpi_gcc_1.2.4.sh

)

-- JazcekBraden - 30 Jan 2008

• matsim-bandwidths.pdf: bandwidths of different mpi implementations on matsim

• matsim-latency.pdf: Latencies of MPI on matsim