Centre Commun de Calcul Intensif

Mèwbalaou

mewbalaou.univ-ag.fr

mewbalaou.univ-ag.fr



System Architecture and Configuration

mewbalaou.univ-ag.fr is an SMP machine with 12 1.15 GHz EV 7 processors and 12 Gbytes of shared memory, running the Tru64 Unix operating system.

Stay Informed

As a user of mewbalaou, it is imperative that you stay informed of changes to the machine's environment. Refer to this document frequently. In addition, important system status information is posted to the C3I's general bulletins.

You will also periodically receive email from C3I with information about mewbalaou. In order to insure that you receive this email, you should make sure that your email forwarding is set properly.

Access to mewbalaou

Getting an allocation on mewbalaou

There are two types of grants available on mewbalaou: development grants and production grants. Development grants are appropriate as precursors to large requests or for work which will exploit the unique architectural capabilities of mewbalaou, chiefly its large shared memory and memory bandwidth. Production grants are large awards for users with extensive computational requirements. Further information about applying for mewbalaou grants is available online.

Connecting to mewbalaou

Use ssh to connect to mewbalaou.
    ssh mewbalaou.univ-ag.fr -l username
You will next be prompted for your mewbalaou password. If you enter your password successfully you will be connected to mewbalaou.

Changing your mewbalaou password

Use the passwd command to change your mewbalaou password.

ACCOUNTING

Accounting procedures have not been determined yet.

Storing Files

File Systems

File systems are file storage spaces directly connected to a system. There are currently three such areas available to you on mewbalaou.
/uag/username

 

 
 
 
 
 
 
 

This is your home directory. The 'username' will be replaced by your userid. You can also refer to this directory as $HOME. You have a 1 Gbyte quota for your home directory. Your home directory is backed up.

/uag/big/username

 

 
 
 
 
 
 
 

This directory provides you access to a large amount of working disk space. The word 'username' will be replaced by your username. You can also refer to this directory as $BIG. There are no quotas and there is no wiper, so you must police your own disk usage. $BIG is not backed up. Thus, you should copy as soon as possible $BIG files that you want to your home system.

Transferring Files

Scp

The scp program can be used to transfer files between your remote machine and your mewbalaou home directory.

The format for the scp command is

    scp source-filename target-filename
where the filename on the remote system, whether it is the target or the source, must be specified as
    username@system:filename
For example, to copy a file to your home directory on mewbalaou when you are logged in to your home system use a command such as
    scp filename username@mewbalaou.univ-ag.fr:/uag/username/filename
If you are logged in to mewbalaou and you want to copy over a file from your home system to mewbalaou, use a command such as
    scp username@remote-system:filename  filename
For more information on the scp command, see the scp man page.

Scp is part of the ssh distribution. C3I provides a list of sites that distribute ssh.

Compilers

Compilers for Fortran90, C, Java, and Gnu and Compaq C++ are available.

Compile your program by executing one of the following sets of statements.

C programs: cc program-name.c
C++ programs: cxx program-name.C for HP compiler or c++ program-name for Gnu compiler
Fortran90 programs: f90 program-name.f90

Please note that the C compiler expects a .c extension (lower case) while the Compaq C++ compiler expects a .C (upper case) extension.

The make command executes /bin/make. To use Gnu make use the gmake command.

We recommend that you use the options

    -O -fast -tune ev7 -arch ev7
when you compile. You must use the options -tune ev7 and -arch ev7 if you compile on the front end because the front end processers are not the same as mewbalaou's compute processors.

Communication libraries

MPI

The MPI message passing library is available on mewbalaou. To enable your programs to use MPI, you must include the MPI header file in your source and link to the MPI libraries when you compile.

For Fortran, use this include directive in your source:

    include 'mpif.h'
and compile with a command like:
    f90 program.f90 -lmpi
For C, use this include directive:
    #include <mpi.h>
and compile with a command similar to:
    cc program.c -lmpi

OpenMP

The OpenMP library for shared memory communication is available on mewbalaou. MPI will use shared memory for communication on mewbalaou so you do not need to use OpenMP to do this.

If you want to use OpenMP you should use the -omp option when compiling Also, before you execute your program set the environment variable OMP_NUM_THREADS to ${PBS_PPN}. This variable is set by the system to the number of processors you request with your qsub comand. If you do this each thread will run on its own processor.

To run an OpenMP code we have found that you must increase the value of vmem and pvmem from their default values to at least 1 Gbyte. See the discussion of batch processing below for more information on qsub, vmem and pvmem.

Running a Job

Scheduling policies

The Portable Batch Scheduler (PBS) controls all access to mewbalaou's processors, for both batch and interactive jobs. PBS on mewbalaou currently has only one queue. Interactive and batch jobs compete in this queue for scheduling.

The priority for this queue is FIFO, but if a job will not fit either because its memory request or its processor request cannot be met, the next job in the queue that will fit will run.

Send email to support@cphpc3i.univ-ag.fr if you have computation needs that cannot be met by these scheduling policies.

mewbalaou's scheduling policies will probably change frequently. Please check this document often for policy and system updates.

Batch access

Use the qsub command to submit a job to PBS. A PBS job script consists of PBS directives, comments and executable commands.

A sample job script is

    #!/bin/csh
    #PBS -l walltime=5:00:00
    #PBS -l nodes=1:ppn=4
    #PBS -l vmem=4gb
    #PBS -l pvmem=4gb
    #PBS -j oe

    set echo

    #execute program
    dmpirun -np ${PBS_PPN} ./a.out

    #cleanup in case dmpirun failed
    mpiclean
The first line in the script cannot be a PBS directive. Any PBS directive in the first line is ignored. Here, the first line identifies which shell should be used.

The next five lines are PBS directives.

#PBS -l walltime=5:00:00
The first directive requests a wall clock time limit of 5 hours. Specify the time in the format HH:MM:SS. Only two digits can be used for hours, minutes and seconds. Do not use leading zeros. The default walltime is 30 minutes. The maximum time you can request is 168 hours (1 week).
#PBS -l nodes=1:ppn=4
The second directive requests 4 processors. The value for nodes must always be 1. The default value for ppn is 1, while the maximum is 16.
#PBS -l vmem=4gb
This directive specifies the maximum amount of memory your entire job can use at one time. The maximum value for vmem is 57 Gbytes, while the default is 3800 Mbytes.
#PBS -l pvmem=4gb
This directive specifies the maximum amount of memory any of your processes can request at one time. The maximum value for pvmem is 57 Gbytes, while the default is 3800 Mbytes.
#PBS -j oe
The final PBS directive combines your stdout and stderr output into one file, in this case stdout. This will make your program easier to debug.
The remaining lines in the script are comments or command lines.
set echo
This command causes your batch output to display each command next to its corresponding output. This will make your program easier to debug. If you are using the Bourne shell or one of its descendants use 'set -x' instead of 'set echo'.
Comment lines
The other lines in the sample script that begin with '#' are comment lines. The '#' for comments and PBS directives must begin in column one of your script file. The remaining lines in the sample script are executable commands.
dmpirun
The dmpirun command must be used to launch your executable if your executable is an MPI code. The PBS_PPN variable has its value set automatically from the value you specified for ppn. We recommend that you set the values of ppn and np in this way so that you will have one process on each of your processors. Dmpirun cannot read directly from stdin but you can use input redirection. If your executable is not an MPI code, you can just specify the executable name to run it.
mpiclean
The mpiclean command must be executed to clean up system resources properly after a failed dmpirun execution.
After you create your script you must make it executable with the chmod command.
    chmod 755 yourscript.job
Then you can submit it to PBS with the qsub command.
    qsub yourscript.job
Batch output, including your job's stdout and stderr output, is returned to the directory from which you issued the qsub comand after your job finishes.

You can also specify the PBS directives as command-line options to qsub. Thus, you could omit the PBS directives in the sample script above and submit the script with

    qsub -l walltime=5:00:00 -l nodes=1:ppn=4 -l vmem=4gb -l pvmem=4gb -j oe
Command line options override PBS directives included in your script.

The -M and -m options can be used to have the system send you email when your job undergoes specified state transitions. See the man page for qsub for more information on these options and other options to the qsub command.

Interactive access via qusb -I

A form of interactive access is available by using the -I option to qsub. An example command using qsub -I is
    qsub -I -l walltime=10:00 -l nodes=1:ppn=2
This command requests interactive access to 2 processors for 10 minutes.

The system will respond with a message similar to

    qsub: waiting for job 54.mewbalaou.psc.edu to start
Your qsub request will wait until it can be satisfied. If you want to cancel your request you should type ^C. You will have to try your interactive job at a later time. The qstat command can be used to show how many jobs are running on the system.

When your job starts you will receive the message

    qsub: job 54.mewbalaou.psc.edu ready
and then the OS command prompt. You can use the -M and -m options to qsub to have the system you email when your job has started.

At this point any commands you enter will be run on your processors as if you had entered them in a batch script. For example, to run an MPI code you must enter a dmpirun command.

Stdin, stdout, and stderr are all connected to your terminal, although you will still need to use input direction for your MPI code to read stdin.

When you are finished with your interactive session type ^D. The system will respond

    qsub: job 54.mewbalaou.psc.edu completed
When you use qsub -I you are charged for the total time that you hold your processors and your memory whether you are computing or not. Thus, as soon as you are done running executables you should type ^D.

Interactive access can be used to compile your mewbalaou programs.

Monitoring and Killing Jobs

Qstat

The qstat command is used to display the status of the PBS queue. It includes running and queued jobs. The -f and -a options to qstat provide you with more extensive status listings. See the man page for qstat for more details.

Qdel

The qdel command is used to kill queued and running jobs.

Qalter

The qalter command is used to alter queue options for queued and running jobs.

Reporting a Problem

You have several options for reporting problems on mewbalaou.

Acknowledgement in Publications

C3I requests that a copy of any publication (preprint or reprint) resulting from research done on C3I systems be sent to the C3I manager

Publications resulting from work done on mewbalaou should include a credit similar to

  The computations were performed on the HP GS1280 system at the 
Centre Commun de Calcul Intensif de l'Université des Antilles et de la Guyane.

Other Documentation



© Centre Commun de Calcul Intensif (C3I), Campus de Fouillole Guadeloupe, Université des Antilles et de la Guyane.
URL: http://www.univ-ag.fr/c3i/
Revised: Monday, 26-June-2003 20:59:15 EDT