Skip to content

LAMMPS

Building on DCS

Draft instructions for building on DCS with GPU support. Last tested with stable version (7 Aug 2019).

Load modules

module load xl_r module load spectrum-mpi module load cuda/10.1

Build LAMMPS GPU library

cd src make lib-gpu args=" -a sm_70 -p double -b -c /usr/local/cuda-10.1"

Edit makefile

cp MAKE/Makefile.mpi MAKE/Makefile.dcs

#Add these lines to MAKE/Makefile.dcs CUDA_HOME = /usr/local/cuda-10.1 gpu_SYSINC = gpu_SYSLIB =  -lcudart -lcuda gpu_SYSPATH = -L$(CUDA_HOME)/lib64

Build LAMMPS

make yes-asphere make yes-kspace make yes-gpu make dcs

Test

./lmp_dcs  -sf gpu -pk gpu 2 -in ../examples/flow/in.flow.pois

Output should list accelerators used

- Using acceleration for lj/cut: -  with 1 proc(s) per device. -------------------------------------------------------------------------- Device 0: Tesla V100-SXM2-16GB, 80 CUs, 15/16 GB, 1.5 GHZ

Running on the DCS

You must have passwordless SSH keys setup for mpirun to work.

More background information is available in the Slurm and DCS Supercomputer articles.

Create a file named run.sh with the following contents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#!/bin/bash -x

if [ "x$SLURM_NPROCS" = "x" ]
then
  if [ "x$SLURM_NTASKS_PER_NODE" = "x" ]
  then
    SLURM_NTASKS_PER_NODE=1
  fi
  SLURM_NPROCS=`expr $SLURM_JOB_NUM_NODES \* $SLURM_NTASKS_PER_NODE`
else
  if [ "x$SLURM_NTASKS_PER_NODE" = "x" ]
  then
    SLURM_NTASKS_PER_NODE=`expr $SLURM_NPROCS / $SLURM_JOB_NUM_NODES`
  fi
fi

srun hostname -s | sort -u > /tmp/hosts.$SLURM_JOB_ID
awk "{ print \$0 \"-ib slots=$SLURM_NTASKS_PER_NODE\"; }" /tmp/hosts.$SLURM_JOB_ID >/tmp/tmp.$SLURM_JOB_ID
mv /tmp/tmp.$SLURM_JOB_ID /tmp/hosts.$SLURM_JOB_ID

cat /tmp/hosts.$SLURM_JOB_ID

module load xl_r
module load spectrum-mpi
module load cuda/10.1

case=YOURPATH/examples/flow/in.flow.pois
exe=YOURPATH//src/lmp_dcs

mpirun -hostfile /tmp/hosts.$SLURM_JOB_ID -np $SLURM_NPROCS $exe -sf gpu -pk gpu $SLURM_NTASKS_PER_NODE -in $case > $SLURM_JOB_ID.out

rm /tmp/hosts.$SLURM_JOB_ID

Submit the job using sbatch, ex: sbatch --gres=gpu:4 -n 8 -N 2 -t numberOfMinutes -p nameOfJobQueue ./run.sh

In this example, 2 nodes will be used to run a total of 8 processes. Therefore, 4 processes will run on each node, each accessing its own GPU. Generally, each processes should have its own GPU for MPI applications.


Last update: May 18, 2020