Skip to content

LAMMPS

Building on DCS

Draft instructions for building on DCS with GPU support. Last tested with stable version (7 Aug 2019).

Load modules

 module load xl_r
 module load spectrum-mpi
 module load cuda/10.1 

Build LAMMPS GPU library

 cd src
 make lib-gpu args=" -a sm_70 -p double -b -c /usr/local/cuda-10.1"

Edit makefile

 cp MAKE/Makefile.mpi MAKE/Makefile.dcs

 #Add these lines to MAKE/Makefile.dcs 
 CUDA_HOME = /usr/local/cuda-10.1
 gpu_SYSINC =
 gpu_SYSLIB =  -lcudart -lcuda
 gpu_SYSPATH = -L$(CUDA_HOME)/lib64

Build LAMMPS

 make yes-asphere
 make yes-kspace
 make yes-gpu
 make dcs

Test

./lmp_dcs  -sf gpu -pk gpu 2 -in ../examples/flow/in.flow.pois

Output should list accelerators used

 - Using acceleration for lj/cut:
 -  with 1 proc(s) per device.
 --------------------------------------------------------------------------
 Device 0: Tesla V100-SXM2-16GB, 80 CUs, 15/16 GB, 1.5 GHZ

Running on the DCS

You must have passwordless SSH keys setup for mpirun to work.

More background information is available in the Slurm and DCS Supercomputer articles.

Create a file named run.sh with the following contents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#!/bin/bash -x

if [ "x$SLURM_NPROCS" = "x" ]
then
  if [ "x$SLURM_NTASKS_PER_NODE" = "x" ]
  then
    SLURM_NTASKS_PER_NODE=1
  fi
  SLURM_NPROCS=`expr $SLURM_JOB_NUM_NODES \* $SLURM_NTASKS_PER_NODE`
else
  if [ "x$SLURM_NTASKS_PER_NODE" = "x" ]
  then
    SLURM_NTASKS_PER_NODE=`expr $SLURM_NPROCS / $SLURM_JOB_NUM_NODES`
  fi
fi

srun hostname -s | sort -u > /tmp/hosts.$SLURM_JOB_ID
awk "{ print \$0 \"-ib slots=$SLURM_NTASKS_PER_NODE\"; }" /tmp/hosts.$SLURM_JOB_ID >/tmp/tmp.$SLURM_JOB_ID
mv /tmp/tmp.$SLURM_JOB_ID /tmp/hosts.$SLURM_JOB_ID

cat /tmp/hosts.$SLURM_JOB_ID

module load xl_r
module load spectrum-mpi
module load cuda/10.1

case=YOURPATH/examples/flow/in.flow.pois
exe=YOURPATH//src/lmp_dcs

mpirun -hostfile /tmp/hosts.$SLURM_JOB_ID -np $SLURM_NPROCS $exe -sf gpu -pk gpu $SLURM_NTASKS_PER_NODE -in $case > $SLURM_JOB_ID.out

rm /tmp/hosts.$SLURM_JOB_ID

Submit the job using sbatch, ex: sbatch --gres=gpu:4 -n 8 -N 2 -t numberOfMinutes -p nameOfJobQueue ./run.sh

In this example, 2 nodes will be used to run a total of 8 processes. Therefore, 4 processes will run on each node, each accessing its own GPU. Generally, each processes should have its own GPU for MPI applications.


Last update: June 15, 2020