Programming on Franklin: Other Topics

• Vendor Documentation
• Fortran Hybrid MPI/OpenMP Example
• C/MPI Main Calling Fortran Subroutine
• C++/MPI Main Calling Fortran Subroutine
• C/MPI Main Calling Fortran/OpenMP Subroutine

Vendor Documentation on Interlanguage Programming and OpenMP

Please see the PGI User's Guide (PDF), chapter 10, "Inter-Language Calling."

This chapter describes inter-language calling conventions
for C, C++ and Fortran programs using the PGI compilers.
The following sections describe how to call a Fortran function or
subroutine from a C or C++ program and how to cal a C or C++ function
from a Fortran program.

Table 10-1 covers Fortran and C/C++ data type compatibility, and Table 10-2 describe complex type representations. Section 10.8 is an example of a Fortran main program calling a C function, and Section 10.9 is an example of a C main program calling a Fortran function. Section 10.12 is an example of a Fortran main program calling a C++ function, and Section 10.13 is an example of a C++ main program calling a Fortran function. The examples include invocations of the pgcc, pgCC and pgf95 compilers.

OpenMP for Fortran is described in Chapter 5 of the PGI User's Guide, and OpenMP for C/C++ is described in Chapter 6. The environment variable OMP_NUM_THREADS is used to control the number of threads in an OpenMP section. Specifying more threads than processors may result in performance degradation.

For small, serial codes to be run on the service nodes (e.g., login nodes) with a standard SuSE Linux environment, the examples in the PGI User's Guide mentioned above should be followed.

Fortran Hybrid MPI/OpenMP Example

This example is based on a code at the OpenMP.org site. The code "solves a finite difference discretization of Helmholtz equation...using Jacobi iterative method." This code has been modified to use a constant set of input values and to compute ther over-relaxation parameter based on the MPI rank. The code is run with one MPI task per node. With the default one thread per task, the time for the computation is about twice that with two threads per task. Because Franklin nodes only have two cores, allowing more than two threads with one MPI task per node does not improve performance.

Fortran Source Code

Batch Job Script

% cat runjac

#PBS -N jac
#PBS -q debug
#PBS -l mppwidth=4
#PBS -l mppnppn=1
#PBS -l mppdepth=2
#PBS -l walltime=00:10:00
#PBS -e jacobijob.out
#PBS -j eo

cd $PBS_O_WORKDIR

ftn -o jac -mp=nonuma -Minfo=mp jac-openmp.f

setenv OMP_NUM_THREADS 2
time aprun -N 1 -n 4 ./jac

Output File

C/MPI Main Calling Fortran Subroutine

Here is a full example of a parallel C code using MPI which calls a Fortran subroutine. The Fortran subroutine sets a number of data values based on the MPI task ID. The code is compiled and run using the batch system.

C Source Code

% cat cmainmpi.c

#include "mpi.h"

int main(int argc, char* argv[]) {
   char bool1, letter1;
   int numint1, numint2;
   float numfloat1;
   double numdoub1;
   short numshor1;
   extern void forts_();
   int myid, numprocs;
 
   MPI_Init(&argc,&argv);
   MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
   MPI_Comm_rank(MPI_COMM_WORLD,&myid); 

   forts_(&bool1, &letter1, &numint1, &numint2, &numfloat1,
          &numdoub1, &numshor1, &myid, 1);

   printf(" Hello from proc %d of %d; data values: %s %c %d %d %3.1f %.0f %d\n",
      myid, numprocs, bool1?"TRUE":"FALSE", letter1, numint1,
      numint2, numfloat1, numdoub1, numshor1); 

   MPI_Finalize();
}

Fortran Source Code

% cat forts.f

      subroutine forts(bool1,letter1,numint1,numint2,numfloat1,
&                      numdoub1,numshor1,numtask)
      logical*1 bool1
      character letter1
      integer numint1, numint2, numtask
      double precision numdoub1
      real numfloat1
      integer*2 numshor1
      
      bool1 = .true.
      if (numtask .ne. 2*(numtask/2)) bool1 = .false.
      letter1 = "y"
      numint1 = 11*numtask
      numint2 = numtask
      numdoub1 = 902
      numfloat1 = 39.6 + (0.1*numtask)
      numshor1 = 299
      
      return
      end

Batch Job Script

% cat runhello

#PBS -N hellojob
#PBS -q debug
#PBS -l mppwidth=4
#PBS -l walltime=00:01:00
#PBS -e hellojob.out
#PBS -j eo

cd $PBS_O_WORKDIR

cc -c cmainmpi.c
ftn -o cftnmpi -Mnomain cmainmpi.o forts.f
aprun -n 4 ./cftnmpi

Output File

% cat hellojob.out

Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
/opt/xt-pe/2.0.24b/bin/snos64/cc: INFO: linux target is being used
/opt/xt-pe/2.0.24b/bin/snos64/ftn: INFO: linux target is being used
forts.f:
 Hello from proc 1 of 4; data values: FALSE y 11 1 39.7 902 299
 Hello from proc 0 of 4; data values: TRUE y 0 0 39.6 902 299
 Hello from proc 3 of 4; data values: FALSE y 33 3 39.9 902 299
 Hello from proc 2 of 4; data values: TRUE y 22 2 39.8 902 299
Application 149009 resources: utime 0, stime 0

C++/MPI Main Calling Fortran Subroutine

Here is a full example of a parallel C++ code using MPI which calls a Fortran subroutine. The Fortran subroutine sets a number of data values based on the MPI task ID. The code is compiled and run using the batch system.

C++ Source Code

% cat cpmain.C

#include <mpi.h>
#include <iostream>
using namespace std;

extern "C" { extern void forts_(char *,char *,int *,int *,
            float *,double *,short *); }

int main (int argc, char* argv[])
{
 char bool1, letter1;
 int numprocs, myid;
 float numfloat1;
 double numdoub1;
 short numshor1;

 MPI_Init(&argc, &argv);
 MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
 MPI_Comm_rank(MPI_COMM_WORLD,&myid);

 forts_(&bool1,&letter1,&myid,&numprocs,&numfloat1,&numdoub1,&numshor1);

 cout << " bool1 = ";
 bool1?cout << "TRUE ":cout << "FALSE ";
 cout << "; letter1 = " << letter1 << "; myid = " << myid << " numprocs = " << numprocs;
 cout << " myid/numprocs  = " << numfloat1 << endl;
 cout << " numdoub1 = " << numdoub1 << " numshor1 = " << numshor1 << endl;

 MPI_Finalize();
}

Fortran Source Code

% cat forts.f

       subroutine forts
     .    (bool1,letter1,numint1,numint2,numfloat1,numdoub1,numshor1)

       logical*1 bool1
       character letter1
       integer numint1, numint2
       double precision numdoub1
       real numfloat1
       integer*2 numshor1

       bool1 = .true. 
       if (numint1 .ne. 2*(numint1/2)) bool1 = .false.
       letter1 = "v" 
       numdoub1 = 902
       numfloat1 = (1.0*numint1)/(1.0*numint2)
       numshor1 = 299

       return
       end

Batch Job Script

% cat runftncpp

#PBS -N ftncpp
#PBS -q debug
#PBS -l mppwidth=4
#PBS -l walltime=00:01:00
#PBS -e ftncpp.out
#PBS -j eo

cd $PBS_O_WORKDIR

ftn -c forts.f
CC -o cppftn forts.o cpmain.C -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -lpgftnrtl

aprun -n 4 ./cppftn

Output File

% cat ftncpp.out

Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
/opt/xt-pe/2.0.24b/bin/snos64/ftn: INFO: linux target is being used
/opt/xt-pe/2.0.24b/bin/snos64/CC: INFO: linux target is being used
 bool1 = FALSE ; letter1 = v; myid = 1 numprocs = 4 myid/numprocs  = 0.25
 numdoub1 = 902 numshor1 = 299
 bool1 = TRUE ; letter1 = v; myid = 0 numprocs = 4 myid/numprocs  = 0
 numdoub1 = 902 numshor1 = 299
 bool1 = FALSE ; letter1 = v; myid = 3 numprocs = 4 myid/numprocs  = 0.75
 numdoub1 = 902 numshor1 = 299
 bool1 = TRUE ; letter1 = v; myid = 2 numprocs = 4 myid/numprocs  = 0.5
 numdoub1 = 902 numshor1 = 299
Application 173761 resources: utime 0, stime 0

C/MPI Main Calling Fortran/OpenMP Subroutine

Here is a full example of a parallel C code using MPI which calls a Fortran subroutine. The Fortran subroutine includes a multi-threaded parallel segment and sets a number of data values based on the MPI task ID and number of OpenMP threads. The code is compiled and run using the batch system.

The environment variable OMP_NUM_THREADS is used to control the number of threads in the OpenMP segment.

Use of OpenMP requires a particular set of options on the Fortran compiler line: ftn -mp=nonuma -Minfo=mp ....

The source code for the C main program is the same as shown above. The Fortran routine, the batch script, and the output for the mixed C/Fortran and mixed MPI/OpenMP example are shown below.

Fortran Source Code

% cat forts.f

      subroutine forts(bool1,letter1,numint1,numint2,numfloat1,
&                      numdoub1,numshor1,numtask)
      logical*1 bool1
      character letter1
      integer numint1, numint2, numtask
      integer nthreads, tid
      integer  OMP_GET_NUM_THREADS, OMP_GET_THREAD_NUM
      double precision numdoub1
      real numfloat1
      integer*2 numshor1

      bool1 = .true.
      if (numtask .ne. 2*(numtask/2)) bool1 = .false.
      letter1 = "y"
      numint1 = 11*numtask
      numint2 = numtask
      numdoub1 = 902
      numfloat1 = 39.6 + (0.1*numtask)
!$OMP PARALLEL PRIVATE(nthreads, tid)
       tid = OMP_GET_THREAD_NUM()
       IF (tid .EQ. 0) THEN
          nthreads = OMP_GET_NUM_THREADS()
          numshor1 = nthreads
       ENDIF
!$OMP END PARALLEL
      
      return
      end

Batch Job Script

% cat runhello

#PBS -N hellojob
#PBS -q debug
#PBS -l mppwidth=2
#PBS -l mppnppn=1
#PBS -l mppdepth=2
#PBS -l walltime=00:05:00
#PBS -e hellojob.out
#PBS -j eo

cd $PBS_O_WORKDIR

cc -c cmainmpi.c
ftn -o cftnmpi -Mnomain -mp=nonuma -Minfo=mp cmainmpi.o forts.f

setenv OMP_NUM_THREADS 2
aprun -n 2 -N 1 ./cftnmpi

Output File

% cat hellojob.out

Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
/opt/xt-pe/2.0.44a2/bin/snos64/cc: INFO: linux target is being used
/opt/xt-pe/2.0.44a2/bin/snos64/ftn: INFO: linux target is being used
forts.f:
forts:
    19, Parallel region activated
    24, Parallel region terminated
 Hello from proc 0 of 2; data values: TRUE y 0 0 39.6 902 2
 Hello from proc 1 of 2; data values: FALSE y 11 1 39.7 902 2
Application 4904044 resources: utime 0, stime 5