Subject: hpff-doc: Comments on SPMD-to-HPF

From: Thomas Brandes <[email-masked]>
Date: Fri, 13 Sep 1996 12:24:10 +0200

---------------------------------------------------------------------------
[email-masked] is a mailing list for HPF 2.0 language specification
authors and editors.  Instructions for adding or deleting yourself from this
list appear at the bottom of this message.
---------------------------------------------------------------------------

Comments on SPMD-to-HPF
=======================

Though I believe that the topic is one of the most exciting features, the
current proposal is too inconsistent and still in a very early draft stage

Here are some detailed comments:

A) currently I would not like to see the strong connections to MPI.

For the example in F.4.5 I see a lot of problems:

!   Set up the process groups and communicators, one for each block

    call MPI_COMM_SPLIT(MPI_COMM_WORLD,ZERO_TO_KM1,DUMMY,COMM_A,IERROR)
    call init_HPF(COMM_A)

    call MPI_COMM_SPLIT(MPI_COMM_WORLD,K_TO_PM1,DUMMY,COMM_B,IERROR)
    call init_HPF(COMM_B)

    if (isMember(COMM_A)
!       TO_MAPPED_ARRAY( ) converts the global array reference to a
!       MAPPED_ARRAY.  k>1 arguments may correspond to TO_MAPPED_ARRAY(A)
        call alloc_mapped_arrayREAL(TO_MAPPED_ARRAY(A), 2, A_EXT, DECOMP)
    if (isMember(COMM_B)
        call alloc_mapped_arrayREAL(TO_MAPPED_ARRAY(B), 2, B_EXT, DECOMP)

How does the realization of alloc_mapped_array know on which processor
array A and B will be allocated? The communicator is not given as an
argument!

So: just propose an HPF_INIT without any arguments and separate this
    initializaton completely from using any processor arrays (in fact
    the proposed mechanisms are not quite understandable, see also below).

B) The use of macros and subroutines should be pointed out more clearly

  Macros: MAPPED_ARRAY(A)
          TO_MAPPED_ARRAY(A)

          BLOCK
          CYCLIC
          SERIAL
          CYCLIC(n)    ! attention with parameter n

  Attention: names, especially SERIAL, should be changed to e.g.
             BLOCK_DISTRIBUTION, CYCLIC_DISTRIBUTION, SERIAL_DISTRIBUTION
             
C) Subroutines (be more consistent between TTT and _TTT):

          HPF_INIT   (attention: later INIT_HPF is used)
          alloc_mapped_array_TTT
          copy_mapped_array_TTT
          copy_mapped_array_sec_TTT

Remark: you propose an optional argument ALIGN for alloc_mapped_array_TTT.
        This restricts the use to Fortran~90. Better: using two different
        routines.

   alloc_aligned_array_TTT (A, NDIM, ALIGN)

   This might also be useful as the EXTENTS must be the same in any case.

D) How works the mechanism to call HPF routines ?
          
!       These calls to HPF execute in different processor groups; their
!       respective arguments were allocated with different communicators

        if (isMember(COMM_A))
            residA = smooth(A)

        if (isMember(COMM_B))
            residB = smooth(B)

Obviously there is never a definition of A and B. How can they be passed
to the routine smooth? Must 'smooth' be defined as an HPF routine.

E) What about using existing functionality ?

The proposal says that 

   GLOBAL_ALIGNMENT, ...., MY_PROCESSOR are incldued.

But: these routines rely on Fortran 90. What can be used in other
     languages like Fortran 77, C, C++ ? 

Idea: There are Fortran 77 versions of these routines. Probably it might
      be possible to find versions that seem appropriate for every language.

F) How to access distributed arrays in the SPMD program ?

Another important issue is that it is absolutely unclear how to access
the local parts of the arrays A and B within the SPMD program. How
can they be initialized ? Can they only be accessed via HPF programs ?

The routines copy_mapped_array_<sec_>TTT (...) will work only on 
mapped arrays. How can interact mapped arrays with other data in the SPMD
program?

G) Interactions between FORTRAN_LOCAL routines 

As calling global HPF from SPMD seems to be similiar to calling F77 code from 
global HPF, there should be some connections that should be pointed out
clearly.

PROPOSAL:
=========

The following proposal describes how to call SPMD code from HPF and vice
versa.

The essential ideas are:

  - Macros are used to define the interface between global HPF code
    and SPMD code to overcome implementation-specific rules for 
    calling conventions. These macros are used for SPMD subroutines that
    are called from HPF and for calls of HPF programs from SPMD code.

  - Macros are used to define/access data structures used for mapped arrays
    within SPMD code

I. Writing Local Routines

A local routine can be invoked from a global HPF program.

       SUBROUTINE DOIT (A)
       REAL, A(:,:)
!HPF$  DISTRIBUTE A(BLOCK,BLOCK)
       INTERFACE
          EXTRINSIC(HPF_LOCAL) SUBROUTINE INIT (A, S)
          REAL A(:,:), S
!HPF$     DISTRIBUTE A(BLOCK,BLOCK)
          END SUBROUTINE INIT
       END INTERFACE
       ...
       call INIT (A,1.0)
       ...
       END

Hint: An interface is required if there is any implicit redistribution at 
the subroutine boundary.

The code realizing INIT can be any SPMD code (also in other languages)
that has to fit the calling conventions.

II. Writing Local SPMD Code in FORTRAN 77

If the local code is not written in HPF, there must be some conventions
how the local code is called from global routines. This is obviously
implementation-specific.

One solution could be to call local code written in other languages by
some new rules (other extrinsic models). The main problem is here how
to get access to the desriptors of mapped data.

An SPMD program that is called from an HPF program has to be defined
in the following way:

      SUBROUTINE INIT HPFM_ARGS(`HPFM_IS_MAPPED(V),HPFM_IS_SEQ(S)')
      HPFM_ENVIRONMENT
      HPFM_DECLARE_MAPPED_ARRAY(V,2,REAL)
      HPFM_DECLARE_SEQ(S)
      REAL S

For the interface the following macros will be used:

  HPFM_ARGS      : has to procede the dummy argument list.
  HFPM_IS_MAPPED : has to procede every argument that corresponds to 
                   a mapped array.
  HPFM_IS_SEQ    : has to procede every argument that corresponds
                   to sequential data, especially scalar values.

  HPFM_ENVIRONMENT : is used to insert some compiler specific definitions 
                     (might be empty).

  HPFM_DECLARE_MAPPED_ARRAY: is used to declare the dummy arguments that 
                             will be used for passing HPF distributed arrays.
  HPFM_DECLARE_SEQ : is used to declare the dummy arguments
      that might additionally be used for passing sequential data. The 
      sequential data has in any case to be defined as a usual argument.

This mechanism guarantees that for calling INIT in an HPF routine it
can be considered like calling a usual HPF routine. An interface is not
required unless other HPF conventions require this (e.g. if a redistribution
at the subroutine boundary is required).

Note:

    There are approved extensions for FORTRAN\_LOCAL routines that
    require an explicit interface in the HPF program. By the interface,
    the HPF compiler knows whether it has to pass a pointer or a descriptor
    of a mapped array. This approach is more general.

III. Calling Global HPF Routines from SPMD Programs

HPF invocations will import scalars and global arrays allocated from
the SMPD  program. The primary concern here in how to
deal with global arrays.

      call sub HPFM_ARGS(`HPFM_MAPPED(V), HPFM_SEQ(S)')

For calling global HPF routines the following macros will be used:

  HPFM_ARGS:   has to procede the dummy argument list.
  HPFM_MAPPED: has to procede every actual argument that
               corresponds to a mapped array.
  HPFM_SEQ:    has to procede every actual argument that
               corresponds to sequential data, especially if 
               scalar values are passed.

IV. Mapped Arrays

Until now, the SPMD program can access only the data passed by global
HPF routines via arguments.

Now, an SPMD program can also define mapped arrays and 
mapped data. These mapped arrays can later be passed to global
HPF routines. Currently, the possibilities for defining mapped
arrays are restricted, especially the possibilities for alignment
are restricted.

Mapped arrays are defined via macros (implementation-specific data
structures for mapped data). Data can be allocated via special
subroutines (this is as Appendix F of HPF2.0 currently proposes, but
some cleanings might be required).

      HPFM_ENVIRONMENT
      HPFM_DECLARE_MAPPED_ARRAY(V,2,REAL)

      INTEGER EXT(4), DECOMP(2)

      EXT = (/1,100,1,100/)
      DECOMP = (/HPFM_BLOCK_DISTRIBUTION, HPFM_SERIAL_DISTRIBUTION/)

      ! HPFM_MAPPED_ARRAY(V) finds correct arguments for passing V

      call alloc_mapped_array (HPFM_MAPPED_ARRAY(V), 2, EXT, DECOMP)

Syntax/Semantic of alloc_mapped_array and alloc_aligned_array have
still to be defined more clearly.

V. Sharing Data between Global and Local Routines

Obviously one can access in local routines only the data that has been
passed by arguments. There is no possibility to access any global data.

This restriction might be very restrictive especially for porting existing
FORTRAN 77 applications. Therefore we propose that global and local routines
can share data of a COMMON block as long as the data is not mapped.

       SUBROUTINE DOIT (A)
       REAL, A(:,:)
!HPF$  DISTRIBUTE A(BLOCK,BLOCK)
       COMMON /DATA/ A1, A2
       REAL A1, A2
       ...
       call INIT (A,1.0)
       ...
       END

       EXTRINSIC(HPF_LOCAL) SUBROUTINE INIT (A,S)
       REAL A(:,:), S
!HPF$  DISTRIBUTE A(BLOCK,BLOCK)
       COMMON /DATA/ A1, A2
       REAL A1, A2
       A = S
       END SUBROUTINE INIT

Furthermore, we propose that distributed arrays of COMMON blocks
can also be accessed via the same mechanism as dummy arguments.

!     global HPF code

      SUBROUTINE DOIT ()
      COMMON /DATA/ A(N,N)
!HPF$ DISTRIBUTE A(BLOCK,BLOCK)
      ...
      call INIT ()
      ...
      END

!     local SPMD code, here in FORTRAN 77

      SUBROUTINE INIT HPFM_ARGS(`')

      COMMON /DATA/ HPFM_MAPPED_COMMON(A)
      HPFM_DECLARE_MAPPED_ARRAY(A,2,REAL)
      ...
      END 

VI. Access of Mapped Data in SPMD Code

A new macro allows to access the first local element of a mapped array.

     HPFM_MAPPED_FIRST(v)

This macro can be used in the following way:

      HPFM_DECLARE_MAPPED_ARRAY(V,2,REAL)
      ... ! V is allocated or dummy argument
      CALL SPMD_SHAPE (SHP, HPFM_MAPPED_ARRAY(V))
      ! local shape is SHP(1) x SHP(2)
      CALL DO (HPFM_MAPPED_FIRST(V), SHP(1), SHP(2), ..)

      SUBROUTINE DO_IT (V,N1,N2,...)
      INTEGER N1, N2
      REAL V(N1,N2), S
      DO J=1,N2
         DO I=1,N1
            V(I,J) = ...
         END DO
      END DO
      END

VII. Runtime Support

Change F77 routines to SPMD routines.

   SPMD_GLOBAL, SPMD_GLOBAL_DISTRIBUTION, SPMD_GLOBAL_TEMPLATE

   SPMD_ABSTRACT_TO_PHYSICAL, SPMD_PHYSICAL_TO_ABSTRACT,
   SPMD_LOCAL_TO_GLOBAL, SPMD_GLOBAL_TO_LOCAL

   SPMD_LOCAL_BLKCNT, SPMD_LOCAL_LINDEX, SPMD_LOCAL_UINDEX

   SPMD_GLOBAL_SHAPE, SPMD_GLOBAL_SIZE

   SPMD_SHAPE, SPMD_SIZE

   SPMD_MY_PROCESSOR

   SPMD_SUBGRID_INFO

Probably they can be called from Fortran 90, FORTRAN 77, C, and C++ programs.

VIII. HPF Initialization

If the main program is an SPMD progran and not an HPF program
a call to a procedure HPF_INIT is required to establish
various state needed by the HPF run time system, that will, for
example, enable the processor to determine `who it is.'

     PROGRAM SPMD_TEST
     ...
     CALL HPF_INIT ()
     ...
     CALL HPF_EXIT ()
     END PROGRAM

IX. Example of Mixed Programming

      PROGRAM TEST

      ! defintions used within other macros

      HPFM_ENVIRONMENT

      ! define data structures for a distributed array

      HPFM_DECLARE_MAPPED_ARRAY(V,2,REAL)

      REAL S

      INTEGER N

      INTEGER EXT(4), DECOMP(2)

      ! initializes HPF environment (is a subroutine)

      call HPF_INIT()

      EXT = (/1,100,1,100/)
      DECOMP = (/HPFM_BLOCK_DISTRIBUTION, HPFM_SERIAL_DISTRIBUTION/)

      ! HPFM_MAPPED_ARRAY(V) finds correct arguments for passing V

      call alloc_mapped_array_REAL (HPFM_MAPPED_ARRAY(V), 2, EXT, DECOMP)

      ! arguments to an HPF routine are specially handled
      ! HPFM_CALL_ARGS(...) translates argument list correctly
      ! HPFM_MAPPED(V) if V is a HPF mapped array
      ! HPFM_SEQUENTIAL(S) if S is a usual Fortran argument

      call sub HPFM_ARGS(`HPFM_MAPPED(V), HPFM_SEQ(S)')

      print *, 'S = ', S

      ! terminate HPF execution correctly (is subroutine)

      call HPF_EXIT ()

      END

      ! HPF routine compiled by the HPF compiler

      subroutine sub (V, SUM1)

      real V(:,:)
      real SUM1
!hpf$ distribute V (Block,*)

      ! here we call a routine that will be realized in Fortran 77

      call init (V,3.0)

      SUM1 = sum(V)
      end

/* SPMD program written in C */

      void init HPFM_ARGS(`HPFM_IS_MAPPED(v),HPFM_IS_SEQ(s)')
      HPFM_DECLARE_MAPPED_ARRAY(v,2,float)
      HPFM_DECLARE_SEQ(s)
      float s;

      { int shp[2];
        float *ptr;

        spmd_shape (shp, HPFM_MAPPED_ARRAY(v))
        ptr = HPFM_MAPPED_FIRST(v);

        for (j=0; j<n2; j++)
          for (i=0; i<n1; i++)
             *ptr++ = s;

      } /* init */

      _______    _   _       _____   Thomas Brandes
     / _____/   / | / |     / ___ |  Forschungszentrum Informationstechnik
    / / ____   /  |/  |    / /  / /  Schloss Birlinghoven
   / / /_  /  / /   / |   / /  / /   D-53754 Sankt Augustin
  / /___/ /  / /|  /| |  / /__/ /    Tel: +49-2241-14-2492   Fax: 2181
  \______/  /_/ |_/ |_| /______/     Email: [email-masked]
                                     http: //www.gmd.de/SCAI/people/brandes.html

---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------