Subject: hpff-doc: f77-local-ext.tex, for inclusion in hpf-local-ext.tex

From: Carol Munroe <[email-masked]>
Date: Sat, 5 Oct 96 17:15:20 EDT

---------------------------------------------------------------------------
[email-masked] is a mailing list for HPF 2.0 language specification
authors and editors.  Instructions for adding or deleting yourself from this
list appear at the bottom of this message.
---------------------------------------------------------------------------
(Except for the name at the top of this file, changed from
fortran-local-ext.tex to f77-local-ext.tex, here and in a version of
hpf-local-ext.tex, this is the same file I posted yesterday to Mary and
Chuck, just reposted to the list for completeness. -- Carol)

%file [FORMERLY fortran-local-ext.tex]
%file f77-local-ext.tex

\section{Fortran 77 Language Bindings}
% \label{ch-fortran-local-ext}   [CHANGED]
\label{ch-f77-local-ext}

Fortran 77 is essentially a subset of ANSI/ISO standard Fortran, so most
considerations relating to HPF calling Fortran also apply to HPF calling
Fortran 77 extrinsic routines.  However, two chief differences between
Fortran and Fortran~77 complicate the specification of any {\tt
EXTRINSIC(LANGUAGE='F77')}, interface from HPF, especially for the local
model:

\begin{itemize}
\item Arguments are passed differently.  In Fortran~77,
     arguments are typically passed between subprograms
     by address (reference).  That is, no other information
     about the actual argument is passed --- for example,
     data type, size, distribution, etc.  In contrast, HPF
     implementations often pass by variables by descriptor in order to
     make such information available to the subprogram.

\item Very different information about how array
     elements are to be assigned to specific memory
     locations is available to Fortran~77 and HPF programmers.

     In Fortran~77, the declaration of an array prescribes
     exactly the mapping of array elements to the linear
     sequence of storage locations.  In HPF, the
     mapping of array elements to different processors
     may be controlled (e.g., with {\tt DISTRIBUTION} and
     {\tt ALIGN} directives) and queried (e.g., with
     {\tt HPF\_ALIGNMENT}, {\tt HPF\_DISTRIBUTION}, and {\tt HPF\_TEMPLATE})
     but there is absolutely no information about how
     array elements on a given processor are organized
     within local, serial memory. Even in Fortran 90, assumed
     shape dummy arrays, for example, do not have to follow the same storage
     and sequence association rules as Fortran 77 arrays do.

     Indeed, different HPF compilers may organize the data
     locally in different manners --- perhaps including
     border cells for ``stencil'' optimizations, or global
     padding to ensure equal-size subgrids on all processors.
     Certainly, different HPF compilers are not {\it bound} to
     organize local data in any particular manner, and some
     may choose imaginative orderings in such cases as SMP's,
     for example.
\end{itemize}

\subsection{Special Considerations for F77\_LOCAL}

An {\tt EXTRINSIC(F77\_LOCAL)} interface is defined that extends the {\tt
HPF\_LOCAL} and {\tt FORTRAN\_LOCAL} extrinsic interfaces to meet the needs
of Fortran~77 programmers.

This {\tt EXTRINSIC} type uses the syntax for calling extrinsic subprograms
described above. It can be described more precisely as an {\tt
EXTRINSIC(LANGUAGE='F77', MODEL='LOCAL')} interface. The basic conventions
for transferring control between global and local routines passing
described previously in Section~\ref{ch-hpf-local-ext} also apply.

However, the differences in argument passing and data distribution between
these two languages, as well as the different possible motivations for
using such an interface, can be better addressed by allowing additional
options for passing data and distribution information. These options are
provided with the help of {\tt LAYOUT} and {\tt PASS_BY} attributes.

\subsection{Argument Passing to F77\_LOCAL Procedures}

A typical Fortran 77 implementation passes arguments by reference, usually
by passing the base address of the location of the first data element, and
such arguments may also be assumed to be sequence associated.  These facts
make it most practical for the default method of passing a distributed data
structure from {\tt HPF} to an {\tt F77\_LOCAL} routine by passing the base
address of that section of local memory that has been allocated to it.
To allow for sequence association of actual and dummy arguments, data
should also be reordered or compressed or both, if necessary, on all
processors. This is the safest method of passing distributed data to
an {\tt EXTRINSIC(F77\_LOCAL)} routine, and hence it should be the 
default one. However, it tends to have the greatest performance costs.

A second argument passing option is to pass distributed array data 'as is'
from a global HPF routine to the local F77 ones, not guaranteeing sequence
association of the dummy arguments in order to avoid unwanted local data
motion that might be required to compress or reorder the elements of an
array local to a processor. In other words, it should be possible to do no
more data motion than if the same argument were being passed to another HPF
procedure. The guarantee of a sequence associated dummy argument is
sacrificed for the possible gains in performance available because the
local components of the actual argument are not reordered or compressed.
The local programmer must be able to use the implementation-dependent
ordering created by the global HPF program.

A third option that can be useful to permit {\tt HPF\_LOCAL}-style local
programming from an {\tt EXTRINSIC(F77\_LOCAL} procedure call is to pass an
array via a descriptor or handle, as is typically done in HPF
implementations or for Fortran 90 assumed shape arrays. The local routine
may not access elements of this dummy argument directly but may only pass
it on to special utility routines, perhaps to obtain local or global
distribution information.

The following attributes suffice to support the above three alternate form
of passing data to an {\tt EXTRINSIC(F77\_LOCAL} routine:

\begin{itemize}

\item {\tt LAYOUT('F77\_ARRAY')}
     indicates that the rectangular configuration
     should be FORTRAN~77 sequence associated in local,
     serial memory.

     For example, many compilers
     add border elements for ``stencil''
     optimizations or pad array allocations
     on particular processors so that all processors
     allocate equal amounts of memory for each array.
     Local reordering eliminates such padding and
     provides FORTRAN~77 sequence association for actual data
     values.

     Any local reordering is in addition to
     any global remapping that may be dictated by
     {\tt DISTRIBUTION} or {\tt ALIGN} directives
     in the {\tt INTERFACE} block.

     If no {\tt LAYOUT()} attribute is specified,
     then {\tt LAYOUT('F77\_ARRAY')} is assumed.

\item {\tt LAYOUT('HPF\_ARRAY')}
     indicates that an array argument is passed just as it would be to
     a global HPF procedure, with no local reordering of the data.

     This option is desirable when the
     programmer decides that the overhead of local
     reordering should be eliminated or that certain
     characteristics of the global HPF compiler's ordering
     (border cells, equal-size allocations among processors,
     etc.) should be preserved at the local level.  It forces
     the local programmer to access the local data in
     whatever implementation-dependent style the global HPF
     compiler employs.

\end{itemize}

Furthermore, each argument in the {\tt INTERFACE} block can also have its
{\tt PASS_BY} attribute specified to indicate whether the data is passed by
reference, for Fortran 77-style access, or via a special handle, perhaps a
descriptor used for HPF variable passing, that permits the global HPF
caller to pass special distribution information for use within the local
Fortran 77 routine.

\begin{itemize}

\item {\tt PASS_BY('*')}
      indicates that the local procedure should be able to access the
      dummy argument locally as an F77-style variable, passed by reference.

\item {\tt PASS_BY('HPF\_HANDLE')}
     indicates that the local routine should receive a reference to global
     descriptor that can be used with special accesss functions to obtain
     useful distribution information.

\end{itemize}

Thus the default dummy argument attributes are {\tt LAYOUT('F77_ARRAY'), a
guarantee of sequence association, and {\tt PASS_BY('*')}, an indication
that data is being passed via a pointer to its location.

\begin{implementors}

In addition to argument passing and data reordering options, a good {\tt
EXTRINSIC(F77\_LOCAL} implementation should address the problem of
declaring arbitrary sized local subgrids and accessing their elements
without being able to describe them as assumed-shape arrays, as in HPF.
Dealing with the local results of global data distributions within each
local procedure initiated by an extrinsic procedure call can also be
difficult without Fortran~90 array inquiry functions and the inquiry
subroutines in the HPF Library.  Special inquiry routines, callable
globally or locally, such as the proposed library of Fortran 77 function
interfaces in Annex~\ref{extrinsic-f77-library} are recommended as
supplements to the {\tt EXTRINSIC(F77\_LOCAL)} procedure interface in order
to permit more flexible and efficient use of a broad range of possible
global HPF data distributions.

\end{implementors}

\subsection{F77\_LOCAL Programming Examples}

\subsubsection{{\tt LAYOUT('F77\_ARRAY')} and {\tt PASS_BY('*')}}

This example illustrates F77\_LOCAL programming using the default {\tt
LAYOUT('F77\_ARRAY')} and {\tt PASS_BY('*')} attributes, and the use of
inquiry routines from the local level using the {LAYOUT(`HPF\_ARRAY')}
attribute.

\begin{itemize}
\item{HPF caller}

\begin{verbatim}
              program example

        ! declare the data array and a verification copy
              integer, parameter :: nx = 100, ny = 100
              real, dimension(nx,ny) :: x, y
        !hpf$ distribute(block,block) :: x, y

        ! the global sum will be computed
        ! by forming partial sums on the processors
              real partial_sum(number_of_processors())
        !hpf$ distribute partial_sum(block)

        ! local subgrid parameters are declared per processor
        ! for a rank-two array
              integer, dimension(number_of_processors(),2) ::
             & lb, ub, number
        !hpf$ distribute(block,*) :: lb, ub, number

        ! define interfaces

              interface

                extrinsic(f77_local) subroutine local1
             &   ( lb1, ub1, lb2, ub2, x , x_desc )
                integer, dimension(:) :: lb1, ub1, lb2, ub2
                real,dimension(:,:),layout('HPF_ARRAY') :: x
                real,dimension(:,:),layout('HPF_ARRAY'),     &
                       pass_by('HPF_HANDLE')           :: x_desc 

        !hpf$ distribute(block) :: lb1, ub1, lb2, ub2
        !hpf$ distribute(block,block) :: x, x_desc
                end

                extrinsic(f77_local) subroutine local2(n,x,r)
                integer n(:)
                real x(:,:), r(:)   
! defaults: layout('F77\_ARRAY'), i.e., sequential, column-major storage;
!           pass_by('*'),         i.e., pass by reference (local address)
        !hpf$ distribute n(block)
        !hpf$ distribute x(block,block)
        !hpf$ distribute r(block)
                end

              end interface

        ! determine result using only global HPF

              ! initialize values
              forall (i=1:nx,j=1:ny) x(i,j) = i + (j-1) * nx

              ! determine and report global sum
              print *, 'global HPF result: ',sum(x)

        ! determine result using local subroutines

              ! initialize values ( assume stride = 1 )
              call HPF_subgrid_info( y, ierr, lb=lb, ub=ub )
              if (ierr.ne.0) stop 'error!'
              call local1( lb(:,1), ub(:,1), lb(:,2), ub(:,2), y , y )

              ! determine and report global sum
              number = ub - lb + 1
              call local2 ( number(:,1) * number(:,2) , y , partial_sum )
              print *, 'F77_LOCAL result #1 : ',sum(partial_sum)

              end
\end{verbatim}

\item{FORTRAN~77 callee}

\begin{verbatim}
              subroutine local1( lb1, ub1, lb2, ub2, x , descrx )

              real x ( lb1 : ub1 , lb2 : ub2 )
              integer descrx ( * )

        ! get the global extent of the first axis
        ! this is an HPF_LOCAL type of inquiry routine with an ``F77_'' prefix
              call F77_global_size ( nx , descrx , 1 )

        ! initialize elements of the array
              do j = lb2, ub2
                do i = lb2, ub2
                  x(i,j) = i + (j-1) * nx
                end do
              end do

              end

              subroutine local2(n,x,r)

        ! here, the correspondence to the global indices is not important
        ! only the total size of the subgrid is passed in
              real x(n)

              r = 0.
              do i = 1, n
                r = r + x(i)
              end do

              end
\end{verbatim}
\end{itemize}

\subsubsection{\tt LAYOUT('HPF\_ARRAY')} and {\tt PASS_BY('HPF\_HANDLE')}

This example performs only the initialization of the above example.  It
illustrates use of the {\tt LAYOUT('F77\_ARRAY')} attribute to pass an HPF
distributed array without remapping, as well as use of {\tt
PASS_BY('HPF\_HANDLE')} to pass an HPF-style descriptor or handle for use
in the F77_LOCAL subgrid inquiry function. It also illustrates the
addressing of data in terms of ``embedding arrays.''

\begin{itemize}
\item{HPF caller}

\begin{verbatim}
              program example

              integer, parameter :: nx = 100, ny = 100
              real, dimension(nx,ny) :: y
        !hpf$ distribute(block,block) :: y

        ! local subgrid parameters are declared per processor
        ! for a rank-two array
              integer, dimension(number_of_processors(),2) ::
             & lb, ub, lb_embed, ub_embed
        !hpf$ distribute(block,*) :: lb, ub, lb_embed, ub_embed

        ! define interfaces

              interface

                extrinsic(f77_local) subroutine local1(
             &   lb1, ub1, lb_embed1, ub_embed1,
             &   lb2, ub2, lb_embed2, ub_embed2, x, x_desc )
                integer, dimension(:) ::
             &   lb1, ub1, lb_embed1, ub_embed1,
             &   lb2, ub2, lb_embed2, ub_embed2
         !             By default, x is passed by reference
                real, dimension(:,:), layout('hpf_array') :: x
         !             x_desc is passed by its descriptor or 'handle'
                real, dimension(:,:), layout('hpf_array'),
             &                    pass_by('hpf_handle') :: x_desc
        !hpf$ distribute(block) :: lb1, ub1, lb_embed1, ub_embed1
        !hpf$ distribute(block) :: lb2, ub2, lb_embed2, ub_embed2
        !hpf$ distribute(block,block) :: x
                end

              end interface

        ! initialize values
        ! ( assume stride = 1 and no axis permutation )

              call HPF_subgrid_info( y, ierr,
             & lb=lb, lb_embed=lb_embed,
             & ub=ub, ub_embed=ub_embed)
              if (ierr.ne.0) stop 'error!'

              call local1(
             & lb(:,1), ub(:,1), lb_embed(:,1), ub_embed(:,1),
             & lb(:,2), ub(:,2), lb_embed(:,2), ub_embed(:,2), y, y )

              end
\end{verbatim}

\item{Fortran~77 callee}

\begin{verbatim}
              subroutine local1(
             & lb1, ub1, lb_embed1, ub_embed1,
             & lb2, ub2, lb_embed2, ub_embed2, x, x_desc )

        ! the subgrid has been passed in its "embedded" form
              real x ( lb_embed1 : ub_embed1 , lb_embed2 : ub_embed2 )

        ! this argument is used only as input to inquiry functions
              integer x_desc

        ! get the global extent of the first axis
        ! this is an HPF_LOCAL type of inquiry routine with an ``F77_'' prefix
              call F77_global_size(nx,x_desc,1)

        ! otherwise, initialize elements of the array
        ! loop only over actual array elements
              do j = lb2, ub2
                do i = lb2, ub2
                  x(i,j) = i + (j-1) * nx
                end do
              end do

              end
\end{verbatim}
\end{itemize}

---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------