From owner-hpff-external Fri Apr 26 10:13:45 1996 Received: (from daemon@localhost) by cs.rice.edu (8.7.1/8.7.1) id KAA23335 for hpff-external-out; Fri, 26 Apr 1996 10:13:45 -0500 (CDT) Received: from mail.think.com (Mail1.Think.COM [131.239.33.245]) by cs.rice.edu (8.7.1/8.7.1) with SMTP id KAA23329 for ; Fri, 26 Apr 1996 10:13:42 -0500 (CDT) Received: from Godot.Think.COM by mail.think.com; Fri, 26 Apr 96 11:13:40 -0400 From: Carol Munroe Received: by godot.think.com (4.1/Think-1.2) id AA01508; Fri, 26 Apr 96 11:13:39 EDT Date: Fri, 26 Apr 96 11:13:39 EDT Message-Id: <9604261513.AA01508@godot.think.com> To: hpff-external@cs.rice.edu Subject: hpff-external: HPFF proposal for an EXTRINSIC(F77_LOCAL) proposal Sender: owner-hpff-external Precedence: bulk --------------------------------------------------------------------------- hpff-external@cs.rice.edu is a mailing list for discussion of external interfaces between HPF and other languages/systems. Instructions for adding or deleting yourself from this list appear at the bottom of this message. --------------------------------------------------------------------------- A Thinking Machines colleague has prepared a proposal for EXTRINSIC(F77_LOCAL) that I am about to send out in latex form. Since it's so close to the meeting time, I thought I should send it to hpff-core as a whole, in hopes that more people will see it before the meeting, but it would be particularly useful to get readers from this subgroup of course. We are very interested in feedback, as soon as possible, since it is so late in the year already. If there are any early comments or suggestions for revisions (it is certainly not cast in stone), we might have time to revise the proposal before the meeting as well. Carol Munroe --------------------------------------------------------------------------- To (un)subscribe to this list, send mail to hpff-external-request@cs.rice.edu. Leave the subject line blank, and in the body put the line (un)subscribe --------------------------------------------------------------------------- From owner-hpff-external Fri Apr 26 11:58:37 1996 Received: (from daemon@localhost) by cs.rice.edu (8.7.1/8.7.1) id LAA26437 for hpff-external-out; Fri, 26 Apr 1996 11:58:37 -0500 (CDT) Message-Id: <199604261658.LAA26437@cs.rice.edu> Date: Fri, 26 Apr 1996 10:55:15 -0500 (CDT) From: Carol Munroe Cc: eugene@think.com, hamel@think.com Subject: hpff-external: HPFF proposal for an F77_LOCAL extrinsic interface Sender: owner-hpff-external Precedence: bulk --------------------------------------------------------------------------- hpff-external@cs.rice.edu is a mailing list for discussion of external interfaces between HPF and other languages/systems. Instructions for adding or deleting yourself from this list appear at the bottom of this message. --------------------------------------------------------------------------- The following is a proposal (prepared here at Thinking Machines) that I would like to have considered for next week's meeting. I apologize for presenting it at the last minute, especially since it may seem long, but the essence of the interface is described in the first few pages, while over half of it consists of examples and specifications for utility routines. When you look this over, please note that there is an important option left open concerning how descriptor information should be made available to local array inquiry routines, similar to the F90 array inquiry intrinsics and the HPF_LOCAL_LIBRARY ones, assuming that at least some such routines should be provided. The question is whether or not to have users pass descriptor handles explicitly, in light of possible ambiguities resulting from passing local array data by address only. The main method presented in this proposal hides the descriptors even when local inquiry functions are used, but an alternate method of passing them explicitly, when arrays will be used in inquiry functions, is also suggested in a highlighted paragraph on page four. I would appreciate feedback on this topic and on the proposal as a whole. the sooner the better, to have a sense of what HPFF members really want in this interface, since there are very few meetings remaining to present this in. ******cut here for f77_local.tex********************************************* \documentstyle[twoside,11pt]{article} \pagestyle{myheadings} \sloppy \footheight=-.60in \headheight=0in \textwidth=5.50in \topmargin=-.20in \oddsidemargin=0.5in \evensidemargin=0.5in \itemsep=.05in \topsep=.05in \parsep=.05in \parskip=.10in \textheight=9in \newcounter{dofigures} \setcounter{dofigures}{0} \newcounter{dotables} \setcounter{dotables}{0} \newcommand{\havefigures}{\setcounter{dofigures}{1}} \newcommand{\havetables} {\setcounter{dotables}{1}} \newcommand{\cccdocid}[5]{ \bibliographystyle{alpha} % ******************** Title Page ******************** \vspace*{.5in} \centerline{\Large \bf {#1}} \vspace*{.02in} \centerline{{#2}} \vspace*{.02in} \centerline{{#3}} \vspace*{.02in} \centerline{{#4}} \vspace*{.02in} \centerline{{#5}} \vspace*{.5in} % ******************** List of Figures ******************** % Note: Remove the following line if there are no figures % \ifodd\value{dofigures} \listoffigures \newpage \fi % \ifodd\value{dotables} \listoftables \newpage \fi % \markboth{#1} {#3} % \pagenumbering{arabic} } % ******************** Special macros ******************** %\catcode`^^Z=9 % Make TeX ignore IBMPC EOF character \newcommand{\ital}[1]{{\it #1}} \newcommand{\type}[1]{{\tt #1}} \newcommand{\bold}[1]{{\bf #1}} \newcommand{\dfn}[1]{{\it #1}\index{#1}} \newcommand{\comment}[1]{} \def\fineprint{\relax} \def\stopfine{\relax} \newcommand{\beginprog}{\begin{tabbing} xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=\kill } \newcommand{\tb}{\>\tt} \newcommand{\finprog}{\end{tabbing}} \newcommand{\progtxt}[1]{``{\tt #1}''} \newcounter{save_enum} \newcommand{\hilite}[1]{ \item {\bf #1\\} } \newcommand{\morelater}{\centerline{$\ldots$\ital{Fill this in later}$\ldots$}} % ******************** Start of Document ******************** \begin{document} \cccdocid {EXTRINSIC(F77\_LOCAL) Interface} {Ver 1.1.2} {Eugene Loh} {Thinking Machines Corporation} {April, 1996} \begin{abstract} The legacy of FORTRAN~77 serial programming and the power and popularity of message-passing programming on parallel machines makes it incumbent on HPF to support a standard \type {EXTRINSIC(F77\_LOCAL)} interface. Here, an F77\_LOCAL interface is proposed. It preserves the basic nature of the HPF\_LOCAL interface but is also designed to be very straightforward for FORTRAN~77 programmers. \end{abstract} \section{Introduction} The HPF standard specifies the nature of \type {EXTRINSIC} interfaces in general and for HPF\_LOCAL in particular. Nevertheless, specification of an F77\_LOCAL interface poses challenges due chiefly to two differences between HPF and FORTRAN~77: \begin{itemize} \item Arguments are passed differently. In FORTRAN~77, arguments are typically passed between subprograms by address (reference). That is, no other information about the actual argument is passed --- for example, data type, size, distribution, etc. In contrast, HPF implementations often pass by descriptor in order to make such information available to the subprogram. \item Very different information about how array elements are to be assigned to specific memory locations is available to FORTRAN~77 and HPF programmers. In FORTRAN~77, the declaration of an array prescribes exactly the mapping of array elements to the linear sequence of storage locations. In HPF, the mapping of array elements to different processors may be controlled (e.g., with DISTRIBUTION and ALIGN directives) and queried (e.g., with HPF\_ALIGNMENT, HPF\_DISTRIBUTION, and HPF\_TEMPLATE) but there is absolutely no information about how array elements on a given processor are organized within local, serial memory. Indeed, different HPF compilers may organize the data locally in different manners --- perhaps including border cells for ``stencil'' optimizations, or global padding to ensure equal-size subgrids on all processors. Certainly, different HPF compilers are not \ital {bound} to organize local data in any particular manner, and some may choose imaginative orderings in such cases as SMP's, for example. \end{itemize} One could argue that there already exists a well-defined interface from HPF to F77\_LOCAL if the programmer goes through HPF\_LOCAL. Quite likely, this would involve SEQUENCE directives in the HPF\_LOCAL code. While the formal elegance of going to F77\_LOCAL only through HPF\_LOCAL is attractive, it is insufficient in several respects and it is certainly clumsy for FORTRAN~77 programmers. A direct F77\_LOCAL interface for global HPF is proposed here. Its characteristics include: \begin{itemize} \item Preservation of the basic nature of the HPF\_LOCAL interface. \item A simplified mechanism for calling local routines that eliminates much of the boilerplate code that is inherent to an HPF\_LOCAL approach. \end{itemize} The proposal is consistent with HPF's specification with regards to \type {EXTRINSIC} interfaces --- particularly, with regards to local procedures --- and is consistent with HPF\_LOCAL when appropriate. For example, before calling and upon returning from the \type {EXTRINSIC} routine, the HPF code should synchronize all processors. As necessary, the compiler should copy incorrectly mapped arguments to temporaries as indicated by INTERFACE blocks before invocation of and after return from an F77\_LOCAL subroutine, just as for any other extrinsic or non-extrinsic routine. Such remapping deals with what array elements are assigned to which processors. Local routines should not share names of COMMON blocks with global routines. A local routine should not have alternate returns. Any explicit message-passing calls within a local routine must be resolved before control is returned to the global caller. And so on. Otherwise, this proposal focuses on the differences between HPF\_LOCAL and this F77\_LOCAL interface. \section{PROPOSAL} This proposal may be presented in four parts: \begin{itemize} \item Argument passing. \item HPF-style utility subroutines. \item Subgrid-inquiry routine. \item Message-passing interface. \end{itemize} \subsection{Argument passing.} An argument is passed from HPF to an F77\_LOCAL routine by passing the base address of that section of local memory that has been allocated to it. If needed, an argument is first replicated to the processors. The restrictions in HPF\_LOCAL suffice to ensure that each physical processor contains a subset of array elements that can be locally arranged in a rectangular configuration. This rectangular configuration is reorganized in local, serial memory according to the \type {MAP\_TO} attribute specified for that argument in the \type {INTERFACE} block for the \type {EXTRINSIC(F77\_LOCAL)} routine: \begin{itemize} \item \type {MAP\_TO([LAYOUT=]F77\_ARRAY)} indicates that the rectangular configuration should be FORTRAN~77 sequence associated in local, serial memory. For example, many compilers add border elements for ``stencil'' optimizations or pad array allocations on particular processors so that all processors allocate equal amounts of memory for each array. Local reordering eliminates such padding and provides FORTRAN~77 sequence association for actual data values. Any local reordering is in addition to any global remapping that may be dictated by \type {DISTRIBUTION} or \type {ALIGN} directives in the \type {INTERFACE} block. If no \type {MAP\_TO()} attribute is specified, then \type {MAP\_TO(F77\_ARRAY)} is assumed. \item \type {MAP\_TO([LAYOUT=]NO\_CHANGE)} indicates that no local reordering of the data should occur. This option is desirable when the programmer decides that the overhead of local reordering should be eliminated or that certain characteristics of the global HPF compiler's ordering (border cells, equal-size allocations among processors, etc.) should be preserved at the local level. It forces the local programmer to access the local data in whatever implementation-dependent style the global HPF compiler employs. \end{itemize} [{\em {Note}}: The syntax of this attribute corresponds to that of Meltzer's ``HPF calling C Interoperability Proposal.'' It should be modified according to the fate of that proposal.] \subsection{HPF-style utility subroutines.} Of course, the global HPF caller may continue to call standard HPF array-inquiry routines and intrinsics. At the local level, FORTRAN~77 versions of particular HPF intrinsics and HPF\_LOCAL\_LIBRARY routines are supported. Since these routines must be called from FORTRAN~77 and not HPF, there are a couple of important distinctions. \begin{itemize} \item The prefix ``F77\_'' is prepended to the utility name. \item All arguments are required. None are optional. The effect of omitting an optional \type {DIM} or \type {PROC} argument may be attained by specifying a value of -1 for that argument. \item Arguments are FORTRAN~77 sequence associated. \item The utilities are subroutines --- none are functions. When the HPF analogs are functions, the F77\_ counterparts return values in a new argument that is prepended to the argument list. [{\em {Note}}: In great part, this distinction arises due to FORTRAN~77's inability to deal with array-valued functions. An important consequence, however, is that local programmers need not \type {INCLUDE} any header files or \type {USE} any modules since no functions, keywords, or interfaces need to be declared.] \item Arguments corresponding to global arrays should use the dummy argument passed in from the HPF caller. [{\em {Note to implementors}}: Note that the F77\_LOCAL routine accepts arrays passed by address rather than by descriptor. Hence, inquiry information is no longer directly available from the argument itself. [One implementation approach would be to construct a table of the descriptors for the arguments that are present in the argument list when the HPF program enters F77\_LOCAL mode, along with the corresponding base addresses that are actually passed to FORTRAN~77. A utility routine that is called from FORTRAN~77 could check this table for the base address and pick up the corresponding descriptor.] [{\bf Note: There is a question as to whether or not any ambiguities could arise. That is, perhaps two arguments in the call from HPF to F77\_LOCAL share the same base address but not the same descriptor --- perhaps because pointers are being used or because the HPF compiler passes certain array sections in-place. It may be necessary to document this restriction and note that an FORTRAN~77 subprogram assumes that distinct dummy arguments do not overlap in memory.} [{\bf These complications arise due to the need to provide programmers with inquiry routines at the local level without exposing implementation-dependent descriptors to them. An alternative is to have local programmers explicitly pass descriptors through to utility routines. For example, instead of the \type {MAP\_TO()} attribute, one might have \type {PASS(F77\_ARRAY)}, \type {PASS(NO\_CHANGE)}, and \type {PASS(DESCRIPTOR)}. The local routine would declare a descriptor object to be \type {INTEGER DESCRX(*)} and the local utility routines would expect such objects to be passed for global arrays. Though programmers would not be expected to manipulate descriptor objects directly, these descriptors would be exposed.} [{\bf Feedback on this point is welcome.}] \end{itemize} Otherwise, these routines should behave the same as their counterparts, without the ``F77\_'' prefaces, would in HPF\_LOCAL. The list of FORTRAN~77, locally supported, HPF-style routines is \begin{itemize} \item \type {F77\_GLOBAL\_ALIGNMENT}, \type {F77\_GLOBAL\_DISTRIBUTION}, \type {F77\_GLOBAL\_TEMPLATE} \item \type {F77\_ABSTRACT\_TO\_PHYSICAL}, \type {F77\_PHYSICAL\_TO\_ABSTRACT} \item \type {F77\_LOCAL\_TO\_GLOBAL}, \type {F77\_GLOBAL\_TO\_LOCAL} \item \type {F77\_LOCAL\_BLKCNT}, \type {F77\_LOCAL\_LINDEX}, \type {F77\_LOCAL\_UINDEX} \item \type {F77\_GLOBAL\_SHAPE}, \type {F77\_GLOBAL\_SIZE} \item \type {F77\_SHAPE}, \type {F77\_SIZE} \item \type {F77\_MY\_PROCESSOR} \end{itemize} [{\em {Note}}: There are no \type {F77\_GLOBAL\_\{L,U\}\_BOUND} routines since it is anticipated that \type {GLOBAL\_\{L,U\}\_BOUND} will be removed from the HPF\_LOCAL\_LIBRARY. Tied to their HPF\_LOCAL counterparts, routines such as \type {F77\_LOCAL\_TO\_GLOBAL} use one-based global indices.] \subsection{Subgrid-inquiry routine.} The HPF\_LOCAL style of programming is to determine bounds of the local portion of a distributed array within the local routine. This complicates indexing tremendously, particularly for multidimensional arrays, for two reasons: \begin{itemize} \item The programmer must perform the multidimensional indexing explicitly, instead of leaving this job to the FORTRAN~77 compiler. \item One must call utility subroutines and then construct global indices if global indices are of interest. \end{itemize} Thus, a globally callable subgrid-inquiry routine, which describes local subgrids in terms of global indices, is provided. There is also a locally callable version. These routines, \type {HPF\_SUBGRID\_INFO} and \type {F77\_SUBGRID\_INFO}, are described in a later section. Only the motivation is discussed here. Consider the HPF code \begin{verbatim} INTEGER X(NX,NY) !HPF$ DISTRIBUTE X(BLOCK,BLOCK) FORALL ( I=1:NX, J=1:NY ) X(I,J) = I + (J-1) * NX \end{verbatim} Writing this in FORTRAN~77 using an HPF\_LOCAL style, one might have \begin{verbatim} SUBROUTINE SUB( X ) REAL X(*) INTEGER L_INDEX(2), G_INDEX(2), L_SHAPE(2) ! DETERMINE NX CALL F77_GLOBAL_SIZE( NX , X , 1 ) ! GET LOCAL SHAPE CALL F77_SHAPE( L_SHAPE , X ) ! IDENTIFY OFFSET OF LOCAL DATA WITHIN GLOBAL ARRAY L_INDEX(1) = 1 L_INDEX(2) = 1 CALL F77_LOCAL_TO_GLOBAL( X , L_INDEX, G_INDEX) ! PERFORM OPERATION III = 0 DO J = 0, L_SHAPE(2) - 1 DO I = 0, L_SHAPE(1) - 1 III = III + 1 X(III) = (I + G_INDEX(1)) + (J + G_INDEX(2) - 1) * NX END DO END DO END \end{verbatim} In contrast, had the subgrid parameters in terms of global indices been passed in from the global caller, one might have written more easily and clearly \begin{verbatim} SUBROUTINE SUB( X , LB1 , UB1 , LB2 , UB2 ) INTEGER LB1 , UB1 , LB2 , UB2 REAL X( LB1:UB1 , LB2:UB2 ) ! DETERMINE NX CALL F77_GLOBAL_SIZE( NX , X , 1 ) ! PERFORM OPERATION DO J = LB2, UB2 DO I = LB1, UB1 X(I,J) = I + (J - 1) * NX END DO END DO END \end{verbatim} \subsection{Message-passing interface.} Many important, local subroutines will deal only with local data. Others, however, will require access to typical message-passing functionality. This proposal takes the approach of the HPF\_LOCAL specification, which is to claim that such communications ``are beyond the scope of this specification.'' This proposal also recommends that no ``generic interface'' to message-passing functionality from local subroutines be adopted by the HPFF. New calls would simply require HPFF to invent and users to learn ``yet another'' message-passing interface. This proposal also \ital {highly} recommends that HPF compiler providers support and F77\_LOCAL programmers use the MPI interface for message-passing calls. \section{SUBGRID INQUIRY} As mentioned above, a globally callable subgrid-inquiry routine, \type {HPF\_SUBGRID\_INFO}, which describes the local subgrids in terms of global indices, is provided. The routine is intended to simplify F77\_LOCAL programming in slightly restricted, but very common, cases. \subsection{``Embedding arrays''} First, a note on ``embedding arrays.'' In this proposal, the phrase ``embedding array'' refers to an array that is formed by taking a multidimensional, rectangular volume of data and extending it in each direction in each dimension (possibly by zero amounts). For example, the two-dimensional volume of data \begin{verbatim} A B C D E F G H I J K L \end{verbatim} may be embedded in any one of \begin{verbatim} * * * * * * * * A B C D A B C D * * A B C D * * E F G H E F G H * * E F G H * * I J K L I J K L * * I J K L * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * A B C D * A B C D * * * * E F G H * E F G H * * * * I J K L * I J K L * * * * * * * * * * * * * * \end{verbatim} While FORTRAN~77 compilers use sequence association to store rectangular volumes of data in serial memory, HPF compilers often embed local, rectangular subgrids in larger arrays and use sequence association for the larger, embedding arrays. The paddings are typically introduced to facilitate ``stencil'' optimizations or ensure equal-size memory allocations across the all processors. Whether due to user-specified alignments or variations among implementations regarding row-major versus column-major orderings, the axes of the embedding array may be ordered differently than those of the data volume. Thus, even if there were no extra padding, sequence association of the embedding array may lead to different storage of the data in serial memory than sequence association of the data volume would. \subsection{Restrictions} The description of local data provided by \type {HPF\_SUBGRID\_INFO} assumes a slightly restricted model over that chosen by HPF for local subprograms: \begin{itemize} \item Restriction 1. Mapping of elements among processors The first restriction is that the subset of array elements that is mapped to any processor must be expressible as an array section of the global array. This case includes most \type {*}, \type {BLOCK}, and \type {CYCLIC} distributions but precludes most \type {CYCLIC(N)} distributions as well as peculiar alignments. \item Restriction 2. Organization of elements on a given processor The second restriction is that the global compiler must store the local subgrid by extending it in each direction in each dimension (perhaps by zero amounts) and sequence associating the larger, ``embedding array.'' This restriction applies \ital {only} in the case that the \type {LB\_EMBED}, \type {UB\_EMBED}, or \type {AXIS\_MAP} arguments to \type {HPF\_SUBGRID\_INFO} are used. These arguments tell an F77\_LOCAL routine how a subgrid, passed from HPF with the \type {MAP\_TO(NO\_CHANGE)} attribute, is ordered by the global HPF compiler. In most cases, however, arrays will be passed to F77\_LOCAL subprograms with the default \type {MAP\_TO(F77\_ARRAY)} attribute so that \type {LB\_EMBED}, \type {UB\_EMBED}, and \type {AXIS\_MAP} will be of no interest and this second restriction will not apply. Instead, the local subgrid will be sequence associated in serial memory. All padding due to embedding will be removed. \end{itemize} If the local programmer does not use \type {HPF\_SUBGRID\_INFO} or \type {F77\_SUBGRID\_INFO}, then neither restriction applies at all. \subsection{HPF-callable subgrid-inquiry subroutine} The subgrid-inquiry subroutine is \type {HPF\_SUBGRID\_INFO (ARRAY, IERR, DIM, LB, UB, STRIDE, LB\_EMBED, UB\_EMBED, AXIS\_MAP)} \begin{itemize} \item \type {ARRAY} is an array of any type, size, shape, or distribution \item \type {IERR} is an error-status return variable, which is zero upon successful return and nonzero otherwise --- say, in the case that the appropriate restrictions are not satisfied \item \type {DIM} is an optional argument indicating the axis along which the parameters are desired --- if no axis is specified, parameters are returned for all axes \item \type {LB}, \type {UB}, \type {STRIDE} The remaining arguments are all optional, \type {INTENT (OUT)}, integer arrays. If a \type {DIM} argument is given, they are rank-one arrays of size \type {NUMBER\_OF\_PROCESSORS()} and \type {BLOCK} distribution. If no \type {DIM} argument is given, there is a second, collapsed axis whose extent is at least the rank of the global array. The arguments \type {LB}, \type {UB}, and \type {STRIDE} use indices of the global array to return the lower bounds, upper bounds, and strides of the array sections that describe the local subgrids. \item \type {LB\_EMBED}, \type {UB\_EMBED}, \type {AXIS\_MAP} These arguments are also optional arguments that should be declared like LB, UB, and STRIDE. They return the lower and upper bounds of the local embedding arrays using indices of the global array and the axes of the embedding array to which the axes of \type {ARRAY} are mapped. Thus, they are of interest only when the programmer will pass the array using the \type {MAP\_TO(NO\_CHANGE)} attribute. \end{itemize} [{\em {Note}}: It is unclear whether local programmers would prefer global indices to be one-based or to be however they declare them in the global caller. In any case, this proposal is predicated on and follows the convention established by HPF\_LOCAL that global indices should be one-based.] Consistent with the HPF model for local subprograms, it is assumed that array axes are mapped independently to axes of the rectangular processor grid. Thus, values for \type {LB}, \type {UB}, and \type {STRIDE} should be determined on an axis-by-axis basis. Results should be the same whether they are queried one-by-one, using a \type {DIM} argument, or all at once. In particular, if an axis does not have any index values mapped to a particular processor, then \type {UB = LB - STRIDE} on that processor for that axis. Values for other axes should be determined independently. If the array has no elements mapped to a particular processor, the choices for \type {UB\_EMBED}, \type {LB\_EMBED}, and \type {AXIS\_MAP} on that processor are somewhat arbitrary. There are certainly no guidelines within HPF --- in contrast to the situation for for \type {LB}, \type {UB}, and \type {STRIDE} --- since HPF only addresses which elements are mapped to which processors, but not how those elements are locally organized on a particular processor. The only strict requirement is that the values of \type {UB\_EMBED} and \type {LB\_EMBED}, given the value of \type {STRIDE}, should give the correct overall size of the locally allocated subgrid, even if none of its elements are actually used. It is suggested that \type {UB\_EMBED}, \type {LB\_EMBED}, and \type {AXIS\_MAP} continue to reflect the axis-by-axis mapping of the array to processors and if no memory is locally allocated due to a particular axis mapping, then \type {UB\_EMBED = LB\_EMBED - STRIDE} for that axis. \subsection{Example} Consider, for example, \begin{verbatim} REAL X(7,5) !HPF$ PROCESSORS P(2,2) !HPF$ DISTRIBUTE X(BLOCK,CYCLIC) ONTO P \end{verbatim} Then the subgrids on the various processors are X(1:4,1:5:2), X(5:7,1:5:2), X(1:4,2:4:2), and X(5:7,2:4:2). The subgrids can all written as array sections of the global array, fulfilling the first restriction. If one passed X to the programmer's F77\_LOCAL routine using the default \type {MAP\_TO(F77\_ARRAY)} attribute, one could call \type {HPF\_SUBGRID\_INFO} to describe the local, sequence-associated subgrids. If X were to be passed in to the programmer's F77\_LOCAL routine with the \type {MAP\_TO(NO\_CHANGE)} attribute, then \type {HPF\_SUBGRID\_INFO} could describe the local subgrids if the compiler used ``embedding arrays''. As examples, \begin{itemize} \item It may allocate only enough room for the subgrids --- in this case, 4x3, 3x3, 4x2, and 3x2 elements per processor, respectively. \item It may allocate same-sized subgrids on each processor. Then the data on each node is embedded in a larger array --- perhaps X(1:4,1:5:2), X(5:8,1:5:2), X(1:4,2:6:2), and X(5:8,2:6:2), respectively, or 4 x 3 elements per processor. \item It may also allocate extra border cells for ``stencil'' optimizations. The allocated arrays on the respective processors might be X(0:5,1:5:2), X(4:9,1:5:2), X(0:5,2:6:2), and X(4:9,2:6:2), or (1+4+1) x 3 elements per processor. \end{itemize} All of these allocations fulfill the second restriction so long as these larger embedding arrays are sequence associated in local, serial memory. For example, processor P(2,2) might have data values X(5:7,2:4:2) embedded in a storage array X(4:9,2:6:2) that is sequence associated in local, serial memory, leading the F77\_LOCAL routine to see \begin{verbatim} MAP_TO(NO_CHANGE) MAP_TO(F77_ARRAY) base address of X ----> * X(5,2) X(5,2) X(6,2) X(6,2) X(7,2) X(7,2) X(5,4) * X(6,4) * X(7,4) * * X(5,4) * X(6,4) * X(7,4) * * * * * * * * * * * * * * * * * \end{verbatim} \subsection{FORTRAN~77-callable subgrid-inquiry subroutine} The locally callable version of \type {HPF\_SUBGRID\_INFO} is \type {F77\_SUBGRID\_INFO (ARRAY, IERR1, IERR2, DIM, LB, UB, STRIDE, LB\_EMBED, UB\_EMBED, AXIS\_MAP)} \begin{itemize} \item all arguments are required --- none is optional \item \type {ARRAY} is a dummy argument passed in from global HPF \item \type {IERR1} is zero upon successful determination of \type {LB}, \type {UB}, and \type {STRIDE} and nonzero otherwise (e.g., subgrids are not expressible as array sections of the global array) \item \type {IERR2} is zero upon successful determination of \type {LB\_EMBED} and \type {UB\_EMBED} and nonzero otherwise (e.g., the global compiler does not organize data locally by sequence associating an ``embedding'' array) \item \type {DIM = -1} behaves the same as omitting the optional \type {DIM} argument in the global version \item the subgrid parameters \type {LB}, \type {UB}, \type {STRIDE}, \type {LB\_EMBED}, \type {UB\_EMBED}, and \type {AXIS\_MAP} should be declared as integer scalars if an axis is specified by the \type {DIM} argument or as rank-one integer arrays of size equal to at least the rank of the \type {ARRAY} if \type {DIM = -1} \end{itemize} \section{PROGRAMMING EXAMPLES} \subsection{\type {MAP\_TO(F77\_ARRAY)}} This example illustrates F77\_LOCAL programming using the default \type {MAP\_TO(F77\_ARRAY)} attribute. \subsubsection{HPF caller} \begin{verbatim} program example ! declare the data array and a verification copy integer, parameter :: nx = 100, ny = 100 real, dimension(nx,ny) :: x, y !hpf$ distribute(block,block) :: x, y ! the global sum will be computed ! by forming partial sums on the processors real partial_sum(number_of_processors()) !hpf$ distribute partial_sum(block) ! local subgrid parameters are declared per processor ! for a rank-two array integer, dimension(number_of_processors(),2) :: & lb, ub, number !hpf$ distribute(block,*) :: lb, ub, number ! define interfaces interface extrinsic(f77_local) subroutine local1 & ( lb1, ub1, lb2, ub2, x ) integer, dimension(:) :: lb1, ub1, lb2, ub2 real x(:,:) !hpf$ distribute(block) :: lb1, ub1, lb2, ub2 !hpf$ distribute(block,block) :: x end extrinsic(f77_local) subroutine local2(n,x,r) integer n(:) real x(:,:), r(:) !hpf$ distribute n(block) !hpf$ distribute x(block,block) !hpf$ distribute r(block) end end interface ! determine result using only global HPF ! initialize values forall (i=1:nx,j=1:ny) x(i,j) = i + (j-1) * nx ! determine and report global sum print *, 'global HPF result: ',sum(x) ! determine result using local subroutines ! initialize values ( assume stride = 1 ) call HPF_subgrid_info( y, ierr, lb=lb, ub=ub ) if (ierr.ne.0) stop 'error!' call local1( lb(:,1), ub(:,1), lb(:,2), ub(:,2), y ) ! determine and report global sum number = ub - lb + 1 call local2 ( number(:,1) * number(:,2) , y , partial_sum ) print *, 'F77_LOCAL result #1 : ',sum(partial_sum) end \end{verbatim} \subsubsection{FORTRAN~77 callee} \begin{verbatim} subroutine local1( lb1, ub1, lb2, ub2, x ) real x ( lb1 : ub1 , lb2 : ub2 ) ! get the global extent of the first axis ! this is an HPF_LOCAL type of inquiry routine with an ``F77_'' prefix call F77_global_size(nx,x,1) ! initialize elements of the array do j = lb2, ub2 do i = lb2, ub2 x(i,j) = i + (j-1) * nx end do end do end subroutine local2(n,x,r) ! here, the correspondence to the global indices is not important ! only the total size of the subgrid is passed in real x(n) r = 0. do i = 1, n r = r + x(i) end do end \end{verbatim} \subsection{\type {MAP\_TO(NO\_CHANGE)}} This example performs only the initialization of the above example. It illustrates use of the \type {MAP\_TO(NO\_CHANGE)} attribute and the addressing of data in terms of ``embedding arrays.'' \subsubsection{HPF caller} \begin{verbatim} program example integer, parameter :: nx = 100, ny = 100 real, dimension(nx,ny) :: y !hpf$ distribute(block,block) :: y ! local subgrid parameters are declared per processor ! for a rank-two array integer, dimension(number_of_processors(),2) :: & lb, ub, lb_embed, ub_embed !hpf$ distribute(block,*) :: lb, ub, lb_embed, ub_embed ! define interfaces interface extrinsic(f77_local) subroutine local1( & lb1, ub1, lb_embed1, ub_embed1, & lb2, ub2, lb_embed2, ub_embed2, x ) integer, dimension(:) :: & lb1, ub1, lb_embed1, ub_embed1, & lb2, ub2, lb_embed2, ub_embed2 real, dimension(:,:), map_to(no_change) :: x !hpf$ distribute(block) :: lb1, ub1, lb_embed1, ub_embed1 !hpf$ distribute(block) :: lb2, ub2, lb_embed2, ub_embed2 !hpf$ distribute(block,block) :: x end end interface ! initialize values ! ( assume stride = 1 and no axis permutation ) call HPF_subgrid_info( y, ierr, & lb=lb, lb_embed=lb_embed, & ub=ub, ub_embed=ub_embed) if (ierr.ne.0) stop 'error!' call local1( & lb(:,1), ub(:,1), lb_embed(:,1), ub_embed(:,1), & lb(:,2), ub(:,2), lb_embed(:,2), ub_embed(:,2), y ) end \end{verbatim} \subsubsection{FORTRAN~77 callee} \begin{verbatim} subroutine local1( & lb1, ub1, lb_embed1, ub_embed1, & lb2, ub2, lb_embed2, ub_embed2, x ) ! the subgrid has been passed in its "embedded" form real x ( lb_embed1 : ub_embed1 , lb_embed2 : ub_embed2 ) ! get the global extent of the first axis ! this is an HPF_LOCAL type of inquiry routine with an ``F77_'' prefix call F77_global_size(nx,x,1) ! otherwise, initialize elements of the array ! loop only over actual array elements do j = lb2, ub2 do i = lb2, ub2 x(i,j) = i + (j-1) * nx end do end do end \end{verbatim} \section{APPENDIX A. NEW HPF UTILITY SUBROUTINES} \subsection{ \type {HPF\_SUBGRID\_INFO (ARRAY, IERR, DIM, LB, UB, STRIDE, LB\_EMBED, UB\_EMBED, AXIS\_MAP)} } \begin{itemize} \item Description. Subgrid-inquiry subroutine. \item \type {ARRAY} is an array of any type, size, shape, or distribution. It is an \type {INTENT (IN)} argument. \item \type {IERR} is a scalar integer. It is an INTENT (OUT) argument. Its return value is zero upon successful return and nonzero otherwise. Errors result if local subgrids cannot be expressed as array sections of \type {ARRAY}. If LB\_EMBED, UB\_EMBED, or AXIS\_MAP is specified, then errors also result if the compiler does not organize the local data in serial memory by sequence associating a larger ``embedding'' array. \item \type {DIM} (optional) is a scalar integer. Is is an \type {INTENT (IN)} argument. \item \type {LB}, \type {UB}, \type {STRIDE}, \type {LB\_EMBED}, \type {UB\_EMBED}, \type {AXIS\_MAP} (all optional) are integer arrays. They are \type {INTENT (OUT)} arguments. If \type {DIM} is specified, they are rank-one arrays of size \type {NUMBER\_OF\_PROCESSORS()} and \type {BLOCK} distribution. If \type {DIM} is not specified, there is a second, collapsed axis whose extent is at least the rank of the global array. The return values are the lower and upper bounds and strides of the array sections describing the local data (in terms of global indices), the lower and upper bounds of the embedding arrays (again, in terms of global indices), and the axes of the embedding arrays to which the axes of \type {ARRAY} are mapped. \end{itemize} \section{APPENDIX B. NEW FORTRAN~77 UTILITY SUBROUTINES} In all of the following: \begin{itemize} \item \type {ARRAY} is some dummy argument passed in from the HPF caller corresponding to some global array or its local subgrid. It is an \type {INTENT (IN)} argument. \item \type {DIM} is a scalar integer. Is is an \type {INTENT (IN)} argument. This argument specifies a particular axis of the global array associated with \type {ARRAY} or, if \type {DIM = -1}, inquiry is for all axes. \item An ``inquiry result'' is an \type {INTENT (OUT)} argument. If \type {DIM = -1}, it is a rank-one array of size equal to at least the rank of the global array associated with \type {ARRAY}, returning information associated with all axes. If \type {DIM} is positive, the ``inquiry result'' is a scalar, returning information only for the axis indicated by \type {DIM}. \item The arguments are defined in the same way as for the corresponding HPF or HPF\_LOCAL routines unless otherwise noted. \end{itemize} \subsection{ \type {F77\_SUBGRID\_INFO (ARRAY, IERR1, IERR2, DIM, LB, UB, STRIDE, LB\_EMBED, UB\_EMBED, AXIS\_MAP)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF subroutine \type {HPF\_SUBGRID\_INFO}. \item \type {IERR1} is a scalar integer. It is an \type {INTENT (OUT)} argument. Its return value is zero if \type {LB}, \type {UB}, and \type {STRIDE} were determined successfully and nonzero otherwise. \item \type {IERR2} is a scalar integer. It is an \type {INTENT (OUT)} argument. Its return value is zero if \type {LB\_EMBED} and \type {UB\_EMBED} were determined successfully and nonzero otherwise. \item \type {LB}, \type {UB}, \type {STRIDE}, \type {LB\_EMBED}, \type {UB\_EMBED}, \type {AXIS\_MAP} are ``inquiry results'' of type integer. They are the lower and upper bounds and strides of the array sections describing the local data (in terms of global indices), the lower and upper bounds of the embedding arrays (again, in terms of global indices), and the axes of the embedding arrays to which the axes of \type {ARRAY} are mapped. \end{itemize} \subsection{ \type {F77\_GLOBAL\_ALIGNMENT (ALIGNEE, LB, UB, STRIDE, AXIS\_MAP, IDENTITY\_MAP, DYNAMIC, NCOPIES)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL subroutine \type {GLOBAL\_ALIGNMENT}. All but the first are \type {INTENT (OUT)} arguments whose return values are as specified by the corresponding HPF routine. \item \type {ALIGNEE} is a dummy argument passed in from global HPF. It is an \type {INTENT (IN)} argument. \item \type {LB}, \type {UB}, \type {STRIDE}, \type {AXIS\_MAP} are integer arrays of rank one. Their size must be at least equal to the rank of the global HPF array associated with \type {ALIGNEE}. \item \type {IDENTITY\_MAP}, \type {DYNAMIC} are scalar logicals. \item \type {NCOPIES} is a scalar integer. \end{itemize} \subsection{ \type {F77\_GLOBAL\_DISTRIBUTION (DISTRIBUTEE, AXIS\_TYPE, AXIS\_INFO, PROCESSORS\_RANK, PROCESSORS\_SHAPE)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL subroutine \type {GLOBAL\_DISTRIBUTION}. All but the first are \type {INTENT (OUT)} arguments whose return values are as specified by the corresponding HPF routine. \item \type {DISTRIBUTEE} is a dummy argument passed in from global HPF. It is an \type {INTENT (IN)} argument. \item \type {AXIS\_TYPE} is a \type {CHARACTER*9} array of rank one. Its size must be at least equal to the rank of the global HPF array associated with \type {DISTRIBUTEE}. \item \type {AXIS\_INFO} is an integer array of rank one. Its size must be at least equal to the rank of the global HPF array associated with \type {DISTRIBUTEE}. \item \type {PROCESSORS\_RANK} is an integer scalar. \item \type {PROCESSORS\_SHAPE} is an integer array of rank one. Its size must be at least equal to the value returned by \type {PROCESSORS\_RANK}. \end{itemize} \subsection{ \type {F77\_GLOBAL\_TEMPLATE (ALIGNEE, TEMPLATE\_RANK, LB, UB, AXIS\_TYPE, AXIS\_INFO, NUMBER\_ALIGNED, DYNAMIC)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL subroutine \type {GLOBAL\_TEMPLATE}. All but the first are \type {INTENT (OUT)} arguments whose return values are as specified by the corresponding HPF routine. \item \type {ALIGNEE} is a dummy argument passed in from global HPF. It is an \type {INTENT (IN)} argument. \item \type {TEMPLATE\_RANK} is a scalar integer. \item \type {LB}, \type {UB}, \type {AXIS\_INFO} are integer arrays of rank one. Their size must be at least equal to the rank of the align-target to which the global HPF array associated with \type {ALIGNEE} is ultimately aligned. \item \type {AXIS\_TYPE} is a \type {CHARACTER*10} array of rank one. Its size must be at least equal to the rank of the align-target to which the global HPF array associated with \type {ALIGNEE} is ultimately aligned. \item \type {NUMBER\_ALIGNED} is a scalar integer. \item \type {DYNAMIC} is a scalar logical. \end{itemize} \subsection{ \type {F77\_ABSTRACT\_TO\_PHYSICAL(ARRAY, INDEX, PROC)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL subroutine \type {ABSTRACT\_TO\_PHYSICAL}. \item \type {INDEX} is a rank-one, \type {INTENT (IN)}, integer array. \item \type {PROC} is a scalar, \type {INTENT (OUT)}, integer. \end{itemize} \subsection{ \type {F77\_PHYSICAL\_TO\_ABSTRACT(ARRAY, PROC, INDEX)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL subroutine \type {PHYSICAL\_TO\_ABSTRACT}. \item \type {PROC} is a scalar, \type {INTENT (IN)}, integer. \item \type {INDEX} is a rank-one, \type {INTENT (OUT)}, integer array. \end{itemize} \subsection{ \type {F77\_LOCAL\_TO\_GLOBAL(ARRAY, L\_INDEX, G\_INDEX)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL subroutine \type {LOCAL\_TO\_GLOBAL}. \item \type {L\_INDEX} is a rank-one, \type {INTENT (IN)}, integer array. \item \type {G\_INDEX} is a rank-one, \type {INTENT (OUT)}, integer array. \end{itemize} \subsection{ \type {F77\_GLOBAL\_TO\_LOCAL(ARRAY, G\_INDEX, L\_INDEX, LOCAL, NCOPIES, PROCS)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL subroutine \type {GLOBAL\_TO\_LOCAL}. \item \type {G\_INDEX} is a rank-one, \type {INTENT (IN)}, integer array. \item \type {L\_INDEX} is a rank-one, \type {INTENT (OUT)}, integer array. \item \type {LOCAL} is a scalar, \type {INTENT (OUT)}, logical. \item \type {NCOPIES} is a scalar, \type {INTENT (OUT)}, integer. \item \type {PROCS} is a rank-one, integer array whose size is at least the number of processors that hold copies of the identified element. \end{itemize} \subsection{ \type {F77\_LOCAL\_BLKCNT(L\_BLKCNT, ARRAY, DIM, PROC)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL function \type {LOCAL\_BLKCNT}. \item \type {L\_BLKCNT} is an ``inquiry result'' of type integer. \item \type {PROC} is a scalar integer. It must be a valid processor number or, if \type {PROC = -1}, the value returned by \type {F77\_MY\_PROCESSOR()} is implied. \end{itemize} \subsection{ \type {F77\_LOCAL\_LINDEX(L\_LINDEX, ARRAY, DIM, PROC)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL function \type {LOCAL\_LINDEX}. \item \type {L\_LINDEX} is a rank-one, integer array of size equal to at least the value returned by \type {F77\_LOCAL\_BLKCNT()}. \item \type {DIM} may not be -1. \item \type {PROC} is a scalar integer. It must be a valid processor number or, if \type {PROC = -1}, the value returned by \type {F77\_MY\_PROCESSOR()} is implied. \end{itemize} \subsection{ \type {F77\_LOCAL\_UINDEX(L\_UINDEX, ARRAY, DIM, PROC)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL function \type {LOCAL\_UINDEX}. \item \type {L\_UINDEX} is a rank-one, integer array of size equal to at least the value returned by \type {F77\_LOCAL\_BLKCNT()}. \item \type {DIM} may not be -1. \item \type {PROC} is a scalar integer. It must be a valid processor number or, if \type {PROC = -1}, the value returned by \type {F77\_MY\_PROCESSOR()} is implied. \end{itemize} \subsection{ \type {F77\_GLOBAL\_SHAPE(SHAPE, ARRAY)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL function \type {GLOBAL\_SHAPE}. \item \type {SHAPE} is a rank-one, integer array of size equal to at least the rank of the global array associated with \type {ARRAY}. Its return value is the shape of that global array. \end{itemize} \subsection{ \type {F77\_GLOBAL\_SIZE(SIZE, ARRAY, DIM)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL function \type {GLOBAL\_SIZE}. \item \type {SIZE} is a scalar integer equal to the extent of axis \type {DIM} of the global array associated with \type {ARRAY} or, if \type {DIM = -1}, the total number of elements in that global array. \end{itemize} \subsection{ \type {F77\_SHAPE(SHAPE, ARRAY)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF intrinsic \type {SHAPE()}, as it would behave as called from HPF\_LOCAL. \item \type {SHAPE} is a rank-one, integer array of size equal to at least the rank of the subgrid associated with \type {ARRAY}. Its return value is the shape of that subgrid. \end{itemize} \subsection{ \type {F77\_SIZE(SIZE, ARRAY, DIM)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF intrinsic \type {SIZE()}, as it would behave as called from HPF\_LOCAL. \item \type {SIZE} is a scalar integer equal to the extent of axis \type {DIM} of the subgrid associated with \type {ARRAY} or, if \type {DIM = -1}, the total number of elements in that subgrid. \end{itemize} \subsection{ \type {F77\_MY\_PROCESSOR(MY\_PROC)} } \begin{itemize} \item Description. FORTRAN~77-callable version of HPF\_LOCAL function \type {MY\_PROCESSOR}. \item \type {MY\_PROC} is a scalar, \type {INTENT (OUT)}, integer. Its value is the identifying number of the physical processor from which this call is made. \end{itemize} \end{document} --------------------------------------------------------------------------- To (un)subscribe to this list, send mail to hpff-external-request@cs.rice.edu. Leave the subject line blank, and in the body put the line (un)subscribe ---------------------------------------------------------------------------