#! /bin/sh
# This is a shell archive.  Remove anything before this line, then unpack
# it by saving it into a file and typing "sh file".  To overwrite existing
# files, type "sh file -c".  You can also feed this as standard input via
# unshar, or by typing "sh <file", e.g..  If this archive is complete, you
# will see the following message at the end:
#		"End of shell archive."
# Contents:  kernel.tex ON_directive_proposal.txt OOC_proposal.txt
#   Reduction_proposal.txt Task_proposal.txt interoperability.tex
#   sep-95-cci
# Wrapped by chk@erato on Mon Sep 18 23:10:32 1995
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
if test -f 'kernel.tex' -a "${1}" != "-c" ; then 
  echo shar: Will not clobber existing file \"'kernel.tex'\"
else
echo shar: Extracting \"'kernel.tex'\" \(19994 characters\)
sed "s/^X//" >'kernel.tex' <<'END_OF_FILE'
X\documentstyle[twoside,11pt]{article}
X
X\pagestyle{myheadings}
X\sloppy
X\footheight=-.60in
X\headheight=0in
X\textwidth=5.50in
X\topmargin=-.20in
X\oddsidemargin=0.5in
X\evensidemargin=0.5in
X
X\itemsep=.05in
X\topsep=.05in
X\parsep=.05in
X\parskip=.10in
X\textheight=9in
X
X\newcounter{dofigures}  \setcounter{dofigures}{0}
X\newcounter{dotables}        \setcounter{dotables}{0}
X\newcommand{\havefigures}{\setcounter{dofigures}{1}}
X\newcommand{\havetables} {\setcounter{dotables}{1}}
X
X\newcommand{\cccdocid}[5]{
X  \bibliographystyle{alpha}
X% ******************** Title Page ********************
X  \vspace*{.5in}
X  \centerline{\Large \bf {#1}}
X  \vspace*{.02in}
X  \centerline{{#2}}
X  \vspace*{.02in}
X  \centerline{{#3}}
X  \vspace*{.02in}
X  \centerline{{#4}}
X  \vspace*{.02in}
X  \centerline{{#5}}
X  \vspace*{.5in}
X
X  % ******************** List of Figures ********************
X  % Note: Remove the following line if there are no figures
X%  \ifodd\value{dofigures} \listoffigures \newpage \fi
X%  \ifodd\value{dotables}  \listoftables  \newpage \fi
X%  \markboth{#1} {#3}
X%  \pagenumbering{arabic}
X  }
X
X% ******************** Special macros ********************
X%\catcode`^^Z=9                % Make TeX ignore IBMPC EOF character
X
X\newcommand{\ital}[1]{{\it #1}}
X\newcommand{\type}[1]{{\tt #1}}
X\newcommand{\bold}[1]{{\bf #1}}
X\newcommand{\dfn}[1]{{\it #1}\index{#1}}
X\newcommand{\comment}[1]{}
X
X\def\fineprint{\relax}
X\def\stopfine{\relax}
X
X
X\newcommand{\beginprog}{\begin{tabbing}
Xxxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=\kill
X}
X\newcommand{\tb}{\>\tt}
X\newcommand{\finprog}{\end{tabbing}}
X\newcommand{\progtxt}[1]{``{\tt #1}''}
X\newcounter{save_enum}
X\newcommand{\hilite}[1]{ \item {\bf #1\\} }
X\newcommand{\morelater}{\centerline{$\ldots$\ital{Fill this in later}$\ldots$}}
X
X
X% ******************** Start of Document ********************
X\begin{document}
X
X\cccdocid
X  {Kernel HPF}
X  {Ver 2.0}
X  {Andrew Meltzer}
X  {Cray Research, Inc.}
X  {August, 1995}
X
X\begin{abstract}
X
XHPF contains many directives which offer ease of use, but
Xthe excessive overhead associated with these directives can severely
Xcompromise performance on one or more (even all for some combination
Xof features) distributed memory parallel computers.
XThis report identifies a kernel of HPF which performs well across many
Xplatforms.  It is an implementors view and may not contain all of the
Xfeatures all applications developers desire, but all of the features
Xwhich are provided enhance performance.
X
X\end{abstract}
X
X\section{Introduction}
X
XHPF contains many directives which offer ease of use, but
Xthe excessive overhead associated with these directives can severely
Xcompromise performance on one or more (even all for some combination
Xof features) distributed memory parallel computers.
XKernel HPF is a high performance subset of HPF.  The features chosen
Xfor the Kernel HPF subset were selected for the following reasons
X\begin{itemize}
X\item They allow high-performance across all platforms.
X\item On no platform are there performance ``surprises,''  Few, if any, 
X      combinations
X      of Kernel HPF features unexpectedly lead the user into low-performance 
X      circumstances although it is still possible for application
X      programmers to write low-performance codes and for vendors to 
X      provide low-performance implementations.
X\item They are easily understood.
X\item They are commonly used.
X\item They are valuable for all platforms.
X\end{itemize}
X
XKernel HPF limits the directives from HPF
Xto a core subset that can be shown to be high performance across a broad 
Xspectrum of machines.   
X
XItems are taken from throughout the entire HPF specification, Kernel HPF 
Xdoes not limit itself to selections from Subset HPF.
X
XThe set of features chosen for Kernel HPF are not a randomly
Xselected set of the available features.  They are chosen based on
Xthe above constraints.  They are also chosen to
Xsimplify the HPF model by reducing the variety of mappings to
Xonly those most commonly used and to simplify
Xthe model by removing a level of abstraction.
X
XThe first part of this paper describes each feature and its Kernel HPF
Xdefinition along with a rationale.
XPart two of this paper steps through the HPF specification, describing 
Xthe changes to the features of each section.
X
XWith the proposed changes to full HPF that allow all remappings to be
Xdone by the caller, an extrinsic environment is no longer necessary for
Xthe kernel because the calling conventions will be the same.  It is
Xrecommended that HPF\_KERNEL subroutines be distinguished by a 
X\begin{verbatim}
X        CHPF$   HPF_KERNEL
X\end{verbatim}
Xdirective.  This directive indicates that the subroutine which follows
Xis a Kernel HPF subroutine.
X
X
X\section{Kernel HPF Features}
X
XThis section describes each of the features of Kernel HPF in detail
Xand gives a rationale for the inclusion or exclusion of each. 
X
X\subsection{The \type {DISTRIBUTE} Directive}
X
XThe {\type {DISTRIBUTE}} directive remains mostly intact, only a few pieces
Xof functionality have been removed.
X\subsubsection{Distribution Formats}
X
XKernel HPF includes only the {\type {BLOCK}} distribution.  It does
Xnot include the {\type {CYCLIC}}, {\type {CYCLIC(N)}}, or {\type {BLOCK(N)}}
Xdistributions.  The effort to determine the locations of the elements
Xin an array with the {\type {BLOCK}} distribution is much less than the
Xother distributions.
X
X%Kernel HPF includes {\type {BLOCK}}, {\type {CYCLIC}}, and 
X%{\type {CYCLIC(N)}} distributions.  It does not include {\type {BLOCK(N)}}
X%distributions.
X
X%{\em{Rationale}}:
X%\begin{itemize}
X%\item  {\type {BLOCK}} is the most common and useful distribution in common
X%       practice.  It is also the most straightforward to implement efficiently.
X%\item  {\type {CYCLIC}} is the next most common distribution.  It allows
X%       maximal load balancing.
X%\item  {\type {CYCLIC(N)}} allows the user to tune the 
X%       communicaton/load-balance ratio for particular applications and 
X%       particular architectures.  It is very important when fine-tuning
X%       for performance.
X%\item  {\type {BLOCK(N)}} is difficult to implement efficiently  and 
X%       it is unclear how a high-performance application could effectively
X%       exploit this feature.
X%\end{itemize}
X
XThere is some sentiment that the kernel should contain more distribution
Xtypes, for example that the {\type {CYCLIC(N)}} distribution be added.
X
X\subsection{\type {DYNAMIC} Distributions}
X
X{\type The {DYNAMIC}} attribute is disallowed in Kernel HPF.
X
X\subsubsection{On-Processor Dimensions}
XThe ``*'' syntax is included in the {\type {DISTRIBUTE}} directive 
X{\em {dist-format-clause}} to indicate on-processor dimensions.
X
X{\em {Rationale}}: This is a convenient technique for increasing locality of
Xreference, a goal for most high performance codes.  It can be accomplished
Xefficiently with reasonable compiler effort.
X
X
X\subsection{The \type {PROCESSORS} Directive}
X
XThe {\type {PROCESSORS}} directive is part of Kernel HPF without change.
X
X{\em {Rationale}}: The processors directive is an important way to
Xindicate the equivalence of mappings.  While most implementations
Xwill probably map arrays with identical distributions and alignments
Xthe same, portability requires that the {\type {PROCESSORS}} directive
Xbe made available.  
X
XThe directive is also perhaps the only way to fine-tune multidimensional
Xdistributions.
X
X
X\subsection{The \type {ALIGN} Directive}
X
XIn Kernel HPF the {\type {ALIGN}} directive is restricted to
Xalignments that are direct;  there may be no offsets or strides
Xwithin an  {\type {ALIGN}}.  Replication and collapsing of dimensions
Xare also disallowed.  In alignment expressions the dimensions may
Xnot be permuted, for example:
X
X\begin{verbatim}
X
X        !HPF$ ALIGN A(I,J) WITH B(J,I)  ! Not allowed in Kernel
X
X\end{verbatim}
X
X{\em {Rationale}}: While these measures initially appear to be 
Xdraconian restrictions on programmability, almost all of the useful
Xfunctionality provided by these features is available through other
Xmeans which are more straightforward to optimize.  Adding strides to the 
X{\type {ALIGN}} directive, among other things, greatly complicates 
Xthe addressing, probably ends up wasting space, and increases the 
Xamount of information which must
Xbe passed across subroutine boundaries. 
X
XIf the {\type {CYCLIC(N)}} distribution is accepted as part of the
Xkernel, almost everything that one wants to do with strided alignments 
Xcan be done by varying the block size.  In both of the following
Xexamples the elements of {\type A} are aligned with the even elements
Xof {\type B}.
X
X    {\em {Example 1a}}
X\begin{verbatim}
X                DIMENSION A(128), B(256)
X        CHPF$   DISTRIBUTE B(CYCLIC(16))
X        CHPF$   ALIGN A(I) WITH B(2*I)  ! Disallowed in Kernel
X\end{verbatim}
X
X    {\em {Example 1b}}
X\begin{verbatim}
X                DIMENSION A(128), B(256)
X        CHPF$   PROCESSORS P(32)
X        CHPF$   DISTRIBUTE A(CYCLIC(8)) ONTO P
X        CHPF$   DISTRIBUTE B(CYCLIC(16)) ONTO P
X\end{verbatim}
X
XIt has been noted however, that the vast majority of codes do not require
Xthis functionality so its lack of inclusion into the HPF kernel would not
Xbe overly restrictive.
X
XOffsets are not allowed in the {\type {ALIGN}} directive of Kernel HPF.  The 
Xeffect of offsets can be
Xgenerally be achieved by changing the bounds of arrays (although this
Xmay be implementation dependent, it is not reaching all that far.)
XThe following two examples will create the same alignment.
X
X    {\em {Example 2a}}
X\begin{verbatim}
X                DIMENSION A(1:100), B(0:99)
X        CHPF$   PROCESSORS P(32)
X        CHPF$   DISTRIBUTE B(BLOCK) ONTO P
X        CHPF$   ALIGN A(I) WITH B(I-1)  ! Disallowed in Kernel
X\end{verbatim}
X
X    {\em {Example 2b}}
X\begin{verbatim}
X                DIMENSION A(1:100), B(0:99)
X        CHPF$   PROCESSORS P(32)
X        CHPF$   DISTRIBUTE A(BLOCK) ONTO P
X        CHPF$   DISTRIBUTE B(BLOCK) ONTO P
X\end{verbatim}
X
XCollapsing of dimensions is disallowed because it can be accomplished 
Xwith the {\type {DISTRIBUTE}} directive.  
X
XReplication is not available with these restrictions, but Kernel HPF
Xis not meant to solve all problems.  Unfortunately to achieve the
Xgoal of adding replication, a whole lot of unwanted baggage would
Xalso be required.   It may be possible to add some syntax to explicitly
Xrecommend replication but it should not be done through the {\type {ALIGN}}
Xdirective.
X
X\subsection{The {\type {TEMPLATE}} Directive}
X
XThe template in a {\type {TEMPLATE}} directive must have the same rank 
Xand extents as the arrays aligned to it.  This, in 
Xessence, reduces the the functionality of templates to a convenient
Xway of specifying that different (identically dimensioned) arrays are 
Xto be distributed identically.  
X
X{\em {Rationale}}: Mis-use of templates is probably the easiest way
Xto turn a high performance code into a low performance one, and it
Xis likely that different compilers will optimize different parts
Xof the complexities of templates due to underlying hardware performance
Xconsiderations. Even when a user finds a complicated template layout
Xto be high performance on one platform it will probably not be high
Xperformance, or even moderately high performance, on all platforms.
X
XAlso, nearly all of the high-performance features of templates
Xare available through other mechanisms.
X
X
X\subsection{The {\type {FORALL}} Statement and Construct}
X
XThese are included in Kernel HPF.
X
X{\em {Rationale}}: {\type {FORALL}} is a powerful and expressive way
Xto code parallelism.  It is true that not all {\type {FORALL}} 
Xstatements and/or constructs can be fully parallelized, but because
Xof their general value and because they will be part of Fortran 95,
Xthey are included in Kernel HPF.
X
X\subsection{{\type {PURE}} Functions and Subroutines}
X
XThese are included in Kernel HPF.
X
X{\em {Rationale}}: {\type {PURE}} functions can do no harm, and allow
Xan increased level of expressiveness.  They simplify the job for
Xthe compiler and are part of Fortran 95.
X
X\subsection{The {\type {INDEPENDENT}} Directive}
X
XThe {\type {INDEPENDENT}} directive is included in Kernel HPF.
X
X{\em {Rationale}}: Directives such as {\type {INDEPENDENT}} have shown
Xtheir worth historically across a broad variety of platforms.  This
Xdirective simplifies the analysis the compiler must do and can inform
Xthe compiler about optimizations that it may do that otherwise
Xare expensive or impossible to deduce.   {\type {INDEPENDENT}} allows 
Xan increased level
Xof optimization.
X
X\subsection{Extrinsics and {\type {HPF\_LOCAL}} and {\type {HPF\_SERIAL}} }
X
XThe general purpose extrinsic mechanism, {\type {HPF\_LOCAL}} and 
X{\type {HPF\_SERIAL}} are all included in Kernel HPF.
X
X{\em {Rationale}}: These allow a program to get at the highest 
Xperforming features of a particular architecture.
X
X
X\subsection{Library}
X
XAll of the HPF library routines are in Kernel HPF.  
X
X{\em {Rationale}}: The intrinsics provide fast, architecture specific
Xversions of many very important parallel functions.  To achieve a
Xsimilar level of performance for these functions a user would have
Xto write non-portable code.
X
X
X\subsection{The {\type {INHERIT}} Directive}
X
XThe  {\type {INHERIT}} directive is not part of Kernel HPF.
X
X{\em {Rationale}}:  The usefulness of {\type {INHERIT}} is greatly
Xlimited by the rest of the restrictions placed on Kernel HPF.  It
Xstill has some use when passing, for example, array sections with
Xstrides but the additional mechanisms that would have to be added
Xelsewhere to handle this case do not justify its inclusion.
X
X\subsection{Subroutine Interfaces}
X
XIn Kernel HPF explicit interfaces are required for any subroutine
Xwhich remaps data.
XDummy arguments from actual argument array sections cannot be 
Xmapped ``{\type {DISTRIBUTE * ONTO *}}.'' Non-unit
Xstrides may be specified when passing array sections as actual
Xarguments.
X
XAssumed shape explicitly mapped dummies are
Xallowed in Kernel HPF, but explicit interfaces must be included where
Xthey are used. Assumed shape dummies may use any mapping syntax.
XThe results of array valued functions may be explicitly mapped.  Assumed
Xsize arrays are not allowed in the Kernel.
X
XTo be exact, an explicit interface is required in each of the following
Xcases:
X\begin{itemize}
X\item A parameter is passed transcriptively or with the inherit attribute.
X\item The mapping of the dummy argument is not the same as the mapping of
X      the actual argument.
X\end{itemize}
X
X
X
X{\em {Rationale}}: Much better control of the remapping can be achieved
Xif the explicit interface is available and much less code to check
Xmappings and remap upon entry is necessary.
X
X
X\section{HPF Specification by Section}
X\subsection{Sections 1 and 2. Overview and Terms and Concepts}
X
XNo changes are needed here for Kernel HPF.
X
X\subsection{Section 3. Data Alignment and Distribution Directives}
X
XThis section contains much of the substance of the HPF specification:
Xthe data alignment and distribution directives.  Kernel HPF severely
Xlimits these directives to only those where high
Xperformance across a broad spectrum of machines can be expected.
X
XThe most restrictions are placed on alignment and templates.  Alignments
Xcannot be permuted, nor can they be offset.  Templates must be identical
Xin rank and size to the arrays aligned to them.
X
X\subsection{Section 3.1 Model}
X
XThe Kernel HPF model is a
Xsmall subset of the HPF model; this subset is so restricted that a
Xsimpler semantic model could have been used. On page 21 of the HPF
Xspecification the model is graphically displayed as a four-level mapping
Xfrom ``arrays or other objects'' to ``group of aligned objects'' to ``abstract
Xprocessors as a user-declared cartesian mesh'' and finally to ``physical
Xprocessors''  In Kernel HPF this model is simplified to three levels by
Xseverly restricting the usefulness of templates.
X
X
X\subsection{Section 3.2. Syntax of Data Alignment and Distribution Directives}
X
XNone of the keywords {\type {REALIGN}}, {\type {REDISTRIBUTE}, 
X{\type {INHERIT}, or {\type {DYNAMIC}} are
Xpart of Kernel HPF. With this exception,
Xsection 3.2 correctly describes the syntax of the data alignment and
Xdistribution directives of Kernel HPF.
X
X\subsection{Section 3.3 {\type {DISTRIBUTE}} and {\type {REDISTRIBUTE}} Directives}
X
X\type {REDISTRIBUTE} and {\type {DYNAMIC}} are not part of Kernel HPF.
X
X\type {BLOCK} is available in Kernel HPF. 
X
XCollapsing of dimensions can only be
Xachieved through distribution, not alignment.  See the following
Xsection for details of the restriction.  The `*' syntax
Xis recognized by Kernel HPF and indicates that the 
Xdimension is on-processor.
X
X\subsection{Section 3.4 {\type {ALIGN}} and {\type {REALIGN}}}
X
X{\type {REALIGN}} is not part of in Kernel HPF
XThe {\type {ALIGN}} directive itself is very
Xrestricted.  The following are additional rules placed on the {\type {ALIGN}}
Xdirective
X
X\begin{itemize}
X\item   Alignments must be direct, i.e., there can be no offsets
X        specified.      For example the following is correct code:
X
X                {\type {CHPF\$           ALIGN A(I,J) WITH B(I,J)}}
X
X         But the following is not:
X
X                {\type {CHPF\$           ALIGN A(I,J) WITH B(I+1,J)}}
X
X\item   The {\em {align-subscript-use}} in the HPF syntax rules should be
X        replaced with {\em {align-dummy}}.
X
X\item   Dimensions may not be permuted.
X
X\item   The {\em {alignee}} and the {\em {align-target}} of 
X        the {\type {ALIGN}} directive must have the same rank and extents.
X
X\item   A `*' is not allowed in either the align-spec or the alignee.
X\end{itemize}
X
X\subsection{Section 3.5. {\type {DYNAMIC}} Directive}
X
XThe {\type {DYNAMIC}} Directive is not part of Kernel HPF.
X
X\subsection{Section 3.6. Allocatable Arrays and Pointers}
X
XAllocatable arrays and pointers in Kernel HPF may be explicitly mapped.
X
X\subsection{Section 3.7. {\type {PROCESSORS}} Directive}
X
XThe {\type {PROCESSORS}} directive is allowed.
X
X\subsection{Section 3.8. {\type {TEMPLATE}} Directive}
X
XTemplates may only be the shape (rank and extent for each dimension) of
Xthe arrays aligned to them. This restriction, in conjunction with the
Xrestrictions on alignment make moot almost all of the examples in
XSection 3.8. The restrictions placed on templates are so severe, in
Xfact, that they reduce the usefulness of templates to a syntactically
Xconvenient way of mapping many identical arrays in a similar fashion.
X
X\subsection{Section 3.9. {\type {INHERIT}} Directive}
X
XThe {\type {INHERIT}} directive is not part of Kernel HPF. 
X
X\subsection{Section 3.10 Alignment, Distribution, and Subprogram Interfaces}
X
XKernel HPF requires interface blocks for any arguments that are re-mapped.
XThis allows the caller to re-map data only when necessary, greatly 
Xsimplifying (and consequently speeding up) the subroutine call interface.
X
XArbitrary array sections may be passed, but must have an explicit
Xinterface and cannot be mapped ``{\type {DISTRIBUTE * ONTO *.}}'' Non-unit
Xstrides may be specified when passing array sections as actual
Xarguments.
X
XKernel HPF retains the distinctions and all of the syntax relating to
Xtranscriptive, prescriptive and descriptive mappings in full HPF.
X
XThe results of array valued functions may be explicitly mapped.
X
X\subsection{Section 4. Data Parallel Statements and Directives}
X
X\subsection{Section 4.1 and 4.2. The {\type {FORALL}} Statement and Construct}
X
XThe {\type {FORALL}} statement and the {\type {FORALL}} construct are part
Xof Kernel HPF.
X
X\subsection{Section 4.3. Pure Procedures}
X
XPure procedures and functions are part of Kernel HPF.  
X
X
X\subsection{Section 4.4. The {\type {INDEPENDENT}} Directive}
X
XThe {\type {INDEPENDENT}} directive is part of Kernel HPF.  The {\type {NEW}}
Xclause is also part of Kernel HPF. 
X
X\subsection{Section 5. Intrinsic and Library Procedures}
X
XKernel HPF should contain all intrinsic and library procedures.  
X
X\subsection{Section 7. Storage and Sequence Association}
X
XThere is no storage or sequence association in Kernel HPF.
X
X\end{document}
END_OF_FILE
if test 19994 -ne `wc -c <'kernel.tex'`; then
    echo shar: \"'kernel.tex'\" unpacked with wrong size!
fi
# end of 'kernel.tex'
fi
if test -f 'ON_directive_proposal.txt' -a "${1}" != "-c" ; then 
  echo shar: Will not clobber existing file \"'ON_directive_proposal.txt'\"
else
echo shar: Extracting \"'ON_directive_proposal.txt'\" \(45801 characters\)
sed "s/^X//" >'ON_directive_proposal.txt' <<'END_OF_FILE'
XThe ON Clause
X
XCharles Koelbel and Rob Schreiber
X
XThe purpose of the ON directive is to allow the programmer to control the 
Xdistribution of computations among the processors of a parallel machine.  
XIn a sense, this is the dual of the DISTRIBUTE and ALIGN directives for 
Xdata.  Modern parallel machines will achieve their best performance if all 
Xoperations are performed by separate processors (as specified by ON) with 
Xeach processor accessing its local data (as specified by DISTRIBUTE).  
XIf either explicit computation partitioning or explicit data mapping is not 
Xpresent, the system must choose one.  Another proposal explains how this 
Xmight be done.
X
XA secondary purpose of the ON directive is to identify data accessed by 
Xthe computation as local to the executing processor.  The compiler can use 
Xthis information to avoid generating communication or to simplify array address 
Xcalculations.  Note that whether a given data element is local depends on 
Xtwo facts:
X	Where the data is stored (i.e. its DISTRIBUTE and ALIGN attributes)
X	Where the computation is executed (i.e. its ON directive)
XFor these reasons, the LOCAL clause is added to the ON directive, which is 
Xeffectively the earliest point in the program where the needed facts might 
Xbe available.  Note that changing the ON directive may invalidate some 
XLOCAL clauses, or may make some new LOCAL clauses true.
X
X
XSyntax
X
XThere are two flavors of the ON directive: a single-statement form, and a 
Xmulti-statement form.  The BNF for both is
X
X	simple-on-directive	IS	ON HOME ( home-expr ) [, LOCAL ( local-var-list ) ]
X	
X	block-on-directive	IS	simple-on-directive BLOCK
X	end-directive	IS	END [ BLOCK ]
X	
X	on-block	IS	
X		block-on-directive
X		block
X		end-directive
X
X	home-expr	IS	variable
X		OR	template-elmt
X		OR	processors-elmt
X	
X	template-elmt	IS	template-name [ ( section-subscript-list ) ]
X	
X	processors-elmt	IS	processors-name [ ( section-subscript-list ) ]
X	
X	local-var	IS	variable
X		OR	ALL = variable-name
X
XNotes:
XConstraints need to be added above...
XAlso, changes are needed in some higher-level syntax (for example, adding 
Xsimple-on-directive as an option to executable-directive).
X
Xhome-expr, template-elmt, and processors-elmt are just auxiliary syntax 
Xcategories.  Note that "variable" is an F90 syntax term that means 
X(roughly) "a reference, including an array element, array section, or 
Xderived type field"; "variable" doesn't include template or processor 
Xelements since those aren't first-class language constructs.  Note also 
Xthat "block" is an F90 syntax term for "a series of statements treated as 
Xa group" - for example, the body of a DO construct.
X
Xsimple-on-directive should be an option under executable-directive (HPF 
Xsyntax term), as redistribute-directive is now.
X
Xon-block most naturally fits as a F90 executable-stmt, since that makes 
Xconstraints about "properly nesting" easier to state.  
X
XAny top-level variables in the LOCAL clause must be explicitly 
Xmapped.  Otherwise, the assertion (see below) is a statement about how the 
Xcompiler works, not about the program.  (Alternately, somebody could 
Xsuggest another definition of LOCAL, I suppose...)
X
X
XRationale:
XThe BLOCK in on-block is necessary so that the parser can distinguish 
Xbetween the one-statement or multi-statement forms.  BEGIN would be equally 
Xgood as a keyword.
X
XHOME is required, even when its argument is a processor.  It seems more 
Xnatural to eliminate the HOME keyword there, but this leads to the
Xfollowing ambiguity:
X
X	INTEGER X(4)	! X(I) will be on processor I
X	!HPF$ PROCESSORS HOME(4)
X	!HPF$ DISTRIBUTE X(BLOCK)
X	X = (/ 4,3,2,1 /)
X	!HPF$ ON HOME(X(2))
X	X(2) = X(1)
X
XIf HOME were not required, where should the computation be done:
X1. Processor HOME(2) (i.e. the owner of X(2))?
X2. Processor HOME(3) (i.e. the value of X(2), before the assignment)?
X3. Processor HOME(4) (i.e. the value of X(2), after the assignment)?
XThe definition of ON clearly indicates interpretation 1 is correct.  One 
Xcan get the effect of interpretation 2 by the directive
X	!HPF$ ON HOME(HOME(X(2)))
XThere is no way to get the effect of interpretation 3, short of building a 
Xclairvoyant computer.  Introducing reserved keywords into Fortran was 
Xsuggested as a better solution to this problem, but was seen as too large a 
Xchange to the underlying language.
XEnd of rationale.
X
X 
XSemantics
X
XThe ON directive advises the compiler to perform the computation in the 
Xfollowing statement(s) using the processor(s) named in the argument to 
XHOME.  Like the mapping directives ALIGN and DISTRIBUTE, this is advice 
Xrather than an absolute commandment; the compiler may override it if 
Xnecessary.  Also like ALIGN and DISTRIBUTE, the ON directive may affect the 
Xefficiency of computation, but not the final results.
X
XAdvice to implementors:
XIf the compiler may override the user's advice in an ON directive, then 
Xthe compiler should also offer the user an option to force all directives 
Xto be obeyed.
XEnd advice to implementors.
X
XThe single-statement ON directive (i.e.  simple-on-directive) precedes 
Xstatement for which it is advising behavior, and is said to apply to that 
Xstatement.  If the statement is a compound statement (e.g.  a DO loop or an 
XIF-THEN-ELSE construct), then the ON directive also applies to all nested 
Xstatements therein.  Similarly, the block ON directive (i.e.  on-block) 
Xapplies the initial ON clause to all statements up to the matching end 
Xdirective.  
X
XIt is legal to nest ON directives, provided that the processors named by 
Xthe inner ON directive are also named by the outer directive.  Similarly, 
Xan ON directive can apply to a CALL statement where the called subroutine 
Xcontains an ON directive.  In this case, any processor named inside the 
Xsubroutine must be included by the ON directive in the caller.  Note that 
Xthe syntax of on-block automatically ensures that it is properly nested 
Xinside other compound statements, and that compound statements properly 
Xnest inside of it.  As with other Fortran 90 compound statements, transfer 
Xof control to the interior of an on-block from outside the block is 
Xprohibited, while transfers within a block may occur.  However, HPF also 
Xprohibits transfers of control from the interior of a on-block to outside 
Xthe on-block.  Note that this is stricter than Fortran 90.
X
XRationale:
XThis allows a fork-join approach to nested parallelism, which seems to be 
Xnatural to many applications and programmers.  The restrictions about 
Xcontrol flow into and out of an ON block essentially make it a single-entry 
Xsingle-exit region, thus simplifying the semantics considerably.
XEnd Rationale.
X
X
XThe HOME clause can name a program variable, a template, or a processors 
Xarrangement.  For each of these possibilities, it can further specify a 
Xsingle element or multiple elements.  This is translated into the 
Xprocessor executing the computation as follows:
X*	If the HOME clause names a program variable (that is, an array element 
X	or section), then every processor owning any element of the variable 
X	will execute the following statement(s).
X*	If the HOME clause names a template element or section, then every 
X	processor owning any element of the template will execute the following 
X	statement(s).
X*	If the HOME clause names a processors arrangement, then the processor(s) 
X	referenced there will execute the following statements.
XIn all cases, the value(s) of the HOME clause are determined as if the HOME 
Xargument was evaluated when control flow reached the ON directive.  That 
Xis, if the value of a variable used in the HOME clause changes within the 
Xon-block, work is not migrated to a new processor.  If evaluation of the 
Xargument of the HOME clause would change the value of any variable in the 
Xprogram, then the value of that variable becomes undefined after the ON 
Xclause is reached.  Finally, note that the HOME clause only specifies how 
Xcomputation is partitioned among processors; it does not indicate 
Xprocessors that may be involved in data transfer.  
X
XRationale:
XThis is the heart of the ON clause.  
X
XThe "as if" wording and making side effects undefined avoids the following 
Xproblem: ON is a directive.  Therefore, it cannot have side effects.  But 
Ximplementing the ON clause may require evaluating some functions, therefore 
Xcausing the side effects.  An alternative would be to only allow PURE 
Xfunctions in the HOME and LOCAL clauses.
XEnd of rationale.
X
XAdvice to implementors:
XIf the HPF program is compiled into Single-Program-Multiple-Data (SPMD) 
Xcode, then the ON clause can always be implemented (albeit inefficiently) 
Xby having all processors compare their processor id to an id (or list of 
Xids) generated from the HOME clause.  (Similar naive implementations can be 
Xconstructed in other paradigms as well.) If the ON clause will be executed 
Xrepeatedly, for example in a DO loop, it is probably worthwhile to invert 
Xthis process.  That is, instead of all processors executing all the HOME 
Xclause tests, the compiler should determine the range of loop iterations 
Xthat will give a true test.  (See the "Advice to implementors" in the 
XExamples section below for more details.) For example, consider the 
Xfollowing complex case:
X	DO I = 1, N
X		!HPF$ ON HOME( A(MY_FCN(I)) ) BLOCK
X		...
X		!HPF$ END BLOCK
X	END DO
XHere, the generated code can perform an "inspector" (i.e.  a skeleton loop 
Xthat only evaluates the HOME clause of each iteration) to produce a list of 
Xiterations assigned to each processor.  This list can be produced in 
Xparallel, since MY_FCN must be side-effect free; however, distributing it 
Xto all processors may require unstructured communications patterns, 
Xpossibly negating the advantage of parallelism.  In general, more advanced 
Xcompilers will be able to efficiently invert more complex HOME clauses.  It 
Xis recommended that the abilities (and limitations) of a particular 
Xcompiler be documented clearly for potential users.
XEnd advice to implementors.
X
XAdvice to programmers:
XThe argument to HOME in an ON clause can be arbitrarily complex.  This is a 
Xtwo-edged sword; it can express very complicated computation partitioning, 
Xbut the implementation of these partitions may not be efficient.  More 
Xconcretely, it may express a perfectly load-balanced computation, but force 
Xthe compiler to serialize the computation of the HOME clauses.  Although 
Xthe amount of overhead for an ON clause will vary based on the HPF code, 
Xthe compiler, and the hardware, one can expect that compilers will generate 
Xvery good code based solely on array mappings, and progressively worse code 
Xas the complexity of the HOME clause increases.  A rough measure of 
Xthe complexity of an ON directive is the amount of run-time data used to 
Xcompute it; for example, a constant offset is fairly simple, while a 
Xpermutation array is very complex.  See the Examples section below for 
Xmore concrete examples of this phenomenon.
XEnd of advice to programmers.
X
XIf ON clauses are nested, then the innermost HOME clause effectively 
Xcontrols execution of the statement(s).  A programmer can think of this as 
Xsuccessively restricting the set of processors at each level of nesting; 
Xclearly, the last restriction must be the strongest.  Similarly, if ON 
Xclauses appear in a subroutine call chain, then the deepest ON clause is 
Xthe effective controller.
X
XRationale:
XThis reinforces the idea of the fork-join parallelism pointed out earlier.
XEnd of rationale.
X
XAdvice to implementors:
XNotice that nested ON clauses do not present additional problems for the 
Xnaive implementation.  If a processor failed the outer HOME comparison, 
Xthen it would fail any tests inside the ON as well.  Thus, it need not even 
Xmake the tests.
XEnd of advice to implementors.
X
XOperations controlled by an ON clause must follow certain restrictions:
X
X* If an ON directive applies to a DISTRIBUTE directive, then the set of 
Xprocessors named in the HOME clause must include
X	- All processors named in the DISTRIBUTE's ONTO clause if one is present
X	- All processors in the default processors arrangement, if there is no 
X	  ONTO clause
X* If an ON directive applies to a REDISTRIBUTE directive, then the set of 
Xprocessors named in the HOME clause must include 
X	- All processors that stored any element of the distributee before 
X	  the REDISTRIBUTE was encountered
X	- The processors required by an equivalent DISTRIBUTE directive
X* If an ON directive applies to a REALIGN directive, then the set of 
Xprocessors named in the HOME clause must include 
X	- All processors that store any element of the align-with-clause
X* If an ON directive applies to a REALIGN directive, then the set of 
Xprocessors named in the HOME clause must include  
X	- All processors that stored any element of the alignee before 
X	  the REALIGN was encountered
X	- The processors required by an equivalent ALIGN clause
X* If an ON directive applies to an ALLOCATE statement which creates an 
Xexplicitly mapped variable, then the set of processors named in the HOME clause
Xmust include the processors required by the associated DISTRIBUTE or ALIGN 
Xdirective.
X* If an ON directive applies to any other declaration of an explicitly 
Xmapped variable, then the set of processors named in the HOME clause
Xmust include the processors required by the associated DISTRIBUTE or ALIGN 
Xdirective.
X
XIf operations within an ON block do not follow these constraints, then the 
Xprogram is not HPF-conforming.
X
XRationale:
XOperations which allocate memory require the cooperation of all processors 
Xthat will own that memory.  Therefore, ON clauses must schedule those 
Xoperations to execute on the cooperating processors.
XEnd of rationale.
X
XAdvice to implementors:
XThese restrictions ensure that HPF data distribution directives inside 
Xan ON block (either lexically or in a dynamic call chain) can be 
Ximplemented without relying on one-way communication outside of the 
X"current" processing group.
XEnd of advice to implementors.
X
XThe LOCAL clause, if present, is an assertion to the compiler that certain 
Xarray references made within the ON are stored in local memory if the 
Xcomputation is performed by the processor(s) named in the HOME clause.  
XSpecifically, if an element in the local-var list is a variable, array, or 
Xarray section, then that element will be stored on at least one of the 
Xprocessors from the HOME clause.  If the local-var is of the form 
XALL=variable-name, then all references to that variable that the ON 
Xapplies to will be stored on at least one of the named processors.  As with 
Xthe HOME clause, the value(s) of the LOCAL clause are determined as if the 
Xlocal-var-list was evaluated when control flow reached the ON directive, 
Xand possible side effects in the LOCAL clause cause the affected variables 
Xto become undefined.
X
XRationale:
XThe ALL=variable-name form is syntactic sugar for listing every 
Xreference to an array in the ON block.  This is similar in spirit to APR's 
Xgrouping IGNORE directives by variable.  I'm open to better syntax for 
Xsaying this.  (Note that ALL(variable-name) doesn't work, since somebody 
Xmight name a variable ALL.  Grr...)
X
XThe pseudo-evaluation of the LOCAL clause is the same as the HOME clause, 
Xfor the same reason.  Treating the LOCAL clause as naming sets of 
Xvariables, rather than as a list of lexical references, gives a reasonably 
Xfirm  logical basis to what the assertion means.  It is unclear whether 
Xcompilers would rather see this "symbolic" information or longer lists that 
Xneed to be pattern-matched.  It is also unclear which approach would 
Xappeal more to programmers.
XEnd Rationale.
X
XAdvice to programmers:
XSince the
XEnd of advice to programmers.
X
XNote that if the HOME clause specifies more than one processor, then LOCAL 
Xonly asserts that the variables are stored on one of the processors.  For 
Xexample, if a statement is executed on a section of the processors 
Xarrangement, then communication within that section may be needed for some 
Xvariables in the LOCAL clause.  Communication with processors outside of 
Xthe section will not be needed for those variables, however.
X
XRationale:
XThe alternative to this interpretation would be that any variable named in 
Xthe LOCAL clause would be local to all processors, i.e. replicated.   
XWhile that certainly allows more extensive optimizations, it is a less 
Xcommon case.  In addition, it does not seem to capture the intent of ON 
Xdirectives applied to CALL statements or compound statements.  For example,
X	!HPF$ PROCESSORS PROCS(MP,MP)
X	!HPF$ DISTRIBUTE X(BLOCK,BLOCK) ONTO PROCS
X	!HPF$ ON HOME(PROCS(1,1:MP)), LOCAL( X(K,1:N) )
X	CALL FOO( X(K,1:N) )
Xwould presumably call FOO on a row of the processors arrangement, passing 
Xelements of X in place.  This is what the current definition does; if 
XLOCAL meant "local on every processor", the call would force X to be 
Xreplicated.
XEnd Rationale.
X
XThe LOCAL assertion is similar to the INDEPENDENT directive, in that if it 
Xis correct it does not change the meaning of the program.  If the LOCAL 
Xclause in incorrect, the program is not standard-conforming (and is thus 
Xundefined).  Like the INDEPENDENT directive, the compiler may use the 
Xinformation in the LOCAL clause, or ignore it if it is insufficient for the 
Xcompiler's purposes.  If the compiler can detect that the LOCAL clause is 
Xincorrect (i.e.  that a LOCAL variable is definitely nonlocal), it is 
Xjustified in producing a warning.  Unlike the INDEPENDENT directive, 
Xhowever, the truth of the LOCAL clause depends on the mapping of 
Xcomputations (specified with the ON clause) and the mapping of data 
X(specified with DISTRIBUTE and ALIGN clauses); if the compiler overrides 
Xeither of these, then it may not be able to use information in the LOCAL 
Xdirective.
X
XRationale:
XKnowing that a reference is local is valuable information for the 
Xoptimizer.  It is in keeping with the spirit of HPF to phrase this as an 
Xassertion of fact, which the compiler can use as it pleases.  Expressing 
Xit as advice to the compiler seems to have disadvantages.  Some possible ways 
Xthis advice could be phrased, and the counter-arguments, are 
X*	"Don't generate communication for this reference" has great potential 
X	for changing the meaning of the program.  Some programmers want this 
X	capability, but it violates the "correct directives should not change the 
X	meaning of a program" principle of HPF.  Also, once communication is 
X	"turned off" for a reference, it's not clear how to turn it back on.
X*	"Generate communication for this reference" is not a useful directive, 
X	since the compiler has to do this anyway.
X*	"Generate communication for this reference, and place it here" is useful, 
X	since it can override the default placement by the compiler.  It still 
X	has potential for changing program meaning.  It also has the potential to 
X	create programs as complex as message-passing, as programmers try to move 
X	communication out of loops.
XEnd of rationale. 
X
X
XExamples
X
XThe following are valid examples of ON directives.  Most of them are 
X"reasonable" in the sense that they illustrate idioms that programmers 
Xmight want to use, rather than contrived situations.  For simplicity, the 
Xfirst several examples assume the following array declarations:
X
X	REAL A(N), B(N), C(N), D(N)
X	!HPF$ DISTRIBUTE A(BLOCK), B(BLOCK), C(BLOCK), D(BLOCK)
X
XOne of the most commonly requested capabilities for HPF as to control how 
Xloop iterations were assigned to processors.  (Historically, the ON clause 
Xfirst appeared to perform exactly this role in the Kali FORALL construct.) 
XThis can be done by the ON directive, as shown in the following examples:
X
X	!HPF$ INDEPENDENT
X	DO I = 2, N-1
X	  !HPF$ ON HOME(A(I))
X	  A(I) = (B(I) + B(I-1) + B(I+1))/3
X	END DO
X	
X	!HPF$ INDEPENDENT
X	DO J = 2, N-1
X	  !HPF$ ON HOME(A(J+1)) BLOCK
X	    A(J) = B(J+1) + C(J+1) + D(J+1)
X	  !HPF$ END BLOCK
X	END DO
X
XThe ON directive in the I loop advises the compiler to have each processor 
Xrun over its local section of the A array (and therefore B as well).  The 
Xreferences to B(I-1) and B(I+1) must be fetched from off-processor for the 
Xfirst and last iterations on each processor (except for the boundary 
Xprocessors); note that those processors are not mentioned in the HOME 
Xclause.  The ON directive in the J loop advises the compiler to "shift" 
Xcomputations so that each processor does a vector sum of its local sections 
Xof B, C, and D, stores the first element of the result on the processor to 
Xits left, and stores the rest of the result (shifted by one) in A.  It is 
Xworth noting that the directives would still be valid (and minimize 
Xnonlocal data accesses) if the arrays were distributed CYCLIC, although the 
Xnumber of nonlocal references would be much higher.
X
XAdvice to implementors:
XIt is highly recommended that compilers concentrate on optimizing DO loops 
Xwith a single ON clause including the entire loop body.  Schematically, 
Xthe code will be:
X
X	DO i = lb, ub, stride
X		!HPF$ ON HOME(array(f(i))) BLOCK
X		body
X		!HPF$ END BLOCK
X	END DO
X
XWhere array has some data mapping.  Assume the mapping give processor p 
Xthe elements my_set(p).  (In a BLOCK distribution, for example, my_set(p) 
Xis a contiguous range of integers.)  Then the generated code on processor 
Xp should be
X
X	DO i in [lb:ub:stride] intersect f^-1(my_set(p))
X		body
X	END DO
X
X(This schematic does not show where communication or synchronization must 
Xbe placed; that must be derived from analysis of the body.) Moreover, f is 
Xmost likely to be the identity function or a linear function with integer 
Xcoefficients, both of which can be inverted easily.  Given this, techniques 
Xfor iterating through the set can be found in several recent conferences.
XEnd of advice to implementors.
X
XAdvice to users:
XOne can expect the I loop above to generate efficient code for the 
Xcomputation partitioning.  In effect, the compiler will arrange for each 
Xprocessor to iterate over its own section of array A.  The J loop is 
Xslightly more complex, since the compiler must find the inverse of the HOME 
Xclause's subscripting function.  That is, the compiler must solve K=J+1 for 
XJ, where K ranges over the local elements of A.  Of course, in this case 
XJ=K-1; in general, linear functions can be inverted by the compiler.  (It 
Xshould be pointed out, however, that complex combinations of ALIGN and 
XDISTRIBUTE may make the description of K unwieldy, and this may add 
Xoverhead to the inversion process.)
XEnd of advice to users.
X 
X
XSometimes it is advantageous to "split" an iteration between processors. 
XThe following case shows one example of this:
X	
X	!HPF$ INDEPENDENT
X	DO I = 2, N-1
X	  !HPF$ ON HOME(A(I))
X	  A(I) = (B(I) + B(I-1) + B(I+1))/3
X	  !HPF$ ON HOME C(I+1)
X	  C(I+1) = A(I) * D(I+1)
X	END DO
X
XDue to the first ON clause, the reference to A(I) is local in the first 
Xstatement. The second ON clause makes A(I) nonlocal (for some values of I) 
Xthere.  This maximizes the data locality in both statements, but does 
Xrequire data movement between the two.  
X
XAdvice to implementors:
XIf there are several non-nested ON clauses in a loop, then the schematic 
Xabove needs to be generalized.  In essence, the iteration range for each 
Xindividual ON clause must be generated.  A processor will then iterate over 
Xthe union of these ranges. Statements guarded by an ON directive must now 
Xbe guarded by an explicit test.  In summary, the code for
X
X	DO i = lb, ub, stride
X		!HPF$ ON HOME(array1(f1(i)))
X		stmt1
X		!HPF$ ON HOME(array2(f2(i)))
X		stmt2
X	END DO
X
Xon processor p becomes 
X
X	set1 = [lb:ub:stride] intersect f1^-1(my_set1(p))
X	set2 = [lb:ub:stride] intersect f2^-1(my_set2(p))
X	DO i in set1 union set2
X	  IF (i in set1) THEN
X	    stmt1
X	  ENDIF
X	  IF (i in set2) THEN
X	    stmt2
X	  ENDIF	  
X	END DO
X
Xwhere my_set1(p) is the local set for array1, and my_set2(p) is the local 
Xset for array2.  (Again, synchronization and communication must be handled 
Xby other means.)  Code transformations such as loop distribution and loop 
Xpeeling can be used to eliminate the tests in many cases.  They will be 
Xparticularly profitable if there are data dependences between the ON 
Xblocks.
XEnd of advice to implementors.
X
XAdvice to users:
XSplitting an iteration like this is likely to require either additional 
Xtests at runtime or additional analysis by the compiler.  Even if the 
Xcompiler can generate low-overhead scheduling for the individual ON 
Xclauses, combining them is not necessarily low-overhead.  The locality 
Xbenefits must be rather substantial for this to pay off, but there are 
Xcases where multiple ON clauses are valuable.  (All these statements are 
Xparticularly true if one ON block uses data computed in another one.)
XEnd of advice to users.
X
X
XBecause ON clauses nest naturally, they can be useful for expressing 
Xparallelism along different dimensions.  Consider the following examples:
X
X	REAL X(M,M)
X	!HPF$ DISTRIBUTE X(BLOCK,BLOCK)
X	
X	!HPF$ INDEPENDENT, NEW(I)
X	DO J = 1, M
X	  !HPF$ ON HOME(X(:,J)) BLOCK
X	    DO I = 2, M
X	      !HPF$ ON HOME(X(I,J))
X	      X(I,J) = (X(I-1,J) + X(I,J)) / 2
X	    END DO
X	  !HPF$ END BLOCK
X	END DO
X
XEach iteration of the J loop is executed by a column of the processors 
Xarrangement.  The I loop further subdivides the computation, giving each 
Xprocessor responsibility for computing the elements it owns.  Many 
Xcompilers would have chosen this computation partitioning automatically for 
Xsuch a simple example.  However, the compiler might have attempted to fully 
Xparallelize the outer loop, executing each inner loop sequentially on one 
Xprocessor.  (This might be attractive on a machine with very fast 
Xcommunications.) By inserting the ON clauses, the user has advised against 
Xthis strategy, thus trading additional locality for restricted parallelism.  
XNotice that the ON directive neither requires nor implies the INDEPENDENT 
Xassertion.  In both nests, each iteration of the I loop depends on the 
Xpreceeding iteration, but the ON directive can still partition the 
Xcomputation among processors.  The ON directive does not automatically make 
Xa loop parallel.
X
XAdvice to implementors:
X"Dimension-based" nesting, as above, will probably be a common case.  The 
XHOME clauses can be inverted at each level, treating indices from outer 
Xloops as run-time invariants.
XEnd of advice to implementors.
X
XAdvice to programmers:
XNested ON directives will tend to have efficient implementations if their 
XHOME clauses refer to different dimensions of the processors arrangements, 
Xas in the above example.  This minimizes the interaction between the levels 
Xof the loops, simplifying the implementation.
XEnd of advice to programmers.
X
XConsider the following variation on the above example:
X
X	!HPF$ DISTRIBUTE Y(BLOCK,*)
X
X	!HPF$ INDEPENDENT, NEW(I)
X	DO J = 1, M
X	  !HPF$ ON HOME(Y(:,J)) BLOCK
X	    DO I = 2, M
X	      !HPF$ ON HOME(Y(I,J))
X	      Y(I,J) = (Y(I-1,J) + Y(I,J)) / 2
X	    END DO
X	  !HPF$ END BLOCK
X	END DO
X
XNote that the ON clauses have not changed, except for the name of the 
Xarray.  The interpretation is similar to the above, except that the outer 
XON directive assigns each iteration of the J loop to all of the processors.  
XThe inner ON directive again implements a simple owner-computes rule.  The 
Xprogrammer has directed the compiler to distribute a serial computation 
Xacross all the processors.  There are a few scenarios where this is more 
Xefficient than parallelizing the outer loop:
X
X1. Parallelizing the outer loop will generate many non-local references, 
X   since only a part of each column is on any processor. If nonlocal references 
X   are very expensive (or if M is relatively small), this overhead 
X   may outweigh any gain from parallel execution.
X2. The compiler may take advantage of the INDEPENDENT directive to avoid 
X   inserting any synchronization.  This allows a natural pipelined execution.  
X   A processor will execute its part of the I loop for one value of J, then 
X   immediately go on to the next J iteration.  Thus, the first processor 
X   will start on J=2 while the second receives the data it needs (from 
X   processor one) for J=1.  (A similar pipeline would develop in the X 
X   example above.)
X
XClearly, the suitability of these ON clauses will depend on the underlying 
Xparallel architecture.  
X
XAdvice to programmers:
XThis example points out how ON may improve software engineering.  While the 
X"value" of HOME(X(I)) will change if X's mapping changes, its intent will 
Xusually stay the same - run the loop "aligned with" the array X.  Moreover, 
Xthe form of the clauses is portable, and they simplify experimenting with 
Xalternative computation partitioning.  Both qualities are similar to the 
Xadvantages of DISTRIBUTE and ALIGN over low-level data layout mechanisms.
XEnd advice to programmers.
X
XON directives are particularly useful when the compiler cannot accurately 
Xestimate data locality, for example when the computation uses indirection 
Xarrays.  Consider three variations of the same loop:
X
X	REAL X(N), Y(N)
X	INTEGER IX1(M), IX2(M)
X	!HPF$ DISTRIBUTE X(BLOCK), Y(BLOCK)
X	!HPF$ DISTRIBUTE IX(BLOCK), IY(BLOCK)
X	
X	!HPF$ INDEPENDENT
X	DO I = 1, N
X	  !HPF$ ON HOME( X(I) )
X	  X(I) = Y(IX(I)) - Y(IY(I))
X	END DO
X	
X	!HPF$ INDEPENDENT
X	DO J = 1, N
X	  !HPF$ ON HOME( IX(J) )
X	  X(J) = Y(IX(J)) - Y(IY(J))
X	END DO
X
X	!HPF$ INDEPENDENT
X	DO K = 1, N
X	  !HPF$ ON HOME( X(IX(K)) )
X	  X(K) = Y(IX(K)) - Y(IY(K))
X	END DO
X
XIn the I loop, each processor runs over its section of the X array.  Only 
Xthe reference X(I) is guaranteed to be local.  (If M<>N, then IX and IY 
Xhave a different block size than X, and thus a different mapping.)  
XHowever, if it is _usually_ the case that X(I), Y(IX(I)), and Y(IY(I)) are 
Xlocated on the same processor, then this mapping may be the best one 
Xavailable.  If X(I) and Y(IX(I)) are _always_ on the same processor, then 
Xthe LOCAL clause should be added:
X	!HPF$ ON HOME( X(I) ), LOCAL( Y(IX(I)) )
XThis will avoid communication setup overhead on most systems, and there is 
Xlittle chance that the compiler would deduce this automatically.  If both 
Xreferences to Y are _always_ on the same processor as X(I), then further 
Ximprovement is possible and desirable:
X	!HPF$ ON HOME( X(I) ), LOCAL( ALL=Y )
XIn the J loop, references IX(J) and IY(J) are always local.  This is the 
Xmost common array reference class in the loop, so it minimizes the 
Xnumber of nonlocal data references in the absence of any special 
Xproperties of IX and IY.  It may not evenly balance the load among 
Xprocessors; for example, if N=M/2 then half the processors will be idle.  
XAs before, if the values in IX or IY ensure that one of the Y references is 
Xalways local, a LOCAL assertion should be added.  In the K loop, only 
Xreference Y(IX(K)) is guaranteed to be local (because Y and X have the same 
Xdistribution).  However, the values stored in IX and IY may ensure that 
XY(IY(K)) and X(K) always local, a fact that should be noted if true:
X  !HPF$ ON HOME( X(IX(K)) ), LOCAL( Y(IY(K)), X(K) )
XEven if the three REAL values are not always, but merely "usually" on the 
Xsame processor, this may be a good computation partitioning for both 
Xlocality and parallelism.  However, these advantages must be weighed 
Xagainst the cost of computing this partitioning.  Since the HOME clause 
Xdepends on a (presumably large) array of runtime values, substantial time 
Xmay be required to determine which iterations are assigned to each 
Xprocessor.  It should be clear from this discussion that there is no magic 
Xsolution for handling complex computation partitionings; the best answer 
Xis usually a combination of application knowledge, careful data structure 
Xdesign (including ordering of the elements), and efficient compilation 
Xmethodology and runtime support.
X
XAdvice to implementors:
XThe K loop is the situation that the inspector strategy described above is 
Xdesigned for.  If there is an outer loop around any of these examples, and 
Xthat loop does not modify the distribution of X or the values of IX, then 
Xa record of each processor's iterations can be saved for reuse.  The cost 
Xis at worst linear in the sizes of the arrays.
XEnd of advice to implementors.
X
XAdvice to users:
XIt is unlikely that any production compiler will generate low-overhead code 
Xfor K loop above in the near term.  The difference from previous examples 
Xis that the HOME clause is not a function that can be easily inverted by 
Xthe compiler.  Some compilers may choose to execute every iteration on all 
Xprocessors, testing the HOME clause at run-time; others may pre-compute a 
Xlist of iterations for every processor.  Of course, the cost of computing 
Xthe list will be substantial.
X
XIn practice, one would make all the arrays the same size to avoid some of 
Xthe alignment problems above; the example was written this way for 
Xpedagogical reasons, not as an example of good data structure design.
XEnd advice to programmers.
X
XExplicit use of processors arrangements in ON directives is usually 
Xassociated with task parallelism.  Many examples can be found there (I 
Xassume...) The following example illustrates how processors can be used 
Xfor a one-dimensional domain decomposition algorithm:
X
X	!HPF$ PROCESSORS (PROCS(NP))
X	!HPF$ DISTRIBUTE X(BLOCK) ONTO PROCS
X
X	! Compute ILO(IP) = lower bound on PROCS(IP)
X	! Compute IHI(IP) = upper bound on PROCS(IP)
X	DONE = .FALSE.
X	DO WHILE (.NOT. DONE)
X	  !HPF$ INDEPENDENT, NEW( ILO, IHI )
X	  DO IP = 1, NP
X	    !HPF$ ON HOME(PROCS(IP)), LOCAL( X(ILO(IP):IHI(IP)) )
X	    CALL SOLVE_SUBDOMAIN( IP, X(ILO(IP):IHI(IP)) )
X	  END DO
X	  !HPF$ ON HOME(X) BLOCK
X	    CALL SOLVE_BOUNDARIES( X, ILO(1:NP), IHI(1:NP) )
X	    DONE = CONVERGENCE_TEST( X, ILO(1:NP), IHI(1:NP) )
X	  !HPF$ END BLOCK
X	END DO
X
XThe algorithm divides the entire computational domain (array X) into NP 
Xsubdomains, one for each processor.  The INDEPENDENT IP loop performs a 
Xcomputation on each subdomain's interior.  The processors then collaborate 
Xto update the boundaries of the subdomains and test for convergence.  The 
Xsubroutine SOLVE_SUBDOMAIN can use a transcriptive or descriptive
Xmapping for its array argument, placing it on a single processor.  An ON 
Xblock encompassing the entire procedure could then ensure the computation 
Xproceded on a single processor.  Subroutines SOLVE_BOUNDARIES and 
XCONVERGENCE_TEST may well have their own loops similar to the IP loop, with 
Xsimilar LOCAL clauses.  Note that only the lower and upper bound of each 
Xsubdomain is recorded; this allows different processors to process 
Xdifferent-sized subdomains.  However, each subdomain must "fit" into one 
Xprocessor's section of the X array.
X
XAdvice to implementors:
XThe IP loop above is likely to be a common idiom among programmers doing 
Xblock-structured codes.  In general, it can be implemented by inverting the 
XHOME clause as was done above.  In the one-to-one case shown here (probably 
Xvery popular with programmers), it can be implemented by assigning the 
Xprocessor id to the loop index variable and testing the range of the loop 
X(once).
XEnd of advice to implementors.
X
XAdvice to programmers:
XSome compilers will propogate the ON information from the caller to the 
Xcallee at compile time, and some at run time.  Repeating the ON clause in 
Xthe caller and callee will tend to give the compiler better information, 
Xresulting in better generated code.  
X
XAgain, note the usefulness of LOCAL clauses in giving the compiler 
Xinformation.  Few compilers would be able to unravel nontrivial assignments 
Xto ILO and IHI, and no current compiler would even attempt to understand 
Xthe comments in the above code fragment.
XEnd of advice to programmers.
X
XBecause it is an assertion of act, the compiler can draw many inferences 
Xfrom a single LOCAL clause.  For example, consider the following case:
X
X	!HPF$ ALIGN Y(I) WITH X(I)
X	!HPF$ ALIGN Z(J) WITH X(J+1)
X	
X	!HPF$ ON HOME( X(K) ), LOCAL( X(INDX(K)) )
X	X(K) = X(INDX(K)) + Y(INDX(K)) + Z(INDX(K))
X
XThe compiler is justified in making the following assumptions in compiling 
Xthe assignment statement (assuming it honors both the ALIGN directives and 
Xthe ON directive):
X
X* X(K) requires no communication (because of the HOME clause)
X* X(INDX(K)) requires no communication (because of the LOCAL clause)
X* Y(INDX(K)) requires no communication (because Y has the same mapping as X)
X
XThe compiler cannot make any assumption about INDX(K) or Z(INDX(K)) from 
Xthe above code.  There is no indication how INDX is mapped relative to X, 
Xso the ON directive gives no guidance.  Note that the fact that an 
Xexpression (here, X(INDX(K))) is local does not imply that its 
Xsubexpressions (here, INDX(K)) are also local.  Similarly, Z's mapping does 
Xnot determine if Z(INDX(K)) would be local; it indicates that Z(INDX(K)-1) 
Xis local, but that isn't a great help.  If the compiler has additional 
Xinformation (for example, X is distributed by BLOCK and INDX(K) is not near 
Xa block boundary), it might be able to make additional deductions.
X
XAdvice to implementors:
XOne mark of a good compiler will be that it aggressively propagates LOCAL 
Xassertions.  This is likely to significantly reduce communication costs.  
XNote the cases under "Advice to users" below.
XEnd advice to implementors.
X
XAdvice to users:
XOne can expect compilers to differ in how aggressive they are in drawing 
Xthese deductions.  Higher-quality compilers will be able to identify more 
Xreferences as local, and use this information to eliminate data movement.  
XAll compilers should recognize that if an element of one array is local, 
Xthen the same element of any other arrays with the same static mapping 
X(i.e.  arrays ALIGNed together, or with the same DISTRIBUTE pattern and 
Xarray size) will also be local.  That is, any compiler should recognize 
XY(INDX(K)) in the above example as local.  Dynamically changing array 
Xmappings (i.e. REALIGN and REDISTRIBUTE) will tend to limit such 
Xinformation and information propagation.
XEnd advice to users.
X
XAnother example of drawing further inferences from LOCAL clauses is the 
Xfollowing:
X
X	!HPF$ ON HOME(PROCS(IP)), LOCAL( X(ILO(IP):IHI(IP)) )
X	DO I = ILO(IP)+1, IHI(IP)-1
X	  X(I) = (X(I-1) + 2*X(I) + X(I+1)) / 4
X	END DO
X
X(Note that this is the construct form of the DO, so the ON clause applies 
Xto all three lines below it.) Here, I is alwas between ILO(IP) and IHI(IP).  
XTherefore, the two references to X(I) require no communication, since they 
Xalways refer to an element of the LOCAL array section.  Slightly deeper 
Xreasoning reveals that the same is true of X(I-1) and X(I+1).  In short, 
Xthe entire DO loop can be executed without communication on PROCS(IP).
X
XAdvice to implementors:
XAnalysis of ranges with symbolic bounds will pay big dividends for 
Xapplications such as block-structured CFD codes.  Of course, the expense 
Xof doing this analysis must be weighed against its benefits.  See also 
Xthe "Advice to programmers" below.
XEnd advice to implementors.
X
XAdvice to programmers:
XAs with the previous example, one can expect the capabilities of compilers 
Xto differ widely in making these inferences.  Most compilers should 
Xrecognize and use compile-time constants in LOCAL clauses and the 
Xexecutable code.  Many compilers will extend this to variables that are 
Xinvariant within the ON clause and will propogate the bounds of DO loops; 
Xothers will use pattern recognition to achieve some of this effect.  Such 
Xpattern recognition suffices to recognize X(I) in the example.  Detailed 
Xreasoning about symbolic values is beyond the capabilities of most 
Xcompilers.  For example, determining the range of X(I-1) in the above 
Xexample may require symbolic manipulation (or rather aggressive 
Xpattern-matching).
XEnd advice to users.
X
X
X
X--------------------------------------------------------------------------------
X[A related mail message from Marc Baber]
X
XHello,
X
XI've excerpted the following from APR's manuals for consideration as one
Xpossible form of an HPF 2.0 work-distribution directive.  DOPAR can
Xbe interpreted as a dynamically-balanced work partitioning directive
Xon shared memory systems by ignoring the ON reference, or the ON reference
Xcan be used to align the DO with any array/dimension for static balancing
Xand reduction of communication on distributed memory systems.  
X
XThough this syntax does not address some peoples' desire to specify
Xprocessor id's in the ON clause, it does achieve a very natural
Xextension of HPF to align DO loops with array distributions.
X
XNote that DO PAR is not interpreted to mean "run this loop parallel at
Xany cost."  DO PAR simply conveys the programmer's preference about
Xwhich loop in a loop nest should be parallelized first.  The IGNORE
Xdirective (also described below) is used to override the xHPF compiler's
Xdeterminations about iteration dependence.
X
X-Marc
X
X
XAPR Directive Descriptions
X
XSelect Loop
X
XDirective:  CAPR$ DO PAR {ON reference }
X
XSelect the next loop for parallelization.  Applies to both automatic and
Xmanual loop selection modes. The ON clause selects a distributed array as
Xcontrolling the loop distribution.  If there is no ON clause, xhpf will
Xdetermine the best partitioned array reference within the loop to determine
Xdistribution of the loop iterations.
X*     Applies only to the next upcoming DO loop.
XThe Controlling Array  ( ON Clause )
X
XWithout the ON clause on a DO PAR directive, dpf will select a partitioned
Xarray reference within each loop to control the distribution of iterations
Xover the processors.  A loop's iterations are distributed so that
Xreferences to the control array occur in the processor that "owns" those
Xelements of the partitioned array.  dpf tries to choose an array that will
Xresult in the minimum communication when the loop is parallelized.  (See
XChapter 2 discussion of loop distribution.)
X
XThe ON clause on a DO PAR directive indicates that the specified array
Xreference is to control the distribution of the iterations of the loop.
X
XThe ON clause reference syntax:
X
X reference      is      array-name
X        or      array-name(subscripts)
X        or      array-name<subscripts>
X
XWithin subscripts, subscript expressions conforming to the array's
Xsubscripts appear:
X
X:  and  *       indicate an arbitrary subscript
X
Xinitial-expr ~ incr-expr        indicates an integer subscript with a
Xconstant
X                increment ("CII")
X
Xinitial-expr : final-expr {: incr-expr }        indicates a complete range of
X                        values.
X
X(Each of the -expr are arithmetic expressions that may include constants
Xand parenthesized sub-expressions).  Because xhpf only considers a single
Xdimension for array partitioning and loop distribution, only one subscript
Xexpression may appear in the ON clause array as a CII or constant;  and the
Xother subscripts must be :  or  *.
X
XExamples:       CAPR$ DO PAR ON QUOM<:,1~1>
X        CAPR$ DO PAR ON UR<:,:,j-1~1>
X        CAPR$ DO PAR ON WKW<:,:,1>
X
XDon't Select Loop
X
XDirective:  CAPR$ DO NOPAR
X
XIndicates that the next loop should not be selected for parallelization.
X
XIgnore Inhibitors or Communication
X
XDirective:  CAPR$ IGNORE ALL INHIBITORS
X
XIndicates that the next loop, if marked for parallelization by a DO PAR
Xdirective, can be parallelized by ignoring any apparent inhibitors.
X
XDirective:  CAPR$ IGNORE type COM {ON reference }
X
XIndicates that xhpf need not generate certain communication calls in the
Xnext loop marked by a DO PAR.  type indicates the kind of communication
Xbeing referenced.  The ON clause further identifies the communication to
Xspecific arrays.  If there is no ON clause, communication of type  for all
Xarrays in the loop is ignored for this loop.
X
X
XIGNORE type:            referring to:
XPRELOOP         pre-loop fetches of off-processor array elements
XPOSTLOOP                post-loop sends of computed off-processor elements
XALL             all and any of the above
X
X
X*   Any CAPR$ IGNORE directives must immediately follow a DO PAR directive.
X
XIGNORE type COM ON Clause
XEach array reference in a loop has the potential of generating data
Xcommunication calls to fetch or store array elements.  PRELOOP
Xcommunications fetch to the processor all the array elements it uses from
Xthe processors who own them, while POSTLOOP communications distribute all
Xthe computed array elements back to the owning processors.
X
XThe IGNORE type COM directive permits the user to optimize the performance
Xof a parallelized loop by recognizing that certain communications are
Xredundant and could be eliminated.  The ON clause specifies which array
Xreference's communication calls can be ignored.  Unlike the DO PAR ON
Xclause, the syntax of the ON clause on the IGNORE directive must indicate
Xall the subscripting in order to match a particular array reference in the
Xsource code, or just the unsubscripted array name to select all references
Xto that array.
X
XFrom the point of view of the DO PAR directive, the array references in the
Xfollowing loop that can be used as the controlling array for distributing
Xiterations of the loop are shown:    (X paritioned BLOCK, TEMP replicated.)
X
X       CAPR$ DO PAR
X             DO 15 I = 1,N
X                temp(I) = I     !  temp<1~1>
X        15      X(I) = temp(I)  !  X<1~1>
X       CAPR$ DO PAR
X             DO 20 i = 1,N
X                temp(I) = i*i
X        20      X(I) = x(i) + temp(I)
X
X
XThe array references that give rise to communication calls for loop 15 are :
X
X   postloop communication of TEMP<1~1>
X   postloop communication of X<1~1>
X
X
XHowever, values set in temp by loop 15 are immediately replaced in loop 20,
Xeliminating the need to bring all copies of temp up to date after loop 15.
XThe postloop communication of temp can be ignored:
X
X        CAPR$ DO PAR
X        CAPR$ IGNORE POSTLOOP COM ON TEMP
X             DO 15 I = 1,N
X                temp(I) = I
X        15      X(I) = temp(I)
X
X-------------------------------------------------------------------------------
END_OF_FILE
if test 45801 -ne `wc -c <'ON_directive_proposal.txt'`; then
    echo shar: \"'ON_directive_proposal.txt'\" unpacked with wrong size!
fi
# end of 'ON_directive_proposal.txt'
fi
if test -f 'OOC_proposal.txt' -a "${1}" != "-c" ; then 
  echo shar: Will not clobber existing file \"'OOC_proposal.txt'\"
else
echo shar: Extracting \"'OOC_proposal.txt'\" \(17619 characters\)
sed "s/^X//" >'OOC_proposal.txt' <<'END_OF_FILE'
XPreliminary Proposal to provide support for OOC arrays in HPF 
X
XAlok Choudhary, Chuck Koelbel, Ken Kennedy
X
X
XThe following proposal presents a way to support out-of-core array in HPF. 
XThe main objectives are consistency with the "normal" HPF data mapping 
Xdirectives, simplicity, and minimal extensions. 
X
XThe following illustrates an example of declaring out-of-core arrays. The 
Xonly addition here is the directive OUT-OF-CORE. (Other directives and 
Xinformation will follow later in this writeup) 
X
X!HPF$ TEMPLATE TEMP(100,100)
X!HPF$ DISTRIBUTE TEMP(CYCLIC(B),CYCLIC(B)) !HPF$ ALIGN WITH TEMP :: A,B,C
X!HPF$ OUT-OF-CORE: TEMP
X
XThis directive simply says that if nodes had infinite memory then elements 
Xof the array will be in the memory of the processor as described by the 
Xdistribution directives. In other words, the directives describe which 
Xprocessor's memory will an element be brought into when in-core. This is a 
Xlogical and simple extension of the HPF data mapping directives. 
X
XAs usual, directives can be directly applied to arrays (rather than 
Xthrough a template).
X
XRestrictions:
X1) One cannot align an in-core array with an out-of-core array. E.g., 
Xif A(10,10) is an OOC array and B(10) is an in-core array, it is not 
Xpermitted to align B with A (or vice-versa) in any form, because not all 
Xthe elements of A may be in-core, and therefore, it is difficult to 
Xenforce the meaning of align in such cases. 2) Others?
X
X
XAssociation of File(s) with Arrays.
X
XIf the data for an out-of-core array comes from an input file then the 
Xfile name must be specified. This is different from traditional open and 
Xread/write of a file because the user does not explicitly access the file.
X
XFor this we propose a directive
X
X!HPF$ ASSOCIATE (<fn>, [,other parameters]) :: OOC array name 
X
Xe.g.,
X
X!HPF ASSOCIATE (filex) :: A
X
XBy default, the storage order in the file is the fortran storage order and 
Xthe element (1,1) corresponds to the first elment of the file.
X
X[,Other parameters] can be used to specify. 1) Storage order
X2) starting element of the dataset within the file. 
X
X1) is needed if the storage order is anything other than the default 
Xcolumn major.
X
XThere are two possibilities. Use key words such as row major, column 
Xmajor, or chucnks with parameters (such as chunk size in each dimension); 
X
XOR use an explicit order, e.g., (2,1,3) which says that the outermost 
Xdimension is 2 followed by 1 followed by 3. This may not allow chunking. 
X
X2) Starting element of the dataset is necessary if file contains some 
Xmeta data in the beginning (e.g., a description of storage order from 
Xwhich the info in 1 will be derived) of a file, record size etc. 
X
XThis would require an open of a file, read of meta data and close of a 
Xfile before the corresponding ASSOCIATE may be executed. 
X
XNote that it is not allowed to have a file open for regular access 
Xas well as for out-of-core computation because OOC computation provides 
Ximplicit access ot a file and any explicit access presents consistency 
Xproblems.
X
XHowever, the following is allowed.
X
XOPEN the file
XREAD/WRITE/INQUIRE to your heart's content CLOSE the file
X(somewhere down the call chain) declare the OOC array use the array
XRETURN (i.e. exit the scope where the array is declared) now you can 
Xaccess the file again
XThe CLOSE acts as a sync point. Similarly, the array being deallocated 
Xacts as a sync point.
X
XNote that this is necessary if metadats is contained within the file. 
X
X
XIf an OCC array is used purely for scratch purposes, then it is not 
Xnecessary to associate a file with it. Compiler can choose a name and 
Xcreate files in whatever way necessary. HOWEVER, any array with which no 
Xfile name is associated, must not be read before anything is written into 
Xit (for obvious reasons). In fact, it should be a runtime error if there 
Xis a part of the OOC array which has not been assigned any values is read. 
X
XASSOCIATE directive is also an executable (like redistribute) because a 
Xuser may want to process several files using the same array (at different 
Xpoints in time, e.g., pipelined computations). At any point, however, only 
Xone association exists. 
X
X
XE.G.,
X
X!HPF$ TEMPLATE TEMP(100,100)
X!HPF$ DISTRIBUTE TEMP(CYCLIC(B),CYCLIC(B)) !HPF$ ALIGN WITH TEMP :: A,B,C
X!HPF$ OUT-OF-CORE: TEMP
X
XDO i=1,numer_of_frames
X
XASSOCIATE (frame.i) :: A ! Syntax to be decided of course! 
XEXTRACT_NICE_FEATURES(A)
XEND Do
X
XNote one can associate multiple OOC arrays with the same file but the 
Xconsistency (if multiple assignments are done) is user's responsiblity.
X
XTILING
X
X
XEven when OOC distribution is provided, it may require significant 
Xcompiler analysis to figure out in what order to bring data in to be 
Xprocessed. User, based on her knowledge of computation (philosophy of HPF) 
Xcan provide this hint. 
X
XFor this purpose, a directive IO_DISTRIBUTE can be used, which essentially 
Xprovides a hint as to how data should be fetched into the memory. 
X
XFor example, say each processors "owns" chunk of OOC data after a 
X(block,block) distribution. How data is to be fetched from whichin this 
Xchunk (bunch of rows, bunch of columns, two-dim blocks) can be described 
Xby IO_DISTRIBUTE.
X
XNote that IO_DISTRIBUTE will also help in reorganizing data into local 
Xfiles of each processor wrt to storage order within each file, if the 
Ximplementation chooses to do such a reorganization. 
X
XHowever, note that block size here would depend on the amount of memory 
X(tile size) available (and not on the NOP). The syntax for this size 
Xspecification within IO_DISTRIBUTE needs to be resolved. 
X
XOne way is to do it explicitly; e.g., IO_DISTRIBUTE A(5,10), meaning 
Xblocks of 5X10 elements...
X
XBut I believe this is a detail.
X---------------------------------------------------------------- 
X
XThe above discussion is based on the following questions. Some of them are 
Xnot answered (in terms of a concrete proposal) 
X
X1. Type of array: is the given array in-core or out-of-core? What is the 
Xdistribution of the array?
X2. If out-of-core, is there a corresponding file? 3. If there exists a 
Xfile, is the file persistent (input/output) or temporary?
X4. Information about array mapping
X1. Multiple arrays mapped to the same file 2. Multiple files mapped to the 
Xsame array 5. Information about File Mapping
X1. File Ordering
X2. File Distribution Information
X6. Hints about tiling
X1. Available Memory
X2. Tiling Parameters
X
X* Execution Model
XCompiler has to decide underlying execution model. Two possible models are
X1. Local Placement Model (OOC data in local space) 2. Global Placement 
XModel (OOC data in Shared Space) 
X
X
X--------------------------------------------------------------------------------
X[A related message from Rob Schreiber]
X
X
X        We would like to start a discussion on this.
X
X        thanks
X        Alok, Chuck, Ken
X
XOkay.   Here goes.
X
X
X        The following proposal presents a way to support out-of-core array in HPF.
X        The main objectives are consistency with the "normal" HPF data mapping
X        directives, simplicity, and minimal extensions.
X
X        The following illustrates an example of declaring out-of-core arrays.
X        The only addition here is the directive OUT-OF-CORE. (Other directives
X        and information will follow later in this writeup)
X
X        !HPF$ TEMPLATE TEMP(100,100)
X        !HPF$ DISTRIBUTE TEMP(CYCLIC(B),CYCLIC(B))
X        !HPF$ ALIGN WITH TEMP :: A,B,C
X        !HPF$ OUT-OF-CORE: TEMP
X
X(and OOC could be an attribute in a combined directive:
X
X!HPF$ TEMPLATE, DISTRIBUTE(BLOCK,*), OUT_OF_CORE :: T
X
X)
XMy prediction is that for at least N years, all HPF compilers will ignore 
Xthe OOC attribute.  For machines with virtual memory and demand paging, N 
Xwill not fit in a 32 bit integer.
X
X        This directive simply says that if nodes had infinite memory then
X        elements of the array will be in the memory of the processor as described
X        by the distribution directives. In other words, the directives describe
X        which processor's memory will an element be brought into when in-core.
X        This is a logical and simple extension of the HPF data mapping directives.
X
X        As usual, directives can be directly applied to arrays (rather than through
X        a template).
X
X        Restrictions: 
X        1) One cannot align an in-core array with an out-of-core array. E.g.,
X           if A(10,10) is an OOC array and B(10) is an in-core array, it is
X           not permitted to align B with A (or vice-versa) in any form, because
X           not all the elements of A may be in-core, and therefore, it is
X           difficult to enforce the meaning of align in such cases.
X
XSince there is no IN_CORE directive, there is no need for a restriction.
XAlignment to an OOC template makes an object OOC.
X(Principle of the excluded middle:  P .or. .not. P).
X
X        Association of File(s) with Arrays.
X
X        If the data for an out-of-core array comes from an input file then 
X        the file name must be specified. This is different from traditional
X        open and read/write of a file because the user does not explicitly access the
X        file.
X
X        For this we propose a directive
X
X        !HPF$ ASSOCIATE (<fn>, [,other parameters]) :: OOC array name
X
X        e.g., 
X
X        !HPF ASSOCIATE (filex) :: A
X
X        By default, the storage order in the file is the fortran storage order
X        and the element (1,1) corresponds to the first elment of the
X        file.
X
XThis seems to me to be a completely absurd idea.  The whole point of
XOOC should be to leave the file format, etc., hidden from the user and
Xup to the implementation; this freedom is the value of it.  If there is
Xinitial data on a file, then open the file and read it into the array,
Xas an idiom for the translation of file formats from the world-visibile
Xfortran file holding the data to the completely internal file holding
Xthe array.
X
XWill you also need to OPEN the file?   If not, how do you get the
Xinformation ordinarily provided by the user in the OPEN statement?
XHow do you handle errors?
X
X
X        [,Other parameters] can be used to specify.
X        1) Storage order
X        2) starting element of the dataset within the file.
X
X        1) is needed if the storage order is anything other than the default
X           column major.
X
X           There are two possibilities. Use key words such as row major, column
X        major, or chucnks with parameters (such as chunk size in each dimension);
X
X         OR use an explicit order, e.g., (2,1,3) which says that the outermost
X        dimension is 2 followed by 1 followed by 3. This may not allow chunking.
X
X        2) Starting element of the dataset is necessary if file contains some
X           meta data in the beginning (e.g., a description of storage order from
X           which the info in 1 will be derived) of a file, record size etc.
X
X           This would require an open of a file, read of meta data and close
X           of a file before the corresponding ASSOCIATE may be executed.
X
X        Note that it is not allowed to have a file open for regular access
X         as well as for out-of-core computation because OOC computation
X         provides implicit access ot a file and any explicit access
X        presents consistency problems.
X
XWhat do you mean "have a file open for OOC access"?   Does ASSOCIATE
Ximplicitly do this to the file?  If so, you are implying that the file
Xis used to hold the array, and is possibly modified if the array is
Xmodified?   So it contains more than the initial data?
X
XIf so, this is really screwy.   For efficiency, the file(s) that hold
Xan OOC array should be
Xa) local to a processor or an SMP pmultiprocessor;
Xb) unformatted
Xc) direct access
Xd) with medium-sized records, whose size is implementation dependent.
X
XSo these notions of row-major, etc, are silly.   They're at the wrong
Xconceptual level, like putting Orthogonal Recursive Bisection into the
Xlanguage (as a distribution function).  Just let the user read his
Xinitial data into the array using fortran I/O!
X
XDo you know of an OS that asks the user how it should implement
Xdemand paging?   What page size, page replacement strategy, ... ?
XThe whole point should be to make this work transparently.
X
X
X        However, the following is allowed.
X
X                OPEN the file
X                READ/WRITE/INQUIRE to your heart's content
X                CLOSE the file
X                (somewhere down the call chain) declare the OOC array
X                use the array
X                RETURN (i.e. exit the scope where the array is declared)
X                now you can access the file again
X        The CLOSE acts as a sync point.  Similarly, the array being
X        deallocated acts as a sync point.
X
X        Note that this is necessary if metadats is contained within the file.
X
XSo it seems that OOC arrays can be persistent files.   In that case, they must be
Xactual FORTRAN files.   So you lose many possible advantages; the system cannot do
Xanything that is "not Fortran".
X
X        If an OCC array is used purely for scratch purposes, then it is not
X        necessary to associate a file with it. Compiler can choose
X        a name and create files in whatever way necessary. HOWEVER,
X        any array with which no file name is associated, must not be
X        read before anything is written into it (for obvious reasons).
X        In fact, it should be a runtime error if there is a part of the
X        OOC array which has not been assigned any values is read.
X
XWhy isnt it just uninitialized data?
X
X        ASSOCIATE directive is also an executable (like redistribute) because
X        a user may want to process several files using the same array
X        (at different points in time, e.g., pipelined computations).
X        At any point, however, only one association exists. 
X
X
X        E.G., 
X
X        !HPF$ TEMPLATE TEMP(100,100)
X        !HPF$ DISTRIBUTE TEMP(CYCLIC(B),CYCLIC(B))
X        !HPF$ ALIGN WITH TEMP :: A,B,C
X        !HPF$ OUT-OF-CORE: TEMP
X
X         DO i=1,numer_of_frames
X
X            ASSOCIATE (frame.i) :: A ! Syntax to be decided of course!
X            EXTRACT_NICE_FEATURES(A)
X         END Do
X
X        Note one can associate multiple OOC arrays with the same file but
X        the consistency (if multiple assignments are done) is user's
X        responsiblity.
X
XAt the same time?  Hey, I see it now.   ASSOCIATE is just EQUIVALENCE
Xfor files!
X
XBut it's executable!   Cool!   Hey, why not just make HPF an
Xinterpreted language?
X
X
X                                TILING
X
X
X        Even when OOC distribution is provided, it may require significant
X        compiler analysis to figure out in what order to bring data in
X        to be processed. User, based on her knowledge of computation
X        (philosophy of HPF) can provide this hint.
X
X        For this purpose, a directive IO_DISTRIBUTE can be used, which essentially
X        provides a hint as to how data should be fetched into the memory.
X
X        For example, say each processors "owns" chunk of OOC data after
X        a (block,block) distribution. How data is to be fetched from whichin this
X        chunk (bunch of rows, bunch of columns, two-dim blocks) can be
X        described by IO_DISTRIBUTE.
X
X        Note that IO_DISTRIBUTE will also help in reorganizing data into local
X        files of each processor wrt to storage order within each file, if the
X        implementation chooses to do such a reorganization.
X
X        However, note that block size here would depend on the amount of memory
X        (tile size) available (and not on the NOP). The syntax for this
X        size specification within IO_DISTRIBUTE needs to be resolved.
X
X        One way is to do it explicitly; e.g., IO_DISTRIBUTE A(5,10), meaning
X        blocks of 5X10 elements...
X
X        But I believe this is a detail.
X        ----------------------------------------------------------------
X
X        The above discussion is based on the following questions. Some of them
X        are not answered (in terms of a concrete proposal)
X
X(My opinions):
X
X         1. Type of array: is the given array in-core or out-of-core? What is
X        the distribution of the array?
XOOC is an attribute of templates and the arrays aligned to them.
X         2. If out-of-core, is there a corresponding file?
XNot visible.
X         3. If there exists a file, is the file persistent (input/output) or
X        temporary?
Xtemporary.
X         4. Information about array mapping
X            1. Multiple arrays mapped to the same file
X            2. Multiple files mapped to the same array
XNeither.
X         5. Information about File Mapping
X            1. File Ordering
XInvisible to the user.
X            2. File Distribution Information
XThe OOC template can have and HPF distribute attribute.
X         6. Hints about tiling
X            1. Available Memory
X            2. Tiling Parameters
XLeave it to the compiler for now.  We have no idea how to give useful advice.
X
X        * Execution Model
X         Compiler has to decide underlying execution model. Two possible
X        models are
X          1. Local Placement Model (OOC data in local space)
X          2. Global Placement Model (OOC data in Shared Space)
XImplementation dependent.
X
X
X--   Rob
X
XPS -- I have a fortran 77 Out Of Core linear systems solver (using
Xplain old Fortran I/O) sitting around.  Want to compare
Ximplementations?   I could make a version using an OOC array.
X
XOf course, if there was no paging and VM, this would be worthwhile
Xin many cases, even if slow.   Except maybe for scientific supercomputing!
XI think Cray never needed to add VM and paging because users didn't want
Xor believe in it.   So why does HPF?
END_OF_FILE
if test 17619 -ne `wc -c <'OOC_proposal.txt'`; then
    echo shar: \"'OOC_proposal.txt'\" unpacked with wrong size!
fi
# end of 'OOC_proposal.txt'
fi
if test -f 'Reduction_proposal.txt' -a "${1}" != "-c" ; then 
  echo shar: Will not clobber existing file \"'Reduction_proposal.txt'\"
else
echo shar: Extracting \"'Reduction_proposal.txt'\" \(25887 characters\)
sed "s/^X//" >'Reduction_proposal.txt' <<'END_OF_FILE'
X
XProposed HPF Extension for Reduction Operations in Independent Loops 
X
XRob Schreiber and Chuck Koelbel, and Joel Saltz 
X
XSeptember 8, 1995
X
X
X1. Rationale
X
XIt is often the case that a data parallel computation cannot be expressed 
Xin HPF as an INDEPENDENT loop because several of the loop iterations 
Xupdate one or more variables. In such cases, parallelism is possible 
Xbecause the order of updates is immaterial to the final result. This is 
Xmost often the case with accumulations, such as the following loop. 
X
XDO I = 1, 1000000000
XX = X + COMPLICATED_FCN(I)
XEND DO
X
XThis loop can run in parallel as long as its iterations make their 
Xmodifications to the shared variable (X) in an atomic manner. Alternately, 
Xthe loop can be run in parallel by making updates to temporary copies of 
XX, with a (short) final phase to merge these changes. In either case, the 
Xcomputation is conceptually parallel but not allowed by the original 
Xdefinition of INDEPENDENT.
X
XWhen programming this kind of computation to run in parallel in HPF 
XVersion 1, the programmer must store update information in a temporary 
Xarray whose size is equal to the number of loop iterations, and then use a 
Xreduction intrinsic or XXX_SCATTER library function after the loop. The 
Xproblems with this approach are twofold: the temporary array may be 
Xexcessively large; and the reduction operation has to be intrinsic.
X
XFor these reasons, we have added a reduction feature to HPF that allows 
Xloops that update shared variables (shared means that they are updated by 
Xmore than one loop iteration) to be labeled as independent. Interestingly, 
Xwith this extension it is feasible to write general purpose _SCATTER and 
Xreduction functions, that can work with user-defined operators.
X
X2. Technical Proposal
X
X2.1 Add a REDUCE Directive
X
XH499 reduce-directive	is !HPF$ REDUCE
X
XConstraint: The first non-comment, non-blank line following a 
Xreduce-directive an assignment-stmt of a special form, called a reduction 
Xstatement. 
X
X2.2 Reduction Statments
X
XH498 reduction-stmt	is assignment-stmt
X
XA reduction statement is any statement immediately preceded by a reduce 
Xdirective. 
X
XConstraint: The form of a reduction-stmt must be 
X
Xreduction-variable = reduction-variable reduction-op expr 
X
Xor
Xreduction-variable = &
Xreduction-function(reduction-variable, expr) 
X
XConstraint: The two reduction-variable references that occur in a 
Xreduction statement must be lexically identical. 
X
XH498.1 reduction-variable is variable
X
XH498.2 reduction-op	is	intrinsic-op
Xor defined-binary-op
X
XH498.3 reduction-function is function-name 
X
XConstraint: A reduction-function must be PURE. 
X
XConstraint: A reduction-op may not be a rel-op; and it may not be power-op 
X(i.e. **). 
X
XConstraint: A reduction-stmt must occur in the range of an INDEPENDENT DO 
Xloop in which the reduction variable does not have the NEW attribute. 
XAddition constraints on the location of a reduction statement are covered 
Xin Section 2.4. 
X
XThe operator or function that follows the reduction variable reference 
Xafter the = sign is the reduction operator. 
X
XAny variable that appears on the left hand side of a reduction statement 
Xis said to be a reduction variable. 
X
X2.3 Allow a reduction-clause in the independent directive: 
X
XH413 independent-directive is INDEPENDENT [, new-clause ] [, 
Xreduction-list] 
X
XH413.1 reduction-clause	is
XREDUCTION(reduction-variable-list; COMBINE = combining-op; IDENTITY = 
Xvalue) 
X
XH413.2 combining-op	is	intrinsic-op
Xor defined-binary-op
Xor reduction-function
X
XConstraint: The combining operator must be a binary operator or a function 
Xof two arguments, defined for two objects of the type of each of the the 
Xreduction variables, and producing an object of that same type (it may not 
Xbe a rel-op); it may not be the power operator **. The only intrinsic 
Xoperators and functions that are allowed are commutative; they are listed 
Xbelow (in Section 3).
X
XConstraint: The value in the identity clause must be an expression 
Xconformable with each of the reduction variables, and for which the 
Xassignment var = value is well-defined for each of the reduction 
Xvariables; or it may the name of a PURE function of one argument that can 
Xaccept every member of the reduction variable list as an argument and that 
Xproduceds a result conformable with its argument and for which the 
Xassignment var = value(var) is well-defined.
X
XThe reduction clause is mandatory for user-defined reduction operators; it 
Xis optional for intrinsic operators applied to a reduction variable of 
Xintrinsic type. 
X
XIn either case, it must appear in the correct INDEPENDENT directive. This 
Xis the INDEPENDENT directive that labels the outermost DO construct in 
Xwhich the
Xreduction variable does not occur in a NEW clause, and which contains no 
XINDEPENDENT directive in which the reduction variable occurs in a NEW 
Xclause. See Section 2.4.
X
XExamples:
X
XThere may be more than one reduction clause in the INDEPENDENT directive: 
X
X!HPF$ INDEPENDENT, REDUCTION(X, COMBINE=+, IDENTITY=0), & !HPF$ & 
XREDUCTION(Y, COMBINE=MAX, IDENTITY= (-HUGE(Y))) DO I = 1, N
X!HPF$ REDUCE
XX = X + A(I)
X!HPF$ REDUCE
XY = MAX(Y,A(I))
XEND DO
X
XThe reduction variable may be of derived type, and the reduction operator 
Xmay be an overloaded operator:
X
Xinterface operator (+)
Xtype (bignum) function rank0_big_plus(x,y) type (bignum) x,y
Xend function rank0_big_plus
Xend interface
X
Xinterface zero
Xreal function rank0_real_zero(x)
Xreal x
Xend function rank0_real_zero
X
Xreal function rank1_real_zero(x)
Xreal x(:), rank1_real_zero(size(x))
Xend function rank1_real_zero
X
Xdouble precision function rank0_dbl_zero(x) double precision x
Xend function rank0_dbl_zero
X
Xtype (bignum) function rank0_big_zero(x) type (bignum) x
Xend function rank0_big_zero
Xend interface
X
Xreal x0, x1(10)
Xdouble precision d
Xtype (bignum) b
X
X!HPF$ INDEPENDENT, REDUCTION(x0, x1, d, b; combine = +; identity = zero) 
Xdo ...
X...
X!hpf$ reduce
Xb = b + <bignum-expr>
X...
X!hpf$ reduce
Xb = b - <bignum-expr>
Xenddo
X
X
X2.4 Restrictions on the Lexical Location of Reduction Statements, and on 
Xthe Use of Reduction Variables.
X
XIn this discussion, we use the term *range* (of a DO construct) and 
X*active* (an active DO construct) in their technical sense as defined by 
XFortran 90. The range of a DO construct consists of the statements 
Xlexically bounded by the DO statement and the terminal statement of the 
Xconstruct. Thus, a subprogram invoked from inside a DO loop is not in its 
Xrange, although the CALL statement or function reference that invokes it 
Xis. On the other hand, the DO construct is active from the time the DO 
Xstatement is encountered until the time that the matching execution of the 
Xterminal statement is executed, or the program stops, or control is 
Xtransferred out of the construct in one of the other ways allowed for 
XINDEPENDENT loops (see page 302 of the Handbook and Section 4.4 of the HPF 
XStandard.) 
X
XWhen a reduction operation is executed, some nest of DO constructs will be 
Xactive; this is guaranteed by the third constraint in Section 2.2. The 
Xreduction variable is said to be *protected* in some, and possibly all of 
Xthe active nest; the rule is this: it is protected in the outermost DO 
Xconstruct that is INDEPENDENT, has no NEW clause that names the reduction 
Xvariable, and does not *contain* an independent
Xdirective in which the reduction variable occurs in a NEW clause (see 
XExample 2, below). It is protected in all of this DO construct, including 
Xany other control structures nested within it. 
X
XHPF requires that any reduction statment occur in the range of the DO 
Xconstruct in which its variable is protected: it must be lexically 
Xcontained in the body of the outermost loop in which it is protected. 
X
XA protected reduction variable may only occur in reduction statments, and 
Xonly in references to the left of the assignment operator and the matching 
Xreference to the right of the assignment operator. Outside of the loop in 
Xwhich it is protected, however, it is treated as an ordinary variable.
X
X2.4.1 Example loop nests.
X
XWhen loops are nested, a reduction variable may be protected in an 
Xindependent outer loop even though the reduction operations in which it 
Xoccurs are nested inside an inner loop. Moreover, the inner loop and any 
Xintervening loops may or may not be independent. 
X
X! Nested Loop Example 1. Inner loop is sequential 
X
Xx = 10
Xouter: do while (x < 1 000 000)
X!hpf$ independent
Xmiddle: do i = 1, n
Xinner: do j = 1, m
X!hpf$ reduce
Xx = x + j
X!	illegal to refer to x here, except in a reduction statement
Xenddo inner
X!	illegal to refer to x here, except in a reduction statement
Xenddo middle
Xprint *, x
Xenddo outer
X
XSince the variable x occurs in a reduction statement in the body on the 
Xindependent
Xmiddle loop, it is a protected reduction variable throughout that loop, 
Xincluding inside the inner loop. The outermost loop is not INDEPENDENT, 
Xand so x is not protected in that part of its range not included in the 
Xmiddle loop. 
X
XThis remains true if the inner loop is independent. On the other hand, a 
Xvariable that occurs in a new clause may not be a reduction variable in 
Xthe body of a loop, although it may be in the body of a contained loop:
X
X! Nested Loop Example 2. Outer loop new clause. 
X
X!hpf$ independent
Xouter: do k = 1, 100
X!hpf$ independent, new x
Xmiddle: do i = 1, n
Xx = 10
X!hpf$ independent
Xinner: do j = 1, m
X!hpf$ reduce
Xx = x + j**2
X! illegal to refer to x here, except in a reduction statement enddo inner
Xy(i) = x
Xenddo middle
Xenddo outer
X
XHere, x is a protected reduction variable only in the inner loop. 
X
X
XThe reduction statment may not occur in a subprogram called from the body 
Xof the independent loop. To get this effect, return the updating value 
Xfrom the subprogram and perform the update with a reduction statement in 
Xthe lexical loop body.
X
Xprogram example
Xinteger x, xinc, i
X
Xx = 10
X!hpf$ independent, new(xinc), reduction(x, combine=+, identity = 0) do i = 
X1, n
Xcall sub(i, xinc)
X!hpf$ reduce
Xx = x + xinc
Xenddo
X
Xsubroutine sub(i, xinc)
Xinteger i, xinc, local_b
Xlocal_b = (i-1)**2
Xxinc = i * local_b
Xend subroutine sub
X
X2.5 Relaxation of the HPF Standard's requirement for INDEPENDENT loops. 
X
XIn Section 4.4, the first proviso on what may not occur in the loop is 
Xmodified to: 
X
X* Any two operations that assign to the same object interfere with each 
Xother,
X(begin added text)
Xunless the object is assigned a new value in a reduction statement. (end 
Xadded text)
X
XThus, the user's assertion implied by INDEPENDENT about the behavior of 
Xthe program is weakened, and no longer applies to reduction variables that 
Xare protected in the DO construct. The value of such a variable on exit 
Xfrom the loop is processor dependent, although it is constrained by the 
Xmodel implementation mechanism given in Section 3. 
X
X
X2.6 Reduction Operators May Be Mixed.
X
XIn most cases, only one operator will be used in the reduction statements 
X(if there are more than one) that update a given reduction variable. It is 
Xsensible, however, to use different operators, such as + and -, together 
Xon the same reduction variable: mathematically, subtraction is just 
Xaddition of the additive inverse. The language makes no restriction on the 
Xmix of operators the programmer chooses to use.
X
X3. Implementation and Semantics
X
XFollowing an independent loop, the value of any reduction variable that 
Xwas protected in the loop is not completely defined by HPF. One possible 
Xvalue is that which would have been computed by sequential execution of 
Xthe loop, but others are also possible.
X
XWhile HPF does not explicitly define
Xthe set of possible values, it does give a precise implementation 
Xmechanism that serves to define this set. 
X
XSince no reference to a protected reduction variable can occur outside of 
Xa reduction statement in the loop body, it is not necessary to define the 
Xvalues that these variables may have inside the loop. 
X
XIn this discussion, the term processor means a single physical processor 
XOR a group of physical processors that together sequentially execute some 
Xor all of the iterations of an independent loop. 
X
XThe value on exit from the loop is to be computed as follows. Each 
Xprocessor makes a local copy of the reduction variable, and initializes it 
Xto the identity value specified in the reduction-clause if any, or the the 
Xidentity element for the intrinsic reduction operator if there is no 
Xreduction-clause.
XThe identity elements for the intrinsic operators are as follows: 
X
XOperator	Identity Element
X-------	----------------
X+	0
X-	0
X*	1
X/	1
X.and.	.true.
X.or.	.false.
X.eqv.	.true.
X.neqv.	.false.
X
X//	''
X
XThe following intrinsics may be used as reduction-functions, with the 
Xindicated identity elements:
X
X
XOperator	Identity Element
X-------	----------------
XIAND(I,J)	(all ones)
XIOR(I,J)	0
XIEOR(I,J)	0
XMIN(X,Y)	-HUGE(X) where x has the same type and kind type
Xparameter as the reduction variable
XMAX(X,Y)	+HUGE(X) where x has the same type and kind type
Xparameter as the reduction variable
XMATMUL(A,B)	The identity matrix of the same shape, type, and kind type
Xparameter as A, provided the shape of A is square, i.e.
XRANK(A) is 2 and SIZE(A,1) is the same as SIZE(A,2). As a
Xconsequence of the form A = MATMUL(A, B) of the reduction
Xoperation, it must also be true that
XSIZE(A,2) = SIZE(B,1) = SIZE(B,2).
X
X
XEach processor performs a subset of the loop iterations; when it 
Xencounters a reduction statement, it updates its
Xown copy of the reduction variable. A processor is free to perform its 
Xloop iterations in any order; furthermore, it may start an iteration, 
Xsuspend work on it, do some or all of the work of other iterations, and 
Xresume work on the suspended iteration. However, any update of a local 
Xreduction variable occurs through the execution of a reduction statement, 
Xand reduction statements *are* executed atomically.
X
XThe final value of the reduction variable is computed by combining the 
Xlocal values with the value of the global reduction variable on entry to 
Xthe loop, using the combining operator. The ordering of this reduction is 
Xlanguage processor dependent, just as it is for the intrinsic reduction 
Xfunctions (SUM, etc.) 
X
X3.1 Loop nests; protection of reduction variables 
X
XWhen loops are nested, a reduction variable may be protected in an 
Xindependent outer loop even though the reduction operations in which it 
Xoccurs are nested inside an inner loop. Moreover, the inner loop and any 
Xintervening loops may or may not be independent. The general rule is that
Xit is protected while is the *outermost* independent loop in which it does 
Xnot appear in a NEW clause is active.
XIt is therefore protected throughout all iterations of any independent 
Xloop in which it occurs in a reduction statment, unless it appears in a 
XNEW clause in the independent directive that applies to the loop. 
X
XFor example:
X
X! Nested Loop Example 1. Inner loop is sequential 
X
Xx = 10
X!hpf$ independent
Xdo i = 1, n
Xdo j = 1, m
X!hpf$ reduce
Xx = x + j
X!	illegal to refer to x here, except in a reduction statement
Xenddo
X!	illegal to refer to x here, except in a reduction statement
Xenddo
Xprint *, x
X
XSince the variable x occurs in a reduction statement in the body on the 
Xindependent outer loop, it is a protected reduction variable throughout 
Xthat loop, including inside the inner loop.
X
XThis remains true if the inner loop is independent. On the other hand, a 
Xvariable that occurs in a new clause may not be a reduction variable in 
Xthe body of a loop, although it may be in the body of a contained loop:
X
X! Nested Loop Example 2. Outer loop new clause. 
X
X!hpf$ independent, new x
Xdo i = 1, n
Xx = 10
X!hpf$ independent
Xdo j = 1, m
X!hpf$ reduce
Xx = x + j**2
X!	illegal to refer to x here, except in a reduction statement
Xenddo
Xy(i) = x
Xenddo
X! In contrast to the example above, the value of x is not defined after 
Xthe loop, ! although it can be redefined here
X
XHere, x is a protected reduction variable only in the inner loop. 
X
XThe reduction statment may not occur in a subprogram called from the body 
Xof the independent loop. To get this effect, return the updating value 
Xfrom the subprogram and perform the update with a reduction statement in 
Xthe lexical loop body.
X
Xprogram example
Xinteger x, xinc, i
X
Xx = 10
X!hpf$ independent, new(xinc), reduction(x, combine=+, identity = 0) do i = 
X1, n
Xcall sub(i, xinc)
X!hpf$ reduce
Xx = x + xinc
Xenddo
X
Xsubroutine sub(i, xinc)
Xinteger i, xinc, local_b
X! Since local_b is automatic it is effectively NEW 
Xlocal_b = (i-1)**2
Xxinc = i * local_b
Xend subroutine sub
X
X4. Examples.
X
XFor intrinsic operators and scalars, the reduction-clause is unnecessary. 
X
X!hpf$ independent
Xdo i = 1, n
X...
X!hpf$ reduce
Xx = x + <expr1>
X...
X!hpf$ reduce
Xx = x - <expr2>
Xenddo
X
XThe reference may be to a scalar array element: 
X!hpf$ independent
Xdo i = 1, n
X!hpf$ reduce
Xx(indx(i)) = x(indx(i)) + <expr>
Xenddo
X
X
XIt may be to a section of an array:
X
X!hpf$ independent
Xdo i = 1, n
X!hpf$ reduce
Xx(i: i+4) = x(i: i+4) + <expr> ! as many as 5 updates to elements of x 
Xenddo
X
XNote that the compiler, if it distributes iterations of this loop in a 
Xblock-wise manner, will not need to make a private copy of the entire 
Xarray x. 
X
XConsider the loop:
X
X!hpf$ independent
Xdo i = 1, n
X!hpf$ reduce
Xx(ix(i)) = <expr>
Xenddo
X
XThis is syntactically disallowed. The effect that is apparently wanted 
Xhere, namely to store in x(k) some unspecified one of the subset of 
Xinstances of <expr> for which ix(i) == k (a copy_scatter) can be achieved 
Xby using the second form of reduction statement.
X
XIn the following example the user decides to accumulate sparse local 
Xupdates to a distributed dense matrix.
X
Xtype real_sparse_matrix
Xinteger nrows, ncols, nnz ! shape, number of nonzeros real, pointer :: 
Xvals(:)
Xend type real_sparse_matrix
X
Xinterface operator (+)
Xfunction full_plus_sparse(x,y)
Xreal x(:,:)
Xtype (real_sparse_matrix) :: y
Xend function full_plus_sparse
X
Xfunction sparse_plus_sparse(x,y)
Xtype (real_sparse_matrix) :: x, y
Xend function sparse_plus_sparse
Xend interface
X
Xfunction spzeros(nr, nc) result(res)
Xtype (real_sparse_matrix) res
Xinteger nr, nc
Xres%nrows = nr
Xres%ncols = nc
Xres%nnz = 0
Xallocate(res%vals(0))
Xend function spzeros
X
Xfunction elt(i, j, x) result(res)
Xtype (real_sparse_matrix) res
Xinteger i,j
Xreal x
Xres%nrows = nr
Xres%ncols = nc
Xres%nnz = 1
Xallocate(res%vals(1))
Xres%vals(1) = x
Xend function elt
X
Xreal gs(n,n)
X!hpf$ distribute (block, block) :: gs
Xtype (real_sparse_matrix) :: s
X
Xs = spzeros(n,n)
X!hpf$ independent, reduction(s; combine = +, identity = spzeros(n,n)) do i 
X= 1, num_updates
X...
X!hpf$ reduce
Xs = s + elt(row, col, val) ! adds in elt at position (row, col) enddo
Xgs = gs + s	! full_plus_sparse
X
X
X--------------------------------------------------------------------------------
X[A response from David Presberg]
X
XQuick reaction concerning the simplest reduction cases after having
Xonly read about 1/6th of the text:
X
X - it is good that the "reduction-clause" is optional for the simple
Xcases: "... intrinsic operators applied to a reduction variable of
Xintrinsic type."  BUT that leaves the status of reductions formed from
X(Fortran) intrinsic functions in doubt as read with the mandate for
X"reduction-clause" for "...user-defined reduction operators". I.e., no
Xmention of optional/mandatory for "reduction-function"-using forms. Note
Xthat your first example in section 2.3 uses MAX(...) --- clearly not a
X"user-defined reduction operator" nor an "intrinsic operator".
X
X  [<Formalistic-aside> The proper Fortran standardeze phrase
X  would be '"intrinsic-operator" of an "intrinsic binary operation"'.
X  There is English pseudo-syntax in F90 Section 7.1.2 that covers that
X  topic.  It is there because the actual syntax doesn't slice the topic in
X  a convenient way to allow talking collectively about all binary
X  operators. </Formalistic-aside>]
X
X - having seen the !HPF$ REDUCTION early in the text, I wondered
Ximmediately about the need for the "reduction-clause" when I
Xencountered it.  Not having read further ahead (yet) I don't know if
X_both_ are required...  But I note that the simplest cases (e.g., that
Xintrinsic operator case and the one that is the easy generalization
Xfrom "x = max( x, expr )" ) are very easily discovered by any compiler
Xwith the technology for doing the rest of HPF.  (I cite an early
X1970's automatic parallelizing (extended) Fortran 77 compiler for a
XSIMD machine -- David L. and Carl O. stop grimacing.)  Thus for that
Xsimple case, neither the (statement form) directive nor the clause
Xought to be required.
X
X - there is a serious flaw in the design of the "reduction-clause":
Xthe "; IDENTITY = value" phrase immediately leads to some (potential)
Xsemantic difference in the HPF as processed by an HPF system and the
X(serial) Fortran as processed by a non-HPF Fortran system.  
X
X==> Please drop that initialization phrase in favor of:
X    
X    (a) tabulating in the HPF standard the "zero"-element-value for
X    each of the intrinsic operators and suitable binary intrinsic
X    functions, and
X    
X    (b) thinking a bit harder about how to convey that same idea for
X    "defined-binary-op"s or "reduction-function"s (particularly
X    user-defined functions).  Perhaps you'll have to settle on e.g. "0
X    of the correct data type" for these cases.
X
X(Is there text later in the discussion about an initial value of the
Xreduction variable prior to the INDEPENDENT loop with the reduction?
XSlap me down if I didn't read far enough before reacting.)
X
X
XHave fun next week.
X
X-- Pres
X
X
X--------------------------------------------------------------------------------
X[And replies^2 from Rob Schreiber]
X
X
X        Major items:
X
X         - I am still not comfortable with the " ; IDENTITY = value" clause.
X        What if the user codes:
X
X               program AbsurdInitialization
X               x=0
X        !HPF$ INDEPENDENT, REDUCTION( x; COMBINE = +; IDENTITY = 2. )
X               do i=1,100
X        !HPF$    REDUCE
X                 x = x + log(i)
X               enddo
X               print *,x
X               end
X
X        Should they be allowed to state an absurd "zero" value for an
X        intrinsic operation?  
XAbsurd, hardly.  The semantics are well defined.   It prints a
Xprocessor dependent approiximation to 2*number_of_processors() +
Xsigma(i=1 to 100) log(i)
X
XI have changed it, however, to forbid combine and identity phrases 
X(clauses?) in the optional reduction clause for an intrinsic operators.  
XMaybe we should just bar the whole REDUCTION clause on the independent 
Xdirective for intrinsic reductions.  Should we define the term "intrinsic 
Xreduction" in the text?
X
X        (I wonder if the keyword would be more intuitive
X        as "ZERO" rather than "IDENTITY".)  
XAs in, like, 
X!HPF$ INDEPENDENT, REDUCTION(X, COMBINE = *, ZERO = 1)
XThe mind boggles.
X
X        If each processor initializes its
X        local value of x to 2., it would seem that the HPF interpretation of
X        this code (if run on more than one processor) would give wildly
X        different results than a sequential F90 interpretation.
XYes, a very BAD processor dependent approximation to the sequential result.
XThis is disconcerting, but I dont see any way around this for user defined
Xoperators, except to REQUIRE that the user create a value "foo" for
Xhis operator bar(x,y) that has the property that bar(foo,y) = y for all y.
XShould we.
X
X        I agree that you need a hook for the "zero" value of a user-defined
X        operator or reduction function, but I think you also need either:
X
X         (a) constraint language about the defined state, _and_ _defining_
X        _value_ of the reduction variable prior to entry to the loop nest with
X        the reduction, or
XDo you mean what I said above about the "identity element" property of the
Xreduction operator and the identity value?
X
X         (b) prohibit use of the clause for the trivial cases you have
X        enumerated in section 3.
XThose are the intrinsic cases, where we could certainly make this prohibition.
X
X         - Again, I think you've surrounded the syntax with so many lexical
X        restrictions that any undergraduate senior-level compiler course
X        project could find and deal with the reductions with absolutely _no_
X        directives!!  Could you at least relax the form of the reduction
X        statement to include:
X
X          var = expr op var
X
X        and maybe even:
X
X          var = expr1 op var op expr2   ! with constraint that the 2 ops are indentical
X
XNo problem here semantically, I guess;   not my call.
X
X
X         - It would seem that one cannot:
X
X         ...
X        !HPF$ REDUCE
X                  var = var .op. f(x)
X         ...
X        !HPF$ REDUCE
X                  var = var .op. f(x)
X         ...
X
X        Why not?
XOh?  Where is this rule?
X
X
X         - I think the constraints on types of lists of variables and on other
X        things that should agree in type, etc., are a little slack.
XGot to show me a bug, however, before I can fix them.
X
X
X        Bottom line:  For user-defined operators and reduction functions, and
X        defined-type reduction variables, the clause on the INDEPENDENT and
X        the REDUCTION directive are (probably) necessary.  I think you really
X        _only_ have to define *protected* for the other cases without any
X        clauses and directives, and then you can do the relaxation of
X        constraints on the body of INDEPENDENT for the simple situations and
X        not have to require any extra stuff for those simple cases.
X
XCan you be a little clearer?   If i understand you, in the case of
Xreductions with intrinsic operators, you suggest no reduction clause on
Xthe loop and no reduce directive on the operation.   I like this idea;
Xhowever, we have, in that case, to prohibit mixed operators, so that
Xthe combine operator and identity value are implied by the (one class
Xof) reduction operator/function that occurs in the loop.
X
XRob
END_OF_FILE
if test 25887 -ne `wc -c <'Reduction_proposal.txt'`; then
    echo shar: \"'Reduction_proposal.txt'\" unpacked with wrong size!
fi
# end of 'Reduction_proposal.txt'
fi
if test -f 'Task_proposal.txt' -a "${1}" != "-c" ; then 
  echo shar: Will not clobber existing file \"'Task_proposal.txt'\"
else
echo shar: Extracting \"'Task_proposal.txt'\" \(15783 characters\)
sed "s/^X//" >'Task_proposal.txt' <<'END_OF_FILE'
XProposal for Task Parallelism in HPF
X
XHere is a modified task parallelism proposal. This proposal is not quite 
Xformal since the final wording will be somewhat dependent on SUBGROUP and 
XON features which are under development. On the bright side, I think it is 
Xrelatively easy to read and better debugged than earlier proposals.
X
XJaspal and Bwolen
X
XAssumptions of functionality available from related features: 
X
X1) A way to group processors into subgroups P1, P2, P3 and distribute 
Xvariables onto them, i.e. a1, a2, a3.
X
X2) An ON construct that directs execution on groups of processors 
XP1, P2, P3 etc. for a block of code.
X
X
XPROPOSAL:
X
XA "task region" is a single entry, single exit region delimited by (say) 
XTASK REGION .... END TASK REGION. A task region can have blocks of code 
Xthat are directed to execute ON a processor subgroup. All other code 
Xexecutes on all available processors, referred to as ALL. 
X
XThe following restrictions must hold for the code inside a task region. 
X
XA code block directed to execute ON a subgroup must be a single entry 
Xsingle exit region.
X
XA code block directed to execute ON a subgroup P may access a variable 
Xlocation not mapped to P only if that variable location is: 
X
Xa) accessed exclusively in code directed to execute ON P. 
XOR
Xb) not written to in the task region.
X
X[There are no other access constraints: Code executing on ALL processors 
Xhas unrestricted access to all variable locations. A code block directed 
Xto execute ON a subgroup P has unrestricted access to all variable 
Xlocations mapped to P]
X
X
XAn I/O operation in a code section directed to execute ON a subgroup may 
Xnot ``interfere'' with an I/O operation in a code section not explicitly 
Xdirected to execute on that subgroup. The interference of I/O operations 
Xis detailed in Section 4.4 (INDEPENDENT). 
X
XFor a subroutine call inside an ON block, "all available processors" are 
Xprocessors in the corresponding subgroup. This is the number that is used 
Xfor mapping the parameters of the subroutine. [This part will become more 
Xspecific after the syntax etc. of creating subgroups is decided. There 
Xshould probably be a system inquiry function for the number of processors 
Xin the current subgroup, if NUMBER_OF_PROCESSORS() is supposed to return 
Xthe total number of processors for the program] 
X
XCOMPILATION/EXECUTION MODEL
X
XThe execution model for a subgroup is to unconditionally execute code ON 
Xit, unconditionally skip code ON others, and participate in the execution 
Xof common code (on ALL processors) as normal data parallel code.
X[The access restrictions guarantee that the results will be consistent 
Xwith pure data parallel execution. A processor group cannot be "invisibly" 
Xwriting to a location being accessed by ALL or another processor group, 
Xand vice versa]
X
XFollowing is one model for accessing variables in a task region: 
X
XAccesses to variables owned by other processors is cooperative, i.e. the 
Xowner sends the value, and the user receives it, with one exception - when 
Xcode ON a subgroup has to access a variable not mapped to it, it use a 
Xremote fetch/deposit. (It can also cache remote locations locally in the 
Xsubgroup for the duration of the execution of the task region since 
Xcomputation not ON that subgroup cannot access it) 
X
XEXAMPLE: 2DFFT
X
XSequential:
X
X
Xreal, dimension(n,n) :: a1, a2
X
Xdo while(.true.)
Xread (unit = iu, end = 100) a1
Xcall rowfft(a1)
Xa2 = a1
Xcall colfft(a2)
Xwrite (unit = ou) a2
Xcycle
X100	continue
Xexit
Xenddo
X
X
XPipelined Data/Task Parallel HPF
X
Xreal dimension(n,n) :: a1,a2
Xboolean done1
X!hpf$ disjoint processor groups P1, P2 (Syntax TBA) !hpf$ distribute 
Xa1(block,*) onto P1
X!hpf$ distribute a2(*,block) onto P2
X!hpf$ distribute done1 onto P1
X
X!hpf$ TASK REGION
Xdone1 = .false.
Xdo while (.true.)
X!hpf$	ON HOME(P1) BLOCK
Xread (unit = iu,end=100) a1
Xcall rowfft(a1)
Xgoto 101
X100	done1 = .true.
X101	continue
X!hpf$	END BLOCK
X
Xif (done1) exit
Xa2 = a1
X
X!hpf$	ON HOME(P2) BLOCK
Xcall colfft(a2)
Xwrite(unit = ou) a2
X!hpf$	END BLOCK
Xenddo
X!hpf$ END TASK REGION
X
X
XThe data parallel code on the two processor groups will look something 
Xlike this, after the task region is compiled.
X
XProcessor Group P1:
X
Xreal dimension(n,n) :: a1
X!hpf$ distribute a1(block,*)
Xboolean done1
X
Xdone1 = .false.
Xdo while (.true.)
Xread (unit = iu,end=100) a1
Xcall rowfft(a1)
Xgoto 101
X100 done1 = .true.
X101 continue
X_send(done1,P2)
Xif (done1) exit
X_send(a1,P2)
Xenddo
X
X
XProcessor group P2:
X
Xreal dimension(n,n) :: a2
X!hpf$ distribute a2(*,block)
Xboolean local_done1
X
Xdo while (.true.)
X_receive(local_done1,P1)
Xif (local_done1) exit
X_receive(a2,P1)
Xcall colfft(a2)
Xwrite(unit = ou) a2
Xenddo
X
X
X
X
XCOMMENTS:
X
X1) The main difference with the simple parallel section/region, 
X(or using an INDEPENDENT do loop to achieve parallel sections), is that 
Xtask regions presented can have code that executes on ALL processors also. 
XIf it has no such code, it is similar to parallel section/region. However, 
Xallowing other code makes this construct more general, and implicitly 
Xallows pipelining in particular. At the same time, existence of code in 
XALL can constrain parallelism due to data dependence, and in the worst 
Xcase no task parallelism may exist. 
X
X2) No explicit control dependence constraints are required. Inside an ON 
Xblock, any variable being read (or used for control flow) cannot be 
Xwritten by any other processor group - it can be only written by ALL 
Xprocessors, in which case the control flow from the subgroup must also 
Xreach that point. 
X
XOutside an ON block, all processor groups execute all control flow (and 
Xother) statements. If a subgroup skips a control construct because it is 
Xnot involved( i.e. its variables are not involved and there is no code 
Xinside the scope of the control construct that is directed to execute ON 
Xit) and continues to execute its next ON block, the constraints ensure 
Xthat it cannot write to a location that is used for managing control flow.
X
X--------------------------------------------------------------------------------
X[Reply from Chuck Koelbel]
X
X>Here is a modified task parallelism proposal. This proposal is not quite 
X>formal since the final wording will be somewhat dependent on SUBGROUP and 
X>ON features which are under development. On the bright side, I think it is 
X>relatively easy to read and better debugged than earlier proposals.
X
XI tend to agree, and will make sure that it gets printed for the next 
Xmeeting. 
X
XA couple details follow...
X
X>The following restrictions must hold for the code inside a task region. 
X>
X>A code block directed to execute ON a subgroup must be a single entry 
X>single exit region.
X
XGood news, this is part of the ON proposal already. (Albeit phrased as "no 
Xjumps into or out of the region)
X
X>A code block directed to execute ON a subgroup P may access a variable 
X>location not mapped to P only if that variable location is: 
X>
X>a) accessed exclusively in code directed to execute ON P. 
X>OR
X>b) not written to in the task region.
X
XShouldn't the second constraint be
Xb) not written to by code in another ON block in the task region. ?
X
XAnother way of asking this is, "Aren't the sequential regions also in the 
Xtask region?"
X
X>COMPILATION/EXECUTION MODEL
X
X>The execution model for a subgroup is to unconditionally execute code ON 
X>it, unconditionally skip code ON others, and participate in the execution 
X>of common code (on ALL processors) as normal data parallel code.
X>[The access restrictions guarantee that the results will be consistent 
X>with pure data parallel execution. A processor group cannot be 
X>"invisibly" writing to a location being accessed by ALL or another 
X>processor group, and vice versa]
X
XRight motivation. But if processor groups can't write to locations read by 
XALL, how is data going to flow from group to group? 
X
X>The data parallel code on the two processor groups will look something 
X>like this, after the task region is compiled.
X
XDefinitely "might" look something like this. I expect many compilers, 
Xespecially those on scalable machines, to produce SPMD code with IF 
Xstatements (checking local data, like myproc()) about where the ON 
Xdirectives are.
X
X>2) No explicit control dependence constraints are required. Inside an ON 
X>block, any variable being read (or used for control flow) cannot be 
X>written by any other processor group - it can be only written by ALL 
X>processors, in which case the control flow from the subgroup must also 
X>reach that point. 
X
X>Outside an ON block, all processor groups execute all control flow (and 
X>other) statements. If a subgroup skips a control construct because it is 
X>not involved( i.e. its variables are not involved and there is no code 
X>inside the scope of the control construct that is directed to execute ON 
X>it) and continues to execute its next ON block, the constraints ensure 
X>that it cannot write to a location that is used for managing control flow.
X
XIn the following sequence, does P1 execute the GOTO? It doesn't involve 
Xany data mapped to P1, nor is it directed ON P1. 
X
X!HPF$ DISTRIBUTE A1(BLOCK) ONTO P1
X!HPF$ DISTRIBUTE A2(BLOCK) ONTO P2
X
X!HPF$ TASK REGION
XIF (A2(2) > 0) THEN
X!HPF$ ON HOME(P2) BLOCK
X... do something with A2 ...
X!HPF$ END BLOCK
XGOTO 100
XEND IF
X!HPF$ ON HOME(P1) BLOCK
X... do something with A1 ...
X!HPF$ END BLOCK
X100 CONTINUE
X!HPF$ END REGION
X
XI think I know what you mean. Probably just need to be a little more 
Xcareful about the execution model, in particular, what does "normal 
Xdata-parallel model" mean?
X
X
XChuck
X
X
X--------------------------------------------------------------------------------
X[Reply^2 from Jaspal Subhlok]
X
X
XYou point out some text on access restrictions that seems to be in error. 
XI think the restrictions may be more subtle than they appear, but the text 
Xis what is intended. I will add a general motivation section along the 
Xlines of the next paragraph in the next revision - please take a look and 
Xmail again if you think something needs to be changed. 
X
X
XTASKING MODEL: Subgroups ``normally'' read and write only to/from the 
Xvariables that are mapped to them. The code in ALL has unrestricted access 
Xto all variables, and data is exchanged between subgroups by copying the 
Xvariables of one subgroup to the variables of another subgroup in the ALL 
Xcode.
X
XIn some circumstances, a subgroup may want to access a variable NOT mapped 
Xto it (e.g. a common block). Such access is allowed, but it must not cause 
Xa data dependence during the entire duration of the execution of the task 
Xregion. (Even dependence between a subgroup and ALL are not allowed since 
Xthat would imply that when ALL is executing no other subgroup can execute 
Xas there is no easy way for a compiler to figure out what variable a 
Xsubgroup may access ). A sufficient condition for "no dependences" is that 
Xsuch accesses should only be to variables that are ``read only'' in the 
Xtask region, or ``read and written'' only by code ON only one subgroup.
X
X
XOther points are well taken and I will add corrections/clarifications in 
Xthe next revision. Yes, the compiled code is just one way to do it, and 
Xmeant only for illustration of the task constructs. Every processor 
Xexecutes control constructs in ALL unless the compiler can determine that 
Xit is not necessary. In general, a GOTO is certainly executed by 
Xeverybody. The "normal data-parallel model" is supposed to mean that 
Xexecution follows the normal HPF semantics (some say no such thing exists 
X:) but I am not going to fix that) without any notion of task parallelism 
Xconstraints.
X
XLet me know if there are unresolved things or other comments. 
X
X
X
Xjaspa
X
X
X--------------------------------------------------------------------------------
X[Reply^3 from Chuck Koelbel] 
X
X
XSounds to me like you're addressing my concerns. Maybe it would also help 
Xto explicitly mention that a processor group always has write access to 
Xdata mapped to it. Also, note that "normal data-parallel mode" on some 
Xmachines may mean GET/PUT access; copying data in this way in ALL may not 
Xbe sufficient for the synchronization that you need. 
X
XChuck
X
X
X
X--------------------------------------------------------------------------------
X[Reply^4 from Jaspal Subhlok]
X
X
X>Sounds to me like you're addressing my concerns. Maybe it would also help 
X>to explicitly mention that a processor group always has write access to 
X>data mapped to it.
X
XActually it is mentioned that processor groups have unrestricted access to 
Xtheir variables, but only as a comment. Excerpt: 
X
X[There are no other access constraints: Code executing on ALL processors 
Xhas unrestricted access to all variable locations. A code block directed 
Xto execute ON a subgroup P has unrestricted access to all variable 
Xlocations mapped to P]
X
XPerhaps it should be in the main text.
X
X>Also, note that "normal data-parallel mode" on some machines may mean 
X>GET/PUT access; copying data in this way in ALL may not be sufficient for 
X>the synchronization that you need. 
X
X
XIf GET/PUT access is used, the compiler has to include explicit 
Xsynchronization like barriers for normal data parallel processing. The 
Xcompiler has to use "subset barriers" to be able to exploit task 
Xparallelism. One concern is that if a full barrier is used around ALL 
Xsections, then the program may be oversynchronized, so subgroup barriers 
Xshould be used in ALL as needed. e.g. if a variable in group1 is copied to 
Xa variable in group2, correct use of subgroup barriers will allow group3 
Xprocessors to continue if there is no other code in ALL that needs them.
X
XAnyway, I think this is about efficient compilation in GET/PUT model and I 
Xdon't think it changes the specification or the basic execution model.
X
Xjaspal
X
X
X--------------------------------------------------------------------------------
X[Reply^5 by Chuck Koelbel]
X
X>>Sounds to me like you're addressing my concerns. Maybe it would also 
X>>help to explicitly mention that a processor group always has write access 
X>>to data mapped to it.
X>
X>Actually it is mentioned that processor groups have unrestricted access 
X>to their variables, but only as a comment. Excerpt: 
X>
X>[There are no other access constraints: Code executing on ALL processors 
X>has unrestricted access to all variable locations. A code block directed 
X>to execute ON a subgroup P has unrestricted access to all variable 
X>locations mapped to P]
X>
X>Perhaps it should be in the main text.
X
XI think so. It's an important point.
X
X>>Also, note that "normal data-parallel mode" on some machines may mean 
X>>GET/PUT access; copying data in this way in ALL may not be sufficient for 
X>>the synchronization that you need. 
X>
X>...
X>Anyway, I think this is about efficient compilation in GET/PUT model and 
X>I don't think it changes the specification or the basic execution model.
X>
X>jaspal
X
XAgreed. Is it true that your execution model assumes that the owners of A 
Xand B both participate in A=B (at least in the sense that they do some 
Xsynchronization)? If so, I'd feel better if this were explicitly stated 
Xsomewhere. Yes, in some sense the synchronization is also implied in 
Xstraight HPF - but here the model is explicitly multi-threaded, as opposed 
Xto the single-threaded model in HPF-default mode. 
X
XChuck
X
X--------------------------------------------------------------------------------
X[Jaspal Subhlok gets the last word]
X
X
XOK, it will probably make things clearer to state that owners of A and B 
Xboth participate in an A=B. But the main point is that execution model in 
XALL is same as regular HPF, (with extra information that subgroup 
Xcomputations can be assumed to only read and modify subgroup variables 
Xfrom ALL's perspective), and it should be stated in those terms, else we 
Xhave to detail the behavior in all scenarios. Perhaps some illustration 
Xwill make things easier. 
X
Xjaspal
END_OF_FILE
if test 15783 -ne `wc -c <'Task_proposal.txt'`; then
    echo shar: \"'Task_proposal.txt'\" unpacked with wrong size!
fi
# end of 'Task_proposal.txt'
fi
if test -f 'interoperability.tex' -a "${1}" != "-c" ; then 
  echo shar: Will not clobber existing file \"'interoperability.tex'\"
else
echo shar: Extracting \"'interoperability.tex'\" \(20332 characters\)
sed "s/^X//" >'interoperability.tex' <<'END_OF_FILE'
X\documentstyle[twoside,11pt]{article}
X
X\pagestyle{myheadings}
X\sloppy
X\footheight=-.60in
X\headheight=0in
X\textwidth=5.50in
X\topmargin=-.20in
X\oddsidemargin=0.5in
X\evensidemargin=0.5in
X
X\itemsep=.05in
X\topsep=.05in
X\parsep=.05in
X\parskip=.10in
X\textheight=9in
X
X\newcounter{dofigures}  \setcounter{dofigures}{0}
X\newcounter{dotables}        \setcounter{dotables}{0}
X\newcommand{\havefigures}{\setcounter{dofigures}{1}}
X\newcommand{\havetables} {\setcounter{dotables}{1}}
X
X\newcommand{\cccdocid}[5]{
X  \bibliographystyle{alpha}
X% ******************** Title Page ********************
X  \vspace*{.5in}
X  \centerline{\Large \bf {#1}}
X  \vspace*{.02in}
X  \centerline{{#2}}
X  \vspace*{.02in}
X  \centerline{{#3}}
X  \vspace*{.02in}
X  \centerline{{#4}}
X  \vspace*{.02in}
X  \centerline{{#5}}
X  \vspace*{.5in}
X
X  % ******************** List of Figures ********************
X  % Note: Remove the following line if there are no figures
X%  \ifodd\value{dofigures} \listoffigures \newpage \fi
X%  \ifodd\value{dotables}  \listoftables  \newpage \fi
X%  \markboth{#1} {#3}
X%  \pagenumbering{arabic}
X  }
X
X% ******************** Special macros ********************
X%\catcode`^^Z=9                % Make TeX ignore IBMPC EOF character
X
X\newcommand{\ital}[1]{{\it #1}}
X\newcommand{\type}[1]{{\tt #1}}
X\newcommand{\bold}[1]{{\bf #1}}
X\newcommand{\dfn}[1]{{\it #1}\index{#1}}
X\newcommand{\comment}[1]{}
X
X\def\fineprint{\relax}
X\def\stopfine{\relax}
X
X
X\newcommand{\beginprog}{\begin{tabbing}
Xxxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=\kill
X}
X\newcommand{\tb}{\>\tt}
X\newcommand{\finprog}{\end{tabbing}}
X\newcommand{\progtxt}[1]{``{\tt #1}''}
X\newcounter{save_enum}
X\newcommand{\hilite}[1]{ \item {\bf #1\\} }
X\newcommand{\morelater}{\centerline{$\ldots$\ital{Fill this in later}$\ldots$}}
X
X
X% ******************** Start of Document ********************
X\begin{document}
X
X\cccdocid
X  {HPF calling C Interoperability Proposal}
X  {Ver 1.1}
X  {Andrew Meltzer}
X  {Cray Research, Inc.}
X  {September, 1995}
X
X
X\section{Introduction}
X
XDefining an interoperability mechanism is as much a problem 
Xof recognizing the limitations of interlanguage communication as it is of 
Xsolving the technical interlanguage calling issues. 
X
XThis document contains a proposal for an HPF calling C interoperability
Xmechanism and a rational for the limitations and techniques used in the
Xproposal.   It also contains a set of representative calls from the
XX Windows interface for context.
X
X
X\subsection{Defining the Bounds of the Problem}
X
XThere are a number of ways in which the interoperability question has
Xbeen viewed:
X
X\begin{enumerate}
X\item\label{i1} HPF should be able to call any C routine and be able to
X           represent or convert any C type. 
X
X\item\label{i2} HPF should be able to deal with any C type that has a
X           general approximation in Fortran.
X
X\begin{enumerate}
X\item\label{i2a}   The compiling system should handle all type
X                   conversions.  This includes remapping structures
X                   and transposing arrays.
X
X\item\label{i2b}   The compiling system should automatically convert
X                   all ``basic'' types to their closest equivalent in the
X                   other language.
X\end{enumerate}
X
X\item\label{i3} HPF programs calling C do not need any special
X           mechanisms.  It is the problem of the application programmer
X           to make sure that values are passed properly.
X\end{enumerate}
X
X
XOption \ref{i3} is currently the standard method, but since most application
Xprogrammers are looking for more ease-of-use, this option is not sufficient.
X
XOption \ref{i1} is an enormous effort and might impose a large runtime
Xpenalty.  To achieve this goal, Fortran would need to be extended to have
Xsome new types (bits, unsigned values, and perhaps unions) and the
Xcompiling system would need to transpose arrays on calls and 
Xconvert complicated structures to their C layouts.  
X
XOption \ref{i2a} is unreasonable for some of the reasons that option 
X\ref{i1} is unreasonable.  This sort of type conversion/mapping is 
Xoften not
Xdesired and may impose a huge and unavoidable runtime penalty on the
Xuser of the interface.  Consequently I believe that option \ref{i2b} is as
Xmuch as a compiling system should offer.  Users would be required to
Xunderstand layout of arrays in memory for each language and the way
Xstructures are constructed.
X
XThe solution proposed below is a variety of option \ref{i2b} which 
Xincorporates some of the features of \ref{i2a}, but only at the application 
Xprogrammer's discretion.
X
X
X\section{The X Window Interface}
X
XThe ability to call the X Windows library has often been mentioned as a
Xgood test of the sufficiency of an interlanguage interface for HPF and C.  
XI have picked a few representative library routines to demonstrate
Xthe problems that must be dealt with.
X
X\begin{verbatim}
X        char * XOpenDisplay(char *display_name)
X\end{verbatim}
X
XClearly any interface will have to deal with a conversion from Fortran
Xcharacter strings to the {\tt char *} of C.  Somewhat more complicated might
Xbe a call like
X
X\begin{verbatim}
X        XGetIconName(Display *display, Window w, char **icon_name)
X\end{verbatim}
X
XFor this call, as in most, the display value is simply a pointer
Xand need not be inspected by the HPF
Xcompiler.  {\tt Window} is an ID (some integer type), and {\tt icon\_name} is the
Xresult value and is consequently a pointer to a {\tt char *}.   A problem
Xwhich must be addressed is that of pointers to pointers in C (though in
Xthis case one might reasonably consider {\tt char *} to be the 
Xcharacter string type of C so that a pointer to it 
Xis the equivalent of a pointer to a
XFortran character string.)  In many other calls, pointers to structure
Xpointers are the return value, for example {\tt size\_list} in the following:
X
X\begin{verbatim}
X        XGetIconSizes(Display *display, Window w,
X                      XIconSize **size_list, int *count)
X\end{verbatim}
X
X
XFinally a call with some simpler parts:
X
X\begin{verbatim}
X        XStoreNamedColor(Display *display, Colormap cmap, char *colorname,
X                         unsigned long pixel, int flags)
X\end{verbatim}
X
X{\tt Display} has been previously discussed. {\tt Colormap} 
Xcan be stored in an 
X{\tt int} type of the proper size. The {\tt char *} type has 
Xbeen discussed.  {\tt Unsigned long} should have 
Xsome equivalent representation for Fortran 90 on any 
Xgiven compilation system, and finally {\tt int} is some size C integer.
X
X
X\section{Interlanguage Calling Proposal}
X
XThis section describes the syntax and semantics for the interlanguage
Xcalling sequence.  The interlanguage calling sequence is 
Ximplemented with the the extrinsic mechanism.
XA new extrinsic type
X\begin{verbatim}
X        EXTRINSIC (C) 
X\end{verbatim}
Xis added to the language.  The extrinsic declaration interface block must
Xbe visible to the compilation system at the time the call to the C function
Xis compiled.
X
XIt is important to fully understand what the {\tt EXTRINSIC}
Xmechanism is.  The {\tt EXTRINSIC} mechanism ``allows an HPF programmer
Xto declare a calling interface to a non-HPF subprogram'' (HPF Version 1.1,
XAnnex A.)  In this case the {\tt EXTRINSIC} mechanism assists the programmer
Xwhen calling a C function or subroutine by redefining the semantics
Xof a function or subroutine call.  Within the {\tt EXTRINSIC} the
Xcode need not conform to HPF semantics, though it is important from an
Ximplementation standpoint that it be easily parsed by an HPF parser.  
XThis proposal extends the semantics of the {\tt EXTRINSIC} mechanism
Xonly a small amount.
X
XThis proposal also extends the HPF language with a new intrinsic function
X(although it is already in common usage):
X\begin{verbatim}
X        LOC
X\end{verbatim}
X
X{\tt LOC} is used at the call site to indicate that the address of the
Xactual argument is required.
X
XWithin the {\tt EXTRINSIC (C)} environment a new attribute is defined:
X
X {\tt MAP\_TO([C\_TYPE= ]} {\it c type specifier } {\tt [, [LAYOUT= ]}{\it layout specifier}{\tt ])}
X
XThis attribute may have one or two specifiers, {\tt C\_TYPE} a 
Xrequired specifier,
Xand {\tt LAYOUT} an optional specifier.  {{\tt C\_TYPE} specifies the 
XC type that the HPF type will be mapped to, and {\tt LAYOUT} 
Xindicates the memory layout of the data being passed.  If no {\tt LAYOUT}
Xargument is given it defaults to {\tt NO\_CHANGE}.  The following are
Xthe layout specifiers an implementation is required to provide:
X
X\begin{verbatim}
X        Name               Function
X        ----               --------
X        C_ARRAY            Convert the array to a C memory layout
X        HPF_ARRAY          Convert the array to an HPF memory layout
X        NO_CHANGE          Do no rearrangement of the array memory layout
X\end{verbatim}
X
X
XThe following are the type specifiers that an implementation is required
Xto provide.
X
X\begin{verbatim}
X        Name                            C type
X        ----                            ------
X        NO_CHANGE                       no remapping is done
X        INT                             int
X        LONG                            long
X        SHORT                           short
X        CHAR                            char
X        FLOAT                           float
X        DOUBLE                          double
X        LONG_DOUBLE                     long double
X        CHAR_PTR                        indicates that the string will be 
X                                        converted to a null terminated char *
X
X\end{verbatim}
X
XThe vendor may also add other type specifiers.
X
XIf a C function using the {\tt EXTRINSIC (C)} interface
Xis called from within an HPF program then the interface
Xuses the same rules as {\tt HPF\_SERIAL} to remap data.  It may also be called
Xfrom within an {\tt HPF\_LOCAL} environment, in which case a different
Xinstance of the interface is
Xinvoked (a different instance of the C function is called) on each processor.
X
XHPF is not case sensitive, C is case sensitive.  
XThis proposal accepts the suggested addition of a {\tt NAME} clause
Xto be added to the {\tt EXTRINSIC} statement.  If this is added I suggest
Xthat it is an ideal way to distinguish the case of the C function.
X
X {\tt NAME(`}{\it function name}{\tt')}
X
XFor example:
X
X\begin{verbatim}
X       EXTRINSIC (C) FUNCTION CASE() NAME("CaSe")
X\end{verbatim}
X
X
X
X\subsection{Explanation of the proposal}
X
XThe declaration {\tt EXTRINSIC (C)} indicates to the compilation
Xsystem how to convert types (in C parlance use a cast to convert types) 
Xto the types that the called C program requires.  It does this with
Xthe {\tt MAP\_TO} attribute.  The implementation must also pass
Xthese arguments by value, or as is appropriate for the vendors implementation
Xof C.  
X
XThe vendor must provide the set of required type specifiers
Xand may add as many additional type specifiers as it wishes.
XThe vendor may also optionally provide the extrinsic interface for 
Xthe C system or library routines as well.
X
XTo handle pointers and pointers to pointers this proposal takes a
Xmechanism from common practice, the {\tt LOC} intrinsic.  If, for
Xexample, a C function required an argument to be passed as a 
Xpointer to a pointer:
X
X\begin{verbatim}
X       void cfunc(int **pp);
X\end{verbatim}
X
Xthen the corresponding call (with, of course, the proper extrinsic
Xdeclaration visible) would be:
X
X\begin{verbatim}
X       INTEGER PP
X       CALL CFUNC(LOC(LOC(PP)))
X\end{verbatim}
X
XSince the called C function requires a pointer to a 
Xpointer to be passed, using the {\tt LOC} mechanism here does not
Xhinder portability.
X
XIn addition, HPF must recognize a new attribute within {\tt EXTRINSIC (C)},
Xthe {\tt MAP\_TO} attribute.  This attribute is the core of the mechanism.
XIt allows the user to specify the conversions necessary to facilitate
Xthe interlanguage calls.   The attribute has one or two specifiers.  The first
Xspecifier indicates the type of the C data that the HPF data must be
Xconverted to.  The second specifier indicates (for an array) whether
Xand how to remap the storage.  {\tt HPF\_ARRAY} indicates that the 
Xruntime system should re-structure an array in memory (this is only
Xuseful for a return value from a called C routine) to conform to the 
Xdeclared layout of the array that will contain the result of the function.
X{\tt C\_ARRAY} indicates that the
Xruntime system should re-structure an array in memory to conform to
Xthe system's default C style layout (this is only useful for arguments.)
XFor example, a user would
Xwrite the following within the {\tt EXTRINSIC (C)} interface to ensure 
Xthat the compiler remaps the array {\tt a}
Xto the system's default C array layout:
X
X\begin{verbatim}
X         INTEGER, MAP_TO(C_TYPE=INT, LAYOUT=C_ARRAY)  a(100, 10, 4)
X\end{verbatim}
X
XThe {\tt LAYOUT} argument is
Xnot strictly required to make the interface work, but its functionality
Xis often enough utilized that there should be a portable way of specifying
Xthe transformations.  
X
XIt is important to emphasize that the extrinsic environment does more
Xthan just type checking.  In the presence
Xof {\tt EXTRINSIC (C)} and the {\tt MAP\_TO} attribute, type conversion 
Xis done.   The implementation 
Xmight warn a user about a real to integer type conversion, but in 
Xgeneral the interface will use the types specified by the {\tt MAP\_TO}
Xattribute in the interface as target 
Xtypes for the call and do whatever it takes to make the types passed 
Xin fit.   This only applies to the system's basic types, however.
XIf the {\tt MAP\_TO} attribute is not used then the storage is passed
Xthrough the interface as is, though it is passed by value rather than
Xreference if the implementation defines a C interface in such a way.
X
XThe vendor must provide an {\tt EXTRINSIC (C)} to interface 
Xwith the native associated C compiler, but may also feel compelled 
Xto provide an extrinsic {\tt (GNU\_C)} or {\tt (CRI\_C)} as well, if 
Xthe C compiler provided by those
Xvendors is available.  
X
XFor arrays and user defined types nothing is done by default in 
Xthe interface, though for arrays the user may specify some transformations using
Xthe {\tt LAYOUT} specifier.  It
Xis the responsibility of the user to make sure that a structure will
Xlook exactly like the implementation's C structure (unless a specifier
Xfor the structure has been provided by the vendor.)  This 
Xmay mean OR'ing bits together into integers, adding padding, switching 
Xthe order of data in memory, etc.  It is also the user's responsibility to 
Xensure that
Xthe row/column major properties of all arrays are transposed properly, etc.,
Xfor the call when {\tt LAYOUT} is not used.
X
XType checking between the declared arguments in the extrinsic interface 
Xblock and the calling routine follow Fortran 90 rules:  types must match
Xexactly.  The conversions that occur are between the declarations in the
Xinterface block and the types specified by the {\tt MAP\_TO} attribute.
X
X\subsection {Types}
X
X\begin{itemize}
X\item      {\bf Unsigned:} \linebreak
X           Fortran does not have unsigned types.  Attempting to call a C
X           routine which requires all the bits of a given integer size to
X           be set properly will require a Fortran programmer to deal with
X           negative numbers.  This cannot be avoided because of
X           representational limits in the Fortran language and is not
X           really an interface issue.  Unsigned types can be dealt with
X           as integers. 
X
X\item      {\bf Bitfields:} \linebreak
X           Same as unsigned, but Fortran 90 does have routines that can 
X           manipulate bits in integer types.
X
X\item      {\bf char*:} \linebreak
X           There is a 99\% solution which is viable.  The HPF program
X           can treat a null as the end of a string and the interface can
X           therefore easily deal with the string conversion problem for
X           scalars, adding
X           a null when calling the C routine (the programmer would be
X           required to ensure that there is space to add it) and removing
X           the terminating null on return.  Character strings such as
X           ``a$\backslash$0b$\backslash$0c'' for example may not be 
X           passed through this interface.  Anything other than scalar
X           data (e.g. arrays and structures) is the user's problem.
X
X\item      {\bf Enums:} \linebreak
X           The solution to this is implementation dependent because
X           of the varied ways in which a C implementation can deal with them.
X           The vendor could probably define a set of 
X           type specifiers that would cover all of the implementation's ranges
X           if it did not map enums to int.
X
X\item      {\bf Unions:} \linebreak
X           These are not really a problem.  If the HPF side wants to
X           access fields of some part of the union, it can just declare
X           something the size of the largest part of the union putting
X           in pads where necessary.   Or the {\tt SELECT CASE} construct could
X           be used and the Fortran application might be able to simulate
X           unions.  This falls under the category of user required mappings.
X
X\item      {\bf Pointers:} \linebreak
X           The {\tt LOC} intrinsic
X           is used to solve the pointer problem.  This 
X           allows the caller to pass the
X           address of an argument rather than the argument itself.
X           {\tt LOC(LOC())} is used to indicate a pointer to a pointer.
X           In this case the compiling system automatically allocates 
X           a temporary to hold the new storage required.  The implementations
X           only need to deal with C style pointers.  The data that the 
X           pointer points to must be contiguous.  In Fortran terminology,
X           the target of the pointer must be scalar or if it points to 
X           an array the user must ensure that the array is contiguous.
X\end{itemize}
X        
X\section{Example}
X
XNote that this example is for a 64-bit machine.
X
XFor this example the application programmer will be passing a user 
Xdefined type as well
Xas a number of basic types. The type below is defined by the 
Xapplication programmer.  The 
Xapplication programmer has determined that
Xthe data for the call to the C program should be laid out as this
XFortran implementation lays out {\tt MY\_TYPE} in memory.  The proper
Xlayout of this user defined structure is the responsibility of the 
Xapplication programmer.
X
X\begin{verbatim}
XMODULE USER_DEFINED_THIS
X  TYPE MY_TYPE
X    INTEGER                int1
X    REAL(KIND=KIND(1.0))   real1
X  END TYPE MY_TYPE
XEND MODULE USER_DEFINED_THIS
X
X\end{verbatim}
X
XThe interface below is the actual interface for the call to a function
Xnamed {\tt cfunc}.  
X
X\begin{verbatim}
XINTERFACE
X   EXTRINSIC (C) FUNCTION CFUNC(w, x, y, z, a, p) NAME('cfunc')
X      use HPF_C_INTERFACE
X         REAL, MAP_TO(float)                cfunc
X         TYPE (MY_TYPE)                     w
X         INTEGER, MAP_TO(short)             x
X         REAL, MAP_TO(float)                y
X         INTEGER, MAP_TO(char)              z
X         INTEGER(KIND=4), MAP_TO(short)     a(100)
X         INTEGER                            p
X   END FUNCTION
XEND INTERFACE
X\end{verbatim}
X
X
XFinally, the call is shown below with some of the Fortran data declarations
Xfor the types that are passed.   Recall that the interface block above must
Xbe visible to this call.
X
X\begin{verbatim}
XThe call:
X
X   TYPE (MY_TYPE)         w
X   INTEGER                z, x
X   REAL                   y
X   INTEGER (KIND=4)       a(100)
X   INTEGER                p
X    .
X    .
X    .
X
X   r = cfunc(w, x, y, z, a, LOC(p))
X\end{verbatim}
X
XBelow is the prototype for the C routine.  This is not visible to the
XFortran code and is shown here for completeness.
X
X\begin{verbatim}
X
X  float cfunc(struct my_type w, short x, float y, char z,
X              short a[100], int *p);
X
X
X\end{verbatim}
X
XThis call to {\tt cfunc} causes the implementation to pass the
Xarguments {\tt w}, {\tt x}, {\tt y}, {\tt z} by value.  Further, 
X{\tt x} is converted to a {\tt short} and {\tt z} is converted to 
Xa C {\tt char}.  {\tt a} is passed as a pointer (the way arrays in
Xthis implementation's C are passed,) but is left as is in memory
Xsince arrays are not automatically converted.  {\tt p} is passed
Xas a pointer since at the call site {\tt LOC} intrinsic is used.
X
X
X\end{document}
END_OF_FILE
if test 20332 -ne `wc -c <'interoperability.tex'`; then
    echo shar: \"'interoperability.tex'\" unpacked with wrong size!
fi
# end of 'interoperability.tex'
fi
if test -f 'sep-95-cci' -a "${1}" != "-c" ; then 
  echo shar: Will not clobber existing file \"'sep-95-cci'\"
else
echo shar: Extracting \"'sep-95-cci'\" \(47370 characters\)
sed "s/^X//" >'sep-95-cci' <<'END_OF_FILE'
X
X
XHpff-Core
XFollowing is a text version of the active CCI items. There will be a 
Xformatted version of the full text at the meeting this week. 
X
XThese are ONLY the active items, and they are sorted by group. So don't be 
Xsurprised that they don't start with #1 - and don't have all the numbers.
X
XTwo disclaimers:
X
X#1 - I didn't get home with any record of action for a number of the CCI's 
Xdiscussed at the July meeting - These were the "easy" ones discussed in 
Xsubgroups, but not in the full meeting. So unfortunately - They're Back! I 
Xhope someone remembers. 
X
X#2 - In the dump and edit process from my database, MSWORD managed to do a 
Xweird and wonderful mangling job on the file. I think I caught all the 
Xproblems, but if some text seems strange - just wait for the "real" copy 
Xthat will be available at the meeting. 
X
XAnd an offer:
XIf any of you would like the postscript version of the text (will be at 
Xthe meeting), or a copy of the filemaker-pro database (mac), send me a 
Xnote and I'll ship it to you.
X-----------------------------------
XActive CCI - for Sept. meeting
X-----------------------------------
X
X******************************************************** GROUP C ACTIVE ITEMS ******************************************************** 
X
XItem #18
XHenry Zongaro
X05/04/95
XNo action from the July meeting was recorded ... was this resolved or not?
XAction: Chuck, Guy, Henry, Jerry
XTitle: Defined assignment in FORALL
XGroup C
XUpdated: 7/12/95
Xstatus: in progress
X-----------------
XHello,
X
XI was wondering whether there's not a problem with allowing defined
Xassignment to appear within a FORALL. Consider the following example.
X
Xmodule mod
Xinteger :: a(3) = (/1,2,3/)
Xcontains
Xpure subroutine def_assign(lhs, rhs)
Xinteger, intent(inout) :: lhs
Xcharacter, intent(in) :: rhs
X
Xlhs = a(ichar(rhs)+1)
Xend subroutine def_assign
Xend module mod
X
Xprogram p
Xuse mod
Xinterface assignment(=)
Xmodule procedure def_assign
Xend interface
X
Xforall (i = 1:2) a(i) = char(i) ! A sneaky way of passing "i" 
X! to def_assign
Xend program p
X
XThe rules of forall specify that the right-hand side and the indices of 
Xthe left-hand side are evaluated, in any order, prior to assignment, which 
Xalso takes place in any order. In the above example, we have 
X
Xa(1) = char(1)
Xa(2) = char(2)
X
Xas the two defined assignments which take place. Inside of def_assign, 
Xthere's a host-associated reference to a, so what ends up happening is the 
Xfollowing:
X
Xa(1) = a(2)
Xa(2) = a(3)
X
XThe order in which these assignments occurs affects the result. The value 
Xof a after the forall statement is executed could be (/2,3,3/) or 
X(/3,3,3/).
X
XBasically, the problem is that in defined assignment, completely 
Xevaluating the right-hand side for all active combinations does not 
Xnecessarily let the compiler precompute everything which might also appear 
Xon the left-hand side.
X
XThanks,
X
XHenry
X------------------
XDISCUSSION AT MEETING:
X
XCCI #18 ...Semantics of a forall are to evaluate all rhs and then store in 
Xall lhs, but if the assignment operator is user defined this is under user 
Xcontrol, not compiler control. The user's definition might make the order 
Xof assignment important. Guy queried how an array assignment was handled 
Xin this case. Jf a forall are to evaluate all rhs and then store in all 
Xlhs, but if the assignment operator is user defined this is under user 
Xcontrol, not compiler control. The user's definition might make the order 
Xof assignment important. Guy queried how an array assignment was handled 
Xin this case. Jerry will take this question to X3J3 about the status with 
Xrespect to elemental functions. Guy pointed out that for WHERE F90 forbids 
Xthe defined assignment. Chuck presented a proposal that sounded promising: 
Xthat the evaluation is as if the rhs were assigned into a temp using the 
Xdefined assignment operator and then the lhs is a direct copy of the 
Xalready evaluated values. There was discussion of issues like the type of 
Xthe temp (same as lhs or rhs?). Action Chuck, Guy, Henry, Jerry to 
Xcirculate proposed wording for this definition.
X
X===================================================== 
X===================================================== Item #25
XMatt Rosing
X6/27/95
XStatus: in progress
XNo action from July meeting - was this resolved? Group C
XTitle: Use of pure and independent
X
XQuestion:
X
XI have some questions about HPF semantics and implementation. These are 
Xbased on the HPF definition I have (dated 5/93). 
X
XI'm trying to determine if HPF can be used to implement a code we have and 
XI have questions about spmd execution within HPF. The code has two phases, 
Xthe first fits the data parallel model very well and the second does not. 
XWithin the second phase, independent operations are done on sections of a 
Xdistributed array. These operations modify the array.
X
XThere are apparently two methods to describe non-data parallel code in 
XHPF. This includes functions and subroutines declared pure, and loops 
Xdeclared independent. There are about five pages of constraints on the use 
Xof pure functions and it's not clear that I can use them. The main reason 
XI believe they won't work is that pure functions are not allowed to have 
Xside effects and our code has each independent operation modifying a part 
Xof a distributed array. Is it true that pure functions can't be used for 
Xthis?
X
XIndependent do loops, however, appear to have fewer constraints. The only 
Xconstraint seems to be that multiple loop iterations can not write to the 
Xsame location (I'm ignoring IO). I have a few questions to see how far I 
Xcan push this:
X
X1) Can an iteration modify data belonging to another processor? 
X
X2) If so, how do implementations handle the one-sided nature of the 
Xcommunication generated by this? The reason I ask is that, if done naively 
Xby buffering remote writes until the loops are done, this could require as 
Xmuch space for buffers as there is space for the data structure being 
Xoperated on.
X
X3) Is there any limitation on the types of routines that can be called 
Xfrom within an independent loop?
X
X4) How is the scheduling done? If there are just a bunch of subroutine 
Xcalls within the body of the loop, how does the compiler figure out which 
Xprocessor does what?
X
X5) Is it possible to dynamically schedule the loops on the processors for 
Xload balancing? Although none of the iterations would interfere, the 
Xscheduling mechanism would.
X
XSome of these questions are probably outside the scope of the language 
Xdefinition but I would still appreciate answers from any implementors. The 
Xreason being that the resulting performance might depend heavily on the 
Ximplementation.
X
X
XThanks,
X
XMatt
X(m_rosing@pnl.gov)
X
X------------
XChuck Replies 6/29/95
XThere is a new draft of the HPF spec, available through 
Xhttp://www.erc.msstate.edu/hpff/home.html. The changes are minor 
X(basically, corrections and clarifications) from the version you're 
Xlooking at.
XThe entire purpose of PURE is to ensure that those functions do not have
Xany side effects. So, at face value, this is correct. 
XYou could rewrite the functions to return arrays (this assumes each 
Xfunction modifies one array). You could then call the PURE functions, 
Xassigning the returned values into the separate array sections. For 
Xexample, the final code calling the functions might look like 
X
XFORALL ( I=1:NUM_BLOCKS )
XX(ILOW(I):IHI(I)) = FOO( A, B, C )
XEND FORALL
XThis would probably mean a lot of modification for the code, and it might 
Xbreak the current generation of compilers. But it would be one way to use 
XPURE in this context.
X>1) Can an iteration modify data belonging to another processor? 
XYes. Distribution of data has no effect on the semantics of the program. 
XNote that HPF has no notion of the processor that an iteration executes 
Xon. (Of course, the underlying compiler should have some such idea...) 
X
X>2) If so, how do implementations handle the one-sided nature of 
X....
XAs you suggest below, this is something outside the scope of the language 
Xdefinition. Very good compilers will do something efficient (like 
Xstrip-mining the loops) to ensure reasonable-sized buffers. Bad compilers 
Xwill run out of memory and generate core dumps. 
X
X>3) Is there any limitation on the types of routines that can be called 
Xfrom within an independent loop?
XNot in the language. The only limitation is in how called routines behave, 
Xi.e. that they don't violate the independence conditions. 
X
X>4) How is the scheduling done? If there are just a bunch of subroutine 
X....
XAs with 2 above, this is beyond the scope of the language definition. You 
Xare correct that this will be difficult to do well. 
XIncidentally, discussions of specifying scheduling mechanisms for parallel 
Xloops are going on in the HPFF 2 meetings. If you have opinions on the 
Xsubject (and I know you do, Matt :-), please feel free to contribute them. 
XThe final language will be better for it. 
X
X>5) Is it possible to dynamically schedule the loops on the processors 
X....
XA dynamic scheduling mechanism would be a valid HPF implementation. (Of 
Xcourse, it is not the only one.) Independence is not a problem here 
Xbecause any interference is between system variables; INDEPENDENT only 
Xmakes an assertion about the user code. 
XAgain, discussion in the HPFF 2 meetings may be relevant to this question 
Xin the future.
X=====Rik Littlefield replies 6/29/95========== Matt,
XYou write:
X>2) If so, how do implementations handle the one-sided nature of 
X....
XIf you're talking about the electronic structure codes, say creating the 
XFock matrix, then the problem is even worse. There are only O(N^2) data 
Xelements but something like O(N^4) updates to those elements. (At best, I 
Xthink it's O(N^3) with clever tricks.) ========
XNO ACTION FROM JULY MEETING RECORDED _ WAS THIS RESOLVED?
X===================================================== 
X===================================================== Item # 29
XRob Schreiber
X8/3/95
Xstatus: new
XTitle: Calling hpf_local from independent loop Group C & E
X
XHello,
XQuestion I. Can an extrinsic(hpf_local) be invoked in an independent loop?
X
XIn a Forall?
X
XEx:
X
XForall (1 = 1:10) a(i) = f(i,a(i))
X
XNote that part of the calling sequence, as specified in Ver 1.1, appendix 
XA, is
X"The processors are synchronized. In other words, all actions that 
Xlogically precede the call are completed." 
X
XIt seems clear that when this was written it was tacitly assumed that the 
Xcall did not occur in an independent loop or forall. 
X
XPart 2: May ony other kind of extrinsic be called in a forall or 
Xindependent loop?
X
X===================================================== 
X===================================================== 
X
X
X********************************************************* GROUP D ACTIVE 
XITEMS
X******************************************************** 
X
XItem # 8
XYasuharu Hayashi
X04/25/95
Xstatus: in progress
XNO ACTION FROM JULY MEETING RECORDED. WAS THIS RESOLVED OR NOT?
Xaction: study and discuss again in July
XTitle: Dummy assertion asterisk
XGroup D
XUpdated: 7/13/95
X
XQuestion:
XI have a question about the interpretation of the assertion asterisk when 
Xthe template of a dummy argument is a natural template. According to High 
XPerformance Fortran Language Specification November 10 ,1994 Version 1.1 
Xp.51,l.31, 
X"If the dummy argument has a natural template (no INHERIT attribute) 
Xthen things are more complicated. In certain situations the programmer is 
Xjustified in inferring a preexisting distribution for the natural template 
X......"
XWhen a actual argument is a whole array, the text on p.51,l.35 states only
X"In all these situations, the actual argument must be a whole array or 
Xarray section, and the template of the actual must be coextensive with the 
Xarray along any axis having a distribution format other than "*". If the 
Xactual argument is a whole array, then the pre-existing distribution of 
Xthe natural template of the dummy is identical to that of the actual 
Xargument".
XI think this description is ambiguous. For example : 
XPROGRAM EX
XREAL A(10,10),B(5,5)
X!HPF$ PROCESSORS P(5)
X!HPF$ DISTRIBUTE A(BLOCK,*) ONTO P
X!HPF$ ALIGN B(*,I) WITH A(2*I,*)
XCALL SUB(B)
X:
XEND
XSUBROUTINE SUB(BB)
XREAL BB(5,5)
X!HPF$ PROCESSORS P(5)
X!HPF$ DISTRIBUTE *(*,BLOCK(1)) ONTO *P :: BB 
X:
XEND
XIs the assertion asterisks for BB in SUB HPF-conforming ? 
XIsn't it necessary to add the list as follows which shows what assertion 
Xasterisks for a natural template are legal when a actual argument is a 
Xwhole array ? "1.If n th axis of a actual argument which is a whole array 
Xcorresponds to T(n) th axis of the template of it and j > i, T(j) must be 
Xlarger than T(i).
X2.If the situation is not described below ,no assertion about the 
Xdistribution of the natural template of a dummy is HPF-conforming. (a) If 
Xthe alignment of the actual array axis with its template is 
Xcollapsed, then * should appear in the distribution for the corresponding 
Xaxis of the natural template of the dummy. (b) If the actual array is 
Xaligned with the axis of its template by 
Xreplication (or "replication-triplet") and that template axis is 
Xdistributed * ,
Xthen no entry should appear in the distribution for the natural template 
Xof the dummy.
X(c) If the actual array is aligned with the axis of its template by 
Xint-expr and that template axis is distributed * , then no entry should 
Xappear in the distribution for the natural template of the dummy.
X(d) If the alignment of the actual array axis with the axis of 
Xits template is subscript triplet l:u:s and that axis of its template 
Xdistributed *, then * should appear in the distribution for the 
Xcorresponding axis of the natural template of the dummy. (e) If the 
Xalignment of the actual array axis with the axis of 
Xits template is subscript triplet l:u:s and that axis of its template 
Xdistributed BLOCK(n) and LB is the lower bound for that axis of the 
Xtemplate, then BLOCK(n/s) should appear in the distribution for the 
Xnatural template of the dummy, provided that s divides n evenly and that l 
X- LB < s. Question continued:
X(f) If the alignment of the actual array axis with the axis of 
Xits template is subscript triplet l:u:s and that axis of its template 
Xdistributed CYCLIC(n) and LB is the lower bound for that axis of the 
Xtemplate, then CYCLIC(n/s) should appear in the distribution for the 
Xnatural template of the dummy , provided that s divides n evenly and that 
Xl - LB < s." (g) If the alignment of the actual array axis with the axis 
Xof 
Xits template is subscript triplet l:u:s ,s must be positive. 
X
XOr it might be better to forbid the use of any assertion asterisks in 
XDISTRIBUTE directive in case that a dummy argument doesn't have INHERIT 
Xattribute and the corresponding actual argument isn't ultimately aligned 
Xwith itself since it seems that this solution makes things far simplified 
Xand cause little actual inconvenience (the same effect can also be 
Xachieved by ALIGN directive). 
X
XDiscussion: Rob replies ...
XThis example is nonconforming because axis 2 of B is NOT coextensive 
X(one-to-one and onto mapping to) axis 1 of the template to which B is 
Xaligned.
X>... example from original message
XNow let the example be this:
X
X>PROGRAM EX
X>REAL A(10,10),B(5,10)
X>!HPF$ PROCESSORS P(5)
X>!HPF$ DISTRIBUTE A(BLOCK,*) ONTO P
X>!HPF$ ALIGN B(*,I) WITH A(I,*)
X>CALL SUB(B)
X>:
X>END
X
X>SUBROUTINE SUB(BB)
X>REAL BB(5,10)
X>!HPF$ PROCESSORS P(5)
X>!HPF$ DISTRIBUTE *(*,BLOCK(2)) ONTO *P :: BB 
X>:
X>END
XThis example is correct. The replication over the second axis of the 
Xtemplate of the actual is not a problem because that is an axis whose 
Xdistribution format is *. B is not coextensive with that axis because it 
Xhas a one-to-many association with it, but since the template axis has a * 
Xdistribution, coextension is not a requirement. Is this reasonable?
XRob Schreiber
X=============meeting discussion
XCCI #8 ...Semantics of a forall are to evaluate all rhs and then store in 
Xall lhs, but if the assignment operator is user defined this is under user 
Xcontrol, not compiler control. The user's definition might make the order 
Xof assignment important. Guy queried how an array assignment was handled 
Xin this case. Jerry will take this question to X3J3 about the status with 
Xrespect to elemental functions. Guy pointed out that for WHERE F90 forbids 
Xthe defined assignment. Chuck presented a proposal that sounded promising: 
Xthat the evaluation is as if the rhs were assigned into a temp using the 
Xdefined assignment operator and then the lhs is a direct copy of the 
Xalready evaluated values. There was discussion of issues like the type of 
Xthe temp (same as lhs or rhs?). Action Chuck, Guy, Henry, Jerry to 
Xcirculate proposed wording for this definition.
X======================
X
XNO ACTION FROM JULY MEETING RECORDED. WAS THIS RESOLVED OR NOT?
X===================================================== 
X===================================================== Item # 10
XKenth Engo
X01/11/95
Xstatus: in progress
XNO ACTION FROM JULY MEETING RECORDED. WAS THIS RESOLVED OR NOT?
Xwho has action item on this?
XTitle: Permutations in HPF
XGroup D
XUpdated: 07/12/95
X
XQuestion:
XI have a general question about the way HPF deals with permutations of 
Xdata on the different parallel architectures. 
X
XIn many applications in MIMD and SIMD computations today, one often 
Xencounter the need to just perform a permutation of the data distributed 
Xon the parallel computer, i.e. a one-to-one mapping of the data set onto 
Xitself. It is then very important that the routing of the data is done in 
Xsuch a fashion that traffic contetions are eliminated in the 
Xinterconnecting network. One does not want a situation where 10 processors 
Xare communicating data to the very same processor. This would make all but 
Xone processor idle, since no more than one processor is allowed to 
Xcommunicate with the destination processor at the same time.
X
XWhat I have in mind is examplified by the following HPF code: 
XC 8 PROCESSORS AND AN ARRAY OF 32 ELEMENTS C
X!HPF$ PROCESSORS SEDECIM(8)
XREAL CENTURY(32)
XC
XC THE ARRAY IS DISTRIBUTED BY BLOCK.
XC
X!HPF$ DISTRIBUTE CENTURY(BLOCK) ONTO SEDECIM C
XC THE ELEMENTS ARE DISTRIBUTED WITH ELEMENTS 1,2,3,4 ON PROC
XC #1, ELEMENTS 5,6,7,8 ON PROC. 2 AND SO ON. 
X
X
XSuppose one want to redistribute the elements during execution to the 
Xfollowing arrangement
X
XC DATA REDISTRIBUTED CYCLIC ON THE PROCESSORS C
X!HPF$ REDISTRIBUTE CENTURY(CYCLIC) ONTO SEDECIM C
XC THE ELEMENTS ARE NOW REDISTRIBUTED WITH ELEMENTS 1,9,17,25 ON
XC PROC #1, ELEMENTS 2,10,18,26 ON PROC. 2 AND SO ON. 
X
XDuring this redistribution a permutation of the data set is performed. How 
Xis this permutation implemented, and what is actually done. Is there a 
Xstrategy/theory for generally doing this permutation optimally, that is; 
Xwithout any traffic contetions in the network? 
X
XI will be grateful if someone could answer this email, and possibly send 
Xor give me references to literature or people where I can find out more 
Xabout how HPF implements the permutations. 
X
X
XBest regards,
X
XKenth Engo
X
X
XNotes from May meeting
XCCI # 10 - not cci but a request for implementation practice. 
X
XNothing recorded for July.
X
X===================================================== 
X===================================================== Item # 11
XHenry Zongaro
X02/16/95
Xstatus: in progress
Xaction: needs more research
XTitle: pointer with sequence
XGroup D
XUpdated: 8/14/95
X
XQuestion:
XHello,
X
XThings have been quiet here lately, so I thought I'd send a few questions
Xthat I've been hoarding. All page and line references are relative to the 
X1.0 HPF Language Spec.
X
XI'd like to hear other people's opinions on these, especially the 2nd and 
X3rd items.
X
X1) The response to CCI item 6.3 indicated that variables with the POINTER 
Xattribute must be distributed or aligned. A related question - can they be 
Xgiven the SEQUENCE attribute? Or can a pointer be associated with both 
Xsequential and nonsequential targets? 
X
X------------
XMeeting minutes: needs more research
X========
Xfrom July meeting - subgroup proposal:
Xconforming dimensions of the pointer object and target either must be 
X(unmapped or both be identcally distributed) or both must be sequential. 
X
X
XThe issue is ... does the pointer exist as an instance by itself, or is it 
Xjust assocated with its bound arg. There is a case of pointers to sections 
Xof arrays, so just knowing that a pointer goes a block distributed thing, 
Xone can't talk about pointers to section. Andy Meltzer recalls that was 
Xrelated to the fact that allocatable objects don't have their distribution 
Xuntil after they are assigned.
XA straw poll was taken with the special understanding that a substantial 
Xvote for "abstain" would mean reconsider. The vote was 7-3-10, so this CCI 
Xitem is returned to committee for further clarification of the issue. 
X
X
X================================================= 
X================================================= Item# 17
XA.C. Marshall
X5/03/95
Xstatus: in progress
XNO ACTION FROM JULY MEETING RECORDED. WAS THIS RESOLVED OR NOT?
Xaction: needs more research
XTitle: Defaults for distribution
XGroup D
XUpdated: 07/11/95
X
XQuestion:
XForgive me for being dim (and only having v1.0 of the draft standard) 
Xbut...
X
XLooking at the syntax rules for DISTRIBUTE (p24 & 26) it would appear to 
Xme that:
X
X!HPF$ DISTRIBUTE A(BLOCK) !H303/5/8
X!HPF$ DISTRIBUTE ONTO P :: A !H301/2/6/10 
X
Xare valid but that
X
X!HPF$ DISTRIBUTE A
X!HPF$ DISTRIBUTE :: A
X
Xare not. Is this just me or is this how things are supposed to be, and if 
Xso why is it not possible to use default distribution and processor grid 
Xin the same statement, after all
X
X!HPF$ PROCESSORS P(NUMBER_OF_PROCESSORS()) !HPF$ DISTRIBUTE ONTO P :: A
X
Xis valid and has the same effect.
X
XAdam Marshall
X
X
Xnotes from May meeting: needs more research: ----
XScott Baden and Chuck Koelbel reply 07/07/95 
X
X
XThis is the way things are "supposed to be" Consult the relevant text 
X(page 30, line 20-21, v. 1.1) 
X
X
X... To prevent syntactic ambiguity, the dist-format-clause 
Xmust be present in the statement form [of a distribute spec] 
X
XChuck Koelbel adds that the "syntactic ambiguity" referred to here is due 
Xto the problem of non-significant blanks in Fortran: 
X
X
XConsider
X
X>!HPF$ DISTRIBUTE PRONTO ONTO LOGY
X>!HPF$ DISTRIBUTE PR ONTO ONTOLOGY
X
X
Xor the following example:
X
X>!HPF$ ALIGN TWITHEADS WITH A
X>!HPF$ ALIGN T WITH EADSWITHA
X
X>Disallowed for the same reason.
X
XChuck also continues:
X
X
X>It's not clear that his example has "the same effect". For example, 
Xconsider
X
X>!HPF$ PROCESSORS P(NUMBER_OF_PROCESSORS()) !HPF$ PROCESSORS 
XQ(4,NUMBER_OF_PROCESSORS()/4) !HPF$ DISTRIBUTE :: A
X>Does A have a 1-dimensional or 2-dimensional distribution? (Yeah, this 
Xassumes that NUMBER_OF_PROCESSORS() is divisible by 4...) 
X
X
X>The reason for requiring at least one of the clauses is, "What 
Xinformation
X>are you giving if you leave both out?" In effect, 
X
X>!HPF$ DISTRIBUTE :: A
X
X>would be a no-op, and we didn't think there was a need for that. 
X
X
XScott Baden
XChuck Koelbel
X
X
X======================
X
XNO ACTION FROM JULY MEETING RECORDED. WAS THIS RESOLVED OR NOT?
X===================================================== 
X===================================================== Item # 19
XHenry Zongaro
X05/04/95
XStatus: in progress
XNO ACTION FROM JULY MEETING RECORDED. WAS THIS RESOLVED OR NOT?
XAction: needs more research
XTitle: Mapping function results
XGroup D
XUpdated: 7/13/95
X
XI have a couple of questions related to specification of mappings for 
Xfunction results.
X1) Consider the following program fragment 
Xprogram prog
Xinterface
Xfunction f()
Xinteger f(100)
X!hpf$	processors p(number_of_processors())
X!hpf$	distribute f(block) onto p
Xend function f
Xend interface
X
Xcall sub(f())
Xend program prog
X
Xsubroutine sub(i)
Xinteger :: i(100)
X!hpf$ processors p(number_of_processors()) !hpf$ distribute i *(block) 
Xonto *p
Xend subroutine sub
XIs the above HPF conforming? Does distribution of a function result 
Xvariable affect the distribution of the expression returned? The text on 
Xpage 53, lines 27-28 indicates that the alignment of an expression is, in 
Xgeneral, unpredictable, except in the case of arrays and array sections, 
Xso I believe the answer to my question is "No". However, this is actually 
Xspurred by another question relating to the SEQUENCE directive. 
X
XAccording to page 151, line 47 of the 1.1 HPF Spec., an <association- 
Xname> can be a <function-name>. When I first read this, I thought the 
Xexplicit reference to <function-name> was there to include result 
Xvariables. Now a co-worker has suggested an alternate interpretation, and 
Xwe were wondering which is correct. Her suggestion was that this is trying 
Xto allow something like the following: 
Xprogram p
Xinteger, external :: f
X!hpf$ sequence :: f
X
Xi = f()
Xend program p
XIs this correct? Will this make the result of the function sequential? If 
Xso, that brings up another question:
X
Xprogram p
Xinterface
Xfunction f()
Xinteger :: f(10)
X!hpf$	sequence :: f
Xend function f
Xend interface
X
Xcall sub(f())
Xend program p
X
Xsubroutine sub(a)
Xinteger a(2, 5)
X!hpf$ sequence a
Xend subroutine sub
X
XAccording to page 155, lines 33-35, an array valued expression cannot be 
Xspecified to be sequential, but if specifying f to be a sequential 
Xfunction makes its result value sequential, this would be a contradiction. 
X----------------
X
XMay meeting minutes:
Xneeds more research
X======
XNothing recorded from July meeting.
X
X
X===================================================== 
X===================================================== Item # 21
XMichael Hennecker
X05/11/95
XStatus: in progress
XNO ACTION FROM JULY MEETING RECORDED. WAS THIS RESOLVED OR NOT?
XTitle: Question about derived type mappings and documentation. Group D
XQuestion:
XHello,
X
XI have some questions regarding data mapping of objects of derived type:
X
X(1) Is it possible to DISTRIBUTE / ALIGN objects of derived type, 
Xor are the data mapping attibutes restricted to intrinsic types? 
X
X(2) If mapping of objects of derived type is not possible, shouldn't the 
Xv1.0 and v1.1 specs for HPF_ALIGNMENT, HPF_DISTRIBUTION and HPF_TEMPLATE
X
X"ALIGNEE	may be of any type." (5.7.15, 5.7.16)
X"DISTRIBUTEE may be of any type." (5.7.17) 
X
Xread "may be of any intrinsic type." ?
X
XBest regards,
XMichael
X
X
XReply by CHK 5/15/95
X...
XIt is possible to map objects of derived type. (It is currently not 
Xpossible to map components of derived type objects; this is being 
Xdiscussed in the HPFF 95 meetings.) 
X
XSecond question is moot, given the first answer. 
X
XThanks for asking.
X
XChuck Koelbel
X-------
XNothing recorded from July meeting.
X
X
X===================================================== 
X===================================================== Item # 22
XHenry Zongaro
X6/15/95
XStatus: in progress
XNO ACTION FROM JULY MEETING RECORDED. WAS THIS RESOLVED OR NOT?
XTitle: Implementor note about processor distributions? Group D
X
XQuestion:
XHello,
X
XWe came across something that didn't seem immediately obvious here, and 
Xmight not be immediately obvious to others, so we were wondering whether a 
Xnote to users and/or implementers might be justified.
X
XOn page 30 of the 1.1 Language Spec., it's stated that if the ONTO clause 
Xof a DISTRIBUTE directive is omitted, an arbitrary processor arrangement 
Xis chosen for each distributee. In some cases, there may be no suitable 
Xarrangement; I assume such a program would not be HPF- conforming. For 
Xexample,
X
Xprogram p
Xinteger :: a(10, 10)
X!hpf$ distribute a(block(5), block(5))
Xend program p
X
XHere, the processor arrangement created would have to have an extent of at 
Xleast two in each dimension (which, by the way, constrains how arbitrary 
Xthe selection of a processor arrangement can be), so this program could 
Xnot run in an environment in which the number of processors was fewer than 
Xfour.
X
XDoes a note seem worthwhile here, or do others feel such a case is 
Ximmediately obvious?
X
XThanks,
X
XHenry
X
X
X--------
XNothing recorded from July meeting
X
X
X===================================================== 
X===================================================== Item # 26
XFabien COELHO
X07/01/95
XStatus: in progress
XNO ACTION FROM JULY MEETING RECORDED. WAS THIS RESOLVED OR NOT?
XTitle: Conditional realignment
XGroup D
X
XQuestion:
XHi out there,
X
XIs this kind of thind of thing allowed in HPF ? 
X
X! align A with T
Xif (some runtime condition)
X! realign A with T'
Xendif
X! redistribute T ...
X
Xafter the redistribution, array A mapping is not known. It depends on the 
Xruntime condition. I cannot remember anything that may forbid this. I 
Xguess it is not very nice for the compiler... 
Xshould/could be forbidden ? Or I may be wrong ? 
X
XFabien.
X
XChuck Koelbel replies 7/3/95:
XYes, this is allowed. This was the intent of HPF REALIGN and REDISTRIBUTE
X- to allow the user to make run-time decisions about data mapping. 
XAllowing run-time remapping will indeed require substantial support in the 
Xrun-time system. We discussed this tradeoff, and the concensus was that 
Xusers had valid reasons for wanting this capability, therefore it should 
Xbe in the language. The difficulties with implementation were one reason 
Xthat REALIGN and REDISTRIBUTE were not put in Subset HPF.
X
XIn short, you are right that this is legal and hard to implement. You are 
Xwrong that it is/should be forbidden.
X
XChuck
X
X--------
XNothing recorded from July meeting
X
X
X===================================================== 
X===================================================== Item # 28
XLarry Meadows
X7/25/95
XStatus: in progress
XTitle: combined directive ordering
XGroup D
XUpdated: 8/14/95
X
XQuestion:
XLOST ORIGINAL MESSAGE !!! Ack!
XHere's what was addressed at the July HPFF meeting. 
X
XThe syntax of combined directives allows: 
XALIGH WITH A(*,:) :: B(:)
XIs this a shape-spec-list for array-decl or array-spec for alignment? It 
Xis confusing to users.
X
X
Xcomment from Pres
Xcomment from Pres
XBut (!!!) I surely would _not_ want the discussion of the topic that Larry 
Xintroduces to slip further into more demands for "orderings" of directives 
Xand declarations as per the discussion of CCI #12 
X
X========
XNotes from July meeting:
XProposal: Add syntax rules to allow shape-list in combined directives only 
Xwith template name and processor names 
X
XALIGN (:) WITH A(*,:) :: B
X
XThe full group recommended that the subgroup come back with a more 
Xspecific proposal..
X
X======== proposed text =====
X
XOn page 24, make the following changes:
XLine 6, chage "entity-decl-list" to "hpf-entity-decl-list" 
X
XAfter line 14, add:
X"H303 hpf-entity-decl is hpf-entity [(explicit-shape-spec-list)] 
XH304 hpf-entity is object-name
Xor template-name
Xor processors-name"
X
XAfter line 20, add:
X"Constraint:: If an explicit-shape-spec-list appears, hpf-entity must be a 
Xtemplate-name or processors-name."
X
XOn lines 34-35, change two occurrences of "object-name" to "hpf- entity"
X
XOn line 35, before "If both" insert "If an explicit-shape-spec-list 
Xappears, hpf-entity has the dimension attribute." 
X
X===================================================== 
X===================================================== Item # 30
XRob Schreiber
X8/3/95
XStatus: new
XTitle: collapsing dimensions
XGroup D
X
XQuestion:
XQuestion II. This is really a question for implementors, as much as a 
Xquestion for language lawyers.
XConsider this program
X
Xreal a(100,200), b(100)
X!hpf$ distribute a(block, *)
X!hpf$ distribute b(*)
X
Xforall (row = 1:100) a(row,:) = f(a(row,*)) ...
Xpure function f(x)
Xreal x(:)
Xreal f(size(x))
X!hpf$ distribute *(*) :: x
X!hpf$ align f(:) with x(:)
X
XI cannot find any rule against this. (The issue is whether one may 
Xdistribute an r dimensional object with fewer than r instances of BLOCK or 
XCYCLIC(k) in its dist format list.) What is the mapping of b? And should 
Xone be allowed to describe the mapping of x in this manner, or must one 
Xuse the more cumbersome and specific:pure function f(x, row)
Xinteger row
Xreal x(:)
Xreal f(size(x))
X!hpf$ template t(100,200)
X!hpf$ align f(:) with x(:)
X!hpf$ align x(:) with t(row, :)
X!hpf$ distribute t(block, *)
X-----------
XHello, (from Rob)
XPres asked me to amplify my previous CCI request concerning the directive 
Xdistribute x(*)
X
XHere is some additional commentary:
X
XThe key issue is to let the compiler know what's going on when a 
Xsubroutine is passed an "on one processor only" section of a distributed 
Xarray; the call site is probably in a forall or independent loop. The 
Xobvious syntax is to say, prescriptively: 
X!hpf$ distribute dummy_arg(*)
Xor descriptively:
X!hpf$ distribute dummy_arg*(*)
X
XI was surprised that this is allowed by the HPF syntax: if this 
Xdistribution is specified by the program for an array, that is not a dummy 
Xarg, I don't know what to make of it. Would it mean to replicate the 
Xarray? To store it on one processor of the compiler's choice? To store it 
Xon the "front-end? In shared memory? 
X
XI think a reasonable proposal would be as follows: 
X-------------------------------------------------------------------------------------- 
X
XIn a (re)distribute directive, the number of non-* (i.e. block and 
Xcyclic[(k)]) entries in the dist-format-list must ordinarily be at least 
Xone, and must be the same
Xas the rank of the processors arrangement in the ONTO clause, if present.
XIf, however, the distributee is a dummy argument, then, if the distribute 
Xdirective is descriptive, the requirement of at least one non-* entry in 
Xthe dist-format-list is waived. Thus 
Xreal dummy(:,:)
X!hpf$ distribute dummy *(*,*)
XIs valid for a dummy argument; it asserts that the actual argument will be 
Xdistributed on a single processor.
X(continued)
X
X(continued clarification)
X-------------------------------------------------------------------------------------- 
X((continued clarification)
X-------------------------------------------------------------------------------------- 
X(Advise to language designers:)
XIt's quite likely that a section of a processors arrangement will be 
Xallowed in the ONTO clause of (re)distribute. In that case, one could also 
Xuse the following
X
Xsubroutine act_on_local_info(dummy, iproc) real dummy(:,:)
X!hpf$ processors all_procs(number_of_processors()) !hpf$ distribute dummy 
X*(*,*) onto all_procs(iproc) 
X
XThis would be appropriate in the following contexts: 
X
Xprogram main
Xreal actual_2d(8,16), actual_wide_2d(8,32), actual_3d(8, 16, 10) $hpf$ 
Xprocessors procs(8)
X!hpf$ distribute (*, block) onto procs :: actual_2d, actual_wide_2d !hpf$ 
Xdistribute (*, block, *) onto procs :: actual_2d, actual_wide_2d 
X
X!hpf$ independent
Xdo j = 1, 1
Xcall act_on_local_info( actual_2d(:, j:j), (j+1)/2 ) ! dummy shape is 
X(8,1) call act_on_local_info( actual_wide_2d(:, 2*j-1:2*j), (j+1)/2 ) ! 
Xdummy shape is (8,2)
Xcall act_on_local_info( actual_3d(:, j, :), (j+1)/2 ) ! dummy shape is 
X(8,10) enddo
X-------------------------------------------------------------------------------------- 
XThe alternative to this, as far as I can tell, is to make the programmer 
Xalign the dummy to a template, as follows: 
X
Xsubroutine act_on_located_info(dummy, iproc) real dummy(:,:)
X!hpf$ processors all_procs(number_of_processors()) !hpf$ template, 
Xdistribute onto all_procs :: all_temp(number_of_processors())
X!hpf$ align *(*,*) with all_temp(iproc) :: dummy 
X
Xprogram main
Xreal actual_2d(8,16), actual_wide_2d(8,32), actual_3d(8, 16, 10) $hpf$ 
Xprocessors procs(8)
X!hpf$ distribute (*, block) onto procs :: actual_2d, actual_wide_2d !hpf$ 
Xdistribute (*, block, *) onto procs :: actual_2d, actual_wide_2d 
X
X!hpf$ independent
Xdo j = 1, 16
Xcall act_on_located_info( actual_2d(:, j:j), (j+1)/2 ) ! dummy shape is 
X(8,1) call act_on_located_info( actual_wide_2d(:, 2*j-1:2*j), (j+1)/2 ) ! 
Xdummy shape is (8,2)
Xcall act_on_located_info( actual_3d(:, j, :), (j+1)/2 )	! dummy shape is
X(8,10)
Xenddo
X
X-- Rob
X===================================================== 
X===================================================== Item # 32
XHenry Zongaro
X8/31/95
XStatus: new
XTitle: Changing distribution of SAVE array Group D
X
XQuestion:
XHello,
X
XPage 41, lines 21-34 of the HPF 1.1 document specifies that an array or 
Xtemplate must not be distributed on a processor arrangement at the time 
Xthe arrangement becomes undefined, unless the array or template also 
Xbecomes undefined or the processor arrangement always has identical bounds.
X
XPresumably this was done so that objects with the SAVE attribute would not 
Xchange mappings from one call to the next. Is this rule sufficiently 
Xstrict? Consider the following: 
X
Xprogram p
Xcall sub(5)
Xcall sub(10)
Xend program p
X
Xsubroutine sub(n)
Xinteger, save :: a(10)
X!hpf$	processors proc(2)
X!hpf$	distribute a(block(n)) onto proc
Xend subroutine sub
X
XIn the first call to sub, a is distributed block(5); in the second call, 
Xit is distributed block(10). This is currently permitted because the 
Xbounds of the processor arrangement have not changed. 
X
XMore complicated examples could be drawn involving ALIGN. 
X
XSimilar text appears on page 44, line 43 - page 45, line 3 for templates. 
X
XComments from Rob 9/1/95
X
XWe should tighten the language to prevent this. It's a way to remap, which 
Xshould be done only if the object remapped has the dynamic attribute, and 
Xonly via executable directives. I believe this applies to objects in 
Xmodules, for example:
X
Xmodule mod
Xreal a(10)
Xend module mod
X
Xsubroutine sub(n)
Xuse mod
X!hpf$ processors procs(2)
X!hpf$ distribute a(block(n)) onto procs
Xend subroutine sub
X
XThis should be proscribed; if redistribute is used and if A is dynamic, 
Xhowever,
XI think it's legal. Right?
X
X===================================================== 
X===================================================== Item # 34
XAdriaan Joubert
X9/06/95
XStatus: new
XTitle: Alignment of a single dimension
XGroup D
X
XQuestion:
XHello,
XI am trying to align one dimension of two 2-dimensional arrays, and cannot 
Xfind a way of expressing this in HPF. The problem is the following
X
XPROGRAM MAIN
XREAL, ALLOCATABLE :: A(:,:), Res(:,:)
X!HPF$ DISTRIBUTE A(BLOCK,*)
X!HPF$ DISTRIBUTE Res(BLOCK,*)
X...
XALLOCATE(A(N,M))
XALLOCATE(Res(N,M*2))
XRes = SUB (A)
X...
XCONTAINS
XFUNCTION SUB (A) RESULT(B)
XREAL, INTENT(in) :: A(:,:)
X!HPF$ DISTRIBUTE *(BLOCK,*) :: A
XREAL :: B(SIZE(A,1),SIZE(A,2)*2)
X!HPF$ DISTRIBUTE (BLOCK,*) :: B
X...
XEND FUNCTION SUB
XEND PROGRAM MAIN
X
XSo the 1st dimension of B and A in the subroutine will be distributed in 
Xthe same way. It seems however that compilers can generate faster code, if 
Xthey know how arrays are aligned with one another. Among other things the 
Xcompiler would have to know that B is exactly aligned with Res. In other 
Xwords I would like to add to the main program something like
X
X!HPF$ TEMPLATE :: MYTemp(N,M*2)
X!HPF$ DISTRIBUTE MyTemp(BLOCK,*)
X!HPF$ ALIGN WITH MyTemp(:,I) :: A(:,I)
X!HPF$ ALIGN WITH MyTemp :: Res
Xand in the subroutine
X!HPF$ TEMPLATE :: MYTemp(SIZE(A,1),SIZE(A,2)*2) !HPF$ DISTRIBUTE 
XMyTemp(BLOCK,*)
X!HPF$ ALIGN WITH *MyTemp(:,I) :: A(:,I)
X!HPF$ ALIGN WITH MyTemp :: B
X
Xand then the compiler should be able to figure out that everything is 
Xnicely aligned. But I cannot do this, as I do not know N and M at compile 
Xtime.
X
XIn this case I could probably get away with definitions 
X!HPF$ ALIGN WITH A(:,*) :: Res(:,*)
Xin the main program and
X!HPF$ ALIGN WITH A(:,*) :: B(:,*)
Xin the subroutine. The replication of the second dimension would not 
Xmatter, as it is all on the same processor. 
X
XBut if I have a (BLOCK,BLOCK) distribution for both arrays, but I still 
Xwant to ensure that all elements in the first section of every column are 
Xon the same row of processors, i.e.
XREAL A(20,20), B(20,40)
X
XP1: A(1:10,1:10)	P2: A(1:10,11:20)
XB(1:10,1:20)	B(1:10,21:40)
X
XP3: A(11:20,1:10)	P4: A(11:20,11:20)
XB(11:20,1:20)	B(11:20,21:40)
Xthere seems to be no way of telling the compiler about this with a 
Xdescriptive statement.
X(continued under discussion)
X
X(question continued)
XWell, what about
X!HPF$ ALIGN WITH A(:,I) :: B(:,I*2-1)
X!HPF$ ALIGN WITH A(:,I) :: B(:,I*2)
XBut is this legal? And this can be harder to do if the second dimension is 
X(SIZE(A,2)-1)*SIZE(A,2)/2, as in my case. 
X
XI'd appreciate any help on this one. Understanding the distribution 
Xdirectives seems to get harder the more you know, instead of easier ;-( 
XAdriaan
X=========
XRob comments 9/6/95
XI see you want allocatable templates, a hole in HPF that we know about. 
XBut in your case, no template is needed. There is no reason to specify 
Xreplicationin your alignment. You can use 
X!HPF$ ALIGN A(I,J) WITH RES(I,J)
Xand drop the distribute directive for A in the main program. In the 
Xfunction, you can use a distribute on B and a descriptive alignment of A 
Xto B.
X.....
X..."But is this legal? And this can be harder to do if the second 
Xdimension is
X(SIZE(A,2)-1)*SIZE(A,2)/2, as in my case." ... No, it's not legal. Align 
Xis not a symmetric relation, and the alignment map cannot be many to one 
Xunless it collapses a dimension; at least that's my understanding. In case 
Xthis is not clear, your statement is equivalent to
X!HPF$ ALIGN B(:,2*I-1) WITH A(:,I)
Xwhich only explicitly aligns the odd numbered columns of B! But why align 
Xthe bigger of the two arrays (B) with the smaller (A)? My version of SUB 
Xwould be:
XFUNCTION SUB (A) RESULT(B)
XREAL, INTENT(in) :: A(:,:)
XREAL :: B(SIZE(A,1),SIZE(A,2)*2)
X!HPF$ ALIGN *(I,J) WITH B(I,J) :: A
X!HPF$ DISTRIBUTE B (BLOCK,*)
X...."I'd appreciate any help on this one. Understanding the distribution 
Xdirectives seems to get harder the more you know, instead of easier ;-( 
X"...
XYup. -- Rob
X=======
X>From Adam M. 9/5/95
XAs I understand it Adriaan Joubert wants to tell HPF to set the data up as
Xfollows:
XREAL A(20,20), B(20,40)
XP1: A(1:10,1:10)	P2: A(1:10,11:20)
XB(1:10,1:20)	B(1:10,21:40)
XP3: A(11:20,1:10)	P4: A(11:20,11:20)
XB(11:20,1:20)	B(11:20,21:40)
XI would say (like Rob Schreiber) that you include, in Main, the line 
X!HPF$ ALIGN A(I,J) WITH B(I,J*2-1)
Xor equivalently
X!HPF$ ALIGN A(:,J) WITH B(:,J*2-1)
XPLUS (distribute them together)
X!HPF$ DISTRIBUTE B (BLOCK,BLOCK)
XAnd I would claim that the procedure should be coded as follows: 
XFUNCTION SUB (A) RESULT(B)
XREAL, INTENT(in) :: A(:,:)
XREAL :: B(SIZE(A,1),SIZE(A,2)*2)
X!HPF$ ALIGN *(I,J) WITH B(I,J*2-1) :: A
X!HPF$ DISTRIBUTE *B (BLOCK,BLOCK)
XIf I am wrong - why? - Adam
X
X
X===================================================== 
X===================================================== 
X
X
X********************************************************* GROUP E ACTIVE 
XITEMS and see also Item #29 in Group C 
X******************************************************** 
X
XItem # 31
XRob Schreiber
X8/4/95
XStatus: new
XTitle: number of processors
XGroup E
X
XQuestion:
XWe had a recent discussion of the question "What should 
Xnumber_of_processors()" return when the machine consists of <describe your 
Xown bizarre and nonhomogeneous architecture here>.
X
XLet's say the machine is 2 SMPs (what does SMP stand for?) with 4 
Xprocessors and a common memory, each.
X
XIn my view, the "right" answer in this case is probably 8. Not 2. Not (/2, 
X4/).
XThat's because I want the following code 
X
Xreal x(1000, number_of_processors())
X!hpf$ independent
Xdo i = 1, number_of_processors()
Xcall embarrassingly_parallel( x(:, i) )
Xenddo
X
Xto run in parallel, with one iteration per processor, on all the available 
Xprocessors.
X
X===================================================== 
X===================================================== Item # 33
XHenry Zongaro
X8/31/95
XStatus: new
XTitle: Query of function result distribution Group E
XA second question I have relates to CCI item #20. The HPF_LOCAL_LIBRARY
Xprocedures all specify that the ARRAY argument must be a local dummy 
Xargument associated with a global HPF actual argument. It would seem to me 
Xto be desirable to be able to call these routines with a function result 
Xas well.
XFor example,
X
Xprogram p
Xinterface
Xextrinsic(hpf_local) function f()
Xinteger :: f(100)
X!hpf$	processors proc(number_of_processors())
X!hpf$	distribute f(blohanks,
X
XHenry
Xend function f
Xend interface
Xend program p
X
Xextrinsic(hpf_local) function f()
Xinteger :: f(100/number_of_processors()), g_index 
X
X! Would be nice to be able to call local_to_global to determine ! which 
Xelements of local result correspond to which elements ! of global result
Xcall local_to_global(f, (/1/), l_index)
Xend function f
X
XAny opinions?
X
XThanks,
X
XHenry
X
X
XComment from Rob 9/1/95
X
XSure.
X
X===================================================== 
X===================================================== PREVIOUSLY ANSWERED 
XITEM REFERRED TO IN CCI 33 Item # 20
XHenry Zongaro
X05/04/95
XStatus: waiting for text
XAction: needs more text in document
XTitle: Declaring array-valued functions
XGroup E
XUpdated: 7/13/95
X
XQuestion:
X
X2) This second question is one that Rich Shapiro was asking towards the 
Xend of the October, 94 HPFF meeting. Consider the following program. 
X
Xprogram prog
Xinterface
Xextrinsic(hpf_local) function f()
Xinteger f(10)
X!hpf$	processors p(number_of_processors())
X!hpf$	distribute f(block) onto p
Xend function f
Xend interface
X
Xprint *, f()
Xend program prog
X
Xextrinsic(hpf_local) function f()
Xinteger f(?)
Xend function f
X
XHow should f be declared in the local subroutine? If this program runs on 
Xa number of processors which divides 10, the answer is simple. But what if 
Xthe program is run on four processors? Three processors have three 
Xelements each, but the fourth has one element. There are ways in which you 
Xcould specify this. However, there's no way for the user to know *which* 
Xprocessor is the one which will have one element of the result variable. 
XDummy array arguments of HPF_LOCAL procedures don't have this problem, 
Xbecause they must be assumed shape. Perhaps function results have to be 
Xdistributed in such a way that the same number of elements of the result 
Xwill be mapped onto all processors.
X
XA related problem: how can an array valued function be specified in an
XHPF_LOCAL module?
X
Xextrinsic(hpf_local) module mod
Xcontains
Xfunction f()
Xinteger f(?)
X!hpf$	processors p(number_of_processors())
X!hpf$	distribute f(block) onto p
Xend function f
Xend module mod
X
XThe declaration of f here has to do double duty. It has to provide the 
Xshape of the result of f for any reference to the function, and it has 
Xdescribe the result variable mapped onto a single particular processor. 
XCan array valued functions be permitted to appear in HPF_LOCAL modules?
X
XThanks,
X
XHenry
X
X
XSubgroup report: - only thing group e could com up with was that: 7 a 
X"local" function can only return a scalar 7 the corresponding extrinsic 
Xfunction return value must be either a scalar or a rank one array of size 
Xnumber_of_processors() 7 Anything more general would not be cci ... and 
Xwhat to do is not obvious.
X====
X
XCCI #20 - this poses a non-trivial problem about how the size of a 
Xfunction result is known and / or declared. Fortran 90 doesn't have any 
Xway of saying that the result is "assumed shape/size". After extended 
Xdiscussion, the subgroup decided to propose that a local function can only 
Xreturn a scalar, and that the corresponding extrinsic function return 
Xvalue must be either a scalar or an array that is made up from the scalars 
Xreturned by the locals. In full group discussion, Guy pointed out that it 
Xis the DISTRIBUTE directive that is the problem - it lies. There was a lot 
Xof discussion, with a basic trend that there should be no change, but 
Xrather explanatory text in the document. To bring the issue to close there 
Xwas a vote on the proposal for no change, but with extra text - (choosing 
Xwhatever reason among several that the individual liked best for no 
Xchange)/ This passed 14 - 1 - 6.
END_OF_FILE
if test 47370 -ne `wc -c <'sep-95-cci'`; then
    echo shar: \"'sep-95-cci'\" unpacked with wrong size!
fi
# end of 'sep-95-cci'
fi
echo shar: End of shell archive.
exit 0
