Subject: hpff-external: Intercallability proposal

From: Andy Meltzer <[email-masked]>
Date: Fri, 15 Sep 1995 15:40:44 -0500 (CDT)

---------------------------------------------------------------------------
[email-masked] is a mailing list for discussion of external
interfaces between HPF and other languages/systems.  Instructions for
adding or deleting yourself from this list appear at the bottom of this
message.
---------------------------------------------------------------------------

Following is a latex version of the intercallability document
that I have promised.  Sorry for the delay.

						Andy Meltzer
						[email-masked]

######################
\documentstyle[twoside,11pt]{article}

\pagestyle{myheadings}
\sloppy
\footheight=-.60in
\headheight=0in
\textwidth=5.50in
\topmargin=-.20in
\oddsidemargin=0.5in
\evensidemargin=0.5in

\itemsep=.05in
\topsep=.05in
\parsep=.05in
\parskip=.10in
\textheight=9in

\newcounter{dofigures}  \setcounter{dofigures}{0}
\newcounter{dotables}        \setcounter{dotables}{0}
\newcommand{\havefigures}{\setcounter{dofigures}{1}}
\newcommand{\havetables} {\setcounter{dotables}{1}}

\newcommand{\cccdocid}[5]{
  \bibliographystyle{alpha}
% ******************** Title Page ********************
  \vspace*{.5in}
  \centerline{\Large \bf {#1}}
  \vspace*{.02in}
  \centerline{{#2}}
  \vspace*{.02in}
  \centerline{{#3}}
  \vspace*{.02in}
  \centerline{{#4}}
  \vspace*{.02in}
  \centerline{{#5}}
  \vspace*{.5in}

  % ******************** List of Figures ********************
  % Note: Remove the following line if there are no figures
%  \ifodd\value{dofigures} \listoffigures \newpage \fi
%  \ifodd\value{dotables}  \listoftables  \newpage \fi
%  \markboth{#1} {#3}
%  \pagenumbering{arabic}
  }

% ******************** Special macros ********************
%\catcode`^^Z=9                % Make TeX ignore IBMPC EOF character

\newcommand{\ital}[1]{{\it #1}}
\newcommand{\type}[1]{{\tt #1}}
\newcommand{\bold}[1]{{\bf #1}}
\newcommand{\dfn}[1]{{\it #1}\index{#1}}
\newcommand{\comment}[1]{}

\def\fineprint{\relax}
\def\stopfine{\relax}

\newcommand{\beginprog}{\begin{tabbing}
xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=xxxx\=\kill
}
\newcommand{\tb}{\>\tt}
\newcommand{\finprog}{\end{tabbing}}
\newcommand{\progtxt}[1]{``{\tt #1}''}
\newcounter{save_enum}
\newcommand{\hilite}[1]{ \item {\bf #1\\} }
\newcommand{\morelater}{\centerline{$\ldots$\ital{Fill this in later}$\ldots$}}

% ******************** Start of Document ********************
\begin{document}

\cccdocid
  {HPF calling C Interoperability Proposal}
  {Ver 1.1}
  {Andrew Meltzer}
  {Cray Research, Inc.}
  {September, 1995}

\section{Introduction}

Defining an interoperability mechanism is as much a problem 
of recognizing the limitations of interlanguage communication as it is of 
solving the technical interlanguage calling issues. 

This document contains a proposal for an HPF calling C interoperability
mechanism and a rational for the limitations and techniques used in the
proposal.   It also contains a set of representative calls from the
X Windows interface for context.

\subsection{Defining the Bounds of the Problem}

There are a number of ways in which the interoperability question has
been viewed:

\begin{enumerate}
\item\label{i1} HPF should be able to call any C routine and be able to
           represent or convert any C type. 

\item\label{i2} HPF should be able to deal with any C type that has a
           general approximation in Fortran.

\begin{enumerate}
\item\label{i2a}   The compiling system should handle all type
                   conversions.  This includes remapping structures
                   and transposing arrays.

\item\label{i2b}   The compiling system should automatically convert
                   all ``basic'' types to their closest equivalent in the
                   other language.
\end{enumerate}

\item\label{i3} HPF programs calling C do not need any special
           mechanisms.  It is the problem of the application programmer
           to make sure that values are passed properly.
\end{enumerate}

Option \ref{i3} is currently the standard method, but since most application
programmers are looking for more ease-of-use, this option is not sufficient.

Option \ref{i1} is an enormous effort and might impose a large runtime
penalty.  To achieve this goal, Fortran would need to be extended to have
some new types (bits, unsigned values, and perhaps unions) and the
compiling system would need to transpose arrays on calls and 
convert complicated structures to their C layouts.  

Option \ref{i2a} is unreasonable for some of the reasons that option 
\ref{i1} is unreasonable.  This sort of type conversion/mapping is 
often not
desired and may impose a huge and unavoidable runtime penalty on the
user of the interface.  Consequently I believe that option \ref{i2b} is as
much as a compiling system should offer.  Users would be required to
understand layout of arrays in memory for each language and the way
structures are constructed.

The solution proposed below is a variety of option \ref{i2b} which 
incorporates some of the features of \ref{i2a}, but only at the application 
programmer's discretion.

\section{The X Window Interface}

The ability to call the X Windows library has often been mentioned as a
good test of the sufficiency of an interlanguage interface for HPF and C.  
I have picked a few representative library routines to demonstrate
the problems that must be dealt with.

\begin{verbatim}
        char * XOpenDisplay(char *display_name)
\end{verbatim}

Clearly any interface will have to deal with a conversion from Fortran
character strings to the {\tt char *} of C.  Somewhat more complicated might
be a call like

\begin{verbatim}
        XGetIconName(Display *display, Window w, char **icon_name)
\end{verbatim}

For this call, as in most, the display value is simply a pointer
and need not be inspected by the HPF
compiler.  {\tt Window} is an ID (some integer type), and {\tt icon\_name} is the
result value and is consequently a pointer to a {\tt char *}.   A problem
which must be addressed is that of pointers to pointers in C (though in
this case one might reasonably consider {\tt char *} to be the 
character string type of C so that a pointer to it 
is the equivalent of a pointer to a
Fortran character string.)  In many other calls, pointers to structure
pointers are the return value, for example {\tt size\_list} in the following:

\begin{verbatim}
        XGetIconSizes(Display *display, Window w,
                      XIconSize **size_list, int *count)
\end{verbatim}

Finally a call with some simpler parts:

\begin{verbatim}
        XStoreNamedColor(Display *display, Colormap cmap, char *colorname,
                         unsigned long pixel, int flags)
\end{verbatim}

{\tt Display} has been previously discussed. {\tt Colormap} 
can be stored in an 
{\tt int} type of the proper size. The {\tt char *} type has 
been discussed.  {\tt Unsigned long} should have 
some equivalent representation for Fortran 90 on any 
given compilation system, and finally {\tt int} is some size C integer.

\section{Interlanguage Calling Proposal}

This section describes the syntax and semantics for the interlanguage
calling sequence.  The interlanguage calling sequence is 
implemented with the the extrinsic mechanism.
A new extrinsic type
\begin{verbatim}
        EXTRINSIC (C) 
\end{verbatim}
is added to the language.  The extrinsic declaration interface block must
be visible to the compilation system at the time the call to the C function
is compiled.

It is important to fully understand what the {\tt EXTRINSIC}
mechanism is.  The {\tt EXTRINSIC} mechanism ``allows an HPF programmer
to declare a calling interface to a non-HPF subprogram'' (HPF Version 1.1,
Annex A.)  In this case the {\tt EXTRINSIC} mechanism assists the programmer
when calling a C function or subroutine by redefining the semantics
of a function or subroutine call.  Within the {\tt EXTRINSIC} the
code need not conform to HPF semantics, though it is important from an
implementation standpoint that it be easily parsed by an HPF parser.  
This proposal extends the semantics of the {\tt EXTRINSIC} mechanism
only a small amount.

This proposal also extends the HPF language with a new intrinsic function
(although it is already in common usage):
\begin{verbatim}
        LOC
\end{verbatim}

{\tt LOC} is used at the call site to indicate that the address of the
actual argument is required.

Within the {\tt EXTRINSIC (C)} environment a new attribute is defined:

 {\tt MAP\_TO([C\_TYPE= ]} {\it c type specifier } {\tt [, [LAYOUT= ]}{\it layout specifier}{\tt ])}

This attribute may have one or two specifiers, {\tt C\_TYPE} a 
required specifier,
and {\tt LAYOUT} an optional specifier.  {{\tt C\_TYPE} specifies the 
C type that the HPF type will be mapped to, and {\tt LAYOUT} 
indicates the memory layout of the data being passed.  If no {\tt LAYOUT}
argument is given it defaults to {\tt NO\_CHANGE}.  The following are
the layout specifiers an implementation is required to provide:

\begin{verbatim}
        Name               Function
        ----               --------
        C_ARRAY            Convert the array to a C memory layout
        HPF_ARRAY          Convert the array to an HPF memory layout
        NO_CHANGE          Do no rearrangement of the array memory layout
\end{verbatim}

The following are the type specifiers that an implementation is required
to provide.

\begin{verbatim}
        Name                            C type
        ----                            ------
        NO_CHANGE                       no remapping is done
        INT                             int
        LONG                            long
        SHORT                           short
        CHAR                            char
        FLOAT                           float
        DOUBLE                          double
        LONG_DOUBLE                     long double
        CHAR_PTR                        indicates that the string will be 
                                        converted to a null terminated char *

\end{verbatim}

The vendor may also add other type specifiers.

If a C function using the {\tt EXTRINSIC (C)} interface
is called from within an HPF program then the interface
uses the same rules as {\tt HPF\_SERIAL} to remap data.  It may also be called
from within an {\tt HPF\_LOCAL} environment, in which case a different
instance of the interface is
invoked (a different instance of the C function is called) on each processor.

HPF is not case sensitive, C is case sensitive.  
This proposal accepts the suggested addition of a {\tt NAME} clause
to be added to the {\tt EXTRINSIC} statement.  If this is added I suggest
that it is an ideal way to distinguish the case of the C function.

 {\tt NAME(`}{\it function name}{\tt')}

For example:

\begin{verbatim}
       EXTRINSIC (C) FUNCTION CASE() NAME("CaSe")
\end{verbatim}

\subsection{Explanation of the proposal}

The declaration {\tt EXTRINSIC (C)} indicates to the compilation
system how to convert types (in C parlance use a cast to convert types) 
to the types that the called C program requires.  It does this with
the {\tt MAP\_TO} attribute.  The implementation must also pass
these arguments by value, or as is appropriate for the vendors implementation
of C.  

The vendor must provide the set of required type specifiers
and may add as many additional type specifiers as it wishes.
The vendor may also optionally provide the extrinsic interface for 
the C system or library routines as well.

To handle pointers and pointers to pointers this proposal takes a
mechanism from common practice, the {\tt LOC} intrinsic.  If, for
example, a C function required an argument to be passed as a 
pointer to a pointer:

\begin{verbatim}
       void cfunc(int **pp);
\end{verbatim}

then the corresponding call (with, of course, the proper extrinsic
declaration visible) would be:

\begin{verbatim}
       INTEGER PP
       CALL CFUNC(LOC(LOC(PP)))
\end{verbatim}

Since the called C function requires a pointer to a 
pointer to be passed, using the {\tt LOC} mechanism here does not
hinder portability.

In addition, HPF must recognize a new attribute within {\tt EXTRINSIC (C)},
the {\tt MAP\_TO} attribute.  This attribute is the core of the mechanism.
It allows the user to specify the conversions necessary to facilitate
the interlanguage calls.   The attribute has one or two specifiers.  The first
specifier indicates the type of the C data that the HPF data must be
converted to.  The second specifier indicates (for an array) whether
and how to remap the storage.  {\tt HPF\_ARRAY} indicates that the 
runtime system should re-structure an array in memory (this is only
useful for a return value from a called C routine) to conform to the 
declared layout of the array that will contain the result of the function.
{\tt C\_ARRAY} indicates that the
runtime system should re-structure an array in memory to conform to
the system's default C style layout (this is only useful for arguments.)
For example, a user would
write the following within the {\tt EXTRINSIC (C)} interface to ensure 
that the compiler remaps the array {\tt a}
to the system's default C array layout:

\begin{verbatim}
         INTEGER, MAP_TO(C_TYPE=INT, LAYOUT=C_ARRAY)  a(100, 10, 4)
\end{verbatim}

The {\tt LAYOUT} argument is
not strictly required to make the interface work, but its functionality
is often enough utilized that there should be a portable way of specifying
the transformations.  

It is important to emphasize that the extrinsic environment does more
than just type checking.  In the presence
of {\tt EXTRINSIC (C)} and the {\tt MAP\_TO} attribute, type conversion 
is done.   The implementation 
might warn a user about a real to integer type conversion, but in 
general the interface will use the types specified by the {\tt MAP\_TO}
attribute in the interface as target 
types for the call and do whatever it takes to make the types passed 
in fit.   This only applies to the system's basic types, however.
If the {\tt MAP\_TO} attribute is not used then the storage is passed
through the interface as is, though it is passed by value rather than
reference if the implementation defines a C interface in such a way.

The vendor must provide an {\tt EXTRINSIC (C)} to interface 
with the native associated C compiler, but may also feel compelled 
to provide an extrinsic {\tt (GNU\_C)} or {\tt (CRI\_C)} as well, if 
the C compiler provided by those
vendors is available.  

For arrays and user defined types nothing is done by default in 
the interface, though for arrays the user may specify some transformations using
the {\tt LAYOUT} specifier.  It
is the responsibility of the user to make sure that a structure will
look exactly like the implementation's C structure (unless a specifier
for the structure has been provided by the vendor.)  This 
may mean OR'ing bits together into integers, adding padding, switching 
the order of data in memory, etc.  It is also the user's responsibility to 
ensure that
the row/column major properties of all arrays are transposed properly, etc.,
for the call when {\tt LAYOUT} is not used.

Type checking between the declared arguments in the extrinsic interface 
block and the calling routine follow Fortran 90 rules:  types must match
exactly.  The conversions that occur are between the declarations in the
interface block and the types specified by the {\tt MAP\_TO} attribute.

\subsection {Types}

\begin{itemize}
\item      {\bf Unsigned:} \linebreak
           Fortran does not have unsigned types.  Attempting to call a C
           routine which requires all the bits of a given integer size to
           be set properly will require a Fortran programmer to deal with
           negative numbers.  This cannot be avoided because of
           representational limits in the Fortran language and is not
           really an interface issue.  Unsigned types can be dealt with
           as integers. 

\item      {\bf Bitfields:} \linebreak
           Same as unsigned, but Fortran 90 does have routines that can 
           manipulate bits in integer types.

\item      {\bf char*:} \linebreak
           There is a 99\% solution which is viable.  The HPF program
           can treat a null as the end of a string and the interface can
           therefore easily deal with the string conversion problem for
           scalars, adding
           a null when calling the C routine (the programmer would be
           required to ensure that there is space to add it) and removing
           the terminating null on return.  Character strings such as
	   ``a$\backslash$0b$\backslash$0c'' for example may not be 
           passed through this interface.  Anything other than scalar
           data (e.g. arrays and structures) is the user's problem.

\item      {\bf Enums:} \linebreak
           The solution to this is implementation dependent because
           of the varied ways in which a C implementation can deal with them.
           The vendor could probably define a set of 
           type specifiers that would cover all of the implementation's ranges
           if it did not map enums to int.

\item      {\bf Unions:} \linebreak
           These are not really a problem.  If the HPF side wants to
           access fields of some part of the union, it can just declare
           something the size of the largest part of the union putting
           in pads where necessary.   Or the {\tt SELECT CASE} construct could
           be used and the Fortran application might be able to simulate
           unions.  This falls under the category of user required mappings.

\item      {\bf Pointers:} \linebreak
           The {\tt LOC} intrinsic
           is used to solve the pointer problem.  This 
           allows the caller to pass the
           address of an argument rather than the argument itself.
           {\tt LOC(LOC())} is used to indicate a pointer to a pointer.
           In this case the compiling system automatically allocates 
           a temporary to hold the new storage required.  The implementations
           only need to deal with C style pointers.  The data that the 
	   pointer points to must be contiguous.  In Fortran terminology,
	   the target of the pointer must be scalar or if it points to 
	   an array the user must ensure that the array is contiguous.
\end{itemize}
        
\section{Example}

Note that this example is for a 64-bit machine.

For this example the application programmer will be passing a user 
defined type as well
as a number of basic types. The type below is defined by the 
application programmer.  The 
application programmer has determined that
the data for the call to the C program should be laid out as this
Fortran implementation lays out {\tt MY\_TYPE} in memory.  The proper
layout of this user defined structure is the responsibility of the 
application programmer.

\begin{verbatim}
MODULE USER_DEFINED_THIS
  TYPE MY_TYPE
    INTEGER                int1
    REAL(KIND=KIND(1.0))   real1
  END TYPE MY_TYPE
END MODULE USER_DEFINED_THIS

\end{verbatim}

The interface below is the actual interface for the call to a function
named {\tt cfunc}.  

\begin{verbatim}
INTERFACE
   EXTRINSIC (C) FUNCTION CFUNC(w, x, y, z, a, p) NAME('cfunc')
      use HPF_C_INTERFACE
         REAL, MAP_TO(float)                cfunc
         TYPE (MY_TYPE)                     w
         INTEGER, MAP_TO(short)             x
         REAL, MAP_TO(float)                y
         INTEGER, MAP_TO(char)              z
         INTEGER(KIND=4), MAP_TO(short)     a(100)
         INTEGER                            p
   END FUNCTION
END INTERFACE
\end{verbatim}

Finally, the call is shown below with some of the Fortran data declarations
for the types that are passed.   Recall that the interface block above must
be visible to this call.

\begin{verbatim}
The call:

   TYPE (MY_TYPE)         w
   INTEGER                z, x
   REAL                   y
   INTEGER (KIND=4)       a(100)
   INTEGER                p
    .
    .
    .

   r = cfunc(w, x, y, z, a, LOC(p))
\end{verbatim}

Below is the prototype for the C routine.  This is not visible to the
Fortran code and is shown here for completeness.

\begin{verbatim}

  float cfunc(struct my_type w, short x, float y, char z,
              short a[100], int *p);

\end{verbatim}

This call to {\tt cfunc} causes the implementation to pass the
arguments {\tt w}, {\tt x}, {\tt y}, {\tt z} by value.  Further, 
{\tt x} is converted to a {\tt short} and {\tt z} is converted to 
a C {\tt char}.  {\tt a} is passed as a pointer (the way arrays in
this implementation's C are passed,) but is left as is in memory
since arrays are not automatically converted.  {\tt p} is passed
as a pointer since at the call site {\tt LOC} intrinsic is used.

\end{document}

---------------------------------------------------------------------------
To (un)subscribe to this list, send mail to [email-masked].
Leave the subject line blank, and in the body put the line
(un)subscribe <email-address>
---------------------------------------------------------------------------