ScreenSurfer XML Support

ScreenSurfer XML Support
ScreenSurfer V3, April 2001


Page Links	Overview, Reference, Status Entity

Examples	Simple

This document describes the ScreenSurfer XML-over-HTTP support, and how it integrates with the ScreenSurfer Web Gateway.

Please note that there is corresponding co-server documentation in the ScreenSurfer SurferScript Reference covering the SurferScript coding for Cold Fusion, ASP, XML-over-HTTP and EJB Co-Servers support.

Features
The features of XML-over-HTTP combined with ScreenSurfer include the following:

Full Functionality	Enables a web developer with the ability to call an XML server using HTTP to access all of the data and functionality available in mainframe, AS/400 or Unix/Vax screen-based applications.
High Productivity	The ScreenSurfer XML-over-HTTP was designed and implemented using a simple approach that should integrate with any XML environment. Since the ScreenSurfer scripting is the same across any of the supported Co-server environments, customers can start projects using one of the custom co-server interfaces with the future ability to easily migrate to an all-XML architecture. The ScreenSurfer DevCenter includes the Visual Surfer Wizards which give ScreenSurfer developers a fast, point-and-click interface for identifying and naming screen fields accessed using the XML support.
Granular Pass-Through	The ScreenSurfer XML-over-HTTP support leverages the ScreenSurfer "selective screen enhancement" features that make it so effective as a direct 3270/5250-to-HTML server. Granular pass-through enables whole or portions of a screen to be directly delivered in HTML inside a <![CDATA[ block inside a named entity. Existing highlighting, entry field areas and other characteristics are faithfully reproduced using HTML for those portions of the screen passed-through in this manner. For many inquiry applications that ride on top of an XML infrastructure, this avoids the need for tedious reproductions of already existing data presentation designs.
High Performance	The ScreenSurfer XML-over-HTTP support adds minimal overhead to the task of accessing mainframe or AS/400 transaction data via 3270/5250/VT100 protocols, further reducing costs by optimizing hardware resources. With its own built-in HTTP port, ScreenSurfer avoids unnecessary overhead found in many screen access environments.
Compatible	The ScreenSurfer XML-over-HTTP support is fully compatible with "how things are done" in a XML Environment. Input to ScreenSurfer can be straight HTTP post contents, or it can be an XML document of any structure. For compatibility with other Co-Server requestors, the input should be HTTP post, or should consist of simple entities that are contained in a top-level <WebInput> document tag.
Team Oriented	The design of ScreenSurfer's screen-to-XML allows one or more developers to learn the ScreenSurfer development environment and then "encapsulate" all the complexities of accessing mainframe/AS400 screens in simple, callable transactions. Once created, the transactions are available to any XML developer in a project or an entire organization, using standard XML-over-HTTP access techniques.
Manageable	The runtime ScreenSurfer Web Gateway system provides troubleshooting and administrative features that prevent the ScreenSurfer gateway from being a "closed black-box".

Overview

ScreenSurfer is a general purpose Web Gateway which provides users at Web Browsers transparent access to mainframe, AS/400 or UNIX/VAX VT100-based screen applications. In addition to ScreenSurfer's ability to provide a direct, HTML-driven user interface to screen applications, ScreenSurfer also has an easily implemented Client or Co-Server Interface.

ScreenSurfer's XML-over-HTTP Co-Server Interface enables it to provide access to the data and functionality of mainframe or AS/400 screen-based (3270/5250) applications by accepting XML-based requests over the HTTP protocol and responding with easily parsed XML responses.

The ScreenSurfer Web Gateway is both a specialized Web Server as well as a "back-end" for co-server products such as Cold Fusion , ASP-based Web Servers, XML-over-HTTP environments and Java application servers. ScreenSurfer runs as an NT Service, and can be installed on the same machine as an XML-based Server/Client, or on another machine that is addressable to the XML client using standard IP Sockets. Communication uses the HTTP protocol for compatibility with the growing standards of issuing XML-based requests over HTTP.

Included with ScreenSurfer is a language environment called SurferScript, which basically consists of HTML-oriented text files that are maintained in a single directory. With SurferScript, the ScreenSurfer developer creates callable "transactions" which any XML developer can then call using a very basic form of XML-over-HTTP. For more information on defining callable transactions, see the ScreenSurfer Visual Surfer documentation as well as the ScreenSurfer SurferScript Reference documentation.

A Sample XML Application

The Sample Application requires the use of an XML-compliant browser.

To execute the sample, you simply start your ScreenSurfer server using the default (Samples) project distributed with Version 3.

The sample utilizes our standard Co-server test transaction, which is named AspTest.stml. AspTest returns a number of simple values as well as a sequence of entities contained in a another.

To execute the sample, if on the same server as ScreenSurfer, in an XML-compliant browser, enter in the URL: http://localhost:82/ss_xml/surfer/details This will return a formatted XML document, which you can view in the browser and see the simple structure of ScreenSurfer-generated XML.

Technical Description

The ScreenSurfer Remote Procedure Call

ScreenSurfer supports a dynamic remote procedure call capability originally implemented to support the ScreenSurfer CFX tag for Allaire's Cold Fusion.

This capability enables a ScreenSurfer developer to define named transactions using the SurferScript <TESECTION> tag. A SurferScript transaction can read named input parameters from the caller, interact with host 3270/5250 applications and return one or more recordsets. A recordset may represent a single data entity (such as ShipTo Address) or multiple entities (such as All Open Orders).

The recordsets returned by a ScreenSurfer transaction consist of one or more rows. Each recordset row can contain multiple named fields. A special Base Recordset is a single-row recordset containing a mix of named data fields and named blocks of HTML text. An HTML text block can contain entire screen areas "Passed-through" using ScreenSurfer's granular pass-through capabilities.

As implemented, the XML-over-HTTP support exploits the ScreenSurfer RPC transaction support, enabling an XML service to easily call any defined ScreenSurfer transaction and incorporate any returned recordset data in an application.

The advantage of encapsulating ScreenSurfer functionality in a callable transaction is that it separates the task of Mainframe or AS/400 application and data access "screen-surfing" from the task of writing web or other integration applications. Screen-scraping / Screen-surfing has very specific challenges and characteristics which the ScreenSurfer language and compiler support through a number of high-level declarative constructs.

Using the HTML Result Type the called SurferScript transaction can return ScreenSurfer-formatted screen areas as named fields in the Base Recordset, so that these may be easily included in an outbound HTML data stream. In the XML environment, this HTML is automatically encapsulated in a <![CDATA[ block.

Another advantage of the RPC approach is that the ScreenSurfer NT Server can be on the same or a different machine, as long as the XML application server can access the ScreenSurfer server using standard IP Sockets.

ScreenSurfer XML-over-HTTP Input

Input to a ScreenSurfer XML-over-HTTP transaction can be simple HTTP post values, or a combination of a URN (URL) along with a text/xml package containing an XML document.

Since ScreenSurfer is frequently installed on the same machine as an existing Web server, it is frequently installed listening on port 82. In any examples you see in this document, port 82 is used.

To call the ScreenSurfer XML support over HTTP, use a URL that starts with the special resource identifier: /ss_xml/

Following the /ss_xml/ identifer is the name of the transaction as defined in your ScreenSurfer application. For example, a transaction named search/employees would have a full URL on localhost of:

http://localhost:82/ss_xml/search/employees

If you are passing simple input parameters, such as an employee number, that can be passed simply in the URL. For example:

http://localhost:82/ss_xml/search/employees?empnumber=123456

If, however, you have more extensive input, such as multiple entry fields, then you can pass an XML document. The simplest format, which is compatible with existing co-server scripts is to encapsulate your input parameter values in a top-level tag named <WebInput>. For example:

http://localhost:82/ss_xml/search/employees

In the above examples, all of the input is available to the ScreenSurfer scripts in the standard web. variable expression.

Input Document Traversal with the Web:: Expression

New in Version 3 of ScreenSurfer is support for traversal of XML documents using the :: (double colon) operator. While in the initial release this only provides read-only access to HTTP input documents, over time it will be extended to a full read/write capability of in-memory XML dynamic documents, including full indexing of entity arrays (see next section).

The syntax of the :: operator is simple. The following input document is used as an example:

 <?xml version="1.0" ?> 
 <Person>
  <FirstName>Joe</FirstName>
  <LastName>Johnson</LastName>
  <Address>
   <Line1>10 This and That Lane</Line1>
   <City>Darien</City>
   <State>Connecticut</State>
   <Zip>06820</Zip>
  </Address>
 </Person>

In sequence, to read all the values described above, the following web:: expressions would be used in SurferScript. Note that ScreenSurfer document traversal does not include the root document name, as in-general that will be identified by the variable holding the handle of the document (such as a variable named "Person"). However, if the root document needs to be verified, the special entity name "Root$" is provided. For each expression, its value is provide in quotes:

Web::Root$ "Person", Web::FirstName "Joe", Web::LastName "Johnson" Web::Address.Line1 "10 This and That Lane", Web::Address.City "Darien", Web::Address.State "Connecticut", Web::Address.Zip "06820"

Indexed Entities

In addition to the ability to traverse document entities using the dot-notation off of an XMLDOC variable type, multiple entities within a single branch can be easily navigated using the entity index notation.

The Index notation is a simple use of square brackets surrounding an integer expression (the integer expression can be any valid SurferScript expression that includes math operators, functions and variables).

Web::Address.Line[0], Web::Address.Line[1]

The Count$ Attribute The special attribute Count$ can be used to determine the number of entities that exist within a particular XML document branch.

The following example demonstrates the use of both Count$ and index expressions:

XML Document Indexed Access

<TESet OrderDetailCount=web::Orders.LineItem[].Count$>
<TELoop Index 0 OrderDetailCount-1>
 <TECall Order.ProcessDetailLine
   web::Orders.LineItem[Index].ItemNumber
   web::Orders.LineItem[Index].Quantity
   web::Orders.LineItem[Index].Price>
</TELoop>

The above loop iterates through all the LineItem Entities, calling the section Order.ProcessDetailLine and passing three child entities/attributes for each LineItem.

Special Index Navigation Expressions Other indexing features include the following special index navigation expressions:

[] = Same index as last reference (first element if first access)
[++] = Next entity in current branch
[--] = Previous entity in current branch

Example using index walking:

XML Document Indexed Access

<!-- process up to 100 line items-->
<TELoop Count 1 100>
 <TECall Order.ProcessDetailLine
   web::Orders.LineItem[].ItemNumber
   web::Orders.LineItem[].Quantity
   web::Orders.LineItem[].Price>
   <teif web::Orders.LineItem[++].ItemNumber=="">
    <tebreak>
   </teif>
</TELoop>

The above loop iterates through all the LineItem Entities, calling the section Order.ProcessDetailLine and passing three child entities/attributes for each LineItem. When a same index reference is used for the first reference on a document branch, it references the first entity.

XML Entity Reference Optimization

The SurferScript compiler will optimize your references to XML document contents automatically.

In the above example, the compiler automatically recognizes that the lines:

All have a common parent entity (Orders.LineItem[Index]) and resolves all the references so that there is no additional navigation inside the parsed XML branches (Quantity and Price will be searched and queried within the same indexed branch node as ItemNumber without any additional searching).

This optimization provides significant, automatic performance benefits, since the normal navigation of the parsed XML document would entail searching at each branch node for that node's reference.

Due to unpredictable results, the compiler will not optimize references following the execution of any of the following tags: TEIF, TEELIF, TEELSE, /TEIF, TECALL, TEPUTSECTION, TESECTION, /TESECTION. You can implement your own optimization when these tags are present by using the XML reference shorthand syntax.

XML Reference Shorthand Syntax
The XML Reference shorthand syntax allows you to "skip" XML entity names, and just include the separating dots in your expression. When this is used, at runtime the number of dots from the start of the expression is used to "index-in" to a node for the last accessed entity/attribute (the shorthand is only recognized if the dots start the expression; you cannot use the shorthand in the middle of a entity reference sentence).

The following example demonstrates forcing optimization with shorthand:

XML Document Indexed Access

<TESet OrderDetailCount=web::Orders.LineItem[].Count$>
<TELoop Index 0 OrderDetailCount-1>
 <TECall Order.ProcessDetailLine
   web::.LineItem[Index].ItemNumber
   web::..Quantity
   web::..Price>
</TELoop>

The above loop iterates through all the LineItem Entities, calling the section Order.ProcessDetailLine and passing three child entities/attributes for each LineItem.

The compiler understands the .. shorthand and will access the current XML document reference at the last node visited (each dot represents a node).

The advantages of the shorthand are that if a TECALL or other tag that contains no XML references (or references to a different XMLDOC variable), then you can force effective optimization with the shorthand.

The shorthand can also be employed if you are coding at a frantic pace and wish to cut-down on your keystrokes...but the resulting code really doesn't read as nicely as full, verbose references :)

The Standard XML-over-HTTP Output Document

When ScreenSurfer returns XML over the HTTP connection, the standard root document is <Results>. See User Schema Output for creating user-defined output schemas.

Contained in the root Results document is each value as returned by the author of the co-server transaction. Entities will always be either direct children of the Results root document (in co-server terms these are members of the "base" recordset), or members of a Table entity that is a child of the Results root document.

The difference between being direct children of the root document or of an additional node is determined by the use of the <TERESULT TABLE...> tag. If a set of <TERESULT ROW...> tags is initiated in the SurferScript side with a <TERESULT TABLE...> tag, then those row values will be children of a tag named with the child name followed by the fixed text Table.

For example, if a ScreenSurfer row named Item has three values, Name, Date and Amount, if a single row of Item is returned in a simple <TERESULT ROW Item>, the result document will look like:

 <?xml version="1.0" ?> 
 <Results>
  <Name>Joe Johnson</Name>
  <Date>12/20/2001</Date>
  <Amount>12.23</Amount>
 </Results>

However, if the same row is preceeded with a <TERESULT TABLE Item>, followed by two TEResult rows of Item, the Results will look like:

 <?xml version="1.0" ?> 
 <Results>
  <ItemTable>
   <Item>
    <Name>Joe Johnson</Name>
    <Date>12/20/2001</Date>
    <Amount>12.23</Amount>
   </Item>
   <Item>
    <Name>Pete Patterson</Name>
    <Date>06/05/2000</Date>
    <Amount>500.16</Amount>
   </Item>
  </ItemTable>
 </Results>

The User-Defined XML-over-HTTP Output Document

Using some special <TEResult> tag properties, you can produce XML documents which follow any schema you desire. Additional information can be found under the description of the TEResult Tag

Defining your Own Root Document

To define your own root document (to be used instead of "<Results>"), preceed all output in your script with the tag:

<TEResult XMLPush name="MyRootDocumentName"> where MyRootDocumentName is the name of your document's root name.

Opening an Entity/Node

To open a new entity with no attributes, use:

<TEResult XMLPush name="EntityName">

To open a new entity that includes attributes, use:

<TEResult Row row="RowName" XMLPush="Yes" XMLAttrs="Yes" [name="OptionalNameOverride"]> (The Row that contains the attributes must have the values in the SurferScript variables set before executing this tag).

Closing an Entity/Node

To close an entity opened using the XMLPush directive, use:

<TEResult XMLPop>

Output a Row of Data

To output a row of data from a <TEDeclare ROW> or <TEDeclare ScreenMap>, you can use the "standard" approach, which will open an entity with the row name, then inside, for each attribute, open an entity, output the data value (as the body), then close the entity. To output a row using this "standard" approach, simply use:

<TEResult Row row=NameOfRow> -- Will open an entity for row, then do open, data, close for each variable in the row.

To output a row as a single entity with attributes use: <TEResult Row row=NameOfRow XMLAttrs="Yes">

TERESULT ROW XML Examples

<TEDECLARE ROW Name=Detail ItemCode Text 20, Quantity Integer, Price Float> ... <TESet Detail.ItemCode="ABCDEF"> <TESet Detail.Quantity=20> <TESet Detail.Price="34.56> ... The XML output for <TEResult Row Detail>: <Detail> <ItemCode> ABCDEF </ItemCode> <Quantity> 20 </Quantity> <Price> 34.56 </Price> </Detail> The XML output for <TEResult Row Detail XMLAttrs="Yes">: <Detail ItemCode="ABCDEF" Quantity="20" Price="34.56"/> The XML output for <TEResult Row Detail XMLAttrs="Yes" XMLPush="Yes">: <Detail ItemCode="ABCDEF" Quantity="20" Price="34.56"> ...additional rows or other results now part of this entity ...close with <TEResult XMLPop>
This shows how the different attributes in the TEDeclare ROW can be used to control the output of XML entities containing the data in the row.

The Status Entity

The Status entity is a special result set that is always returned by the ScreenSurfer transaction. It is a direct child of the Results root document, and in-turn contains a simple sequence of child entities.

The children of the Status entity may be used in addition to the base entities and optional recordset contents that correspond to data from the ScreenSurfer 3270/5250 application screens. Generally the Status fields are "housekeeping" fields used for managing the environment of accessing host application data with XML-over-HTTP through ScreenSurfer.

Status Fields returned by XML-over-HTTP call

SessionKey A special text string which may be set in subsequent calls in order to provide an association between the client's session user and the host screen session providing the current application data. The SessionKey maintains the state of a given connection across calls.

ScreenID Returns a string containing the name of the most recently displayed host screen. Useful when synchronization is necessary, although this should normally be handled by the ScreenSurfer transaction logic.

RetCode Return code, 0 means no error

FailComponent Only present when RetCode is non-zero.
Contains the name of the ScreenSurfer component producing an error although this may also be set by the ScreenSurfer application programmer to indicate some other application-related error.

FailMessage Only present when RetCode is non-zero.
Contains a textual message describing the error represented by RetCode.

Extending the prior sample to include the Status Entity, the full returned document would look like:

 <?xml version="1.0" ?> 
 <Results>
  <ItemTable>
   <Item>
    <Name>Joe Johnson</Name>
    <Date>12/20/2001</Date>
    <Amount>12.23</Amount>
   </Item>
   <Item>
    <Name>Pete Patterson</Name>
    <Date>06/05/2000</Date>
    <Amount>500.16</Amount>
   </Item>
  </ItemTable>
  <Status>
   <ScreenID>EmpStatus.VSRecog</ScreenID> 
   <Handler>default</Handler> 
   <RetCode>0</RetCode> 
   <FailComponent /> 
   <FailMessage /> 
   <FailSection /> 
  </Status>
 </Results>

XML-over-HTTP to ScreenSurfer Error Management

Since host communications can create situations similar to database issues, it is important to understand how to detect and work with any errors that may occur during the calls to ScreenSurfer.

The primary approach is to use scripting logic and check the returned codes in either the base recordset (application level) or the status recordset (system level). Each of these can provide a quick check of a success code (like the status "retcode" value being zero, which it should be on a clean call), and provide an avenue for reporting errors.

Generally, calling a common routine to display relevant contents of the Status recordset to the user would be similar to displaying un-expected database errors.

SessionKey	A special text string which may be set in subsequent calls in order to provide an association between the client's session user and the host screen session providing the current application data. The SessionKey maintains the state of a given connection across calls.
ScreenID	Returns a string containing the name of the most recently displayed host screen. Useful when synchronization is necessary, although this should normally be handled by the ScreenSurfer transaction logic.
RetCode	Return code, 0 means no error
FailComponent	Only present when RetCode is non-zero. Contains the name of the ScreenSurfer component producing an error although this may also be set by the ScreenSurfer application programmer to indicate some other application-related error.
FailMessage	Only present when RetCode is non-zero. Contains a textual message describing the error represented by RetCode.