Message Modeling
1
Unit objectives
After completing this unit, you should be able to:
• Explain the concepts of message models and how they are used to help
message transformation
• List the parsers that are available for use within IBM Integration Bus
• Create and modify a DFDL model
• Use importers to create data models
• Choose the appropriate message validation options
• Organize and administer message models
• Reference message models in message flows
Message Modeling
• Most messages require a model at run time to parse the message bit stream from the
physical data and serialize to create the physical bit stream from the model
• Modeling accelerates the development of transformations at design time in the IBM
Integration Toolkit
– Provides a source and a target for graphical mappings
– Assists when editing transformation programs
• Validation can be used for checking message correctness when parsing
• Instant documentation for programmers, analysts, and integration specialists
IBM Integration Bus message modeling
Parser types
Parsers understand the format of messages
• Programmatic parsers
– Knowledge of the message format is encoded in a program
– Each message format needs a new parser program
• Descriptive parsers
– Knowledge of the message format is encoded in a model
– A general-purpose parser program uses the model when parsing
– Model can be used as-is or generated into code
• IBM Integration Bus uses both approaches
– XMLNSC, JSON, and BLOB are programmatic parsers
– DFDL, MRM, and XMLNSC with an XML schema are model-driven parsers
Model definition files
• Describe the logical structure of the message
• Describe the physical formats of the message bit stream during transmission
• Can be imported from WSDL, an XML schema, DTD, COBOL, or C
• Contains a message model with messages, elements, types, and so on
• Can link DFDL schema files
• Based on XML schema standards
Model definition files
• Describe the logical structure of the message
• Describe the physical formats of the message bit stream during transmission
• Can be imported from WSDL, an XML schema, DTD, COBOL, or C
• Contains a message model with messages, elements, types, and so on
• Can link DFDL schema files
• Based on XML schema standards
Data Format Description Language (DFDL)
• An open standard from the Open Grid Forum (OGF)
• Uses XML technology and concepts
– W3C XML Schema subset and type system describe the logical format of the data
– Annotations within the XSD describe the physical representation of the data
– XPath notation references fields within the data
• Describes any data format
– Textual and binary
– Commercial record-oriented
– Scientific and numeric
– Modern and legacy data
– Industry standards
• Supports round-tripping
– Read and write data in the described format from the same description
DFDL data support
• Text data types such as strings, numbers, zoned decimals, calendars, and Booleans
• Binary data types such as integers, floats, BCD, packed decimal, calendars, and
Booleans
• Pattern languages for text, numbers, and calendars
• Ordered, unordered, and floating content
• Default values on parsing and serializing
• Nil values for handling out-of-band data
• XPath 2.0 expression language, including variables to model dynamic data
• Speculative parsing to resolve choices and optional content
• Fixed and variable arrays
• Hide elements in the data
DFDL data support continued
• Calculate element values
• Validation to XML Schema 1.0 rules
• Scoping mechanism to allow common property values to be applied at multiple
points
• User-defined variables in DFDL expressions
• Delimited binary data
• Fields lengths that come from regular expressions
Example: Delimited text data
Separators, initiators (tags), and terminators are all examples of DFDL delimiters
Example: Delimited text data
DFDL support in IBM Integration Bus
• DFDL models (schema files) are stored in IBM Integration Bus projects
• DFDL domain and the model-driven parser are available in all nodes
• IBM Integration Toolkit has built in tools for creating DFDL models
– DFDL model debugger to test the parsing and serialization of data by using a model
• DFDL models are deployed to the integration server
DFDL usage
• Use DFDL to model:
– Binary data from COBOL, C, PL/I, ASM programs
– Text data with delimiters such as CSV
– Text industry standards such as SWIFT, HL7, EDIFACT, and X12
– Binary industry standards such as ISO8583 and TLog
• Do not use DFDL to model:
– XML data that can use XML parsers and have an XML schema or DTD
– JSON that can use JSON parsers
– Serialization formats where the wire format is never available to the consumer and access
uses APIs
• Do not use DFDL expressions to implement complex validation rules
DFDL object model
• DFDL properties are placed only on an element, simpleType, sequence, and
choice objects
• Asterisk (*) indicates a zero-to-many relationship
DFDL schema editor
• Use the DFDL schema editor to create, edit, and test DFDL schema files
• DFDL schema editor is started when you open an existing DFDL schema
file or when you create a DFDL schema file by using the New Message
Model wizard
• With the DFDL schema editor, you can:
– Edit DFDL schema files
– Populate empty DFDL schema files with DFDL schema objects
– Parse the DFDL schema files against sample input data
– Serialize a logical instance document against your DFDL schema file
Creating a new DFDL schema file from wizard
• You can use the New Message Model wizard to generate a DFDL schema
file by modeling an existing data file
– Comma-separated value (CSV) data
– Record-oriented data
– Custom text or binary data
– COBOL data structures
Creating a DFDL model
Creating a DFDL model by using guided authoring
Creating a DFDL model by using the editor (1 of 2)
Creating a DFDL model by using the editor (2 of 2)
Created DFDL schemas
Helper DFDL schemas
• DFDL properties do not have built-in defaults, so if an object needs a property, a
value must be supplied
• New Message Model wizard automatically creates a helper DFDL schema with
values that are set for most DFDL properties, and adds it as an import into the user
DFDL schema
– Different helper DFDL schema for each type of data such as COBOL or CSV
– Helper DFDL schemas are created only one time per IBM Integration Bus application
– Helper DFDL schemas are created as read-only but can be edited by using the DFDL editor, if
required
– Helper DFDL schemas must be deployed with user DFDL schema
DFDL schema editor (1 of 3)
DFDL schema editor (2 of 3)
DFDL schema editor (3 of 3)
DFDL schema editor toolbar actions
• Test Parse Model: Runs the parser against a data file with the model
• Test Serialize Model: Runs the serializer against logical instance data
• Hide properties / Show properties: Hides or reveals the element Properties pane
• Show all sections / hide empty sections: Hides or reveals schema definition sections that
contain no information
• Focus on selected: When Show all sections is selected, the focus is on the currently
selected item
• Show quick outline: Displays a summary outline menu of the DFDL schema
• Create logical instance: Creates a logical instance for the Test Serialize Model
Testing DFDL schema
• Parse a data file by using the model as it is defined at the time
• Test the schema definition on real data, and modify it if needed
• IBM DFDL editor uses parser error markers to highlight errors
– Static schema error when a schema is saved
– Parser error when test parsing data
– Serializer error when test serializing a logical instance
• Click Test Parse Model
– Opens the DFDL Test perspective in the IBM Integration Toolkit
– Specify the input file to use for testing; either a resource that is in the application, or one from
the file system
Testing a DFDL model within the editor (1 of 4)
Testing a DFDL model within the editor (2 of 4)
Testing a DFDL model within the editor (3 of 4)
Testing a DFDL model within the editor (4 of 4)
Test parse logical instance
Test parse failure
Debugging a DFDL model test failure (1 of 2)
Debugging a DFDL model test failure (2 of 2)
Trace console
Test serialize model
DFDL points of uncertainty
• DFDL parser is a recursive-descent parser; it uses look-ahead when it encounters
“points of uncertainty,” such as:
– A choice
– An optional element
– A variable array of elements
• Parser attempts to parse data until an object is either “known to exist” or “known
not to exist”
– When a parsing error occurs, the parser attempts to suppress the error, backtrack, and try
another alternative
– Use a discriminator annotation to assert that an object is “known to exist” to prevent
unnecessary backtracking during parsing
– Use an initiator to assert that an object is “known to exist”
DFDL further details
• DFDL home page:
[Link]
• DFDL 1.0 specification:
[Link]
• DFDL schemas on GitHub
[Link]
• IBM DeveloperWorks: Get Started with Data Format Description Language
[Link]
Assigning the DFDL parser to a message
• IBM Integration Bus uses [Link] to determine the parser and the
dictionary for parsing (In) and for serialization (Out)
• Which information is needed?
– Message domain
– Message
Creating an XML message model
Parsing XML data
• In most cases, use the XMLNSC parser
– Fast memory-saving parser
– Builds a compact message tree with options to preserve the white space
– Supports opaque parsing where an opaque element (an element in a message that a message
flow never references) and its subtree are parsed as a single string; reduces parse time
– Can operate both as a programmatic and a model-driven parser
– An XML schema is created from the message set and deployed to the integration node
– Optional validation compliant with the XML Schema V1.0 specification
• Use XMLNS only if the message tree must conform as closely as possible to the XML
data model (for example, by using certain Xpath expressions), or to preserve the
inline DTDs
Configuring nodes for an XMLNSC parser (1 of 2)
Configuring nodes for an XMLNSC parser (2 of 2)
Message validation
• What can be validated?
– Runtime data is validated against a dictionary or an XML schema
– Content: Cardinality, complex type content, composition, and data type
– Value: Data fields conform to the value constraints in a message model
• When does validation occur?
– Validate an incoming message in the XXinput node
– Validate a constructed output message in the XXoutput node
– Validate immediately or request validation when a message is converted to a bitstream
• Failure action choices
– Throw an Exception on the first validation failure and stop validation
– Log all validation failures in the ExceptionList and throw an Exception after completion of a parse or
a write operation
– Log all validation failures in Usertrace or LocalErrorLog and continue
Parsing and validation
Organizing message models
• Integration Bus application provides runtime isolation so that the visibility of a resource is
restricted to the application in which it is contained
• Use applications to ensure that updates to one group of deployed resources do not affect
another group
• Consider the use of shared libraries to share message models across multiple teams,
projects, or integration nodes
Organizing message models
• Integration Bus application provides runtime isolation so that the visibility of a resource is
restricted to the application in which it is contained
• Use applications to ensure that updates to one group of deployed resources do not affect
another group
• Consider the use of shared libraries to share message models across multiple teams,
projects, or integration nodes
Unit Summary
Having completed this unit, you should be able to:
• Explain the concepts of message models and how they are used to help message
transformation
• List the parsers that are available for use within IBM Integration Bus
• Create and modify a DFDL model
• Use importers to create data models
• Choose the appropriate message validation options
• Organize and administer message models
• Reference message models in message flows
Checkpoint questions
1. True or false: You must deploy the DFDL “Helper” schema with the user DFDL
schema at run time.
2. True or false: When writing a message, the DFDL parser generates a DFDL
formatted bit stream from a DFDL domain logical message tree.
Checkpoint questions
1. True or false: You must deploy the DFDL “Helper” schema with the user DFDL
schema at run time.
Answer: True.
2. True or false: When writing a message, the DFDL parser generates a DFDL
formatted bit stream from a DFDL domain logical message tree.
Answer: False. The DFDL serializer generates a DFDL formatted bit stream.
Exercise – Creating a DFDL model
Exercise objectives
• Creating DFDL model for Record-oriented, tagged, delimited text
• Refining the model to specify correct data types
• Apply pattern validation on fields
• Test parse message