Print PRINT
PDF

Step by step Flat file parsing in webMethods 8.2

 

For the Parsing a Flat File we need to know about below concepts of Flat File i.e.

-> Flat File Dictionary

-> Flat File Schema

-> Flat File Document

-> Flow service for parsing Flat File


Introductions of Flat File:

  • Flat file is a complex hierarchical data in a record–based storage format.
  • The metadata of a flat file is separated from the data and contained in a flat file schema.
  • A single record of application data is externalized as a set of records without any structural information.
  • The application receiving the flat file must have information about the structure of the flat file, for this we are creating flat file schema, to read the flat file.
  • All Flat File have a list of records which containing fields and composites.

 

Types of Flat Files which Support by webMethods

The WmFlatFile package can communicate all types of flat files but can process only some types of flat files. The WmFlatFile package can process flat files in which:

· The records in the flat file are defined using one of the following methods:

· Delimiters- all record in the flat file is define by a delimiter.

· Fixed length- all record is a fixed number of bytes.

· Variable length- all record is define by two bytes that indicate the length of the record.

· Each distinct record structure has a record identifier, or if no record identifier is any, then record must be parsed using a default record definition.

· If the flat file contains record identifiers, the record identifiers must be located in the same location in all records in the file.

· The WmFlatFile package then can identify fields in these records based on :

· Delimiters- all field is separated by a delimiter, and the Nth delimited field is identified in the record.

· Byte position- all field is defined by one is the number of bytes from the beginning of the record and second is the field length.

 

Flat File Schema:

To communicate using flat files, we have to create a flat file schema that contains a particular flat file’s structural information, including how to identify records and separate those records into fields.

Flat File Constraints:

Flat file Schemas contain information about the document, record, and field levels of flat files. A flat file schema can define three types of constraints:

Structural Constraints: its describes the sequence of records, composites, and fields in a flat file.

For example, a flat file schema might describe Employee Details:


Content type constraints : its describe the possible values or minimum/maximum values for elements in a flat file.

For example, if there is a mobile number field might have a content constraint specifying that its value must be exactly ten digits.

Conditional constraints describe the valid combinations of fields and subfields allowed in records and composites.

For example, you can use the Required (R) conditional validator.

Flat File Schema Used to Parse Records in different type:

The WmFlatFile package provides the pub.flatFile:convertToValues service, which uses a flatfile schema to parse flat files inbound to the Integration Server. Record Parsers: The convertToValues service uses a record parser (Delimited, Fixed Length, or Variable Length) to parse the records in a flat file.A record parser breaks a flat file into individual records. In the WmFlatFile package, we can choose from one of its four record parsers:

•    Delimited Record Parser. This parser expects a specific delimiter to indicate the end of a
Record. For a record delimiter, you can specify A character, Hexadecimal value, Octal value,    Unicode characters. The occurrence of the record delimiter signals the end of one record and the beginning of the next record.

Example- No length describe for field, only record, field ,sub filed delimiter define like
Here record delimiter is ‘new line’ and field delimit is ‘*’


Name*Adderess*Age
Name2*Adderess2*age2

 


·Fixed Length Record Parser. This parser splits a file into records of the same pre-specified length. We can define parser measures records as in terms of lengths or position.

Example-In this field length is fixed like:

Field one have length 0 to10
second field have length 11 to 25
third field have 26 to 29

and record size is 30

123456789123456789123456789123
name      address       age


After 30 character its take next record.

·Variable Length Record Parser. This parser expects each record to be preceded by two bytes that indicate the length of the record. We can describe record may be a different length.

·EDI Document Type Record Parser. This parser is used only for EDI flat files and provides additional functionality needed to properly parse EDI documents.

Record Identifiers: After each record in a flat file has been parsed, each record must be identified. This is done using the record identifier. Identifying the record provides the definition of that record, as defined in the flat file schema using the Flat File Schema Editor.

We can choose from one of two methods of record identification:

·Starts at position record identifiers compare the value that occurs in the record, at the specified offset, to all the record names defined in the flat file schema.

·Nth Field record identifiers use the value of the specified field as the record identifier. These identifiers count from zero (0). For example, if 1 is specified, the second field is used as the record identifier.

 

Extractors: Extractors take data out of a parsed record and place it in the output of the pub.flatFile:convertToValues service. If extractors are not defined in the flat file schema, the Parser returns a series of records that contain no fields or composites. Fields and composites are extracted from a record based on the position of the field delimiter. Fields also can be extracted from a record based on a sub-string of the original record starting at a particular byte count and ending at a particular byte count.

Undefined Data: In some cases, the data resulting from the pub.flatFile:convertToValues service might contain records that were not recognized, known as undefined data. This could be a result of the complexity of the flat file, or of the lack of detail in the flat file schema.

Default Records:

If the pub.flatFile:convertToValues service cannot recognize a record ,the record will be treated as undefined data unless you specify a default record definition to be used to parse all unrecognized records of a particular type. In fact, if you’re flat file schema does not contain any record identifiers, You must specify a default record. A default record can only be defined in a flat file Dictionary.

Finally, the parsed record is placed in the output data based on the structure defined in the flat file schema. This process is repeated until all records in the flat file have been Parsed.

Flat File Dictionary

A flat file dictionary is simply a repository for elements that you reference from flat file schemas. This allows you to create record definitions in a dictionary that can be used across multiple flat file schemas. Reusing record definitions reduces the amount of memory consumed by a flat file schema.

Flat file dictionaries are created as namespace elements in the Integration Server and contain definitions of records, composites, and fields. When you change a definition in a flat file dictionary that is referenced in multiple flat file schemas, the element definition is updated automatically in all of the flat file schemas.


Advantage of Flat File Dictionary

The decision to define records in a flat file dictionary versus in a flat file schema depends on the type of flat files that you intend to parse.

For Example:The Electronic Document Interchange (EDI) ANSI X12 standard defines a large set of document structures that reuse the same record, field, and composite definitions many times. Defining these records, fields, and composites in a dictionary allows for them to be reused throughout the entire set of EDI ANSI X12 document flat file schemas.

The main advantage of dictionary is Reusing definitions reduces the amount of memory consumed by the webMethods Integration Server.

Flat File Schema Editor

When we select a flat file schema or dictionary on Designer or Developer, the contents of the flat file schema or dictionary is displayed in the Flat File Schema Editor.

The Flat File Schema Editor is divided into the following sections:

Flat File Definition tab. This tab enables you to specify what type of flat file you are parsing or creating.

Flat File Structure tab. This tab enables you to specify the structural, content type, and conditional constraints of your flat file schema.

Properties panel. This panel enables you to configure a default record, allow undefined data, add areas, and set up the permissions for your flat file schema or dictionary.

Flat File Structure Tab

When creating your flat file schema, you will create new records, composites and fields.For adding new element we can see few components like:

Record Definition: Identifies a record definition in a flat file schema.

Record Reference: Identifies a record reference in a flat file schema.

Composite Definition: Identifies a composite definition in a flat file schema.

Composite Reference: Identifies a composite reference in a flat file schema.

Field Definition: Identifies a field definition in the flat file.

Field Reference: Identifies a field reference in a flat file schema

 

 

 

Creating Distionary

In webMethods Developer, click File -> New-> Flat File Dictionary -> Next -> Type the name of the new flat file dictionary -> Finish .


 

To add elements, select the element type to which you want to add and click Record Definition: Define all other record definition properties in the record.



 

Same as above we can define another two element type.

Composite Definition: Define all other composite definition properties in the composite reference instance in the flat file schema.

Field Definition: Define all other field definition properties in the field reference instance in the flat file schema.

After you have added all of the elements to this flat file dictionary,

Dictionary elements Properties :

Validator: Click the Validator field, and then type the desired validation you want to be performed on this Element.

 

 

Check Fields: Select True to report as errors more fields at the end of the record otherwise take it False.

Alternate Name: Use this field to provide a more meaningful name that can be helpful when mapping the record.

Description: Describe the purpose of the record, composite, or field. This description will appear in the comments section of the document type .

ID Code: The IDCode, which is provided by your SEF file, is used by the WmEDI package.

DataType: Select the appropriate data type.

Format Service: Click the Format Service field to select the service you want to invoke for a particular field when convertToValues or convertToString is called, and then click OK

 

Creating Schema

In webMethods Developer, click File -> New -> Flat File Schema -> Next -> Type the name of the new flat file schema -> Finish ->->->in the menu bar.

 


 

On the Flat File Definition -> Record Parser-> select the type of parser ->

Delimiter: Each record is separated by a delimiter.

Fixed Length: Each record is of a fixed length .

Variable Length: The first two bytes of the input data stream contains the length of each record.

EDI Document Type: The EDI Document Type option is displayed for you to view existing EDI flat file schemas.



When we specify delimiters, use the specific syntax.

Specify Flat File Properties:

View the Properties panel :

Under Default Record -> configure the following properties:

Set : click Set to select the default record for this flat file schema from a flat file dictionary. This is optional field.

Delete : click Delete to delete the default record for this flat file schema.


Under Settings->

Allow Undefined Data :This property helps control the generation of undefined data errors at any location in the flat file schema.

If we want to allow undefined data select True otherwise select False.

Areas : This property enables you to add and edit areas for this flat file schema.

For example, you may have an address record that needs to specify the shipping address and billing address in two different areas .you would assign the appropriate area to a record to identify in which section of the document the record belongs.

Floating Records:

This is Optional property. The name of the record that is defined in the schema as a floating record. You can use the Floating Record property to designate any single record of a given flat file schema to be a floating record. By designating a floating record, you enable that record to appear in any position within a flat file without causing a parsing validation error. If you do not use this property, validation errors will occur if the record structure of an inbound document does not match the record structure defined in its flat file schema.

Define the Structure of the Flat File Schema

·click the Flat File Structure tab of the Flat File Schema Editor.

·To add the first record, click the name of the flat file schema in the Name column, and then click . The Select New ->·Select one of the following.

Record Definition : Select to create a record definition to be used one time in this flat file schema. After selecting Record Definition, click Next.



 

Record Reference : Select to create a reference -> Record Reference -> Next -> select the dictionary -> Next -> Select the element-> Finish


 

 


·To add more elements, select the element to which you want to add a child and click

·If you are adding a record definition, type the Name of the record in the Enter

Record Definition Name(s) dialog box.


 

·If you are adding a composite definition, specify the Name, Position, and Mandatory properties in the Enter Composite Definition Name.

 


·If you are adding a field definition, specify the Extractor Type, Name, Start, End, and Mandatory properties in the Enter Field Definition Name.


Properties Description :

List of the Records properties and Field Properties-


 

 

Mandatory –If it is selected and the convertToValues service validate variable is set to true, errors will be generated if the composite does not appear in the flat file.

Max Repeat - Maximum number of times this record can repeat within the flat file.You can select Unlimited or type any positive integer.

Ordered - Select True to require that the child records of this record appear in the flat file in the order that they appear in the flat file schema.

Position - Reserved for future use.

Validator - Click the Validator field, and then type the desired validation you want to be performed on this Element.

Check Fields - Select True to report as errors more fields at the end of the record otherwise take it False.

Alternate Name- Use this field to provide a more meaningful name that can be helpful when mapping the record.

Description - Describe the purpose of the record, composite, or field. This description will appear in the comments section of the document type .

ID Code -The IDCode, which is provided by your SEF file, is used by the WmEDI package.

DataType - Select the appropriate data type.

Extractors -Extractors represent the location of a field within a record, or of a subfield within a composite. You extract fields from a composite using a Nth field extractor, while you extract fields from a record using Nth field, ID Node, or Fixed Position extractors.

Flat File Parsing:

Now we start on parsing of flat file after created Dictionary and schema. We have to know first this services that the WmFlatFile package provides to process flat files: pub.flatFile:convertToValues and pub.flatFile:convertToString. For more descriptions:

Processing Flat Files.


·Flat File (ffData):its contain Flat file input with type of String, InputStream, or ByteArray.

·Conversion/Validation Services: This is the most impotent step to parse Flat file to IS document.

WmFlatFile package provides services pub.flatFile:convertToValues that parse, validate, and convert inbound flat files to IS documents and if requirement like target wants the output in Flat file only not IS document then after performing all operation we have to convert IS output document into Flat file and then sent it to Target application. For this we have service pub.flatFile:convertToString that parse, validate, and construct outbound documents from IS documents.

·Flat File Schema : Flat File schema containing the structure of the flat file.

·IS Doc (ffValues) :An IS document represents the structure and values of your flat file for mapping or other processing. All inbound documents must be converted to IS documents before being processed further.

Step 1: Create Appropriate Flat File Schema

Create a flat file schema to be used for conversion and validation of the document. Here we have first create a dictionary and then same set in schema.



 

Step2:Receive the Flat File

Invoke the pub.file:getFile service to retrieve the file from the local file system.

 

 

Step3 : Parse Flat File Data to an IS Document

To parse a flat file using a flat file schema, and to convert the data into IS documents. We can call the convertToValues service.

 


 

Step 4: Process the IS Document

Now we can process IS document .As per the requirements we can map the data.

·In the Developer Navigation Panel, select the saved flat file schema, click the Flat File

Structure tab, and then click Icon ‘Create Document Type’. This creates an IS document type in the same folder as the schema with the same name of Schema name.

·On the Pipeline tab under Service In, in the ffSchema variable specify the location and name of the flat file schema.

·Perform the following to add a Document Reference to the IS document type that is based on the flat file schema.

On the Pipeline tab click under Pipeline Out, Select Document Reference to add a new document reference variable or else choose the IS document type created in step 1 of this procedure .

·On the Pipeline tab under Service Out, map the value of the ffValues variable to the

·document reference variable created in this procedure and save the service.

Handling Large Flat Files

This all parsed file save in system memory. If you receive large files, might be you get some issue related to memory constraints. WmFlatFile package can encounter problems when working with these files because the system does not have enough memory to hold the entire parsed file.

For this we can set the iterator variable in the pub.wmflatFile:convertToValues service to true to process top level records in the schema one at a time. After all child records of the top level record are parsed.

The convertToValues service returns and the iterator moves to the top level of the next record in the schema, until all records are parsed.

This parsing should be done in a repeat step where each time the convertToValues service returns, the results are mapped and dropped from the pipeline to conserve memory. If the results were kept in the pipeline ‘out of memory’ errors might occur.

The service generates an output object ffIterator variable that encapsulates and keeps track of the input records during processing. When all input data has been parsed, this object becomes null. When the ffIterator variable is null, you should exit from the repeat step to discontinue processing.

 


 

 

Test Service extracts the flat file in IS Document:

 


·Step 5: Processing Outbound Flat Files

·We have the pub.flatFile:convertToString service, which uses a flat file schema to create a flat file outbound from a webMethods Integration Server.

·The input of this service is an IS document and a flat file schema, and the output of this service is a flat file.

·For the flow service you are creating or editing, on the Flow Editor of the Developer select the INVOKE pub.flatFile:convertToString flow operation.

·On the Pipeline tab under Service In, in the ffSchema variable specify the location and name of the flat file schema.

·Perform the following to add a Document Reference to the IS document type that is based on the flat file schema.

On the Pipeline tab click under Pipeline Out, Select Document Reference to add a new document reference variable or else choose the IS document type created in step 1 of this procedure .

·On the Pipeline tab, map the document reference variable created in this procedure to the value of the ffValues variable under Service Out and save the service.

·After the flat file is created, you can send or process the file as necessary.

 

 


Setting Access Permissions

Administrators of the Integration Server can set access control list (ACL) permissions to control user access and editing privileges for each flat file schema or flat file dictionary.

To set access privileges for a flat file schema or dictionary

1.Double–click the flat file schema for which you want to set permissions.

2 .In the Properties panel, under Permissions, from the List ACL drop–down box and Write ACL drop–down box, select the ACL level to which you want to grant read access.This enables the ACL you select to view and edit this flat file schema or dictionary.

3. Click to save the flat file schema permissions.

 

 

 

Comments (8)Add Comment
reddy
...
written by reddy, November 07, 2014
Hi ...... I gone your documentary presentation. Its very nice ....

I need a small favor from your end, how do you create flat file schema DT. Just share that screen shots separately to my mail ID ::: annurureddy@gmail.com

The schema I know with record,segment and loop id (and) recordswithnoID. You done additional activity also, In SchemaFlatFileDT i want to create like that as per my requirement.
RAMAKRISHNA KANNEGANTI
...
written by RAMAKRISHNA KANNEGANTI, July 05, 2014
Nice ...!!!
wMDevloper
...
written by wMDevloper, April 28, 2014
ok,i will post variable length flat file example.
Akash
...
written by Akash, April 26, 2014
Can you post with variable length flat file example?
Rajesh
...
written by Rajesh, January 15, 2014
Hello,

Regarding your sample with Large files, you do a getFile outside of the loop in Bytes.
Won't that read the complete document and beat the purpose of the iteration?

Thanks
Sindhuja
...
written by Sindhuja, November 28, 2013
Hi,

We have a specific requirement. Get all the flat files from a file server and send the batch size of 200 records to another ESB . Where can I get information for Batch File processing? or some documentation to understand Batch File processing.

Thanks for your information !
wMDevloper
...
written by wMDevloper, September 19, 2013
smilies/smiley.gif

Ansari
...
written by Ansari, September 18, 2013
Excellent, this resolves all my doubts on Flat files. wMDeveloper you are amazing i am learning wM following your blogs.

Thanks much for uploading the step by step scenarios for beginners.

Ansari.

Write comment
You must be logged in to post a comment. Please register if you do not have an account yet.

busy