Start of change
WebSphere Message Brokers
File: ac55150_.dita
Writer: Kate Hostler

Reference topic

This build: July 31, 2007 21:21:04

FileInput node

This topic describes the FileInput node.

Purpose

Use the FileInput node to receive messages from clients that connect to the broker by creating files in a directory in the broker server's file system.

The FileInput node monitors a directory, and when a suitable file appears, reads and propagates messages based on the file's contents. It can generate one or more records from an input file's contents and establishes the processing environment for each record as a separate message. It can detect the following types of records in the input file's content:
  • A single record in the whole file
  • Multiple fixed length records
  • Multiple records delimited by a line end sequence or a character string that you have specified
  • Multiple records each detected by the parser configured in the properties in the node's Inbound Message Parsing tab
  • Multiple occurrences of a selected message element within the file after it has been parsed

The node removes the file from the directory when the flow successfully completes.

The FileInput node looks in the directory for files matching a pattern in a process known as polling. It then processes all matching files in the directory except those files that are locked by another process or broker flow. The node then waits for a period of time, called the polling interval, before repeating processing any new files that arrive in the directory. In normal use, the node processes files faster than they arrive, so execution consists of repeatedly waiting for the polling interval to expire and then processing all the files that have arrived in the directory. Locked files remain unprocessed in the directory until the lock is released. When the node next processes the files in the directory, it includes the files now unlocked.

The FileInput node processes matching files in sequence according to the file system time stamp for when the file was last modified, oldest first. Do not code flows that rely on this sequence because, for example, files may arrive while one or more flow instances are already processing files and your sequence will become out of step.

After successfully processing a file's contents, the FileInput node can delete the input file, or move it to the archive subdirectory in the same file system. If the flow fails, the node can move the file to a back out subdirectory defined in the same file system, move the file to the archive subdirectory, or delete the file.

The FileInput node handles messages in the following message domains:
  • MRM
  • Start of changeSOAPEnd of change
  • XMLNS
  • Start of changeDataObjectEnd of change
  • XMLNSC
  • JMSMap
  • JMSStream
  • MIME
  • BLOB
  • IDOC
  • XML

If you include an output node in a message flow that starts with a FileInput node, it can be any of the supported output nodes, including user-defined output nodes; you do not need to include a FileOutput node. You can create a message flow that receives messages from files and generates messages for clients that use any of the supported transports to connect to the broker because you can configure the message flow to request that the broker provides any conversion that is necessary.

When the FileInput node propagates a message, it stores the information in the LocalEnvironment.File message tree. If the input file is empty, the message is empty (assuming it passes any validation), and the Size element is zero. The following table lists the LocalEnvironment.File message tree structure.

Table 1.
Element Name Element Data Type Element Attributes
Directory CHARACTER Absolute directory path. Type the path separator character ("/" or "\") according to the operating system on which your flow is executing. The final path separator character is suppressed. On Windows®, start the directory path with the drive letter prefix, for example, "C:".
Name CHARACTER File name and extension.
Size INTEGER Size of the file in bytes.
IsDirectory BOOLEAN Reserved for future use. Always False.
LastModified TIMESTAMP Date and time the file was last modified.
TimeStamp CHARACTER Date and time the input node started processing the file in the coordinated universal time (UTC) zone, as a character string because this is the formatted form used to create archive and backout file names. This is used to timestamp the input file if you configure the Add Timestamp and Move to Archive Subdirectory and Add Timestamp and Move to Backout Subdirectory properties.
The following elements contain data about the detected record if present:
Offset INTEGER Start of the record with the file. The first record starts at offset 0.
Record INTEGER Number of the record within the file. The first record is record number 1.
Delimiter CHARACTER The characters to be used to separate the record from the subsequent record, if Delimited is specified in Record Detection. The delimiter is either the default of a line end sequence, or a character string that you have specified in Custom delimiter.

The FileInput node is contained in the File drawer of the message flow node palette, and is represented in the workbench by the following icon:

FileInput node icon

Using this node in a message flow

The FileInput node can be used in any message flow that needs to accept messages in files. Look at the following sample to see how to use this node:
  • FileInput sample
<<LINK TO FileInput SAMPLE>>

You can view samples only when you use the information center that is integrated with the Message Brokers Toolkit.

Configuring the FileInput node

When you have put an instance of the FileInput node into a message flow, you can configure it. The properties of the node are displayed in the Properties view. All mandatory properties for which you must enter a value (those that do not have a default value defined) are marked with an asterisk.

Configure the FileInput node:

  1. Optional: On the Description tab, enter a short description, a long description, or both. You can also rename the node on this tab.
  2. On the Basic tab, enter the directories and files processed by the FileInput node, together with what to do with any duplicate files encountered.
    1. In Input directory, specify the directory that the FileInput node will monitor. You must predefine this directory on the system which hosts the broker to which you deploy the flow. The name must be specified as an absolute path, for example, on Windows C:\fileinput, or on UNIX® var/fileinput.
    2. In File name or pattern, specify either a file name or a pattern that matches the name of the files you want the FileInput node to process. The pattern should be a valid file name on the system which hosts the broker to which the message flow is deployed. The file name can contain the following wildcard characters:
      Wildcard character Description Example
      * Indicates any sequence of zero or more characters *.xml matches all files with an xml extension
      ? Indicates any single character f??????.csv matches all files with the letter f followed by six characters and a .csv extension
    3. Select Action on successful processing to specify the action the FileInput node takes after successfully processing all the messages in the file. The action can be to move the file to the archive subdirectory, add time stamp and move the file to the archive subdirectory, or to delete the file.
      • If you select Move to Archive Subdirectory, the file will be moved to the archive subdirectory of the monitored directory. The subdirectory name is mqsiarchive. For example, if the monitored directory is /var/fileinput, the archive subdirectory's absolute path is /var/fileinput/mqsiarchive. If this directory does not exist, the broker will create it when it first tries to move a file.
      • If you select Add Timestamp and Move to Archive Subdirectory, the file will be time stamped with the current date and time, and then moved to mqsiarchive, as above.
      • If you select Delete, the file is deleted.
      The FileInput node writes a message to the user trace whenever it processes a file.
    4. Select Replace duplicate archive files if you want to overwrite existing archive files with the same name as the successfully processed input file. If you do not set this option, the node will throw an exception when it tries to move a successfully processed file with the same name into the archive subdirectory.
    5. Draft comment:
      <<info required for recordCodedCharSetIdProperty and recordEncodingProperty - these appear in gui>>recordCodedCharSetIdProperty and recordEncodingProperty
  3. On the Inbound Message Parsing tab, set values for the properties that the node uses to determine how to parse the incoming message.
    1. In Message domain, select the name of the parser that you are using from the list. You can choose from:
      • MRM
      • Start of changeSOAPEnd of change
      • XMLNS
      • Start of changeDataObjectEnd of change
      • XMLNSC
      • JMSMap
      • JMSStream
      • MIME
      • BLOB
      • IDOC
      • XML
    2. If you are using the MRM or IDOC parser, select the correct message set from the list in Message set. This list is populated with available message sets when you select MRM or IDOC as the domain.
    3. If you are using the MRM parser, select the correct message type from the list in Message type. This list is populated with available message types when you select the MRM parser.
    4. If you are using the MRM or IDOC parsers, select the correct message format from the list in Message format. This list is populated with available message formats when you select the MRM or IDOC parsers.
    5. Select the correct message coded character set ID from the list in Message coded character set ID. This list is populated with available message coded character set IDs when you select the parser.
    6. Select the message encoding from the list in Message encoding. For more information about encoding, see Converting data with message flows.
  4. Use the Parser Options tab to specify the timing to be used when parsing a message. The Parse Timing property determines whether on demand parsing is to be used when parsing a message. It also gives you control over the timing of MRM input message validation. The Parse Timing property has no effect on the validation of output messages.
    1. In Parse timing, select the timing you want to apply:
      • On Demand means that validation of a field in the message is delayed until it is parsed by partial parsing.
      • Immediate means that partial parsing is overridden, and everything in the message is parsed and validated.
      • Complete means that partial parsing is overridden, and everything is parsed and validated.
    2. Choose from the following XMLNSC options:
      • Select Use XMLNSC compact parser for XMLNS domain if you want to use less memory. Setting this property causes the outgoing MQRFH2 to specify the XMLNS instead of XMLNSC parser, allowing an external application to remain unchanged. If outgoing messages do not contain MQRFH2 headers, this property has no effect.
      • Select Retain mixed content if you need to preserve formatting in XML messages on output. This property controls whether the XMLNSC parser creates elements in the message tree when it encounters mixed text in an input message. If you select the check box, elements are created for mixed text. If you clear the check box, mixed text is ignored and no elements are created.
      • Select Retain comments if you want the XMLNSC parser to create elements in the message tree when it encounters comments in an input message. If you select the check box, elements are created for comments. If you clear the check box, comments are ignored and no elements are created.
      • Select Retain processing instructions if you want the XMLNSC parser to create elements in the message tree when it encounters processing instructions in an input message. If you select the check box, elements are created for processing instructions. If you clear the check box, processing instructions are ignored and no elements are created.
  5. On the Polling tab, enter the FileInput node's polling interval to specify how frequently the node scans the monitored directory. If you want the FileInput node to monitor the input directory only for selected periods of the day, you should start and stop polling by starting and stopping the message flow.

    A lower polling interval limit can be set on a message broker for this value. If the value on the node is below the lower limit set on the message broker, the value is raised to the broker limit. Set an execution group property on the broker with the following values:

    • Execution group level
    • Object name - Polling
    • Name - MinimumFrequency
    • Value - Time in seconds, default is 1 second

    The FileInput node will scan the input directory when the flow starts and thereafter according to the above interval.

    The value set for the polling interval here can be overridden by the Scan delay property on the FTP tab.

  6. Use the Retry tab to define how retry processing is performed when a flow fails. You can set the following values:
    1. Retry mode is how the node handles a flow failure. Choose from the following:
      • Select Failure for the node to report a failure.
      • Select Short retry then failure for the node to wait the number of seconds specified in the Short retry interval, and then if conditions persist, to report a failure.
      • Select Short retry then long retry for the node to wait the number of seconds specified in Short retry interval, and then if the condition persists, to wait the number of seconds specified in Long retry interval before reporting a failure.
    2. Specify the Short retry count. This is the number of times the system retries waiting the short retry interval and then retrying.
    3. Enter the Short retry interval. This is the length of time to wait between attempts until the number of retries exceeds the Retry threshold.
    4. Enter the Long retry interval. This is the length of time to wait between attempts until a message is successful, the message flow stopped, or the message flow is redeployed. There is a broker property, MinLongRetryInterval, that defines the minimum value that this can take. If the value is lower than the minimum then the broker value is used.
    5. Specify in Action on failing file, what the node does with the input file after all attempts to process its contents fail. Choose from the following:
      • Move to back out subdirectory
      • Delete
      • Add timestamp and move to backout subdirectory
  7. Use the Records and Elements tab to specify how to detect the boundaries of each record in the file, and what to propagate to the Out terminal:
    • In Record Detection:
      • If you want the node to process the whole file as a single record, select Whole File.
      • If you want to split the file's contents into fixed length records (except, perhaps, the final record which may be shorter), select Fixed Length. Specify the length in bytes in Length. This must be a value between zero and 100 MB. You can use fixed length record detection with the BLOB inbound message parser to process a large file in convenient amounts.
      • If the records you are processing are separated by a DOS or UNIX line end or delimiter character, select Delimited, specify the delimiter, and delimiter type.
      • If the file contains one or more structures that can be parsed by the parser specified in Message domain, select Parsed Record Sequence. The node propagates each matching structure as a separate message.
      • If you want the node to process a propagated record element, select Parsed Embedded Records and specify in Propagated record element the part of the record tree you want to process. The default is the body of the tree denoted by $Body.
    • If you specified Fixed Length in Record definition, use Length to specify the required length of the output record. This must be a value between 1 and 100 MB. The default is 80.
    • If you specified Delimited in Record definition, use Delimiter to specify the delimiter to be used. It is either the value that you specify in Custom delimiter or the default value. On UNIX systems, the default is a line feed character (<LF>, X'0A'). On Windows systems, the default is a carriage return character followed by a line feed character (<CR><LF>, X'0D0A').

      The FileInput node considers each occurrence of the delimiter in the input file as the start of a new record. If the first record of the file begins with a delimiter, the node treats the missing file contents preceding that delimiter as a record and propagates an empty record. The delimiter is never included in the propagated message.

    • In Custom delimiter, specify the delimiter character or characters to be used when the default value is not to be used. Specify this as an even-numbered string of hexadecimal digits. The default is X'0A' and the maximum length of the string is 16 bytes.
    • If you specified Delimited in Record definition, use Delimiter type to specify the type of delimiter. Permitted values are:
      • Infix. If you select this, and the final data in the file ends with a delimiter, the node generates an empty record to indicate the delimiter's presence.
      • Postfix. If you specify this, the node does not generate an empty record regardless of whether the last data in the file ends with a delimiter. This is the default.
    • If you specified Parsed Embedded Records in Record Detection, specify a value in Propagated record element. This is a Xpath or ESQL expression which identifies a node or set of nodes in the parsed file content. The FileInput node propagates each matching node as a separate record.

      The default is $Body denoting the whole custom-parsed message. This is equivalent to specifying Whole File on Record Detection.

  8. The broker provides validation based on the message dictionaries for predefined messages. For more information about validation, see Validating messages. For information on how to fill in this tab, see Validation tab properties.
  9. On the FTP tab, select the FTP check box if you want the node to read files from an FTP server using the File Transfer Protocol properties:
    • Specify the name of the configurable service in which the FTP server connection is configured, or the IP address of remote FTP server if no such resource.
    • Specify the client user identification, that is, the name of the user identification used to access the FTP server. If left blank, the system uses the identification defined in the server identification configurable service.
    • Specify the client user password. If left blank, the system uses the password associated with the client user identification.
    • Enter the directory in the FTP server.
    • Specify the transfer mode. Select Binary or Text.
    • Type the scan delay in seconds. The default is 60 seconds. The value set in this property overrides the value set for the polling interval on the Polling tab.
  10. On the Transactions tab, set the transaction mode. Select Yes if you want messages to be received under sync point. Select No if you do not want messages to be received under sync point. The default for this property is Yes.
  11. Optional: On the Instances tab, set values for the properties that control the additional instances that are available for a node. For more details, refer to Configurable message flow properties.

Terminals and properties

The FileInput node terminals are described in the following table.

Terminal Description
Failure The output terminal to which a message is routed if a failure is detected when the message is propagated to the Out flow (such as a message validation failure). Failures routed to this terminal include those caused by the retry processing occurring before the retry propagates the message to the Out flow.
Out The output terminal to which the message is routed if it has been successfully put to the output queue, and if further processing is required within this message flow. If no errors occur within the input node, a message received from an external resource is always sent to the Out terminal first.
Catch The output terminal to which the message is routed if an exception is thrown downstream and caught by this node.

The following tables describe the node properties. The column headed M indicates whether the property is mandatory (marked with an asterisk if you must enter a value when no default is defined); the column headed C indicates whether the property is configurable (you can change the value when you add the message flow to the bar file to deploy it).

The FileInput node Description properties are described in the following table:

Property M C Default Description
Node name No No The node type The name of the node.
Short Description No No None A brief description of the node.
Long Description No No None Text that describes the purpose of the node in the message flow.

The FileInput node Basic properties are described in the following table:

Property M C Default Description
Input directory Yes Yes None The full directory path scanned on each polling interval.
File name or pattern Yes Yes * A file name or simple string containing optional wildcard characters (* and ?) identifying files to process from the input directory.
Action on successful processing Yes No Move to archive subdirectory The action the node takes on the file after successfully processing the contents.
Replace duplicate archive files Yes No False This property controls whether the node replaces existing archive files with the same name as the input file.

The FileInput node Polling property is described in the following table:

Property M C Default Description
Polling Interval Yes Yes 5 The polling interval in seconds

The FileInput node Retry properties are described in the following table:

Property M C Default Description
Retry mode Yes No Failure How the node handles a flow failure.
Short retry count Yes Yes 0 The number of retries.
Short retry interval No Yes 0 The interval in seconds between each retry if the short retry count is not zero.
Long retry interval No Yes 300 The interval between each retry, if the mode is Short retry then long retry and the short retry count is exhausted.
Action on failing file Yes Ye Move to back-out subdirectory The action the node takes with the input file after all attempts to process its contents fail.

The FileInput node Records and Elements properties are described in the following table:

Property M C Default Description
Record detection Yes No Whole file The mechanism used to identify records in the input file.
Length Yes No 80 The length of records when Fixed length record detection is selected.
Delimiter Yes No Line end Record separator bytes when Delimited record detection is selected.
Custom delimiter No No   The delimiter in hexadecimal when Delimited record detection is selected.
Delimiter type Yes No Postfix The position of the delimiter when Delimited record detection is selected.
Propagated record element No No $Body The element to propagate as a record when Parse Embedded Records record detection is selected.

The FileInput node Validation properties are described in the following table:

Property M C Default Description
Validate Yes Yes None This property controls whether validation takes place. Valid values are None, Content and Value, or Content.
Failure action Yes No Exception This property controls what happens if validation fails. You can set this property only if you set Validate to Content or Content and Value. Valid values are User Trace, Local Error Log, Exception, and Exception List.

The FileInput node FTP properties are described in the following table:

Property M C Default Description
FTP No No Cleared This property defines whether the node uses the File Transfer Protocol (FTP) properties listed below and reads files from an FTP server.
Server identification Yes Yes None The name of the configurable service in which the FTP server connection is configured, or the IP address of remote FTP server if no such resource.
Client user identification No Yes None The name of the user identification used to access the FTP server. If left blank, the system uses the identification defined in the server identification configurable service.
Client user password No Yes None Password for the client user identification. If left blank, the system uses the password associated with the client user identification.
Server directory No Yes "." The directory in the FTP server.
Transfer mode No Yes Binary The mode for transfer of files. Select Binary or Text.
Scan delay No Yes 60 The time, in seconds, to wait before repeating the scan of the directory.

The FileInput node Transactions properties are described in the following table:

Property M C Default Description
Transaction mode No Yes Yes This property defines how the node controls transactions. When you include a FileInput node in a message flow, the value that you set for Transaction mode defines whether messages are sent under sync point. The FileInput node is not transactional, and does not itself process files under sync point. If you set this property to Yes (the default), the message is received under sync point (that is, within a WebSphere® MQ unit of work). Any messages subsequently sent by an output node in the same instance of the message flow are put under sync point, unless the output node has explicitly overridden this. If the File Input node backs-out the transaction, all changes made under sync point are backed-out. If you set this property to No, the message is not received under sync point. Any messages subsequently sent by an output node in the flow are not put under sync point, unless an individual output node has specified otherwise. Changes made under sync point can be backed-out if the FileInput node backs-out the transaction. When FTP is enabled, the file transfer is performed outside of sync point, and no attempt is made to restore any file or files that have been transferred, whether the message flow is successful or not.
The FileInput node Instances properties are described in the following table. For a full description of these properties, refer to Configurable message flow properties.
Property M C Default Description
Additional instances pool No Yes Use Pool Associated with Message Flow The pool from which additional instances are obtained.
  • If you select Use Pool Associated with Message Flow, additional instances are obtained from the message flow value.
  • If you select Use Pool Associated with Node, additional instances are allocated from the node's additional instances based on the number specified in the Additional instances property.
Additional instances No Yes 0 The number of additional instances that the node can start if the Additional instances pool property is set to Use Pool Associated with Node. By default, no additional instances are given to the node.
Notices | Trademarks | Downloads | Library | Support | Feedback

Copyright IBM Corporation 1999, 2007Copyright IBM Corporation 1999, 2007. All Rights Reserved.
This build: July 31, 2007 21:21:05

ac55150_.dita This topic's URL is:
End of change