*
Metamerge logo
Search

Advanced Search
*
*
*
* HOME DOCUMENTS & RESOURCES DOWNLOADS EARLY TECH ACCESS SUPPORT FAQ KNOWN ISSUES OLD VERSIONS
*

Deltas and Compute Changes

Deltas

Warning

The Delta mechanism should be avoided if you can.  Not because it does not work or is buggy, but because it introduces a new local repository in order to manage the synchronization process.  The data source which is being scanned for changes becomes the master in a master-slave relationship, and it is then vital that all changes made to the slave be done such that the Delta mechanism is informed of them. Otherwise, the temporary Delta database which Integrator maintains will become inconsistent and the Delta function will fail. 

However, there are some situations where you will either want to integrate as non-intrusively as possible, or be forced to;  For example, with a legacy application that cannot be interfaced directly. In cases like this you may be forced to read a change log generated by the target system, or even a periodic dump of all registered entries. Here is were the Delta feature can used so that Integrator can determine which Entries are new, modified or deleted.

Whenever possible, use the Compute Changes functionality of the Update connector mode instead of the Delta feature.

Overview

Deltas are used to synchronize a slave data source with a master.  The Delta mechanism will know whether Entries or Attributes have been added, changed or deleted in the master and replicate the changes to the slave.  However, if the slave is changed by anyone but the Delta mechanism, the Delta mechanism will never discover it.

You can set Delta settings on Iterator Connectors .  The effect of this is that the first time the Connector iterates through the data, it records each Entry in a local high-speed B-tree database. Then, on successive runs, the Connector will look up each Entry read to see if it has been changed (requiring a modify), is new (resulting in an add), or has gone missing from the input data source (signalling a delete in the slave repository). 

Note that this functionality could be implemented by hand by creating a separate Lookup Connector for any database of choice, and using it to check whether add, modify or delete was necessary.  The Integrator Delta feature simply wraps all this into a checkbox.

Process workflow

The Delta functionality uses a local B-tree database for storing all Entries handled by the Iterator Connector during its runs. During a run of the AssemblyLine each Entry is compared to its record in the local B-tree database (if it exists). Thus each Entry is tracked for existence and changes in its Attributes' values.

All the Entry's Attributes are stored in the local B-tree database. One of the Entry's Attributes is selected as the unique key Attribute - this is configured in MI Admin, Connector's "Configure Delta" dialog. Note that the value of the key Attribute must be unique for each Entry. This Attribute is used to retrieve the local copy of the Entry retrieved by the Iterator Connector in order to determine its state - unchanged, new, modified or deleted. This state is set in the Entry,  and the Entry is delivered to the other Connectors in the AssemblyLine. You can get an Entry's state from within your scripts through the "getOperation()" Entry method (in an Output Connector you will actually have to filter Entries according their state and this means you have to examine the value of "work.getOperation() ").

Below is the sequence of actions performed by MI to determine the state of each Entry obtained from the data source:

1. The Iterator Connector retrieves the Entry from the data source.
2. A search is performed in the local B-tree database for the Entry, based on the specified key Attribute.
    2.1. If no Entry is found in the B-tree database, the Entry processed is marked as new and the "getOperation()" method will return the value of "add".
    2.2. If an Entry with the same key Attribute value is found then:
        2.2.1. If any of the Attributes' values have changed then the Entry processed is marked as modified, and the "getOperation()" method will return the value of "modify".
        2.2.2. If none of the Attributes' values have changed, the Entry is skipped, i.e. not passed to the other Connectors in the AssemblyLine.

If you want the Delta mechanism to process the Entries that have been deleted then you have to turn on the "Read Deleted" option in the Connector's "Configure Delta" dialog. With this option turned on, the following processing occurs after the Iterator Connector has read all Entries from the data source:

1. The local B-tree database is scanned. 
2. Each Entry present in the B-tree database and not retrieved by the Iterator Connector is marked for deletion. This means the "getOperation() " method will return the value of "delete" for these Entries.

Another important Delta option concerning the processing of the deleted Entries is "Remove Deleted". When this option is turned on the deleted Entries, once determined, are removed from the local B-tree database. When the option is turned off, these Entries stay in the B-tree database and will be processed for deletion in further executions of the AssemblyLine.  If you delete Entries in the target system, you will typically want to check "Remove Deleted" so the Entries disappear from the B-tree as well.

How it works

In the B-tree every Entry will get a counter as it is iterated through. This is an internally generated number, which is incremented for each run of the AssemblyLine. For the sake of this explanation, we'll say that it has a value of n. Next time the Iterator connects to the source and starts to iterate through the data again, one of the following three operations takes place:

  1. If the Entry already exists in the B-tree database, then we can check whether its Attributes have changed (or been added/deleted).  The generation number of the Entry in the B-tree store is incremented to n+1.
  2. If the Entry did not exist, then it is added to the B-tree database with generation number n+1
  3. If, after Connector iteration is over, there still exist Entries in the B-tree database with generation number n or less, those Entries have disappeared from the original source.  These Entries are returned one at time by the Iterator if the "Read Deleted" checkbox was ticked.
Example 

A ready to run MI configuration that demonstrates the use of Deltas is included here. Note, however that this demo uses a MS Access database, so in order to run it you must have access to ODBC/Access.

Computed Changes

Computed Changes option is a switch on Update Connectors. It is used to make sure an existing record has changed before the Connector tries to update it. Description of how is this option used can be found in the Connectors concept page, in its "Update mode" section. 

 

*
  Metamerge Integrator version 4.5 ©Copyright Metamerge AS 2000-2002 Last edited 2002-04-30 contact us