The Operation Log


This section describes the mechanism that supports synchronization. This information is not required to use MultiSite, but is helpful when you want to deepen your understanding of the error-recovery facilities described in Troubleshooting MultiSite Operations.

Most changes made to a VOB are recorded as event records in the VOB database. Each event record describes a change. Most changes to a replicated VOB are recorded as entries in an operation log (oplog). These entries store all the information required to replay the changes in another replica:

The exact kind and amount of information varies with the specific operation. For example, an oplog entry for the removal of a label has different, and less, information than an oplog entry for a checkout command.

Note: You can delete a replica’s oplog entries after they have been used to update other replicas. For more information, see Scrubbing Parameters for Replicas.

Tracking Operations for Each Replica

The history of an unreplicated VOB is a linear sequence of operations (Figure 4).

Figure 4 History of Changes to a Database

Within a replica family, changes are tracked for each replica. This is why an oplog entry includes the identity of the replica where the operation originated. Thus, the history of a replica family can be viewed as several stacks of oplog entries. Each stack is represented by a linear sequence of oplog IDs for the operations that originate in that replica.

Figure 5 shows the state of two replicas in a family:

Figure 5 State of a Family

A replica has accurate data only about its own operations. Until it receives update packets, its information about other replicas is out of date. For example, replica boston_hub records 950 local operations, but has received update packets for only 504 sanfran_hub operations. Similarly, replica sanfran_hub records 702 local operations, but has received update packets for only 791 boston_hub operations.

Figure 6 illustrates this scenario, in which each replica is out of date with respect to the operations originating at the other replica.

Figure 6 Out-of-Date Replicas

Picturing a replica family as a set of oplog stacks, shown in Figure 6, makes it easy to understand the synchronization process. For example, an update packet sent from replica boston_hub to replica sanfran_hub consists of increments to the stack for replica boston_hub (operations 792–950). Figure 7 shows the two increments. Because sanfran_hub knows its own state, it needs updates only for operations originating at other replicas. (In certain error-recovery situations, you must reset a replica’s data about its own operations. See Troubleshooting MultiSite Operations.)

Figure 7 Updates Between Two Replicas

Note: By the time the packet is imported at sanfran_hub, additional changes may have been made at boston_hub. Those changes are not included in the update packet.

Oplog IDs and Epoch Numbers

An epoch number is the total number of operations that originated at a particular replica. In Figure 5, the epoch number for boston_hub is 950.

The MultiSite synchronization scheme attempts to minimize the amount of data transmitted among replicas. Each replica keeps track of these epoch numbers:

Table 3 shows how these epoch numbers fall into an epoch number matrix. Each replica maintains its own such matrix, revising its rows as work occurs locally and as it exchanges update packets with other replicas:

The contents of this matrix are reported by the lsepoch command at the boston_hub replica:

multitool lsepoch 
For VOB replica "/vobs/dev":
Oplog IDs for row "boston_hub" (@ minuteman):
 oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950    (boston_hub) 
 oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504    (sanfran_hub) 
Oplog IDs for row "sanfran_hub" (@ goldengate):
 oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=912    (boston_hub) 
 oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504    (sanfran_hub) 

A syncreplica –export command entered at boston_hub uses this matrix as follows to generate an update destined for sanfran_hub:

  1. At the boston_hub replica, the number of local operations is 950 (number in upper left corner of matrix), and the estimate is that the sanfran_hub replica has imported all operations through oplog ID 912 (number in lower left corner).
  2. The update packet that the boston_hub replica sends to the sanfran_hub replica includes boston_hub oplog entries 913-950. After the Boston administrator invokes syncreplica –export, the sanfran_hub row is updated:
  3. multitool lsepoch 
    For VOB replica "/vobs/dev":
    Oplog IDs for row "boston_hub" (@ minuteman):
     oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950    (boston_hub) 
     oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504    (sanfran_hub) 
    Oplog IDs for row "sanfran_hub" (@ goldengate):
     oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950    (boston_hub) 
     oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504    (sanfran_hub) 

Indirect Synchronization

If a family includes more than two replicas, synchronization can occur indirectly. A replica can include nonlocal changes in update packets. For example, if the boston_hub replica exchanges updates with the sanfran_hub and bangalore replicas, it sends bangalore oplog entries that it has received previously from sanfran_hub. These entries may or may not bring replica bangalore up to date on sanfran_hub’s changes. (An update sent from sanfran_hub to bangalore does bring bangalore up to date.)

Note: If a replica does not receive packets directly from some replicas in its family, its rows for those replicas may contain zeros. This is expected behavior.

Table 4 shows replica boston_hub’s epoch number matrix.

Table 4 Three-Row Epoch Number Matrix at Replica boston_hub
 
Operations originated at boston_hub
Operations originated at bangalore
Operations originated at sanfran_hub
boston_hub’s record of its own state
950
653
504
boston_hub’s estimate of sanfran_hub’s state
912
653
504
boston_hub’s estimate of bangalore’s state
709
653
221

The contents of this matrix are reported by the lsepoch command:

multitool lsepoch 
For VOB replica "/vobs/dev":
Oplog IDs for row "boston_hub" (@ minuteman):
 oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950    (boston_hub) 
 oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653    (bangalore) 
 oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504    (sanfran_hub) 
Oplog IDs for row "bangalore" (@ ramohalli):
 oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=709    (boston_hub) 
 oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653    (bangalore) 
 oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=221    (sanfran_hub) 
Oplog IDs for row "sanfran_hub" (@ goldengate):
 oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=912    (boston_hub) 
 oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653    (bangalore) 
 oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504    (sanfran_hub) 

A syncreplica –export command at the Boston site uses this matrix to export an update for the bangalore replica:

  1. At the boston_hub replica, there are 950 local operations (number in upper left corner of matrix), and the estimate is that the bangalore replica has imported all operations through oplog ID 709 (lower left corner).
  2. For operations that originated at the sanfran_hub replica, boston_hub has imported all operations up to oplog ID 504 and estimates that bangalore has imported all operations through oplog ID 221.
  3. The update packet that boston_hub sends to bangalore includes boston_hub operations 710-950 and sanfran_hub operations 222-504. The output of an lsepoch command at the boston_hub replica now looks like this:
  4. multitool lsepoch 
    For VOB replica "/vobs/dev":
    Oplog IDs for row "boston_hub" (@ minuteman):
     oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950    (boston_hub) 
     oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653    (bangalore) 
     oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504    (sanfran_hub) 
    Oplog IDs for row "bangalore" (@ sushi):
     oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=950    (boston_hub) 
     oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653    (bangalore) 
     oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504    (sanfran_hub) 
    Oplog IDs for row "sanfran_hub" (@ goldengate):
     oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=912    (boston_hub) 
     oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=653    (bangalore) 
     oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=504    (sanfran_hub)