This section describes problems that can occur during the import phase of synchronization.
A recoverable error occurs when an update packet is lost and is not applied to your replica. These are the symptoms:
To verify that a packet is missing and determine which operations are needed:
multitool syncreplica –import packet-pathname Sync. packet packet-pathname was not applied to VOB ... - packet depends on changes not yet received Packet requires changes up to 872; VOB has only 756 from replica: sanfran_hub Packet requires changes up to 605; VOB has only 500 from replica: bangalore
In this example, one or more update packets are missing, containing operations 757–872 originally occurring at replica sanfran_hub and operations 501-605 from bangalore. In general, a packet can contain operations from several replicas; the syncreplica –import command fails if operations are missing from any replica.
Locate the missing packets. They may be on media that you forgot to process or in packet files that were not processed because the shipping.conf file on UNIX or the MultiSite Control Panel on Windows specifies the wrong storage bay. If you locate the missing packets, do one of the following things:
If you cannot locate the missing packets, see Recovering from Lost Packets.
Import can fail with the following message:
This error can occur when a replica has been moved and the host-name property has not been updated with the chreplica command.
cleartool describe –fmt "%[replica_host]p\n" replica:importing-replica-name@VOB-tag
For example:
cleartool describe –fmt "%[replica_host]p\n" replica:newyork@/vobs/tests manhattan
If the host name is incorrect, use the chreplica command to change it. At the master replica of the importing replica, enter a chreplica command:
For example:
multitool chreplica –c "change host name" –host brooklyn replica:newyork@/vobs/tests Updated replica information for "newyork".
Send an update packet to the other replicas in the family.
If a syncreplica –import command fails with a message like this one, the packet is corrupted:
Delete the packet and ask the administrator at the sending replica to re-create the packet and send it again (see Recovering from Lost Packets). Then import it.
If a syncreplica –import command fails with one of the following messages, restart the import:
The messages report that multitool was trying to import an operation for an element while another process (for example, a developer using cleartool) was operating on the same element.
If possible, run syncreplica –import from within a view. If it fails again, you see more information about what element it is failing on, and you can look through output from the lshistory command to try to find the conflict.
Import of an rmreplica operation fails if the importing replica records that the removed replica still masters objects. The import fails with an error like the following:
multitool: Error: There are still objects mastered by this replica. multitool: Error: Unable to replay oplog entry 565632: error detected by ClearCase subsystem. 565632: 12 op= rmreplica 13 replica_oid= 48abc01d.123456a7.b890.06:00:08:c4:73:84 (boston_hub.mstr) 14 oplog_id= 23456 15 op_time= 08/07/02 12:35:46 create_time= 08/07/02 12:35:46 16 event comment= "Destroyed replica "boston_hub".
This situation can occur if two replica hosts do not have the same patch level or if an upgrade had problems.
You can use the lsmaster command to determine which objects are believed to be mastered by the removed replica. In this example, the administrator at importing replica sanfran_hub uses the lsmaster command to list the objects replica sanfran_hub believes to be mastered by replica boston_hub:
multitool lsmaster –view admin_view boston_hub@/vobs/dev master replica: boston_hub@/vobs/dev "label type" V2.0 master replica: boston_hub@/vobs/dev "label type" V1.1
In this example, the administrator at replica sanfran_hub uses the lsmaster command to contact all replicas in the VOB family and list the objects they believe to be mastered by replica boston_hub:
multitool lsmaster –view admin_view –inreplicas –all boston_hub@/vobs/dev In replica "bangalore" master replica: boston_hub@/vobs/dev "label type" V2.0 In replica "sanfran_hub" master replica: boston_hub@/vobs/dev "label type" V2.0 master replica: boston_hub@/vobs/dev "label type" V1.1
To resolve this problem, contact Rational Customer Support.
ClearCase versions 4.0 and later include support for a new VOB database schema. If you update one or more replicated VOBs in a family to the new schema (version 54), you do not have to update the other replicas in the VOB family immediately. However, you must update all replicas before one of the updated replicas exceeds the database limit of the previous schema (version 53). If you do not, replicas that have not been updated cannot import synchronization update packets from the updated replica.
When this type of import failure occurs, syncreplica output includes a VOB database error, and an error is written to the db log.
The syncreplica output includes an error like the following:
The db log includes an error like the following:
09/20/96 10:40:49 db_server(19528): Error: DBMS error in "../db__lock.c" line 79 *** db_VISTA database error -909 - file record limit exceeded 09/20/96 10:40:49 db_server(19528): Error: DBMS error 09/20/96 10:40:49 db_server(19528): Error: db_VISTA error -909
To fix this problem, you must convert all replicas in the family to schema version 54. To display the schema version for a VOB replica, use the cleartool describe vob:vob-tag command. To display the schema version of the version of ClearCase installed on your computer, use the cleartool –ver command.
The following error can occur during packet import:
The replica incarnation is the last time the replica was restored (with the restorereplica command). The incarnation is set to 0 when the replica is created and remains 0 until a restoration occurs.
Each replica keeps a record of the incarnation of each other replica in its family. During packet export, the incarnations of the target replicas are recorded in the packet. The syncreplica –import command at the importing replica checks the incarnation in the packet. If the incarnation in the packet is earlier than the importing replica’s own record of its incarnation, the packet is not imported.
If the incarnations are different, the exporting replica does not have a record of the importing replica undergoing restoration. This situation may occur for the following reasons:
Replicas A and B synchronize every day, replicas B and C synchronize once a week, and replicas A and C synchronize once a month.
Replica A is restored from backup and the administrator runs restorereplica. Because replica A’s last synchronization was with replica B, the administrator optimizes the process to require an update packet only from replica B. After the packet is received from replica B, the restoration is complete and replica A resumes normal synchronization.
Because neither replica A nor replica B synchronized with replica C during the restoration process, replica C does not have any information about the restoration, and its record of replica A’s incarnation is not updated.
The next time replica C sends an export packet to replica A, the incarnation in the packet is earlier than replica A’s actual incarnation, and the import fails.
To determine which reason applies to your situation:
cleartool dump replica:name-of-importing-replica@VOB-tag
In the output, look for a line beginning with incarnation=
. This line displays the incarnation time. For example:
cleartool dump replica:boston_hub@/vobs/dev ... incarnation=01-Apr-02.22:40:54UTC ...
Different versions of MultiSite have different packet protocols. When multitool with a newer protocol reads a packet with the older protocol, it prints this message:
This message does not indicate a problem. It means one of the following things:
Table 18 lists the packet protocols for MultiSite versions.
MultiSite version
|
Packet protocol
|
---|---|
3.2, 3.2.1
|
1.2
|
4.0, 4.1, 4.2
|
3
|
2002.05.00
|
4
|
2003.06.00
|
5
|
Processing of an incoming replica-creation or update packet may fail because of these conditions:
Make sure that multiple syncreplica –import commands do not run in the same replica simultaneously. Check the timing of schedule tasks, and adjust them if necessary. (An invocation of the sync_receive script fails if another sync_receive process is running.)
In such cases, fix the problem and reenter the syncreplica –import command.
There are several circumstances in which a replica-creation or update packet is generated but is never applied at one or more of its destinations:
To recover a lost replica-creation packet:
cleartool rename bangalore@/vobs/dev bangalore-old@/vobs/dev Renamed replica from "bangalore" to "bangalore-old".
multitool rmreplica bangalore-old@/vobs/dev Deleted replica "bangalore-old".
The following procedure is simpler, but the rmreplica command may take a long time if you have a large VOB:
The syncreplica –export command assumes successful delivery of the update packet it generates. For example, when replica boston_hub sends an update to replica sanfran_hub, the syncreplica command assumes that the operations originating at boston_hub are imported to the sanfran_hub replica. For simplicity, this example does not reflect the fact that the update packet can also contain operations that originated at other replicas in the VOB family.
But, if the packet is lost, this assumption is invalid, and boston_hub must reset its estimate of the state of replica sanfran_hub. After this correction is made, the next update packet sent from boston_hub to sanfran_hub contains the operations sanfran_hub needs.
To reset the epoch row, use one of the methods described here.
For example, use one of the following commands:
/opt/rational/clearcase/config/scheduler/tasks/sync_export_list –update –replicas sanfran_hub@/vobs/dev
multitool chepoch –actual sanfran_hub@/vobs/dev Entry for bangalore changed from: 985 to 950 Entry for boston_hub changed from: 1400 to 1300 Entry for sanfran_hub changed from: 2562 to 2000
multitool lsepoch sanfran_hub@/vobs/dev For VOB replica "/vobs/dev": Oplog IDs for row "sanfran_hub" (@ goldengate): oid:7ag3b0bc.defa11d0.ba57.00:01:72:73:3c:94=950 (bangalore) oid:87f6265f.72d911d4.a5cd.00:01:80:c0:4b:e7=1300 (boston_hub) oid:0eaa6fc3.737d11d4.adbe.00:01:80:c0:4b:e7=2000 (sanfran_hub)
multitool chepoch sanfran_hub bangalore=950 boston_hub=1300 sanfran_hub=2000 Change oplog IDs in row "sanfran_hub" [no] yes Epoch row successfully set.
cleartool lshistory –long replica:sanfran_hub 30-Jul.14:42:50 Susan Goechs (susan.user@minuteman) export sync from replica “boston_hub” to replica “sanfran_hub” “Exported synchronization information for replica “sanfran_hub”. Row at export was: bangalore=950 boston_hub=1300 sanfran_hub=2000” 23-Jul.17:36:46 Susan Goechs (susan.user@minuteman) export sync from replica “boston_hub” to replica “sanfran_hub” “Exported synchronization information for replica “sanfran_hub”. Row at export was: bangalore=900 boston_hub=800 sanfran_hub=1500” ...
multitool chepoch sanfran_hub bangalore=950 boston_hub=1300 sanfran_hub=2000 Change oplog IDs in row "sanfran_hub" [no] yes Epoch row successfully set.
GOLDENGATE> cleartool lshistory replica:sanfran_hub 01-Aug.07:08 jcole import sync from replica “boston_hub” to replica “sanfran_hub” “Imported synchronization information from replica “boston_hub”. Row at import was: sanfran_hub=2000 boston_hub=1300 bangalore=950” ...
susan@minuteman% multitool recoverpacket –since 01-Aug.01:00 sanfran_hub
Note: With this method, you must adjust the time from the lshistory output for time zone differences and the amount of time elapsed between export and import.
If there are no saved epoch rows for the receiving replica that are as old as the time specified, you must use one of the chepoch procedures.
A recoverable error occurs if syncreplica –import detects that an incoming change is inconsistent with another change that has already been applied to the replica.
Note: In some cases, an inconsistency is resolved by syncreplica –import. For example, a replica receives an update that deletes an element, and then receives an update from another replica that creates a new version on a branch of that element. The create-version operation in the second update is discarded because the element no longer exists.
If two replicas preserve identities and permissions or permissions only, the OS-level permissions of their individual elements are synchronized. However, synchronizing the VOB group lists of the replicas is a manual task that you must perform using cleartool protectvob –add_group.
syncreplica –import generates the following identities-related error messages:
Can't create object with group that is not in the VOB's group list Can’t change to a group that is not in the VOB’s group list
These messages indicate that a group was added to the sending replica’s VOB group list, and someone created a new element in that group or reassigned an existing element to that group. Then, the change was sent to a replica whose VOB group list has not been updated.
These messages may also indicate that the sending replica and/or receiving replica were created incorrectly as identities and permissions preserving.
If the replicas are intended to be identities and permissions preserving, follow these steps to recover from this kind of error:
elem_fstat= ino: 0; type: 2; mode: 0444; uid: 1037; gid: 20 . . name_p= "aux_util.c" nsdir_ver_oid= ed2549e2.97f411cd.b3c8.08:00:69:06:4d:f6 (/vobs/dev/src@@/main/ev2/CHECKEDOUT.572)
These lines indicate that the element’s pathname in the sending replica is /vobs/dev/src/aux_util.c. Note also that its group ID (GID) is 20
.
cleartool protectvob –add_group 20 /vobstg/dev.vbs
Note: If the administrators at the sites of identities- and permissions-preserving replicas have not informed one another of changes in the shared user/group namespace, you may need to adjust the password and group databases before entering the protectvob command.
If one or both of the replicas should not be identities and permissions preserving, follow these steps:
multitool chreplica –npreserve boston_hub@/vobs/dev Updated replica information for "boston_hub".
multitool syncreplica –import –receive Applied sync. packet /opt/rational/clearcase/shipping/ms_ship/incoming/sync_sanfran_hub_ 18-Jan-02.16.54.14_386_1 to VOB /net/minuteman/vobstg/dev.vbs
To avoid this problem in the future, use the procedure described in the section Gathering Identities Information.
An object mastered by one replica can depend on an object mastered by another replica. For example, an element and one of its branches are dependent objects, but these objects can be mastered by different replicas. As a result, certain kinds of inconsistent changes can be made at different replicas. The inconsistency is detected by syncreplica –import, causing it to fail with a recoverable error.
For example, if a type object is deleted in another replica, your replica may refuse to import this change because a trigger type in your replica depends on the deleted type object. During import, the following error message is displayed:
The syncreplica –import command resolves naming conflicts among type objects or replica objects created at two or more replicas. For example, a branch type object named v1.0_bugfix is created at two different replicas. At some point, an invocation of syncreplica –import detects the conflict. (This may occur at one of the replicas that created the branch types, or at some other replica.)
syncreplica –import resolves the conflict by renaming the incoming object. In this example, branch type v1.0_bugfix is renamed to boston_hub:v1.0_bugfix, indicating that boston_hub was the replica at which the type was created. syncreplica –import displays the following message:
Intervention is not required at this point unless branch types or replicas are renamed. (Renaming of branch types affects config specs, and renaming of replicas affects synchronization scripts.) However, if you do not rename the objects, different replicas have different names for the same object. In this example, the boston_hub replica calls a branch type v1.0_bugfix, but at least one other replica calls the same type object boston_hub:v1.0_bugfix.
The administrators of the various replicas involved in such a conflict must coordinate the renaming of all the objects involved, to guarantee that all objects have the same name in all replicas. Here is a general procedure:
Note: The name that caused the original conflict can be reused. One replica (and only one) can change the name to its original value:
cleartool rename brtype:boston_hub:v1.0_bugfix v1.0_bugfix
When this change is propagated to other replicas, it undoes any previous conflict-avoidance name changes, for example, by renaming boston_hub:v1.0_bugfix to v1.0_bugfix. (The propagation of this change must wait until after the other rename commands have been run in the other replicas and propagated throughout the VOB family, to make the name v1.0_bugfix available again.)