Simultaneous dump capture from multiple CICS regions across a sysplex is possible, considerably aiding problem determination in XCF/MRO environments where many CICS regions are running. Two types of situation, in particular, benefit from this:
Dump data from all CICS regions related to the region issuing the dump request is usually required in order to fully diagnose and solve the problem. This dump data from related regions would also need to be captured at the same time as the dump taken on the region issuing the dump.
Before CICS® for MVS/ESA 4.1, such automatic and simultaneous dump data capture of CICS data was impossible.
You need MVS/ESA 5.1, the MVS workload manager, and the XCF facility in order to collect dump data in this way. The MVS images in the sysplex must be connected via XCF. The CICS regions must be using MRO supported by the CICS Transaction Server for z/OS®, Version 3 Release 1 interregion communication program, DFHIRP.
It is possible to collect dump data simultaneously from all related CICS regions in a sysplex. Related CICS regions are those containing one or more tasks which have unit of work identifiers, in the form of APPC tokens, that match the unit of work identifiers in the CICS region which initially issued the dump request.
The CICS regions must be connected via XCF/MRO. Connections using VTAM® ISC are not eligible to use the related dump facility.
The function is controlled by the DUMPSCOPE option on each CICS dump table entry. You can set this option to have either of the following values:
The DUMPSCOPE option is available on the following master terminal and system programming commands:
If the DUMPSCOPE option is set to RELATED in the CICS region issuing the dump request, a request for a system dump is sent to all MVS images in the sysplex that run related CICS regions.
The local MVS image running the CICS region that initiated the dump request has two dumps - one of the originating CICS region, the other containing the originating CICS region and up to fourteen additional related CICS regions from the local MVS image.
When a dump is requested, the DUMPSCOPE option is tested only in the CICS region issuing the original dump request. If the requesting CICS region has DUMPSCOPE defined as RELATED for the dump code, then all related CICS regions are dumped even if they have DUMPSCOPE defined as LOCAL for the dump code.
There is a maximum of fifteen address spaces in an SDUMP. If there are more than fifteen related CICS regions on an MVS image, then not all of them will be dumped. Related CICS regions may also fail to be dumped if they are swapped out when the dump request is issued. You should consider whether to make certain CICS regions non-swappable as a result.
The MVS console operator may want to issue a dump request simultaneously to several CICS regions in the sysplex. There may be a problem in the sysplex where one or more CICS regions is hanging and, to fully diagnose and solve the problem, dump data captured at the same time is required.
Without this facility, such simultaneous dump data capture across multiple CICS regions in the sysplex is impossible.
Use the following command at the console:
DUMP COMM=()
R x,REMOTE=(SYSLIST=*),PROBDESC=(SYSDCOND,SYSDLOCL,(DFHJOBN,jobnames))
where:
See the MVS System Commands manual, GC28-1626, for a full description of all command options.
If you adopt a suitable naming convention for your CICS regions, this can be used to define suitable generic jobnames to determine which CICS regions to dump. See the System/390 MVS Sysplex Application Migration manual for recommendations on naming conventions. If you follow the recommendation in this manual, the generic job name for all CICS regions in the sysplex would be ‘CICS*’.
You may sometimes encounter a CICS problem that involves SMSVSAM. If you need to submit such a problem to IBM®, in particular where CICS or a CICS transaction is in a hung state while accessing VSAM data sets in RLS mode, include a dump of the following:
You can obtain such a dump by using the following command:
/DUMP COMM=(comment to describe the problem)
When MVS responds, reply to WTOR number nn with the following:
/R nn,JOBNAME=(CICS-job-name,SMSVSAM,CATALOG,GRS),DSPNAME='SMSVSAM'.*,
REMOTE=(SYSLIST=*('SMSVSAM','CATALOG','GRS'),DSPNAME,SDATA,
END
/R nn,JOBNAME=(CICS-job-name,SMSVSAM,CATALOG,GRS), CONT
/R nn,DSPNAME='SMSVSAM'.*,REMOTE=(SYSLIST=*('SMSVSAM', CONT
/R nn,'CATALOG','GRS'),DSPNAME,SDATA, END
MVS support for remote SDUMPs is available only for images running MVS/ESA 5.1 or later and which are connected by XCF. Related CICS SDUMPs are produced only for CICS regions which are MRO connected using XCF. DFHIRP must be at CICS Transaction Server for z/OS, Version 3 Release 1 level. Connections using VTAM ISC are not eligible to use the related dump facility.
If you are unable to produce related system dumps when there are related CICS regions across MVS images, ensure that the regions are MRO connected.
Use CEMT I IRC to ensure that interregion communication is available. If IRC is not available it may be started using the CEMT S IRC OPEN command. Failure to start IRC results in DFHIRxxx messages which may be used to identify the source of the problem.
During IRC start processing, CICS attempts to join XCF group DFHIR000. If this fails, return code yyy is given in the DFHIR3777 message.
The following MVS console commands may be used to monitor activity in the sysplex:
An example of a response to this command looks like this:
08.14.16 DEV5 d xcf
08.14.16 DEV5 IXC334I 08.14.16 DISPLAY XCF 602
SYSPLEX DEVPLEX5: DEV5
This response tells you that MVS image DEV5 has joined sysplex DEVPLEX5.
16.21.36 DEV5 d xcf,group
16.21.36 DEV5 IXC331I 16.21.36 DISPLAY XCF 877
GROUPS(SIZE): COFVLFNO(1) SYSDAE(2) SYSGRS(1)
SYSIGW00(1) SYSIGW01(1) SYSIKJBC(1)
SYSMCS(6) SYSMCS2(3) SYSWLM(1)
WINMVSG(1)
16.30.45 DEV5 d xcf,couple
16.30.45 DEV5 IXC357I 16.30.45 DISPLAY XCF 883
SYSTEM DEV5 DATA
INTERVAL OPNOTIFY MAXMSG CLEANUP RETRY CLASSLEN
42 45 500 60 10 956
SSUM ACTION SSUM INTERVAL WEIGHT
N/A N/A N/A
SYSPLEX COUPLE DATA SETS
PRIMARY DSN: SYS1.XCFDEV5.PXCF
VOLSER: SYS001 DEVN: 0F0E
FORMAT TOD MAXSYSTEM MAXGROUP(PEAK) MAXMEMBER(PEAK)
08/16/93 08:05:39 8 10 (10) 87 (6)
ALTERNATE DSN: SYS1.XCFDEV5.AXCF
VOLSER: SYS001 DEVN: 0F0E
FORMAT TOD MAXSYSTEM MAXGROUP MAXMEMBER
08/16/93 08:05:41 8 10 87
Use the CICS master terminal command CEMT I CONNECTION to display the status of the connections. ‘XCF’ is displayed for every acquired connection using MRO/XCF for communications.
I CONNECTION
STATUS: RESULTS - OVERTYPE TO MODIFY
Con(FORD) Net(IYAHZCES) Ins Acq Xcf
Con(F100) Net(IYAHZCEC) Ins Acq Irc
Con(F150) Net(IYAHZCED) Ins Acq Irc
Con(GEO ) Net(IYAHZCEG) Ins Acq Xcf
Con(GMC ) Net(IYAHZCEB) Ins Acq Xcf
Con(JIM ) Net(IYAHZCEJ) Ins Acq Xcf
Con(MARY) Net(IYAHZCEM) Ins Acq Xcf
Con(MIKE) Net(IYAHZCEI) Ins Acq Xcf
+ Con(RAMB) Net(IYAHZCEE) Ins Acq Xcf
SYSID=CHEV APPLID=IYAHZCET
RESPONSE: NORMAL TIME: 01.28.59 DATE: 06.11.94
PF 1 HELP 3 END 7 SBH 8 SFH 9 MSG 10 SB 11 SF
CICS initialization issues an MVS CSVDYNEX request to add DFHDUMPX as an MVS IEASDUMP.QUERY exit. If you need to change the status of the exit, use the MVS console command SETPROG EXIT.
If you have determined that XCF communication is in use, you can verify that the CICS SDUMP exit has been established using the following MVS commands:
D PROG,EXIT,MODNAME=DFHDUMPX
08.16.04 DEV5 CSV463I MODULE DFHDUMPX IS NOT ASSOCIATED ANY EXIT
D PROG,EXIT,EN=IEASDUMP.QUERY
08.17.44 DEV5 CSV463I NO MODULES ARE ASSOCIATED WITH EXIT IEASDUMP.QUERY
This example indicates that the exit has not been established.
The following displays indicate that DFHDUMPX is active as an IEASDUMP exit with one CICS Transaction Server for z/OS, Version 3 Release 1 region active in one MVS image.
D PROG,EXIT,MODNAME=DFHDUMPX
01.19.16 DEV5 CSV461I 01.19.16 PROG,EXIT DISPLAY 993
EXIT MODULE STATE MODULE STATE MODULE STATE
IEASDUMP.QUERY DFHDUMPX A
D PROG,EXIT,EN=IEASDUMP.QUERY
01.19.46 DEV5 CSV462I 01.19.46 PROG,EXIT DISPLAY 996
MODULE DFHDUMPX
EXIT(S) IEASDUMP.QUERY
You may issue MVS dump commands from the console to verify that remote dumping is available within the MVS image, without an active CICS region.
11.29.59 DEV5 dump comm=(NO CICS)
11.29.59 DEV5 *03 IEE094D SPECIFY OPERAND(S) FOR DUMP COMMAND
11.36.49 DEV5 r 03,remote=(syslist=*),probdesc=(sysdcond,
sysdlocl,(dfhjobn,iyahzcet))
11.36.49 DEV5 IEE600I REPLY TO 03 IS;REMOTE=(SYSLIST=*),
PROBDESC=(SYSDCOND,SYSDL
11.36.52 DEV5 IEA794I SVC DUMP HAS CAPTURED:
DUMPID=001 REQUESTED BY JOB (*MASTER*)
DUMP TITLE=NO CICS
In the next example, the messages from SDUMP indicate that one dump of the master address space has been taken.
*11.37.03 DEV5 *IEA911E COMPLETE DUMP ON SYS1.DUMP03
*DUMPID=001 REQUESTED BY JOB (*MASTER*)
*FOR ASID (0001)
*REMOTE DUMPS REQUESTED
*INCIDENT TOKEN: DEVPLEX5 DEV5 06/28/1994 11:36:49
Another test is to issue the dump command specifying the CICS XCF group.
11.42.33 DEV5 dump comm=(STILL NO CICS)
11.42.33 DEV5 *05 IEE094D SPECIFY OPERAND(S) FOR DUMP COMMAND
11.43.27 DEV5 r 05,remote=(grplist=dfhir000(*)),
probdesc=(sysdcond,sysdlocl,(dfhjobn,iyahzcet))
11.43.28 DEV5 IEE600I REPLY TO 05
IS;REMOTE=(GRPLIST=DFHIR000(*)),PROBDESC=(SYSD
11.43.31 DEV5 IEA794I SVC DUMP HAS CAPTURED:
DUMPID=002 REQUESTED BY JOB (*MASTER*)
DUMP TITLE=STILL NO CICS
The messages from SDUMP indicate that one dump of the master address space has been taken.
*11.43.42 DEV5 *IEA911E COMPLETE DUMP ON SYS1.DUMP03
*DUMPID=002 REQUESTED BY JOB (*MASTER*)
*FOR ASID (0001)
*REMOTE DUMPS REQUESTED
*INCIDENT TOKEN: DEVPLEX5 DEV5 06/28/1994 11:43:28
To verify that the remote dumping function works on the local system, use the following commands:
11.45.57 DEV5 dump comm=(TEST REMOTE FUNCTION ON LOCAL SYSTEM
11.45.57 DEV5 *06 IEE094D SPECIFY OPERAND(S) FOR DUMP COMMAND
11.46.57 DEV5 r 06,remote=(grplist=*(*)),probdesc=(sysdlocl)
11.46.59 DEV5 IEE600I REPLY TO 06 IS;REMOTE=(GRPLIST=*(*)),
PROBDESC=(SYSDLOCL)
11.47.00 DEV5 IEA794I SVC DUMP HAS CAPTURED:
DUMPID=003 REQUESTED BY JOB (*MASTER*)
DUMP TITLE=TEST REMOTE FUNCTION ON LOCAL SYSTEM
11.47.17 DEV5 IEA794I SVC DUMP HAS CAPTURED:
DUMPID=004 REQUESTED BY JOB (DUMPSRV )
DUMP TITLE=TEST REMOTE FUNCTION ON LOCAL SYSTEM
The messages from SDUMP indicate two dumps were taken, one for the master address space and a second which contains ASIDs 0101, 0012, 0001, 0005, 000B, 000A, 0008, 0007. Note that the same incident token is used for both dumps.
*11.47.39 DEV5 *IEA911E COMPLETE DUMP ON SYS1.DUMP03
*DUMPID=003 REQUESTED BY JOB (*MASTER*)
*FOR ASID (0001)
*REMOTE DUMPS REQUESTED
*INCIDENT TOKEN: DEVPLEX5 DEV5 06/28/1994 11:46:57
*11.47.59 DEV5 *IEA911E COMPLETE DUMP ON SYS1.DUMP04
*DUMPID=004 REQUESTED BY JOB (DUMPSRV )
*FOR ASIDS(0101,0012,0005,0001,000A,000B,0008,0007)
*REMOTE DUMP FOR SYSNAME: DEV5
*INCIDENT TOKEN: DEVPLEX5 DEV5 06/28/1994 11:46:57
The following example lists the MVS console messages received when the CICS master terminal command CEMT P DUMP is issued from CICS APPLID IYAHZCET executing in ASID 19 on MVS image DEV6. IYAHZCET has at least one related task in the CICS region executing in ASID 1B on MVS DEV6 and ASIDS 001A, 001C, 001B, 001E, 001F, 0020, 001D, 0022, 0024, 0021, 0023, 0028, 0025, 0029, 0026 on MVS image DEV7.
- 22.19.16 DEV6 JOB00029 +DFHDU0201 IYAHZCET ABOUT TO TAKE SDUMP. DUMPCODE: MT0001
- 22.19.23 DEV7 DFHDU0214 DFHDUMPX IS ABOUT TO REQUEST A REMOTE SDUMPX.
- 22.19.23 DEV6 DFHDU0214 DFHDUMPX IS ABOUT TO REQUEST A REMOTE SDUMPX.
22.19.27 DEV6 JOB00029 IEA794I SVC DUMP HAS CAPTURED:
DUMPID=001 REQUESTED BY JOB (IYAHZCET)
DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCET CODE=MT0001 ID=1/0001
22.19.43 DEV6 JOB00029 IEA794I SVC DUMP HAS CAPTURED:
DUMPID=002 REQUESTED BY JOB (DUMPSRV )
DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCET CODE=MT0001 ID=1/0001
- 22.19.43 DEV6 JOB00029 +DFHDU0202 IYAHZCET SDUMPX COMPLETE. SDUMPX RETURN CODE X'00'
*22.21.00 DEV6 *IEA911E COMPLETE DUMP ON SYS1.DUMP03
*DUMPID=001 REQUESTED BY JOB (IYAHZCET)
*FOR ASID (0019)
*REMOTE DUMPS REQUESTED
*INCIDENT TOKEN: DEVPLEX1 DEV6 06/10/1994 22:19:16
*ID = DUMP : APPLID IYAHZCET DUMPCODE MT0001 /1/0001
The dump in SYS1.DUMP03 on DEV6 was taken as a result of the CEMT request on IYAHZCET.
*22.21.15 DEV6 *IEA911E COMPLETE DUMP ON SYS1.DUMP04
*DUMPID=002 REQUESTED BY JOB (DUMPSRV )
*FOR ASIDS(0019,001B)
*REMOTE DUMP FOR SYSNAME: DEV6
*INCIDENT TOKEN: DEVPLEX1 DEV6 06/10/1994 22:19:16
*ID = DUMP : APPLID IYAHZCET DUMPCODE MT0001 /1/0001
The dump in SYS1.DUMP04 on DEV6 was taken as a remote dump by MVS dump services as a result of the CEMT request on IYAHZCET. Note that the incident token and ID are the same.
22.22.35 DEV7 JOB00088 IEA794I SVC DUMP HAS CAPTURED:
DUMPID=003 REQUESTED BY JOB (DUMPSRV )
DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCET CODE=MT0001 ID=1/0001
*22.25.58 DEV7 *IEA911E COMPLETE DUMP ON SYS1.DUMP05
*DUMPID=003 REQUESTED BY JOB (DUMPSRV )
*FOR ASIDS(001A,001C,001B,001E,001F,0020,001D,0022,0024,0021,0023,0028,
*0025,0029,0026)
*REMOTE DUMP FOR SYSNAME: DEV6
*INCIDENT TOKEN: DEVPLEX1 DEV6 06/10/1994 22:19:16
*ID = DUMP : APPLID IYAHZCET DUMPCODE MT0001 /1/0001
The dump in SYS1.DUMP05 on DEV7 was taken as a remote dump by MVS dump services as a result of the CEMT request on IYAHZCET. Note that the incident token and ID are the same as those for the dumps produced on DEV6, indicating the originating MVS and CICS IDs.
The following example lists the MVS console messages received when transaction abend SCOP is initiated after having first been added to the transaction dump table in CICS IYAHZCES as requiring related dumps. (CEMT S TRD(SCOP) ADD RELATE).
CICS IYAHZCES (ASID 1A in MVS DEV7) has at least one related task in CICS IYAHZCET (ASID 19 in MVS DEV6).
23.40.41 DEV7 JOB00088 +DFHDU0201 IYAHZCES ABOUT TO TAKE SDUMP. DUMPCODE: SCOP
23.40.49 DEV7 DFHDU0214 DFHDUMPX IS ABOUT TO REQUEST A REMOTE SDUMPX.
23.40.55 DEV7 JOB00088 IEA794I SVC DUMP HAS CAPTURED:
DUMPID=012 REQUESTED BY JOB (IYAHZCES)
DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCES CODE=SCOP ID=1/0008
23.40.49 DEV6 DFHDU0214 DFHDUMPX IS ABOUT TO REQUEST A REMOTE SDUMPX.
23.40.56 DEV6 JOB00029 IEA794I SVC DUMP HAS CAPTURED:
DUMPID=007 REQUESTED BY JOB (DUMPSRV )
DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCES CODE=SCOP ID=1/0008
23.41.11 DEV7 JOB00088 IEA794I SVC DUMP HAS CAPTURED:
DUMPID=013 REQUESTED BY JOB (DUMPSRV )
DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCES CODE=SCOP ID=1/0008
23.41.11 DEV7 JOB00088 +DFHDU0202 IYAHZCES SDUMPX COMPLETE. SDUMPX RETURN CODE X'00'
*23.41.18 DEV6 *IEA911E COMPLETE DUMP ON SYS1.DUMP03
*DUMPID=007 REQUESTED BY JOB (DUMPSRV )
*FOR ASID (0019)
*REMOTE DUMP FOR SYSNAME: DEV7
*INCIDENT TOKEN: DEVPLEX1 DEV7 06/10/1994 23:40:41
*ID = DUMP : APPLID IYAHZCES DUMPCODE SCOP /1/0008
The dump in SYS1.DUMP03 on DEV6 was taken upon receipt of the remote dump request issued from IYAHZCES. Note the incident token and ID are the same as those for dumps produced on DEV7.
*23.41.28 DEV7 *IEA911E COMPLETE DUMP ON SYS1.DUMP03
*DUMPID=012 REQUESTED BY JOB (IYAHZCES)
*FOR ASID (001A)
*REMOTE DUMPS REQUESTED
*INCIDENT TOKEN: DEVPLEX1 DEV7 06/10/1994 23:40:41
*ID = DUMP : APPLID IYAHZCES DUMPCODE SCOP /1/0008
*23.41.38 DEV7 *IEA911E COMPLETE DUMP ON SYS1.DUMP04
*DUMPID=013 REQUESTED BY JOB (DUMPSRV )
*FOR ASID (001A)
*REMOTE DUMP FOR SYSNAME: DEV7
*INCIDENT TOKEN: DEVPLEX1 DEV7 06/10/1994 23:40:41
*ID = DUMP : APPLID IYAHZCES DUMPCODE SCOP /1/0008
The dump in SYS1.DUMP04 on DEV7 was taken as a remote dump by MVS dump services as a result of the request from IYAHZCES. Note the incident token and ID are the same as those for the dumps produced on DEV6, indicating the originating MVS and CICS IDs. A second dump of ASID 1A is taken because the CICS IEASDUMP does not have information indicating that a dump has already been taken for that address space.