CICS dumping in a sysplex

Simultaneous dump capture from multiple CICS regions across a sysplex is possible, considerably aiding problem determination in XCF/MRO environments where many CICS regions are running. Two types of situation, in particular, benefit from this:

Before CICS® for MVS/ESA 4.1, such automatic and simultaneous dump data capture of CICS data was impossible.

You need MVS/ESA 5.1, the MVS workload manager, and the XCF facility in order to collect dump data in this way. The MVS images in the sysplex must be connected via XCF. The CICS regions must be using MRO supported by the CICS Transaction Server for z/OS®, Version 3 Release 1 interregion communication program, DFHIRP.

Automatic dump data capture from related CICS regions

It is possible to collect dump data simultaneously from all related CICS regions in a sysplex. Related CICS regions are those containing one or more tasks which have unit of work identifiers, in the form of APPC tokens, that match the unit of work identifiers in the CICS region which initially issued the dump request.

The CICS regions must be connected via XCF/MRO. Connections using VTAM® ISC are not eligible to use the related dump facility.

The function is controlled by the DUMPSCOPE option on each CICS dump table entry. You can set this option to have either of the following values:

The DUMPSCOPE option is available on the following master terminal and system programming commands:

If the DUMPSCOPE option is set to RELATED in the CICS region issuing the dump request, a request for a system dump is sent to all MVS images in the sysplex that run related CICS regions.

The local MVS image running the CICS region that initiated the dump request has two dumps - one of the originating CICS region, the other containing the originating CICS region and up to fourteen additional related CICS regions from the local MVS image.

When a dump is requested, the DUMPSCOPE option is tested only in the CICS region issuing the original dump request. If the requesting CICS region has DUMPSCOPE defined as RELATED for the dump code, then all related CICS regions are dumped even if they have DUMPSCOPE defined as LOCAL for the dump code.

There is a maximum of fifteen address spaces in an SDUMP. If there are more than fifteen related CICS regions on an MVS image, then not all of them will be dumped. Related CICS regions may also fail to be dumped if they are swapped out when the dump request is issued. You should consider whether to make certain CICS regions non-swappable as a result.

Operator-requested simultaneous dump data capture

The MVS console operator may want to issue a dump request simultaneously to several CICS regions in the sysplex. There may be a problem in the sysplex where one or more CICS regions is hanging and, to fully diagnose and solve the problem, dump data captured at the same time is required.

Without this facility, such simultaneous dump data capture across multiple CICS regions in the sysplex is impossible.

Use the following command at the console:

DUMP COMM=()
R x,REMOTE=(SYSLIST=*),PROBDESC=(SYSDCOND,SYSDLOCL,(DFHJOBN,jobnames))

where:

See the MVS System Commands manual, GC28-1626, for a full description of all command options.

If you adopt a suitable naming convention for your CICS regions, this can be used to define suitable generic jobnames to determine which CICS regions to dump. See the System/390 MVS Sysplex Application Migration manual for recommendations on naming conventions. If you follow the recommendation in this manual, the generic job name for all CICS regions in the sysplex would be ‘CICS*’.

Requesting dumps to resolve SMSVSAM problems

You may sometimes encounter a CICS problem that involves SMSVSAM. If you need to submit such a problem to IBM®, in particular where CICS or a CICS transaction is in a hung state while accessing VSAM data sets in RLS mode, include a dump of the following:

You can obtain such a dump by using the following command:

/DUMP COMM=(comment to describe the problem)

When MVS responds, reply to WTOR number nn with the following:

/R nn,JOBNAME=(CICS-job-name,SMSVSAM,CATALOG,GRS),DSPNAME='SMSVSAM'.*,
      REMOTE=(SYSLIST=*('SMSVSAM','CATALOG','GRS'),DSPNAME,SDATA,
      END
Note:
You can use the CONT option to split this command into parts, as follows:
/R nn,JOBNAME=(CICS-job-name,SMSVSAM,CATALOG,GRS), CONT
/R nn,DSPNAME='SMSVSAM'.*,REMOTE=(SYSLIST=*('SMSVSAM', CONT
/R nn,'CATALOG','GRS'),DSPNAME,SDATA, END

Useful CICS master terminal and MVS console commands in a sysplex

MVS support for remote SDUMPs is available only for images running MVS/ESA 5.1 or later and which are connected by XCF. Related CICS SDUMPs are produced only for CICS regions which are MRO connected using XCF. DFHIRP must be at CICS Transaction Server for z/OS, Version 3 Release 1 level. Connections using VTAM ISC are not eligible to use the related dump facility.

If you are unable to produce related system dumps when there are related CICS regions across MVS images, ensure that the regions are MRO connected.

Use CEMT I IRC to ensure that interregion communication is available. If IRC is not available it may be started using the CEMT S IRC OPEN command. Failure to start IRC results in DFHIRxxx messages which may be used to identify the source of the problem.

During IRC start processing, CICS attempts to join XCF group DFHIR000. If this fails, return code yyy is given in the DFHIR3777 message.

The following MVS console commands may be used to monitor activity in the sysplex:

Use the CICS master terminal command CEMT I CONNECTION to display the status of the connections. ‘XCF’ is displayed for every acquired connection using MRO/XCF for communications.

  I CONNECTION
  STATUS:  RESULTS - OVERTYPE TO MODIFY
   Con(FORD) Net(IYAHZCES)     Ins Acq         Xcf
   Con(F100) Net(IYAHZCEC)     Ins Acq         Irc
   Con(F150) Net(IYAHZCED)     Ins Acq         Irc
   Con(GEO ) Net(IYAHZCEG)     Ins Acq         Xcf
   Con(GMC ) Net(IYAHZCEB)     Ins Acq         Xcf
   Con(JIM ) Net(IYAHZCEJ)     Ins Acq         Xcf
   Con(MARY) Net(IYAHZCEM)     Ins Acq         Xcf
   Con(MIKE) Net(IYAHZCEI)     Ins Acq         Xcf
 + Con(RAMB) Net(IYAHZCEE)     Ins Acq         Xcf
                                                    SYSID=CHEV APPLID=IYAHZCET
   RESPONSE: NORMAL                           TIME:  01.28.59  DATE: 06.11.94
 PF 1 HELP       3 END                    7 SBH 8 SFH 9 MSG 10 SB 11 SF

CICS initialization issues an MVS CSVDYNEX request to add DFHDUMPX as an MVS IEASDUMP.QUERY exit. If you need to change the status of the exit, use the MVS console command SETPROG EXIT.

If you have determined that XCF communication is in use, you can verify that the CICS SDUMP exit has been established using the following MVS commands:

D PROG,EXIT,MODNAME=DFHDUMPX
    08.16.04 DEV5           CSV463I MODULE DFHDUMPX IS NOT ASSOCIATED ANY EXIT

D PROG,EXIT,EN=IEASDUMP.QUERY
   08.17.44 DEV5           CSV463I NO MODULES ARE ASSOCIATED WITH EXIT IEASDUMP.QUERY

This example indicates that the exit has not been established.

The following displays indicate that DFHDUMPX is active as an IEASDUMP exit with one CICS Transaction Server for z/OS, Version 3 Release 1 region active in one MVS image.

 D PROG,EXIT,MODNAME=DFHDUMPX
 01.19.16 DEV5           CSV461I 01.19.16 PROG,EXIT DISPLAY 993
 EXIT             MODULE   STATE MODULE   STATE MODULE   STATE
 IEASDUMP.QUERY   DFHDUMPX   A
 
 D PROG,EXIT,EN=IEASDUMP.QUERY
 01.19.46 DEV5           CSV462I 01.19.46 PROG,EXIT DISPLAY 996
 MODULE  DFHDUMPX
 EXIT(S) IEASDUMP.QUERY

You may issue MVS dump commands from the console to verify that remote dumping is available within the MVS image, without an active CICS region.

 11.29.59 DEV5           dump comm=(NO CICS)
 11.29.59 DEV5          *03 IEE094D SPECIFY OPERAND(S) FOR DUMP COMMAND
 11.36.49 DEV5           r 03,remote=(syslist=*),probdesc=(sysdcond,
  sysdlocl,(dfhjobn,iyahzcet))
 11.36.49 DEV5           IEE600I REPLY TO 03 IS;REMOTE=(SYSLIST=*),
  PROBDESC=(SYSDCOND,SYSDL
 11.36.52 DEV5           IEA794I SVC DUMP HAS CAPTURED:
 DUMPID=001 REQUESTED BY JOB (*MASTER*)
 DUMP TITLE=NO CICS

In the next example, the messages from SDUMP indicate that one dump of the master address space has been taken.

 *11.37.03 DEV5          *IEA911E COMPLETE DUMP ON SYS1.DUMP03
 *DUMPID=001 REQUESTED BY JOB (*MASTER*)
 *FOR ASID (0001)
 *REMOTE DUMPS REQUESTED
 *INCIDENT TOKEN: DEVPLEX5 DEV5     06/28/1994 11:36:49

Another test is to issue the dump command specifying the CICS XCF group.

 11.42.33 DEV5           dump comm=(STILL NO CICS)
 11.42.33 DEV5          *05 IEE094D SPECIFY OPERAND(S) FOR DUMP COMMAND
 11.43.27 DEV5           r 05,remote=(grplist=dfhir000(*)),
  probdesc=(sysdcond,sysdlocl,(dfhjobn,iyahzcet))
 11.43.28 DEV5           IEE600I REPLY TO 05
  IS;REMOTE=(GRPLIST=DFHIR000(*)),PROBDESC=(SYSD
 11.43.31 DEV5           IEA794I SVC DUMP HAS CAPTURED:
 DUMPID=002 REQUESTED BY JOB (*MASTER*)
 DUMP TITLE=STILL NO CICS

The messages from SDUMP indicate that one dump of the master address space has been taken.

 *11.43.42 DEV5          *IEA911E COMPLETE DUMP ON SYS1.DUMP03
 *DUMPID=002 REQUESTED BY JOB (*MASTER*)
 *FOR ASID (0001)
 *REMOTE DUMPS REQUESTED
 *INCIDENT TOKEN: DEVPLEX5 DEV5     06/28/1994 11:43:28

To verify that the remote dumping function works on the local system, use the following commands:

 11.45.57 DEV5           dump comm=(TEST REMOTE FUNCTION ON LOCAL SYSTEM
 11.45.57 DEV5          *06 IEE094D SPECIFY OPERAND(S) FOR DUMP COMMAND
 11.46.57 DEV5           r 06,remote=(grplist=*(*)),probdesc=(sysdlocl)
 11.46.59 DEV5           IEE600I REPLY TO 06 IS;REMOTE=(GRPLIST=*(*)),
  PROBDESC=(SYSDLOCL)
 11.47.00 DEV5           IEA794I SVC DUMP HAS CAPTURED:
 DUMPID=003 REQUESTED BY JOB (*MASTER*)
 DUMP TITLE=TEST REMOTE FUNCTION ON LOCAL SYSTEM
 11.47.17 DEV5           IEA794I SVC DUMP HAS CAPTURED:
 DUMPID=004 REQUESTED BY JOB (DUMPSRV )
 DUMP TITLE=TEST REMOTE FUNCTION ON LOCAL SYSTEM

The messages from SDUMP indicate two dumps were taken, one for the master address space and a second which contains ASIDs 0101, 0012, 0001, 0005, 000B, 000A, 0008, 0007. Note that the same incident token is used for both dumps.

 *11.47.39 DEV5          *IEA911E COMPLETE DUMP ON SYS1.DUMP03
 *DUMPID=003 REQUESTED BY JOB (*MASTER*)
 *FOR ASID (0001)
 *REMOTE DUMPS REQUESTED
 *INCIDENT TOKEN: DEVPLEX5 DEV5     06/28/1994 11:46:57
 *11.47.59 DEV5          *IEA911E COMPLETE DUMP ON SYS1.DUMP04
 *DUMPID=004 REQUESTED BY JOB (DUMPSRV )
 *FOR ASIDS(0101,0012,0005,0001,000A,000B,0008,0007)
 *REMOTE DUMP FOR SYSNAME: DEV5
 *INCIDENT TOKEN: DEVPLEX5 DEV5     06/28/1994 11:46:57

The following example lists the MVS console messages received when the CICS master terminal command CEMT P DUMP is issued from CICS APPLID IYAHZCET executing in ASID 19 on MVS image DEV6. IYAHZCET has at least one related task in the CICS region executing in ASID 1B on MVS DEV6 and ASIDS 001A, 001C, 001B, 001E, 001F, 0020, 001D, 0022, 0024, 0021, 0023, 0028, 0025, 0029, 0026 on MVS image DEV7.

 - 22.19.16 DEV6 JOB00029  +DFHDU0201 IYAHZCET ABOUT TO TAKE SDUMP. DUMPCODE: MT0001
 - 22.19.23 DEV7           DFHDU0214 DFHDUMPX IS ABOUT TO REQUEST A REMOTE SDUMPX.
 - 22.19.23 DEV6           DFHDU0214 DFHDUMPX IS ABOUT TO REQUEST A REMOTE SDUMPX.
   22.19.27 DEV6 JOB00029  IEA794I SVC DUMP HAS CAPTURED:
   DUMPID=001 REQUESTED BY JOB (IYAHZCET)
   DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCET CODE=MT0001   ID=1/0001
 
   22.19.43 DEV6 JOB00029  IEA794I SVC DUMP HAS CAPTURED:
   DUMPID=002 REQUESTED BY JOB (DUMPSRV )
   DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCET CODE=MT0001   ID=1/0001
 
 - 22.19.43 DEV6 JOB00029  +DFHDU0202 IYAHZCET SDUMPX COMPLETE. SDUMPX RETURN CODE X'00'
 
  *22.21.00 DEV6          *IEA911E COMPLETE DUMP ON SYS1.DUMP03
  *DUMPID=001 REQUESTED BY JOB (IYAHZCET)
  *FOR ASID (0019)
  *REMOTE DUMPS REQUESTED
  *INCIDENT TOKEN: DEVPLEX1 DEV6     06/10/1994 22:19:16
  *ID = DUMP : APPLID IYAHZCET DUMPCODE MT0001  /1/0001

The dump in SYS1.DUMP03 on DEV6 was taken as a result of the CEMT request on IYAHZCET.

 *22.21.15 DEV6          *IEA911E COMPLETE DUMP ON SYS1.DUMP04
 *DUMPID=002 REQUESTED BY JOB (DUMPSRV )
 *FOR ASIDS(0019,001B)
 *REMOTE DUMP FOR SYSNAME: DEV6
 *INCIDENT TOKEN: DEVPLEX1 DEV6     06/10/1994 22:19:16
 *ID = DUMP : APPLID IYAHZCET DUMPCODE MT0001  /1/0001

The dump in SYS1.DUMP04 on DEV6 was taken as a remote dump by MVS dump services as a result of the CEMT request on IYAHZCET. Note that the incident token and ID are the same.

  22.22.35 DEV7 JOB00088  IEA794I SVC DUMP HAS CAPTURED:
  DUMPID=003 REQUESTED BY JOB (DUMPSRV )
  DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCET CODE=MT0001   ID=1/0001
 
 *22.25.58 DEV7          *IEA911E COMPLETE DUMP ON SYS1.DUMP05
 *DUMPID=003 REQUESTED BY JOB (DUMPSRV )
 *FOR ASIDS(001A,001C,001B,001E,001F,0020,001D,0022,0024,0021,0023,0028,
 *0025,0029,0026)
 *REMOTE DUMP FOR SYSNAME: DEV6
 *INCIDENT TOKEN: DEVPLEX1 DEV6     06/10/1994 22:19:16
 *ID = DUMP : APPLID IYAHZCET DUMPCODE MT0001  /1/0001

The dump in SYS1.DUMP05 on DEV7 was taken as a remote dump by MVS dump services as a result of the CEMT request on IYAHZCET. Note that the incident token and ID are the same as those for the dumps produced on DEV6, indicating the originating MVS and CICS IDs.

The following example lists the MVS console messages received when transaction abend SCOP is initiated after having first been added to the transaction dump table in CICS IYAHZCES as requiring related dumps. (CEMT S TRD(SCOP) ADD RELATE).

CICS IYAHZCES (ASID 1A in MVS DEV7) has at least one related task in CICS IYAHZCET (ASID 19 in MVS DEV6).

  23.40.41 DEV7 JOB00088  +DFHDU0201 IYAHZCES ABOUT TO TAKE SDUMP. DUMPCODE: SCOP
  23.40.49 DEV7           DFHDU0214 DFHDUMPX IS ABOUT TO REQUEST A REMOTE SDUMPX.
  23.40.55 DEV7 JOB00088  IEA794I SVC DUMP HAS CAPTURED:
 
  DUMPID=012 REQUESTED BY JOB (IYAHZCES)
  DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCES CODE=SCOP     ID=1/0008
 
  23.40.49 DEV6           DFHDU0214 DFHDUMPX IS ABOUT TO REQUEST A REMOTE SDUMPX.
  23.40.56 DEV6 JOB00029  IEA794I SVC DUMP HAS CAPTURED:
  DUMPID=007 REQUESTED BY JOB (DUMPSRV )
  DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCES CODE=SCOP     ID=1/0008
 
 
  23.41.11 DEV7 JOB00088  IEA794I SVC DUMP HAS CAPTURED:
  DUMPID=013 REQUESTED BY JOB (DUMPSRV )
  DUMP TITLE=CICS DUMP: SYSTEM=IYAHZCES CODE=SCOP     ID=1/0008
 
  23.41.11 DEV7 JOB00088  +DFHDU0202 IYAHZCES SDUMPX COMPLETE. SDUMPX RETURN CODE X'00'
 
 
 *23.41.18 DEV6          *IEA911E COMPLETE DUMP ON SYS1.DUMP03
 *DUMPID=007 REQUESTED BY JOB (DUMPSRV )
 *FOR ASID (0019)
 *REMOTE DUMP FOR SYSNAME: DEV7
 *INCIDENT TOKEN: DEVPLEX1 DEV7     06/10/1994 23:40:41
 *ID = DUMP : APPLID IYAHZCES DUMPCODE SCOP    /1/0008

The dump in SYS1.DUMP03 on DEV6 was taken upon receipt of the remote dump request issued from IYAHZCES. Note the incident token and ID are the same as those for dumps produced on DEV7.

    *23.41.28 DEV7          *IEA911E COMPLETE DUMP ON SYS1.DUMP03
    *DUMPID=012 REQUESTED BY JOB (IYAHZCES)
    *FOR ASID (001A)
    *REMOTE DUMPS REQUESTED
    *INCIDENT TOKEN: DEVPLEX1 DEV7     06/10/1994 23:40:41
    *ID = DUMP : APPLID IYAHZCES DUMPCODE SCOP    /1/0008
 
    *23.41.38 DEV7          *IEA911E COMPLETE DUMP ON SYS1.DUMP04
    *DUMPID=013 REQUESTED BY JOB (DUMPSRV )
    *FOR ASID (001A)
    *REMOTE DUMP FOR SYSNAME: DEV7
    *INCIDENT TOKEN: DEVPLEX1 DEV7     06/10/1994 23:40:41
    *ID = DUMP : APPLID IYAHZCES DUMPCODE SCOP    /1/0008

The dump in SYS1.DUMP04 on DEV7 was taken as a remote dump by MVS dump services as a result of the request from IYAHZCES. Note the incident token and ID are the same as those for the dumps produced on DEV6, indicating the originating MVS and CICS IDs. A second dump of ASID 1A is taken because the CICS IEASDUMP does not have information indicating that a dump has already been taken for that address space.

Related Concepts
Cross-system multiregion operation (XCF/MRO)
System dumps

Related Tasks
Formatting and interpreting dumps

Related References
The transaction dump table
The system dump table
[[ Contents Previous Page | Next Page Index ]]