Techniques for investigating waits

You can investigate waits in a CICS® system by online inquiry, by tracing, or by analysis of the formatted CICS system dump. The last two techniques are, to some extent, complementary.

Online inquiry is the least powerful technique, and it can only tell you what resource a suspended user task is waiting for. This is enough information to locate the failing area, but you often need to do more investigation before you can solve the problem. The advantage of online inquiry is that you can find out about the waiting task as soon as you detect the problem, and so you capture the data early.

Tracing can give you much more detail than online inquiry, but it involves significant processing overhead. It must also be running with the appropriate options selected when the task first enters a wait state, so this usually means you need to reproduce the problem. However, the information it gives you about system activity in the period leading up to the wait is likely to provide much of the information you need to solve the problem.

A CICS system dump can give you a picture of the state of the CICS system at an instant during the wait. You can request the dump as soon as you notice that a task has entered a wait state, so it gives you early data capture for the problem. However, the dump is unlikely to tell you anything about system activity in the period leading up to the wait, even if you had internal tracing running with the correct selectivity when the task entered the wait. This is because the trace table has probably wrapped before you have had a chance to respond. However, the formatted dump might contain much of the information you need to solve the problem.

If you are able to reproduce the problem, consider using auxiliary tracing and dumping in combination.

Investigating waits--online method

Online, you can use either CEMT INQ TASK or EXEC CICS INQUIRE TASK to find out what resource a user task is waiting on. EXEC CICS INQUIRE TASK can be executed under CECI, or from a user program. Whatever online method you use, you need to supply the task ID of the suspended user task.

If the task is suspended, the information that is returned to you includes the resource type and/or the resource name identifying the unavailable resource. CEMT INQ TASK displays the resource type of the unavailable resource in the HTYPE field. The HVALUE field displays the resource name of the unavailable resource. EXEC CICS INQUIRE TASK returns values in the SUSPENDTYPE and SUSPENDVALUE fields which correspond to the resource type and resource name of the unavailable resource.

HTYPE and SUSPENDTYPE, and HVALUE and SUSPENDVALUE correspond to the values in the resource type and resource name fields of the dispatcher task summary.

Table 9 gives a list of all the resource types and resource names that user tasks might be suspended on, and references showing where to look next for guidance about solving the wait.

You probably need a system dump of the appropriate CICS region to investigate the wait. If you do not yet have one, you can get one using CEMT PERFORM SNAP or CEMT PERFORM DUMP--but make sure the task is still in a wait state when you take the dump. You subsequently need to format the dump using keywords for the given resource type. Advice on which keywords to use is given, where appropriate, in the individual sections.

Investigating waits--using trace

You can find detailed information about the suspension and resumption of tasks during a run of CICS by studying the trace table. Tracing must, of course, be running when the task in question is suspended or resumed, and the tracing options must be selected correctly.

When you look at the trace table, you can find trace entries relating to a particular task from the task numbers that the entries contain. Each is unique to a task so you can be sure that, for any run of CICS, trace entries having the same task number belong to the same task.

For general guidance about setting tracing options and interpreting trace entries, see Using traces in problem determination.

Setting up trace for wait problems

Gate DSSR of dispatcher domain provides the major functions associated with the suspension and resumption of tasks. (See How tasks are made to wait.) The level-1 trace points DS 0004 and DS 0005 are produced on entry to, and exit from, the gate.

You need, therefore, to select tracing to capture the DS level-1 trace entries to investigate any wait problem. You need to capture trace entries for other components as well, when you know what functional areas are involved. The functional area invoking the task wait might, for example, be terminal control (TC), or file control (FC). Level-1 tracing is often enough for these components. However, there are cases, such as those relating to VSAM I/O errors where level-2 trace is needed to examine the RPL as it is passed to VSAM.

Next, you need to ensure that tracing is done for the task that has the wait problem. At first select special tracing for just that task, and disable tracing for all other tasks by setting the master system trace flag off. Subsequently, you can select special tracing for other tasks as well if it becomes clear that they are implicated in the wait.

Interpreting trace for wait problems

For new-style trace entries, which include those for point IDs DS 0004 and DS 0005, the function being traced is shown explicitly in the interpretation string. The functions that can cause a task to enter a wait state are identified in the table. Look out for these in particular in the trace entries for any waiting task you are investigating.

Each function has its own set of input and output parameters, and these, too, are shown in the interpretation strings of the formatted trace entries. Input parameters are shown in the trace entries made from point ID DS 0004, and output parameters in the trace entries made from point ID DS 0005.

The values of the parameters can provide valuable information about task waits, so pay particular attention to them when you study the trace table.

Investigating waits--the formatted CICS system dump

If you are suitably authorized, you can request a CICS system dump using CEMT PERFORM DUMP, CEMT PERFORM SNAP, or EXEC CICS PERFORM DUMP. Make sure that the task in question is waiting when you take the dump, so that you capture information relevant to the wait.

You need to use the dump formatting keyword DS to format the dispatcher task summary. You probably need to look at other areas of the dump as well, so keep the dump on the dump data set.

Interpreting the dump for wait problems

The dispatcher task summary gives you information like that shown in Figure 5.

Figure 5. Dispatcher task summary
===DS: DISPATCHER DOMAIN - SUMMARY
 KEY FOR SUMMARY
     TY = TYPE OF TASK            SY=SYSTEM  NS=NON-SYSTEM
     S  = STATE OF TASK           DIS=DISPATCHABLE  SUS=SUSPENDED  RUN=RUNNING  REE=RESUMED EARLY
     P  = PURGEABLE WAIT/SUSPEND  Y=YES  N=NO
     PS = PURGE STATUS            OK=NO PURGE  PU=PURGED  PP=PURGE PENDING
     TT = TIMEOUT TYPE            IN=INTERVAL  DD=DEADLOCK DELAYED  DI=DEADLOCK IMMEDIATE
     ST = SUSPEND TYPE            MVS=WAIT_MVS  SUSP=SUSPEND  OLDC=WAIT_OLDC  OLDW=WAIT_OLDW
     DTA= DISPATCHER TASK AREA
     AD = ATTACHING DOMAIN
     MO = TASK MODE               CO=CONCURRENT  QR=QUASI-REENTRANT  RO=RESOURCE OWNING  SZ=FEPI OWNING
     RP=ONC/RPC OWNING  FO=FILE OWNING
DS_TOKEN KE_TASK  TY S   P PS TT RESOURCE RESOURCE ST   TIME OF      TIMEOUT      DTA      AD ATTACHER MO SUSPAREA XM_TXN_TOKEN
                                 TYPE     NAME          SUSPEND      DUE          (DSTSK)     TOKEN
00000001 06A23900 SY SUS N OK -  ENF      NOTIFY   MVS  17:50:10.056      -       06B69080 DM 06C33640 RO 06C33658
00020003 06A23580 SY SUS N OK -                    SUSP 17:51:23.515      -       06B69180 XM 06C06690 QR 06B69180 06C066900000022C
00040003 06A23200 SY SUS N OK -  TIEXPIRY DS_NUDGE SUSP 17:50:35.325      -       06B69280 TI 003D0003 QR 06B6A530
00820003 06A30900 SY SUS N OK -  ICEXPIRY DFHAPTIX SUSP 17:50:35.326      -       06B89180 XM 06C06360 QR 06B89180 06C063600000006C
00900003 06A30C80 SY SUS N OK -  ICMIDNTE DFHAPTIM SUSP 17:50:29.437      -       06B89880 XM 06C06250 QR 06B89880 06C062500000005C
00940003 06B5FC80 SY SUS N OK -  TCP_NORM DFHZDSP  OLDW 17:51:46.369      -       06B89A80 XM 06C06470 QR 0004FC10 06C064700000008C
00980001 06C3D080 SY SUS N OK IN SMSYSTEM          SUSP 17:50:10.081 17:55:10.081 06B89C80 SM 00000002 QR 06B6A580
0200000B 07128B00 NS SUS Y OK -  ZCIOWAIT DFHZARQ1 SUSP 17:51:39.616      -       06B9C080 XM 06C06580 QR 06B9C080 06C065800000029C
02020009 07135080 NS RUN                                                          06B9C180 XM 06C06140 QR          06C061400000031C

A brief explanation of the summary information is given in the dump. A more detailed explanation is given in the section that follows.

Dispatcher task summary fields

Detailed descriptions of the fields in the dispatcher task summary are given in Table 3. Some of the fields relate to all tasks known to the dispatcher, and some (identified in the table) relate only to suspended tasks. Values are not provided in fields of the latter type for tasks that are not suspended.

Table 3. Descriptions of fields shown in the dispatcher task summary
Field Description
AD The 2-character domain index identifying the domain that attached the task to the dispatcher.
ATTACHER TOKEN A token provided by the domain that attached the task. This token uniquely identifies the task to the attaching domain.
DS_TOKEN A token given by the dispatcher to a domain that attaches a task. It identifies the attached task uniquely to the dispatcher.
DTA An address used internally by the dispatcher.
KE_TASK A value that uniquely identifies to the kernel a task that has been created.
MO The dispatching mode for the task. There is an MVS™ TCB for each mode. All tasks in a given mode are running, or will run next, under the TCB corresponding to the mode. The possible values are:
  • CO--concurrent.
  • FO--file owning.
  • QR--quasi-reentrant.
  • RO--resource owning.
  • RP--ONC RPC
  • SZ--FEPI owning.
P Whether at the time of the suspend call the control block containing the definition of the transaction specified SPURGE(YES) or SPURGE(NO). SPURGE(NO) inhibits deadlock time-out, CEMT SET TASK PURGE, and EXEC CICS SET TASK PURGE.

The possible values are:

  • Y (=YES)--the task is purgeable.
  • N (=NO)--the task is not purgeable.
PS The purge status of the task. The possible values are:
  • OK--the task has not been purged, and there is no purge pending.
  • PU--the task has been purged, either by the dispatcher or by the operator.
  • PP--there is a purge pending on the task.
RESOURCE NAME (suspended tasks only) The name of the resource that a suspended task is waiting for. A value is given only if RESOURCE_NAME has been included as an input parameter on the suspend call.
RESOURCE TYPE (suspended tasks only) The type of the resource that the task is waiting for. A value is given only if RESOURCE_TYPE has been included as an input parameter on the suspend call.
S The state of the task within the dispatcher. Possible values are:
  • DIS--the task is dispatchable. It is ready to run, and it will be dispatched when a TCB becomes available.
  • RUN--the task is running.
  • SUS--the task has been suspended by any of the functions SUSPEND, WAIT_MVS, WAIT_OLDC, or WAIT_OLDW of gate DSSR. For an explanation of these functions, see How tasks are made to wait.
  • REE--the task has been resumed early, possibly because a RESUME request has arrived before the corresponding SUSPEND request. (The SUSPEND/RESUME interface is asynchronous.)
ST (suspended tasks only) The type of function that was invoked to suspend a currently suspended task. Possible values include:
  • MVS--function WAIT_MVS
  • OLDC--function WAIT_OLDC
  • OLDW--function WAIT_OLDW
  • SUSP--function SUSPEND
For a description of the functions, see How tasks are made to wait.
SUSPAREA (suspended tasks only) Either an address used internally by the dispatcher, or an ECB address, or an ECB list address. These are the cases:
  • Address used internally, if the task was suspended by a SUSPEND call.
  • ECB address or ECB list address, if the task was suspended by a WAIT_MVS or WAIT_OLDW call.
  • ECB address, if the task was suspended by a WAIT_OLDC call.
Look at the value given in the ST column to see which one of these descriptions applies.
TIME OF SUSPEND (suspended tasks only) The time when a currently suspended task was suspended.

The format is hh:mm:ss.mmm (hours, minutes, seconds, milliseconds), GMT.

TIMEOUT DUE    (suspended tasks only) The time that a suspended task is due to timeout, if a timeout interval has been specified. A suspended task only times out if it is not resumed before this time arrives.

The format is hh:mm:ss.mmm (hours, minutes, seconds, milliseconds).

TT (suspended tasks only) The time-out type for the task. The possible values, where one is given, are:
  • IN--a time-out interval has been specified for the task.
  • DD--deadlock action is to be delayed when the time-out interval expires.
  • DI--deadlock action is immediate when the time-out interval expires.
TY Whether this is a system task or a non-system task. Possible values are:
  • SY--this is a system task.
  • NS--this is a non-system task.
A non-system task can be either a user written transaction, or a CICS-supplied transaction.

Parameters and functions setting fields in the dispatcher task summary

Many of the values shown in the dispatcher task summary are provided directly by parameters included on calls to and from the dispatcher. If you are using trace, you can see the values of the parameters in the trace entries, and this can be useful for debugging. For details of how you can use trace to investigate waits, see Investigating waits--using trace .

Table 4 shows the parameters that set task summary fields, the functions that use those parameters, and the domain gates that provide the functions. Task summary fields that are not set by parameters are also identified (by none in "Related parameter" column).

Table 4. Parameters and functions that set fields shown in the dispatcher task summary
Field Related parameter Function Input or output Gate
AD DOMAIN_INDEX
INQUIRE_TASK
GET_NEXT
IN
OUT
DSBR
DTA ATTACH_TOKEN CREATE_TASK IN KEDS
DS_TOKEN TASK_TOKEN
ATTACH
CANCEL_TASK
PURGE_INHIBIT_QUERY
SET_PRIORITY
TASK_REPLY
 
GET_NEXT
INQUIRE_TASK
OUT
IN
IN
IN
IN
 
OUT
OUT
DSAT
 
 
 
 
 
DSBR
KE_TASK TASK_TOKEN
CREATE_TASK
CREATE_TCB
PUSH_TASK
TASK_REPLY
TCB_REPLY
OUT
OUT
IN
IN
IN
KEDS
MO MODE
ATTACH
CHANGE_MODE
 
GET_NEXT
INQUIRE_TASK
IN
IN
 
OUT
OUT
DSAT
 
 
DSBR
P PURGEABLE
SUSPEND
WAIT_MVS
WAIT_OLDC
WAIT_OLDW
IN
IN
IN
IN
DSSR
PS none      
RESOURCE NAME RESOURCE_NAME
ADD_SUSPEND
SUSPEND
WAIT_MVS
WAIT_OLDC
WAIT_OLDW
 
GET_NEXT
INQUIRE_TASK
IN
IN
IN
IN
IN
 
OUT
OUT
DSSR
 
 
 
 
 
DSBR
RESOURCE TYPE RESOURCE_TYPE
ADD_SUSPEND
SUSPEND
WAIT_MVS
WAIT_OLDC
WAIT_OLDW
 
GET_NEXT
INQUIRE_TASK
IN
IN
IN
IN
IN
 
OUT
OUT
DSSR
 
 
 
 
 
DSBR
S
(see note 1)
STATE
GET_NEXT
INQUIRE_TASK
OUT
OUT
DSBR
SUSPAREA
(see note 2)
ECB_ADDRESS or
ECB_LIST_ADDRESS
(see note 3)
WAIT_MVS
WAIT_OLDC
WAIT_OLDW
IN
IN
IN
DSSR
TIME OF SUSPEND none      
TASKNO none      
TIMEOUT DUE
(see note 4)
none      
TT
INTERVAL and
DEADLOCK_ACTION
SUSPEND
WAIT_MVS
WAIT_OLDW
WAIT_OLDC
IN
IN
IN
IN
DSSR
TY none      
ATTACHER TOKEN USER_TOKEN
ATTACH
PURGE_INHIBIT_QUERY
TASK_REPLY
 
GET_NEXT
INQUIRE_TASK
OUT
OUT
DSAT
 
 
 
DSBR
ST none      
Notes:
  1. Field S (for STATE) of the dispatcher task summary has a wider range of values than parameter STATE of DSBR functions GET_NEXT and INQUIRE_TASK. Parameter STATE can only have the values READY, RUNNING, or SUSPENDED. For the possible values of field S, see Table 3.
  2. Parameters ECB_ADDRESS and ECB_LIST_ADDRESS only relate to SUSPAREA when the task has been suspended by the WAIT_MVS, WAIT_OLDW, or WAIT_OLDC functions of gate DSSR.
  3. Parameter ECB_LIST_ADDRESS is only valid for functions WAIT_MVS and WAIT_OLDW, and not for function WAIT_OLDC.
  4. If INTERVAL has been specified, the value of TIMEOUT DUE should be equal to INTERVAL + TIME OF SUSPEND.

Related concepts
Dealing with waits
CICS tracing
Related tasks
Formatting system dumps
Selecting trace destinations and related options
[[ Contents Previous Page | Next Page Index ]]