Resolving deadlocks in a CICS region

You can diagnose deadlocks between tasks wanting an exclusive lock on the same resource, such as a record in a non-RLS file, a recoverable transient data queue, or any resource represented by an EXEC CICS® ENQUEUE.

Enqueue deadlocks between tasks occur when each of two transactions (say, A and B) needs an exclusive lock on a resource that the other holds already. Transaction A waits for transaction B to release the resource. However, if transaction B cannot release the resource because it, in turn, is enqueued on a resource held by transaction A, the two transactions are deadlocked. Further transactions may then queue, enqueued on the resources held by transactions A and B.

You can use CEMT online to identify deadlocked transactions, and identify the resources they hold.

The CEMT INQUIRE UOWENQ command displays information about the owners of all enqueues held. More importantly, for deadlock diagnosis purposes, it displays information about the tasks waiting for the enqueues. Note that CEMT INQUIRE UOWENQ can be used only for files accessed in non-RLS mode, because files accessed in RLS mode have their locks managed by VSAM, not by CICS. Deadlock and timeout detection for files accessed in RLS mode is also performed by VSAM.

An example of deadlock diagnosis using CEMT INQUIRE UOWENQ

Consider the following example. The user of task 32 complains that his terminal is locked: he cannot enter data and his terminal does not respond to any prompt.

If you enter CEMT INQUIRE TASK at a different terminal at this time, a display similar to the following might appear:

  INQUIRE TASK
  STATUS:  RESULTS - OVERTYPE TO MODIFY
   Tas(0000025) Tra(CEMT) Fac(T773) Run Ter Pri( 255 )
      Sta(TO) Use(CICSUSER) Uow(AA8E9505458D8C01)
   Tas(0000028) Tra(TDUP) Fac(T774) Sus Ter Pri( 001 )
      Sta(TO) Use(CICSUSER) Uow(AA8E950545CAD227) Hty(ZCIOWAIT) Hva(DFHZARQ1)
   Tas(0000032) Tra(FUPD) Fac(T775) Sus Ter Pri( 001 )
      Sta(TO) Use(CICSUSER) Uow(AA8E950545DAC004) Hty(ENQUEUE ) Hva(FCDSRECD)
   Tas(0000035) Tra(FUPD) Fac(T784) Sus Ter Pri( 001 )
      Sta(TO) Use(CICSUSER) Uow(AA8E950545DBC357) Hty(ENQUEUE ) Hva(FCDSRECD)
   Tas(0000039) Tra(FUPD) Fac(T778) Sus Ter Pri( 001 )
      Sta(TO) Use(CICSUSER) Uow(AA8E97FE9592F403) Hty(ENQUEUE ) Hva(FCDSRECD)
   Tas(0000042) Tra(FUP2) Fac(T783) Sus Ter Pri( 001 )
      Sta(TO) Use(CICSUSER) Uow(AA8E97FE95DC1B9A) Hty(ENQUEUE ) Hva(FCDSRECD)

Task 32 is waiting on an enqueue (Hty = ENQUEUE). You can also see that the task is waiting for a lock on a data set record (Hva = FCDSRECD). But, at this stage, you cannot tell which (if any) task has control of this resource.

If you now enter CEMT INQUIRE UOWENQ at the same terminal, you can see the enqueues held in the system - you can see which tasks own which resources and which tasks are waiting for those resources.

A screen similar to the following might be displayed:

  INQUIRE UOWENQ
  STATUS:  RESULTS
   Uow(AA8E9505458D8C01) Tra(CEMT) Tas(0000025) Act Exe Own
   Uow(AA8E950545CAD227) Tra(TDUP) Tas(0000028) Act Tdq Own
   Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Own
   Uow(AA8E950545DBC357) Tra(FUPD) Tas(0000035) Act Dat Wai
   Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Wai
   Uow(AA8E9505458D8C01) Tra(TSUP) Tas(0000034) Ret Tsq Own
   Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Own
   Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Wai
   Uow(AA8E97FE95DC1B9A) Tra(FUPD) Tas(0000042) Act Dat Own

You can see all the enqueue owners and waiters on the same region on this display. Tasks waiting for an enqueue are displayed immediately after the task which owns the enqueue. Owners and waiters on other regions are not displayed.

You can clarify this display in a busy system by displaying only those resources that the task you are interested in owns and waits for. This is called filtering. You add a filter to the end of the INQUIRE UOWENQ command: CEMT INQUIRE UOWENQ TASK(32).

  INQUIRE UOWENQ TASK(32)
  STATUS:  RESULTS
   Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Own
   Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Wai

You can now see that task 32 owns one enqueue but is also waiting for another.

This display shows one line of information per item, listing:

In order to see more information, press ENTER alongside the item that interests you. If you press ENTER alongside the first entry of the output from CEMT INQUIRE UOWENQ TASK(32), a screen similar to the following might be displayed:

INQUIRE UOWENQ TASK(32)
RESULT
  Uowenq
  Uow(AA8E950545DAC004)
  Transid(FUPD)
  Taskid(0000032)
  State(Active)
  Type(Dataset)
  Relation(Owner)
  Resource(ACCT.CICS640.ACCTFILE)
  Qualifier(SMITH)
  Netuowid(..GBIBMIYA.IYA2T774.n......)
  Enqfails(00000000)

This shows you details of the enqueue that task 32 owns.

Expanding the second entry shows the enqueue that task 32 is waiting for:

INQUIRE UOWENQ TASK(32)
RESULT
  Uowenq
  Uow(AA8E950545DAC004)
  Transid(FUPD)
  Taskid(0000032)
  State(Active)
  Type(Dataset)
  Relation(Waiter)
  Resource(INDX.CICS640.ACIXFILE)
  Qualifier(SMITH)
  Netuowid(..GBIBMIYA.IYA2T774.n......)
  Enqfails(00000000)

Expanding the one-line display is useful because RESOURCE and QUALIFIER fields are then revealed. These identify the physical resource that is related to the enqueue. You can see, from the first entry in this example, that task 32 owns the enqueue on record identifier "SMITH" in the ACCT.CICS640.ACCTFILE data set. You can also see, from the second expanded entry, that task 32 is waiting on an enqueue - for record identifier "SMITH" in the INDX.CICS640.ACIXFILE data set.

So now you need to investigate why task 32 is waiting on the enqueue detailed in the second expanded entry. You need to find out which task owns this enqueue and why it is holding it for such a long time. You can do this by filtering the CEMT INQUIRE UOWENQ command with the RESOURCE and QUALIFIER options. Enter CEMT INQUIRE UOWENQ RESOURCE(INDX.CICS640.ACIXFILE) QUALIFIER(SMITH) This shows the task that owns the enqueue that is being waited on.

INQUIRE UOWENQ RESOURCE(INDX.CICS640.ACIXFILE) QUALIFIER(SMITH)
STATUS:  RESULTS
 Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Own
 Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Wai

This shows you that another task, task 39, owns the enqueue that task 32 is waiting on. In order to find out why task 39 is holding this enqueue, filter the CEMT command again for task 39. Enter CEMT INQUIRE UOWENQ TASK(39).

INQUIRE UOWENQ TASK(39)
STATUS:  RESULTS
 Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Wai
 Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Own

This shows you that task 39 is also waiting for an enqueue. If you expand the entry which indicates the waiting state, you may see a display similar to the following:

INQUIRE UOWENQ TASK(39)
RESULT
  Uowenq
  Uow(AA8E97FE9592F403)
  Transid(FUP2)
  Taskid(0000039)
  State(Active)
  Type(Dataset)
  Relation(Waiter)
  Resource(ACCT.CICS640.ACCTFILE)
  Qualifier(SMITH)
  Netuowid(..GBIBMIYA.IYA2T776.p.nk4..)
  Enqfails(00000000)

This shows you that task 39 is waiting for the enqueue on record "SMITH" in the ACCT.CICS640.ACCTFILE data set. This is the enqueue that task 32 owns. You can now see that the deadlock is between tasks 32 and 39.

If you filter by the RESOURCE and QUALIFIER of this enqueue, you can confirm that this is the case. This also shows that task 35 also waits on the enqueue owned by task 32.

INQUIRE UOWENQ RESOURCE(ACCT.CICS640.ACCTFILE) QUALIFIER(SMITH)
STATUS:  RESULTS
 Uow(AA8E950545DAC004) Tra(FUPD) Tas(0000032) Act Dat Own
 Uow(AA8E950545DBC357) Tra(FUPD) Tas(0000035) Act Dat Wai
 Uow(AA8E97FE9592F403) Tra(FUP2) Tas(0000039) Act Dat Wai

You are now in a position of knowing which transaction(s) to cancel and investigate further.

You can also use the EXEC CICS INQUIRE UOWENQ command or the EXEC CICS INQUIRE ENQ command in your applications. These return all the information that is available under CEMT INQUIRE UOWENQ. If you wish to automate deadlock detection and resolution, these commands are of great benefit.

Related concepts
Transaction Deadlocks
Related tasks
Investigating enqueue waits
Resolving deadlocks in a sysplex
Related references
ENQ
[[ Contents Previous Page | Next Page Index ]]