Contents
Abstract
Preface
What this book is about
Who this book is for
What you need to know to understand this book
How to use this book
Notes about terms used in this book
Softcopy links
Summary of changes
Changes for CICS Transaction Server for z/OS, Version 3 Release 1
Changes for CICS Transaction Server for z/OS, Version 2 Release 3
Changes for CICS Transaction Server for z/OS, Version 2 Release 2
Changes forCICS Transaction Server for z/OS, Version 2 Release 1
Changes for CICS Transaction Server for OS/390, Version 1 Release 2
Changes for CICS Transaction Server for OS/390, Version 1 Release 1
Changes for CICS for MVS/ESA 4.1
Figures
Tables
Approaches to problem determination
Introduction to problem determination
Before you start--preliminary checks
What to do next
Classifying the problem
Using symptom keywords as a basis for classifying problems
Using the symptoms to classify the problem
CICS has stopped running
CICS is running slowly
A task fails to start
A task is running slowly
A task stops running at a terminal
A transaction has abended
You have obtained some incorrect output
A storage violation has occurred
An XRF error has occurred
Distinguishing between waits, loops, and poor performance
Waits
Loops
Poor performance
Poor application design
Classifying problems by functional area
What to do next
Sources of information
Your own documentation
Product information
Source listings and link-edit maps
Abend codes and error messages
Symptom strings
Change log
Dumps
Statistics
Monitoring
Transaction inputs and outputs
Terminal data
Transient data and temporary storage
Passed information
Files and databases
Traces
Dealing with the problem
Dealing with transaction abend codes
Collecting the evidence
Symptom string
What the abend code can tell you
Transaction abend codes -- AICA, ASRA, ASRB, ASRD and AEYD
AICA abends
ASRA abends
ASRB abends
ASRD abends
AEYD abends
Finding where a program check occurred
What type of program check occurred?
Dealing with arithmetic exceptions
Dealing with protection exceptions
Causes of protection exceptions
Transaction isolation
Command protection
Possible causes of protection exceptions referencing CICS DSAs
Protection exceptions referencing the read-only DSAs
Protection exceptions referencing the UDSA and EUDSA
Analyzing the problem further
Abends when CICS is using the DBCTL interface
Worksheet for transaction abends
FEPI abends
Dealing with CICS system abends
The documentation you need
Interpreting the evidence
Looking at the messages
Looking at the exception trace entry
Looking at the symptom string in the dump
Looking at the kernel domain storage areas
Finding which tasks are associated with the error
Finding more information about the error
Using the linkage stack to identify the failing module
Using the PSW to find the offset of the failing instruction
Finding the PTF level of the module in error
Dealing with waits
Techniques for investigating waits
Investigating waits--online method
Investigating waits--using trace
Investigating waits--the formatted CICS system dump
Investigating terminal waits
Terminal waits--first considerations
Terminal waits--a systematic approach
VTAM in use--debugging procedures
Tools you can use for debugging terminal waits when VTAM is in use
Your task is waiting on a physical terminal
Investigating storage waits
Was the request for too much storage?
Is the storage close to being exhausted?
Is fragmentation of free storage causing the GETMAIN request to fail?
Investigating temporary storage waits
Resource type ENQUEUE
Resource type TSAUX
Resource type TSBUFFER
Resource type TSEXTEND
Resource type TSIO
Resource type TSPOOL
Resource type TSQUEUE
Resource type TSSHARED
Resource type TSSTRING
Resource type TSWBUFFR
Investigating enqueue waits
Using a system dump to resolve enqueue waits
EXEC CICS ENQ waits
Investigating interval control waits
Finding the reason for a DELAY request not completing
Using trace to find out why tasks are waiting on interval control
Investigating file control waits
Resource type CFDTWAIT--wait for CFDT request to complete
Resource type CFDTPOOL - wait for CFDT a request slot
Resource type CFDTLRSW - wait for CFDT locking request slot
Resource type FCACWAIT & FCCRWAIT--wait for SMSVSAM cleanup
Resource type FCBFWAIT--waits for VSAM buffers
Resource type FCCAWAIT--waits on the SMSVSAM control ACB
Resource type FCCFQR--wait for SMSVSAM server notification
Resource type FCCFQS--wait for user task to issue quiesce
Resource type FCDWWAIT--wait for VSAM to complete update processing
Resource type FCFSWAIT--wait for file state changes
Resource type FCIOWAIT--wait for VSAM I/O (non-RLS)
Resource type FCIRWAIT--wait for FC environment to be rebuilt
Resource types FCPSWAIT and FCSRSUSP--waits for VSAM strings
Resource type FCQUIES--wait for a quiesce request to complete
Resource type FCRAWAIT--file control to process non-recoverable requests
Resource type FCRBWAIT--file control to process recoverable requests
Resource type FCRDWAIT--wait for a drain of the RLS control ACB
Resource type FCRPWAIT--wait for file control initialization to complete
Resource Type FCRRWAIT--wait for dynamic RLS restart to complete
Resource type FCRVWAIT--wait for VSAM I/O (RLS)
Resource type FCTISUSP--wait for a VSAM transaction ID
Resource type FCXCWAIT--VSAM exclusive control wait
Resource type ENQUEUE--waits for locks on files or data tables
Investigating loader waits
Investigating lock manager waits
Collecting the evidence
Investigating transaction manager waits
Maximum task condition waits
Transaction summary
MXT summary
Transaction class summary
A user task is waiting on resource type FOREVER
Resource type TRANDEF
Resolving deadlocks in a CICS region
An example of deadlock diagnosis using CEMT INQUIRE UOWENQ
Resolving deadlocks in a sysplex
Resolving in-doubt and resynchronization failures
What to do if CICS has stalled
CICS has stalled during initialization
CICS has stalled during a run
CICS has stalled during termination
How tasks are made to wait
The resources on which tasks in a CICS system can wait
Dispatcher waits
Resource name JVM_POOL
Resource name OPENPOOL
Resource name OPEN_DEL
Resource type DSTSKDEF
Resource type SOSMVS
CICS DB2 waits
Resource type CDB2CONN
Resource type CDB2RDYQ
Resource type CDB2TCB
Resource type DB2
Resource type DB2_INIT
Resource type DB2CDISC
Resource type DB2EDISA
DBCTL waits
Connection to DBCTL has failed to complete
A user task is waiting on resource type DBCTL
Disconnection from DBCTL has failed to complete
EDF waits
Log manager waits
Resource type LG_DEFER
Resource type LG_FORCE
Resource type LGDELALL
Resource type LGDELRAN
Resource type LGENDBLK
Resource type LGENDCRS
Resource type LGREDBLK
Resource type LGREDCRS
Resource type LGSTRBLK
Resource type LGSTRCRS
Resource type LGWRITE
Task control waits
Resource type KCCOMPAT
Resource type KC_ENQ
VTAM terminal control waits
Resource type ZC
Resource type ZC_ZCGRP
Resource type ZC_ZGCH
Resource type ZC_ZGIN
Resource type ZC_ZGRP, resource name PSINQECB
Resource type ZC_ZGRP, resource name PSOP2ECB
Resource type ZC_ZGUB
Resource type ZCIOWAIT
Resource type ZCZGET
Resource type ZCZNAC
Resource type ZXQOWAIT
Resource type ZXSTWAIT
Interregion and intersystem communication waits
IIOP waits
Transient data waits
Resource type TD_INIT--waits during initialization processing
Resource type TDEPLOCK-waits for transient data extrapartition requests
Resource types TDIPLOCK, ENQUEUE, TD_READ, Any_MBCB, Any_MRCB, MBCB_xxx, and MRCB_xxx
XRF alternate system waits
XRSTIECB
XRSIAECB
XRSTCECB
XRSRAECB
XRSSSECB
XRSSTECB
CICS system task waits
FEPI waits
Recovery manager waits
Resource type RMCLIENT
Resource type RMUOWOBJ, resource name LOGMOVE
Resource type RMUOWOBJ, resource name EXISTENC
CICS Web waits
Resource type WBALIAS
Dealing with loops
What sort of loop is indicated by the symptoms?
Tight loops and non-yielding loops
Yielding loops
Investigating loops that cause transactions to abend with abend code AICA
The documentation you need
Looking at the evidence
Identifying the loop
Finding the reason for the loop
Investigating loops that are not detected by CICS
The documentation you need
Identifying the loop
Finding the reason for the loop
What to do if you cannot find the reason for a loop
Investigating loops using interactive tools
Modifying your program to investigate the loop
Dealing with performance problems
Finding the bottleneck
Initial attach to the transaction manager
Initial attach to the dispatcher
Why tasks fail to get attached to the transaction manager
Why tasks fail to get attached to the dispatcher
Is the MXT limit preventing tasks from getting attached?
Why tasks fail to get an initial dispatch
Priorities of tasks
Why tasks take a long time to complete
The effect of system loading on performance
The effect of task time-out interval on performance
The distribution of data sets on DASD volumes
A summary of performance bottlenecks, symptoms, and causes
Dealing with incorrect output
Trace output is incorrect
Tracing has gone to the wrong destination
You have captured the wrong trace data
Dump output is incorrect
The dump does not seem to relate to your CICS region
You do not get a dump when an abend occurs
Some dump IDs are missing from the sequence of dumps
You do not get the correct data when formatting the CICS system dump
Incorrect data is displayed on a terminal
The preliminary information you need to get
Tools for debugging terminal output in a VTAM environment
Specific types of incorrect output for terminals
Incorrect data is present on a VSAM data set
An application does not work as expected
General points for you to consider
Your transaction produces no output at all
Are there any messages explaining why there is no output?
Can you use the terminal where the transaction should have started?
No output--what to do if the task is still in the system
No output--what to do if the task is not in the system
Did the task run? Techniques for all tasks
Investigating tasks initiated by ATI
Your transaction produces some output, but it is wrong
The origins of corrupted data
Are records in the file incorrect or missing?
Is the data mapped correctly into the program?
Is the data being corrupted by bad programming logic?
Is the data being mapped incorrectly to the terminal?
Dealing with storage violations
Avoiding storage violations
Two kinds of storage violation
CICS has detected a storage violation
What happens when CICS detects a storage violation
What to do if you cannot find what is overlaying the SAA
Storage violations that affect innocent transactions
A strategy for storage violations affecting innocent transactions
Procedure for resolving storage violations affecting innocent transactions
What to do if you still cannot find the cause of the overlay
Programming errors that can cause storage violations
Storage recovery
Dealing with external CICS interface (EXCI) problems
Dealing with MRO problems
Dealing with log manager problems
Categories of problem
Exceeding the capacity of a log stream
System log
General logs
How CICS checks for the availability of the MVS logger
Some conditions that cause CICS log manager error messages
Message DFHLG0772
Message DFHLG0002
Restarting CICS after a system log failure
Diagnosing problems in the MVS logger
Console messages and dumps
GRS resource contention
Checking coupling facility structure and couple data set status
Checking log stream status
SMF and RMF statistics
Obtaining MVS logger and coupling facility dumps
Restarting the MVS logger address space
Dealing with a corrupt system log
The CICS diagnostic run mechanism
Getting dumps of the MVS logger and coupling facility address spaces
Using traces and dumps in problem determination
Using traces in problem determination
CICS tracing
Trace points
Trace levels
CICS exception tracing
User exception trace entries
CICS XRF tracing
Entry types
Program check and abend tracing
CICS VTAM exit tracing
Controlling CICS VTAM exit tracing
Interpreting CICS VTAM exit trace entries
VTAM buffer tracing
Selecting tracing by transaction
Tracing for selected tasks
The tracing logic used by CICS
Selecting tracing by component
Component names and abbreviations
Defining tracing for JVMs
Selecting JVM trace options
Interpreting JVM trace
Selecting trace destinations and related options
CICS internal trace
CICS auxiliary trace
CICS GTF trace
Setting the tracing status
Setting the tracing status at system initialization
Formatting and interpreting trace entries
Interpreting extended-format CICS system trace entries
Interpreting short-format CICS system trace entries
Interpreting abbreviated-format CICS system trace entries
Interpreting user trace entries
Using dumps in problem determination
Setting up the dumping environment
Detecting and avoiding duplicate system dumps
Where dumps are written
Events that can cause dumps to be taken
The ways that you can request dumps
The occasions when CICS requests a dump
CICS dumping in a sysplex
Automatic dump data capture from related CICS regions
Operator-requested simultaneous dump data capture
Requesting dumps to resolve SMSVSAM problems
Useful CICS master terminal and MVS console commands in a sysplex
Enabling system dumps for some CICS messages
System dump actions with messages DFHAP0001 and DFHSR0001
The dump code options you can specify
Specifying the areas you want written to a transaction dump
Dump table statistics
What happens to a dump request if there is no dump table entry?
The transaction dump table
The system dump table
Dumping a CFDT list structure
Dumping a named counter list structure
Dumping a shared temporary storage list structure
The CSFE ZCQTRACE facility
Formatting and interpreting dumps
Formatting transaction dumps
Formatting system dumps
The DFHIPCSP CICS exit control data
A summary of system dump formatting keywords and levels
The default SDUMP formatting levels
Interpreting transaction dumps
Job log
Symptom string
CICS Transaction Server for z/OS level
Transaction environment summary
Remote abend indicator
PSW
Registers at the time of interrupt
Execution key
Space
Registers at last EXEC command
Task control area (TCA)
Task control area, system area
Transaction work area (TWA)
EXEC interface structure (EIS)
System EXEC interface block (SYSEIB)
EXEC interface user structure (EIUS)
EXEC interface block (DFHEIB)
Kernel stack entries
Common system area (CSA)
Common system area optional features list (CSAOPFL)
Trace table (abbreviated format)
Trace table (extended format)
Common work area (CWA)
Transaction storage
Terminal control table terminal entry (TCTTE)
Program information for the current transaction
Module index
Locating the last command or statement
Last command identification
Last statement identification
Locating program data
Dumps for C/370 programs
Location of COBOL dumped areas
Storage freeze
Formatting a coupling facility data table pool dump
Formatting a named counter pool dump
Formatting a shared temporary storage pool dump
The global trap/trace exit
Establishing the exit
Information passed to the exit
Actions the exit can take
Program check handling
Coding the exit
Working with IBM to solve your problem
IBM program support
When to contact the Support Center
Dealing with the Support Center
What the Support Center needs to know
What happens next
Reporting a FEPI problem to IBM
APARs, fixes, and PTFs
The APAR process
Collecting the documentation for the APAR
General documentation needed for all problems with CICS
Sending the documentation to the change team
Packing and mailing the APAR box
Applying the fix
The APAR becomes a PTF
Appendixes
Appendix A. SDUMP contents and IPCS CICS VERBEXIT keywords
Finding the control blocks from the keywords
Finding the keywords from the control blocks
Appendix B. Summary data for PG and US keywords
PG keyword
PGA (program manager anchor)
System LLE Summary
PGWE Summary
PPTE Summary
PTA Summary
Task LLE Summary
CHCB Summary
CRCB Summary
Task PLCB Summary
US keyword
USXD summary
USUD summary
Bibliography
The CICS Transaction Server for z/OS library
The entitlement set
PDF-only books
Other CICS books
Accessibility
Index
Notices
Trademarks
[[
Contents
Previous Page
|
Next Page
Index
]]