This topic describes lifecycle management tasks for CM
Server.
CM Server uses two J2C managed connection factories (one for ClearCase
and one for ClearQuest) to launch and control backend server processes
that communicate with the ClearCase and ClearQuest core products.
The backend ONCRPC processes (ccrpc processes for ClearCase and cqrpc
processes for ClearQuest) bridge the gap between the J2EE and Websphere
components of the CM Server and the ClearCase and ClearQuest core
components.
The CM Server executes a number of background tasks to help manage
and control the full lifecycle of each launched backend ONCRPC server
process; no administrator intervention is required to enable these
tasks to run.
There is one generic lifecycle task that performs non-product specific
lifecycle management, and product-specific tasks that control the
ClearCase and ClearQuest backend server object behaviors. Each task
uses configurable MBean attributes to help govern the number and longevity
of active server objects to help maintain the system's ability to
keep up with the workload and to prevent performance degradation due
to excessive resource consumption. See Setting available MBean attributes for
information on the MBean attributes and how to configure them.
Generic managed connection factory lifecycle management
The
generic lifecycle management task affects all backend ONCRPC processes
created by the connection factories. This task is launched from within
the CM Server when the CM Server is initially started. The task executes
every two minutes and is responsible for testing each backend ONCRPC
server to ensure that each server's backing process is still functioning
normally and to clean up the remnants of server processes that have
been terminated.
Each time the task executes it obtains a list
of handles to the backend server objects launched by the CM Server
that are marked as RUNNING. Each backend server object is tested to
ensure that a functioning backing process is running. For any backend
process detected to be malfunctioning, the cleanup of the server object
is initiated as follows (log messages record such events):
- The server object is marked as non-functional so that no new work
is given to that server.
- An MBean cleanup notification is queued and within five seconds
the notification is asynchronously received and all cleanup work on
the non-functioning server object is performed.
- The server object is removed from the parent managed connection
factory's server object list
- The MBean associated with that server is unregistered.
Next the task obtains an updated list of STOPPING servers
(servers that are idle are marked this way). For each server object
that has been in the STOPPING state for at least 30 seconds the server
object's backing process is checked to ensure that the process has
actually stopped. If the server object's process has not stopped,
the process is forcibly stopped and removed from the factory's server
list. Servers that have not been STOPPING for at least 30 seconds
are left in the factory's server list and are checked the next time
this task executes.
Product-specific managed connection factory lifecycle
management tasks
In addition to the generic lifecycle management
task, an independent ClearCase task and an independent ClearQuest
task control the product-specific lifecycle management of their respective
backend server processes.
The ClearQuest backend oncrpc server
(referred to as a cqrpc process) is a multithreaded process; there
is typically a small number of cqrpc processes running at any given
time. The ClearCase backend oncrpc server (referred to as a ccrpc
process) is a single-threaded process; there are usually many of these
running at any given time. The product-specific managed connection
factories use configurable MBean attributes to help govern the number
and longevity of active processes to help maintain the system's ability
to keep up with the workload while preventing performance degradation
due to excessive resource consumption.
ClearCase managed connection factory
lifecycle management
There are a few key MBean
attributes used to govern the number of ccrpc processes to enable;
the default settings for these parameters may need to be modified
based on the type of system that the CM Server is installed on as
well as the expected server usage, based on the load and capacity:
- The CcServerFactoryMBean.serverThresholdCount MBean attribute
is the threshold number of CCRPC servers that once reached will trigger
in-line lifecycle management within the ClearCase managed Connection
Factory.
- The CcServerFactoryMBean.maxServerCount MBean attribute
is the maximum number of CCRPC servers that can be created within
the ClearCase managed Connection Factory at one time.
- The CcServerFactoryMBean.maxServersPerCredential MBean
attribute limits client applications from creating too many threads
per single user session.
These values can be adjusted using the wsadmin command-line
utility. See Setting available MBean attributes for information
on the MBean attributes and how to configure them. Parameters such
as memory size, processor speed, and other system attributes govern
how these and other CcServerFactoryMBean values should be set.
When
a ClearCase request comes into the CM Server from a client, an attempt
is made to obtain a backend server object to process the request.
The following in-line checks are performed at each request that arrives:
- The list of ClearCase server objects is checked for a non-busy
server to process the request. A server object is considered busy
if it is already in the middle of processing another request. If an
existing server object is not busy and it matches the credentials
of the request, the request is serviced by that server object.
- If no existing server objects match the credentials or existing
server objects are busy, a new server object needs to be created and
associated with the request; this can be done only if the managed
connection factory has the capacity, as follows:
- If the CcServerFactoryMBean.maxServersPerCredential number
of ccrpc servers has been reached or if the CcServerFactoryMBean.maxServerCount number
of ccrpc servers has been reached, the worker thread assigned to process
the client request enters a wait-and-retry loop; up to TeamServerMBean.maxProcureServerAttempts attempts
will be made for up to a total of TeamServerMBean.procureServerInterval seconds
to acquire a server.
- If a server cannot be acquired then the client request is rejected.
(Otherwise a new server object is created and assigned to the request.)
In addition to the above in-line checks performed as each
request arrives, a background ClearCase-specific lifecycle management
task performs the following subtask:
Every two minutes
a list of all ccrpc servers that have been idle for
CcServerFactoryMBean.idleServerInterval seconds
or more is created and each server in this list is gracefully stopped
as follows:
- The server instance is put into the STOPPING state so that no
new work is given to that server.
- An MBean TERMINATE notification is queued and within five seconds
the notification is asynchronously received and all cleanup of the
server object is performed.
- A shutdown RPC message is sent to the ccrpc server process and
the server instance is set to the STOPPED state.
- The current time is recorded and saved in the server instance
so that the generic CheckServer task knows how long the server has
been in the STOPPED state.
- The server object instance is removed from the connection factory's
server list.
- The MBean associated with that server is unregistered.
ClearQuest managed connection factory lifecycle management
The
ClearQuest lifecycle management consists of one background task that
runs every two minutes, as well as some per-client-request foreground
checks that are made when the requests are processed. The checks are
performed to determine if any critical resource limits have been reached:
- If time-based recycling is enabled (that is, if CqServerFactoryMBean.recycleServerLifetimeLimit is
greater than zero and any cqrpc server object has been running for
at least that amount of time) then the cqrpc server object is marked
as ready to recycle.
- When a client request has completed, a check is made to determine
if the cqrpc server object has processed at least CqServerFactoryMBean.recycleServerOncrpcCallLimit RPC
calls. If so, the cqrpc server object is marked as ready to recycle.
- When a client request arrives and the session count is incremented,
a check is made to determine if the cqrpc server object has created
at least CqServerFactoryMBean.recycleServerHttpSessionLimit HTTP
sessions. If so, the cqrpc server object is marked as ready to recycle;
in this case a new cqrpc server object is started to handle the incoming
request.
The ClearQuest managed connection factory recycles cqrpc
server objects instead of terminating them (as is done by the ClearCase
managed connection factory for ccrpc server objects that are no longer
needed). The reason for this is that cqrpc server objects may contain
items such as a queries or records that are yet to be committed; thus
a grace period is allowed for any pending work to be committed.
Recycling
a cqrpc server objects consists of:
- The server object is put into the STOPPING state (also referred
to as the RECYCLING state) so that no new work is given to that server.
- An MBean RECYCLE notification is queued. The notification is asynchronously
received within five seconds; any READ-ONLY sessions associated with
the server object are marked to be reassociated with a new server
object when the next request for that session arrives.
Sessions
that are in the middle of a READ-WRITE transaction (that is, a transaction
involving the commit of a query or record) are allowed up to CqServerFactoryMBean.recyclingPeriod seconds
for the pending work to be committed. Once the work has been committed
or the CqServerFactoryMBean.recyclingPeriod has been reached,
the server object is put into the STOPPED state, its backing process
is then terminated, and the server object is removed from ClearQuest
managed connection factory's server list.