This topic describes lifecycle management tasks for CM Server.
CM Server uses two J2C managed connection factories (one for ClearCase
and one for ClearQuest) to launch and control backend server processes that
communicate with the ClearCase and ClearQuest core products. The backend ONCRPC
processes (ccrpc processes for ClearCase and cqrpc processes for ClearQuest)
bridge the gap between the J2EE and Websphere components of the CM Server
and the ClearCase and ClearQuest core components.
The CM Server executes a number of background tasks to help manage and
control the full lifecycle of each launched backend ONCRPC server process;
no administrator intervention is required to enable these tasks to run.
There is one generic lifecycle task that performs non-product specific
lifecycle management, and product-specific tasks that control the ClearCase
and ClearQuest backend server object behaviors. Each task uses configurable
MBean attributes to help govern the number and longevity of active server
objects to help maintain the system's ability to keep up with the workload
and to prevent performance degradation due to excessive resource consumption.
See Setting available MBean attributes for information on the MBean attributes
and how to configure them.
Generic managed connection factory lifecycle management
The
generic lifecycle management task affects all backend ONCRPC processes created
by the connection factories. This task is launched from within the CM Server
when the CM Server is initially started. The task executes every two minutes
and is responsible for testing each backend ONCRPC server to ensure that each
server's backing process is still functioning normally and to clean up the
remnants of server processes that have been terminated.
Each time the
task executes it obtains a list of handles to the backend server objects launched
by the CM Server that are marked as RUNNING. Each backend server object is
tested to ensure that a functioning backing process is running. For any backend
process detected to be malfunctioning, the cleanup of the server object is
initiated as follows (log messages record such events):
- The server object is marked as non-functional so that no new work is given
to that server.
- An MBean cleanup notification is queued and within five seconds the notification
is asynchronously received and all cleanup work on the non-functioning server
object is performed.
- The server object is removed from the parent managed connection factory's
server object list
- The MBean associated with that server is unregistered.
Next the task obtains an updated list of STOPPING servers (servers
that are idle are marked this way). For each server object that has been in
the STOPPING state for at least 30 seconds the server object's backing process
is checked to ensure that the process has actually stopped. If the server
object's process has not stopped, the process is forcibly stopped and removed
from the factory's server list. Servers that have not been STOPPING for at
least 30 seconds are left in the factory's server list and are checked the
next time this task executes.
Product-specific managed connection factory lifecycle management
tasks
In addition to the generic lifecycle management task, an independent
ClearCase task and an independent ClearQuest task control the product-specific
lifecycle management of their respective backend server processes.
The
ClearQuest backend oncrpc server (referred to as a cqrpc process) is a multithreaded
process; there is typically a small number of cqrpc processes running at any
given time. The ClearCase backend oncrpc server (referred to as a ccrpc process)
is a single-threaded process; there are usually many of these running at any
given time. The product-specific managed connection factories use configurable
MBean attributes to help govern the number and longevity of active processes
to help maintain the system's ability to keep up with the workload while preventing
performance degradation due to excessive resource consumption.
There are a few
key MBean attributes used to govern the number of ccrpc processes to enable;
the default settings for these parameters may need to be modified based on
the type of system that the CM Server is installed on as well as the expected
server usage, based on the load and capacity:
- The CcServerFactoryMBean.serverThresholdCount MBean attribute is
the threshold number of CCRPC servers that once reached will trigger in-line
lifecycle management within the ClearCase managed Connection Factory.
- The CcServerFactoryMBean.maxServerCount MBean attribute is the
maximum number of CCRPC servers that can be created within the ClearCase managed
Connection Factory at one time.
- The CcServerFactoryMBean.maxServersPerCredential MBean attribute
limits client applications from creating too many threads per single user
session.
These values can be adjusted using the wsadmin command-line
utility. See Setting available MBean attributes for information on the
MBean attributes and how to configure them. Parameters such as memory size,
processor speed, and other system attributes govern how these and other CcServerFactoryMBean
values should be set.
When a ClearCase request comes into the CM Server
from a client, an attempt is made to obtain a backend server object to process
the request. The following in-line checks are performed at each request that
arrives:
- The list of ClearCase server objects is checked for a non-busy server
to process the request. A server object is considered busy if it is already
in the middle of processing another request. If an existing server object
is not busy and it matches the credentials of the request, the request is
serviced by that server object.
- If no existing server objects match the credentials or existing server
objects are busy, a new server object needs to be created and associated with
the request; this can be done only if the managed connection factory has the
capacity, as follows:
- If the CcServerFactoryMBean.maxServersPerCredential number of ccrpc
servers has been reached or if the CcServerFactoryMBean.maxServerCount number
of ccrpc servers has been reached, the worker thread assigned to process the
client request enters a wait-and-retry loop; up to TeamServerMBean.maxProcureServerAttempts attempts
will be made for up to a total of TeamServerMBean.procureServerInterval seconds
to acquire a server.
- If a server cannot be acquired then the client request is rejected. (Otherwise
a new server object is created and assigned to the request.)
In addition to the above in-line checks performed as each request
arrives, a background ClearCase-specific lifecycle management task performs
the following subtask:
Every two minutes a list of all ccrpc servers
that have been idle for
CcServerFactoryMBean.idleServerInterval seconds
or more is created and each server in this list is gracefully stopped as follows:
- The server instance is put into the STOPPING state so that no new work
is given to that server.
- An MBean TERMINATE notification is queued and within five seconds the
notification is asynchronously received and all cleanup of the server object
is performed.
- A shutdown RPC message is sent to the ccrpc server process and the server
instance is set to the STOPPED state.
- The current time is recorded and saved in the server instance so that
the generic CheckServer task knows how long the server has been in the STOPPED
state.
- The server object instance is removed from the connection factory's server
list.
- The MBean associated with that server is unregistered.
The
ClearQuest lifecycle management consists of one background task that runs
every two minutes, as well as some per-client-request foreground checks that
are made when the requests are processed. The checks are performed to determine
if any critical resource limits have been reached:
- If time-based recycling is enabled (that is, if CqServerFactoryMBean.recycleServerLifetimeLimit is
greater than zero and any cqrpc server object has been running for at least
that amount of time) then the cqrpc server object is marked as ready to recycle.
- When a client request has completed, a check is made to determine if the
cqrpc server object has processed at least CqServerFactoryMBean.recycleServerOncrpcCallLimit RPC
calls. If so, the cqrpc server object is marked as ready to recycle.
- When a client request arrives and the session count is incremented, a
check is made to determine if the cqrpc server object has created at least CqServerFactoryMBean.recycleServerHttpSessionLimit HTTP
sessions. If so, the cqrpc server object is marked as ready to recycle; in
this case a new cqrpc server object is started to handle the incoming request.
The ClearQuest managed connection factory recycles cqrpc server
objects instead of terminating them (as is done by the ClearCase managed connection
factory for ccrpc server objects that are no longer needed). The reason for
this is that cqrpc server objects may contain items such as a queries or records
that are yet to be committed; thus a grace period is allowed for any pending
work to be committed.
Recycling a cqrpc server objects consists of:
- The server object is put into the STOPPING state (also referred to as
the RECYCLING state) so that no new work is given to that server.
- An MBean RECYCLE notification is queued. The notification is asynchronously
received within five seconds; any READ-ONLY sessions associated with the server
object are marked to be reassociated with a new server object when the next
request for that session arrives.
Sessions that are in the middle of a
READ-WRITE transaction (that is, a transaction involving the commit of a query
or record) are allowed up to CqServerFactoryMBean.recyclingPeriod seconds
for the pending work to be committed. Once the work has been committed or
the CqServerFactoryMBean.recyclingPeriod has been reached, the server
object is put into the STOPPED state, its backing process is then terminated,
and the server object is removed from ClearQuest managed connection factory's
server list.