CM Server managed connection factory lifecycle management

This topic describes lifecycle management tasks for CM Server.

CM Server uses two J2C managed connection factories (one for ClearCase and one for ClearQuest) to launch and control backend server processes that communicate with the ClearCase and ClearQuest core products. The backend ONCRPC processes (ccrpc processes for ClearCase and cqrpc processes for ClearQuest) bridge the gap between the J2EE and Websphere components of the CM Server and the ClearCase and ClearQuest core components.

The CM Server executes a number of background tasks to help manage and control the full lifecycle of each launched backend ONCRPC server process; no administrator intervention is required to enable these tasks to run.

There is one generic lifecycle task that performs non-product specific lifecycle management, and product-specific tasks that control the ClearCase and ClearQuest backend server object behaviors. Each task uses configurable MBean attributes to help govern the number and longevity of active server objects to help maintain the system's ability to keep up with the workload and to prevent performance degradation due to excessive resource consumption. See Setting available MBean attributes for information on the MBean attributes and how to configure them.

Generic managed connection factory lifecycle management

The generic lifecycle management task affects all backend ONCRPC processes created by the connection factories. This task is launched from within the CM Server when the CM Server is initially started. The task executes every two minutes and is responsible for testing each backend ONCRPC server to ensure that each server's backing process is still functioning normally and to clean up the remnants of server processes that have been terminated.

Each time the task executes it obtains a list of handles to the backend server objects launched by the CM Server that are marked as RUNNING. Each backend server object is tested to ensure that a functioning backing process is running. For any backend process detected to be malfunctioning, the cleanup of the server object is initiated as follows (log messages record such events):
  • The server object is marked as non-functional so that no new work is given to that server.
  • An MBean cleanup notification is queued and within five seconds the notification is asynchronously received and all cleanup work on the non-functioning server object is performed.
  • The server object is removed from the parent managed connection factory's server object list
  • The MBean associated with that server is unregistered.

Next the task obtains an updated list of STOPPING servers (servers that are idle are marked this way). For each server object that has been in the STOPPING state for at least 30 seconds the server object's backing process is checked to ensure that the process has actually stopped. If the server object's process has not stopped, the process is forcibly stopped and removed from the factory's server list. Servers that have not been STOPPING for at least 30 seconds are left in the factory's server list and are checked the next time this task executes.

Product-specific managed connection factory lifecycle management tasks

In addition to the generic lifecycle management task, an independent ClearCase task and an independent ClearQuest task control the product-specific lifecycle management of their respective backend server processes.

The ClearQuest backend oncrpc server (referred to as a cqrpc process) is a multithreaded process; there is typically a small number of cqrpc processes running at any given time. The ClearCase backend oncrpc server (referred to as a ccrpc process) is a single-threaded process; there are usually many of these running at any given time. The product-specific managed connection factories use configurable MBean attributes to help govern the number and longevity of active processes to help maintain the system's ability to keep up with the workload while preventing performance degradation due to excessive resource consumption.

There are a few key MBean attributes used to govern the number of ccrpc processes to enable; the default settings for these parameters may need to be modified based on the type of system that the CM Server is installed on as well as the expected server usage, based on the load and capacity:
  • The CcServerFactoryMBean.serverThresholdCount MBean attribute is the threshold number of CCRPC servers that once reached will trigger in-line lifecycle management within the ClearCase managed Connection Factory.
  • The CcServerFactoryMBean.maxServerCount MBean attribute is the maximum number of CCRPC servers that can be created within the ClearCase managed Connection Factory at one time.
  • The CcServerFactoryMBean.maxServersPerCredential MBean attribute limits client applications from creating too many threads per single user session.

These values can be adjusted using the wsadmin command-line utility. See Setting available MBean attributes for information on the MBean attributes and how to configure them. Parameters such as memory size, processor speed, and other system attributes govern how these and other CcServerFactoryMBean values should be set.

When a ClearCase request comes into the CM Server from a client, an attempt is made to obtain a backend server object to process the request. The following in-line checks are performed at each request that arrives:
  • The list of ClearCase server objects is checked for a non-busy server to process the request. A server object is considered busy if it is already in the middle of processing another request. If an existing server object is not busy and it matches the credentials of the request, the request is serviced by that server object.
  • If no existing server objects match the credentials or existing server objects are busy, a new server object needs to be created and associated with the request; this can be done only if the managed connection factory has the capacity, as follows:
    • If the CcServerFactoryMBean.maxServersPerCredential number of ccrpc servers has been reached or if the CcServerFactoryMBean.maxServerCount number of ccrpc servers has been reached, the worker thread assigned to process the client request enters a wait-and-retry loop; up to TeamServerMBean.maxProcureServerAttempts attempts will be made for up to a total of TeamServerMBean.procureServerInterval seconds to acquire a server.
    • If a server cannot be acquired then the client request is rejected. (Otherwise a new server object is created and assigned to the request.)

In addition to the above in-line checks performed as each request arrives, a background ClearCase-specific lifecycle management task performs the following subtask:

Every two minutes a list of all ccrpc servers that have been idle for CcServerFactoryMBean.idleServerInterval seconds or more is created and each server in this list is gracefully stopped as follows:
  • The server instance is put into the STOPPING state so that no new work is given to that server.
  • An MBean TERMINATE notification is queued and within five seconds the notification is asynchronously received and all cleanup of the server object is performed.
  • A shutdown RPC message is sent to the ccrpc server process and the server instance is set to the STOPPED state.
  • The current time is recorded and saved in the server instance so that the generic CheckServer task knows how long the server has been in the STOPPED state.
  • The server object instance is removed from the connection factory's server list.
  • The MBean associated with that server is unregistered.
The ClearQuest lifecycle management consists of one background task that runs every two minutes, as well as some per-client-request foreground checks that are made when the requests are processed. The checks are performed to determine if any critical resource limits have been reached:
  • If time-based recycling is enabled (that is, if CqServerFactoryMBean.recycleServerLifetimeLimit is greater than zero and any cqrpc server object has been running for at least that amount of time) then the cqrpc server object is marked as ready to recycle.
  • When a client request has completed, a check is made to determine if the cqrpc server object has processed at least CqServerFactoryMBean.recycleServerOncrpcCallLimit RPC calls. If so, the cqrpc server object is marked as ready to recycle.
  • When a client request arrives and the session count is incremented, a check is made to determine if the cqrpc server object has created at least CqServerFactoryMBean.recycleServerHttpSessionLimit HTTP sessions. If so, the cqrpc server object is marked as ready to recycle; in this case a new cqrpc server object is started to handle the incoming request.

The ClearQuest managed connection factory recycles cqrpc server objects instead of terminating them (as is done by the ClearCase managed connection factory for ccrpc server objects that are no longer needed). The reason for this is that cqrpc server objects may contain items such as a queries or records that are yet to be committed; thus a grace period is allowed for any pending work to be committed.

Recycling a cqrpc server objects consists of:
  • The server object is put into the STOPPING state (also referred to as the RECYCLING state) so that no new work is given to that server.
  • An MBean RECYCLE notification is queued. The notification is asynchronously received within five seconds; any READ-ONLY sessions associated with the server object are marked to be reassociated with a new server object when the next request for that session arrives.

    Sessions that are in the middle of a READ-WRITE transaction (that is, a transaction involving the commit of a query or record) are allowed up to CqServerFactoryMBean.recyclingPeriod seconds for the pending work to be committed. Once the work has been committed or the CqServerFactoryMBean.recyclingPeriod has been reached, the server object is put into the STOPPED state, its backing process is then terminated, and the server object is removed from ClearQuest managed connection factory's server list.


Feedback