This document describes the Grid Engine source code modules, and is written for use by software developers. Our goal is to provide you with useful information if you are considering participating in the Grid Engine project or will be using Grid Engine source code. In addition to this source documentation, the user product documentation is necessary for a full understanding of the functional operation of the software. Consequently the man pages are refered to herein where appropriate.

If you have problems with finding functions or browsing through the souce code we suggest the use of grep or preferably ctags.

1. Job life cycle

The job life cycle is a good way to understand to overall system. 
  1. Execds report load information to qmaster
  2. User submits job using qsub command
  3. Qmaster notifies Schedd about new job
  4. Schedd dispatches job to an Execd
  5. Qmaster delivers job to Execd; Execd starts job using Shepherd
  6. At job end Execd notfies Qmaster about job finish
  7. Qmaster feeds jobs resource consumption in accounting database

2. Components

2.1 Daemon components

In the "source/daemons" subdirectory there is one directory for each daemon.

    Qmaster

    The Qmaster (Queue Master) controls the overall behaviour in a cluster. For the purpose of scheduling jobs Qmaster cooperates with Schedd. The Qmaster is the responsible for answering GDI requests from clients and for delivering dispatched jobs to the assigned Execd's. 

    Schedd

    The Schedd (Scheduling Daemon) gets notified via events about all scheduling relevant information. The resulting scheduling decisions are sent as orders to Qmaster. 

    Execd

    The Execd (Execution Daemon) provides Qmaster with information about utilization and availability of resources. Each job sent to Execd by Qmaster is started by writing all relevant information into a couple files describing the job and forking a Sheperd. After Shepherds termination Execd reports details about the job run back to Qmaster. 

    Shepherd

    The Shepherd starts all kinds of jobs according to what he finds in the per-job configuration files written by Execd. If necessary administrator supplied procedures are started before the job is started (prolog, PE start-up), during job run (checkpoint/migrate, terminate/suspend/resume) and after job termination (PE shut-down, epilog). 

    Commd

    Nearly all network communication in a cluster is handled via the Commd (Communication Daemon). Daemons and clients use the commlib (communication library) calls to send and receive messages (sync/async), but also a hostname resolving service is provided by Commd. 

    Shadowd

    The Shadowd (Shadow Daemon) detects failures of the Qmaster and starts a new Qmaster if necessary. 

2.2 Client components

In the "source/clients" subdirectory there is one directory for each client.

    Qsub

    Qsub combines all information provided with the command line and in per-cluster or per-user default settings files in a job object and sends the job via GDI to Qmaster. As response Qsub receives a job ID and prints it. 

    Qrsh

    Qrsh combines all job relevant information in a job object like Qsub does and sends it as an interactive job via GDI to Qmaster. For stdin/stdout/stderr redirection rsh command is forked by qrsh. The rsh command cooperates with the rshd forked by shepherd for jobs of that type. Instead of rsh/rshd also other commands can be used. 

    Qmon

    Qmon is a Motif GUI allowing to administer and monitor the system but also to submit and monitor jobs. Qmon uses GDI to retrieve from Qmaster the whole configuration and to change it. 

    Qdel

    Qdel uses GDI to initiate job termination. 

    Qacct

    Qacct produces accounting reports by scanning the accounting data file. 

    Qlogin

    Qlogin invokes an interactive login session in the current terminal using GDI. 

    Qresub

    Qresub creates a copy from an existing job using GDI. 

    Qselect

    Qselect prints a list of queues matching the specified criteria from the command line. The necessary information about the clusters configuration is fetched from Qmaster using GDI. 

    Qalter

    Qalter modifies an already submitted job using GDI. 

    Qhold

    Qhold puts a hold on a job using Qalter. 

    Qrls

    Qhold releases a hold from a job using Qalter. 

    Qsh

    Qsh submits an interactive X-windows session job using GDI. 

    Qconf

    Qconf allows to add, delete, modify and view the configuration of all kind of configuration objects. Qconf uses GDI to do this. 

    Qhost

    Qhost shows the current status of all hosts in the cluster based on information retrieved using GDI. 

    Qstat

    Qstat shows the current status of all queues in the cluster based on information retrieved using GDI. 

    Qmod

    Qmod allows to modify the state of queues and jobs by using GDI. 

2.3 Libraries

In the "source/libs" subdirectory there is one directory for each library.

    librmon

    This low level library provides functionality to run daemons in a monitoring mode. 

    libuti

    A low level library with several utility functions. This library bases on functionality of librmon. 

    libcull

    The generic list library which is used in nearly all components of the system. The cull library bases on functionality of libuti and librmon. 

    libcomm

    The communication libraryis the interface providing communication via Commd. This library bases on librmon. 

    libgdi

    The Gridengine Database Interface library provides methods to retrieve, add, delete and modify all internal objects hosted by the Qmaster. The GDI library bases on libcomm and libcull. 

    libsched

    This library contains many important high-level functions mainly used by Schedd. Some of them however have turned out to be very helpful also in other clients like Qstat, Qhost or qmon and also in Qmaster. 

    event client interface

    The event client interface allows a client to register at qmaster and receive events about changes in the system more user friendly than with polling. 

2.4 Third-party components

The "source/3rdparty" directory contains modules with different licenses than SISSL. 

    Qmake

    This client is a GNU make enhanced with transparent remote execution. The implementation of transparent remote execution bases on a forked 'qrsh -inherit' command. 

    Qtcsh

    This client program is an extension to the popular csh(1) derivative tcsh. The implementation of transparent remote execution bases on libgdi. 

    Remote tools

    This directory is the home of rsh, rshd, rlogin.

2.5 Utilities

The "source/utilbin" directory contains binaries that are needed during installation of Grid Engine, for diagnosis purposes or for the job samples.

2.6 Dist

The "source/dist" directory contains files and directories which are part of a Grid Engine distribution.

3. Security

The "source/security" directory contains modules handling security.

    Security

    Contains security related modules.

4. Experimental

Each subdirectory of the "source/experimental" directory contains an experimental module. Those modules cannot be expected to work - although some may, at least with a bit of work. The experimental modules are provided here for information purposes mainly and may be helpful if you plan to enhance Grid Engine in a direction similar to one of the modules. Some also can be a valid starting point for such an effort.

This site sponsored by Sun Microsystems, Inc.
Copyright © 2001 Collab.Net, Inc. All rights reserved. Copyright © 2001 Sun Microsystems, Inc. All rights reserved.
Home | Help | Feedback | FAQ | Legalese