Grid Engine FAQ


 

General Overview

Q: What is Grid Engine?

A: Grid Engine is Distributed Resource Management (DRM) software. DRM software aggregates compute power and delivers it as a network service. Grid Engine software is used with a set of computers to create powerful compute farms, which are used in a wide range of technical computing applications such as the development of semiconductors, bioinformatics, mechanical design, software development, oil/gas exploration, and financial analysis; additionally, massively scaling supercomputers use Grid Engine software in a variety of academic and research pursuits. Users enjoy access to large computing capability and organizations enjoy effective utilization of their computing resource investments approaching 100%.

Q: How can Grid Engine can help me?

A: Grid Engine is helpful in three elementary ways. It can:

  • optimally place computing tasks and balance the load on a set of networked computers
  • allow users to generate and queue more computing tasks than can be run at the moment
  • ensure that tasks are executed with respect to priority and to providing all users with a fair share of access over time
Essentially Grid Engine presents users a seamless, integrated computing capability. Grid Engine is used to support a wide variety of requirements; for instance, where users start many interactive and batch tasks as in product design or financial simulations; where sets of repetitious tasks are run as in software QA; where large numbers of users are placing jobs on limited resources as in education environments; and where users are launching parallel applications across massive numbers of processors for applications such as weather simulation.

Q: Why is Sun launching the Grid Engine project?

A: Sun believes the era of distributed, networked computing is just beginning, and Sun wants to accelerate the adoption of the model by providing high quality enabling DRM technology, by facilitating the development of open standards for DRM software, and by enabling solution providers to address customer needs. As the use of DRM software shifts from the "innovator" phase to the mainstream, open source provides the proven collaborative development environment that ensures the widest set of needs are met, and with high quality code. Open source facilitates the development of open standards which are important for interoperability with applications and for ease of integration. Perhaps most importantly the Grid Engine project effectively provides integrators, support organizations and other service providers a independent 'business franchise' in the deployment and maintenance of Grid Engine enabled solutions.

Q: What software is included in the Grid Engine project?

A: The Grid Engine project includes source code files for Sun Grid Engine software, and for an "Enterprise Edition" version of Sun Grid Engine software. The project sources can be compiled into either Grid Engine or Grid Engine Enterprise Edition binaries. Grid Engine binaries are similar to the Sun Grid Engine software product as available from Sun. The Grid Engine Enterprise Edition build features an advanced scheduler with many additional capabilities beyond Grid Engine, and is targeted for use by organizations with requirements to support several departments or projects simultaneously from a single large compute farm or 'campus grid'. Sun is planning to release a product based on Grid Engine Enterprise Edition in late 2000 - early 2001.

Additionally the project includes other experimental goodies, such as

  • Two versions of a Java GUI
  • A Windows NT GUI
  • A Perl interface with a GUI prototype
  • A Windows 2000 rsh based execution host
  • SSL, Kerberos, DCE security frameworks
It should be understood that these may not be usable as is but may require some development time to get them working even in a prototype state (we know, of course, that some of them are closer and others are farther from being usable, but we don't want to make guarantees).

Q: How does software from the Grid Engine project differ from the Sun Grid Engine product distributed by Sun?

A: The Grid Engine project is an open source project, and as such, is focused on development of future versions of Grid Engine software through the community process. At the commencement of the project Sun transitioned away from its previous proprietary development process for its Sun Grid Engine products. Future versions of Sun Grid Engine products distributed by Sun will be based on releases of software from the Grid Engine project. The branded Sun products will be distinguished by the Sun Grid Engine name, and will follow Sun's rigorous testing methodologies, will have matching Sun documentation, and will have associated support contracts available; additionally, these products may have extended capabilities, and may have a product cost. Licensees in the Grid Engine project may do likewise.

Q: What version of software is Sun making available in the Grid Engine project at the launch?

A: Sun made available the version currently under development, identified as version 5.3.

Q: Will the software in the Grid Engine project be fully useable at the launch?

A: Software in the project will be buildable, but as is true for most development efforts in an pre-release phase, the software may not be "ready for prime time" by end users. Software developers and application integrators will find the project documented and accessible. The most recent version of Sun Grid Engine binaries available from Sun may be downloaded at www.sun.com/gridware.

Q: Is Sun making the source for the now-current Sun Grid Engine software versions 5.2.2/5.2.3 available in open source?

A: The source available in the Grid Engine open source project has received numerous modifications in order to prepare it for release as open source software. Organizations interested in obtaining the source for the now current 5.2.2/5.2.3 versions under a Sun source-access (non-open source) license should contact sungridengine_partners@sun.com.

Q: Will Grid Engine software version 5.3 from the Grid Engine open source project interoperate with Sun Grid Engine software version 5.2.2/5.2.3 previously released by Sun?

A: The changes required for open source were such that interoperability could not be maintained.

Q: How will the Grid Engine project facilitate heterogeneous compute farms?

A: Grid Engine software has been ported to many operating systems, including Solaris, Linux, IRIX, Tru64, AIX, HP/ux, and others. The project welcomes those who are interested in implementing new ports or in taking over the maintenance of an existing port. Interoperability test suites included in the project will facilitate maintenance and testing of ports.

Q: Will the Grid Engine project promote open standards for DRM software?

A: Developing open standards for DRM is a basic objective of the project. Today, there are no existing standards for DRM software, which has resulted in increased tasks for users when implementing compute farms. A lack of a standard API for application programs has discouraged ISVs from directly interfacing to the DRM software layer, which results in increased integration tasks. Sun recognizes that the open source movement has facilitated the development of new open standards. The Grid Engine project will work towards development of open standards for DRM software, initially for an API for application programs, though cooperative efforts with ISVs and other producers of DRM software.

Q: Will Sun continue to work on the Grid Engine sources as part of gridengine.sunsource.net?

A: Sun's development team in Regensburg, Germany, will continue their full development efforts through the project. At the commencement of the project, the developers have roles in the project similar to their roles prior to the launch of the project. See the governance page for a description of project roles. The project welcomes the involvement of new users and developers!

Q: I found a bug; how do I report it? Where can I ask a question?

A: Grid Engine is currently using IssueZilla to track bugs. IssueZilla is an open source bug tracking system developed and used by Mozilla. You use IssueZilla to browse or modify existing bugs, or enter new bugs. Start with the Report Bugs page.

Q: Where can I get help with using Grid Engine?

A: Check out the discuss mailing list and archives on the mail lists for questions about the use of Grid Engine software. Also, the end user support page for Sun Grid Engine software at http://supportforum.Sun.com/gridengine/ covers many general user questions and may have more content at the beginning of the project.

Q: How can I learn about configuring and supporting Grid Engine?

A: Sun has a free Sun Grid Engine software web-based training course for system administrators.
 

License FAQ

Q: What is the license for the Grid Engine project?

A: The project uses the Sun Industry Standards Source License, or SISSL. This license is recognized as a free and open license by the Free Software Foundation and the Open Source Initiative respectively. This license is a good license where interoperability and commercial considerations are important. Under the SISSL license a user may do what they like with the source base, such as modifing it and extending it, but the user must maintain compatibility if they want to provide their modified version to others. It is not mandatory for the user to contribute back code changes to the project, though of course this is encouraged. If a user wishes to make available a modified version which is not compatible with the project, the license requires that the licensee must provide a reference implementation of sources which constitute the modification, thereby opening the details of any incompatibility/modification which has been introduced. Additionally the non-compliant modified version cannot use the Grid Engine name.

Q: What are the project standards for?

A: The project standards are designed to facilitate both compatibility and evolution of the technology. As the community works to advance Grid Engine technology, the project standards define a common framework so that components from different users can interoperate. Additionally, this same framework defines the boundaries in which individual users can create proprietary value-added extensions and modifications.

Each release of the software will include a test suite so that users can validate in a uniform way their modified versions for internal operation and for interoperability with reference project binaries. These tests will be specific to a release and will evolve with the technology.

The project standards also specify that the modifications made by users are limited by component interfaces and protocols. Of course, these interfaces and protocols are part of the project, and suggestions and contributions for modifications and enhancements of these interfaces and protocols are welcomed and encouraged.

Q: I want to modify Grid Engine for my own internal use. Do the project standards pertain to me?

A: You may make any modification you wish to Grid Engine code for internal use in your organization. The project standards only apply to those who are providing Grid Engine binaries and/or source to others outside their organization.

Q: I want to redistribute a modified version, but my modification requires a change to an internal interface. What do I do?

A: Offer your modified internal interface as a contribution. This way the community can benefit from an advance in the internal interfaces, while you can keep your modifications on the 'user' side of the interface proprietary if you wish. Of course, the project encourages all contributions!

Q: Are there any licensing issues regarding contributing code to the project?

A: Sun asks that developers grant a license to Sun to use the contributed code in the Grid Engine project under the SISSL license. This grant is done by completing a Contributor Agreement Form.

Q: Is there any cost to using Grid Engine?

A: No! Participation is free.

Q: Can unmodified Grid Engine binaries be redistibuted commercially?

A: The SISSL license supports this. Please read and understand the license when downloading binaries from the download page. Remember that software in the project is in development!

Q: Can I develop and redistribute proprietary products using Grid Engine?

A: The SISSL license supports that you may distribute compiled binaries of your modified source in accordance with the project standards.

Q: Does the Grid Engine project license also give me rights to redistribute the Sun Grid Engine software available from Sun directly?

A: The Grid Engine project does not convey any rights to use or distribute the Sun Grid Engine software available from Sun directly. If you are interested in redistributing the Sun-branded Sun Grid Engine binaries, or in discussing being a Sun partner for Sun Grid Engine software, email gridengine_partners@sun.com.
 

Technical FAQ

Q: Does Grid Engine run on platform xyz?

A: Have a look at our download page for a list of platforms to which Grid Engine has been ported and for a list of platforms where ready-to-use binary distributions are available.

Q: What kind of security does Grid Engine use?

A: Grid Engine can be configured to accept only messages that are coming from special hosts and reserved port. This mechanism provides security comparable to that of rsh. For improved security installations an integration with Kerberos5 and DCE exists. This integration has not been used frequently and does not come with an out-of-the-box installation procedure. It still should be possible to get it working with a little effort if you are particularly concerned about security. In addition, a SSL based security framework prototype exists which represents an opportunity for an interesting and relatively small developement project toward a Grid Engine embedded security solution without a need for Kerberos or DCE.

Q: How can I get Grid Engine in some language other than English?

A: Grid Engine has been internationalized and prepared for localization. The internationalization is based on gettext. There exist test localizations that can be used for additional localization efforts. An overview what has been done and what has to be done can be found in the I18N source description document. So anyone who is interested in helping with l10n is very welcome ;-)

Q: What interfaces does Grid Engine have?

A: Grid Engine has a command line interface that is aligned with the POSIX standard for batch queuing systems 1003.2d, Motif based GUI representing the full Grid Engine client functionality and there is a library interface (GDI). The GDI (Grid Engine Database Interface) is used by most Grid Engine clients to implement their functionality. See the GDI module description for more information. Note that this interface is evolving. One of the objectives of the Grid Engine project is to work with the community to develop a stable API for integration with applications.

Q: Where can I get the POSIX 1003.2d specification?

A: You can buy it from IEEE: http://standards.ieee.org/catalog/posix.html.

Q: What are the minimum recommended requirements for Grid Engine?

A: Solaris:

  • Master host: UltraSPARC, 64M RAM, Solaris 2.6 or later
  • Exec hosts: Sparc running Solaris 2.6 or later
Linux:
  • Master host: Intel or compatible processor with 200Mhz, 64 MB RAM
  • Exec host: recent 2.2.x kernel, with glibc 2.1.3 or later
Fileserver: 150M disk space

Q: What Linux distribution is needed for running Grid Engine?

A: Grid Engine has been developed and tested on RedHat 6.2 and SuSE 6.4/7.0. Customers have reported that Grid Engine also runs on other Linux distributions (Debian etc.) without any problems. It is important that the Linux system runs with a recent 2.2.x kernel and a recent version of glibc (2.1.3 or later). Earlier versions of glibc may cause memory leaks in the Grid Engine daemons. Older versions of KDE and other desktops and window managers may cause problems with the GUI "qmon". Please update your desktop to a recent version.

Q: Does Grid Engine work with the Linux 2.4 kernel?

A: Yes, we received positive feedback for kernel 2.4 systems.

Q: Can I run interactive jobs through Grid Engine?

A: Yes. Through the qrsh command or qtcsh you can run interactive jobs and the standard output will be redirected back to your terminal.

Q: Can Grid Engine keep track of the number of my available software licenses (e.g. compiler licenses) or other resources with a variable availability?

A: Grid Engine can be configured to track licenses and similar resources (so called consumables) either statically (see HOWTO on consumables) or dynamically (see HOWTO on load sensors).

Q: How can I learn more about Grid Engine?

A: Please see the free Sun training course for Sun's productized version of Grid Engine and the Grid Engine documentation link provided on all project pages.

Q: What happens if my master host fails?

A: There is a facility within Grid Engine called master shadowing. One or more hosts are designated to take over in the event the master host fails.  See the documentation on this site.

Q: What programming language does Grid Engine use?

A: Most parts of the core system is written in plain C.
 

Getting Started (Newbies) FAQ

Q: Where can I find Grid Engine source code?

A: You can download the source as one tar.gz archive. If you plan to participate in the development process, you can use CVS to checkout the sources as described on the Project Source page.

Q: Where can I find Grid Engine binaries?

A: Download Grid Engine binaries here.

Q: How do I install Grid Engine?

A: The install procedure is described here.

Q: Can I contribute to the Grid Engine project?

A: Yes - and we encourage you to. In accordance with the proposed Grid Engine project governance process, you can submit code to the project leader as a contribution either as a user, a developer, or as a committer. If your code is accepted to be integrated into the source tree, then you will be asked to sign a Contributor's Agreement. This agreement clarifies that to your knowledge, the code you've submitted was developed by you, and that you have authority to provide it and any related intellectual property (IP) to Sun. Through this agreement you will be licensing your contribution to Sun for use in the project under the SISSL license.
 

Community FAQ

Q: What is the purpose of this site?

A: Gridengine.sunsource.net is a home site for those interested in collaborating on Grid Engine, including developers and end-users. Here you can participate with other like-minded people who share an interest in distributed computing. Whether you are interested in optimizing Grid Engine for your application, helping others to implement and support a compute farm, or installing Grid Engine for your own use, you'll find here a community of people that share your interest. You will also find all the Web-based tools you need to manage your project and collaborate with others.

Q: I am considering participating with the project. What should I do?

A: Check out the getting started page.

Q: What do you mean by "hosted tools"?

A: The Grid Engine project uses a software toolset supplied by CollabNet called SourceCast to enable collaborative software development. These are Web-based tools made available to project owners and developers who become project members. SourceCast combines the best-of-breed open source tools into a Web-based software development suite. Key features of this tool set are security, scalability, extensibility, and customizability.

Q: What are those tools?

A: SourceCast provides simple, powerful Web interfaces for version control, issue tracking, discussions and decision making, automated builds, automated testing, project management and knowledge management. Project owners can configure these tools within each project and grant roles to project participants to fit the needs of the project.

Q: Whom do I contact for help or feedback?

A: Send any questions or comments concerning the SourceCast tools to feedback@sourcecast.com. For Grid Engine related issues, several mailing lists have been installed, see the mail lists page.

Q: What client-side tools will I need to participate?

A: At minimum, you need a Web browser, email client, CVS, and optionally SSH to secure your CVS connections. There might be additional domain specific tools you need depending upon the types of development project you want to contribute to. Source or, where possible, binaries can be downloaded from the download page.

Q: How do I share code?

A: When you join or lead a Grid Engine project, you use CVS version control to check out and work with project files, and then contribute your changes back into the project's repository. When other project members update their working files, they also have access to your code. Everyone on the CVS mailing list also receives automatic e-mail notification when files are updated.

As a project member, you can do code enhancements or contribute patches via IssueZilla, an issue tracking tool for managing project issues (defects, enhancements, features, tasks, and patches). You can also subscribe to project mailing lists, where you can follow and participate in ongoing discussions about the project.