Troubleshooting Guide

Table of contents

Security troubleshooting checklist

Security troubleshooting steps

If you are not able to establish a TLS or SSL connection to the server, check the following:

  1. What kind of certificate are you using?
  2. Where is the certificate stored (for example, is it in the server's keyring)?
  3. Have you added the certificate to the Host On-Demand server's keyring database?   Issue the keyrng command from a command prompt. The syntax is:

    keyrng x connect server_name:port_number

    where:

    Press enter at the password prompt. A list of all the certificates in the server's keyring database is displayed.

  4. If you have added a certificate, check the validity dates on the certificate to determine if the certificate has expired.
  5. If the certificate is still valid, add it to the Host On-Demandserver keyring. Stop and restart the Service Manager.
  6. Add the certificate to the client keyring to make it available to the clients.
  7. Use the keyrng utility to verify the correct certificate and validity dates. For example:
    keyrng CustomizedCAs verify
  8. Use the keyrng command to connect to the server on the 12173 SSL port. For example:
    keyrng servername:12173
  9. Configure a TLS or SSL session on the client to connect to a server on an SSL port (such as 12173).
  10. Try removing the Host On-Demand cached client, deleting temporary internet files, and try again. Deleting the temporary internet files removes the current CustomizedCAs.class.
  11. If possible, verify connectivity by trying to connect. (See Client does not connect for troubleshooting hints.)
  12. Verify that the HODServerKeyDb.* files are present in the \hostondemand\HOD directory. This is the key database file that is used by the Host On-Demand Redirector, and it contains certificates from well known CAs. It also contains the private key and any personal certificates for the server.
  13. If you are using a self-signed certificate, verify that the CustomizedCAs.p12 file (if it exists) and CustomizedCAs.class are present in the \hostondemand\HOD directory. If you are using a self-signed certificate or a certificate issued by an unknown CA, extract the certificate and include it in the CustomizedCAs.p12 (if it exists) and CustomizedCAs.class files. This file resides in the Host On-Demand publish directory so that the clients can get to it. It is downloaded every time the Host On-Demand client is downloaded.
  14. If you are experiencing problems with SSL on the Redirector, verify that the Host On-Demand server key database and the CustomizedCAs.p12 file (if it exists) and CustomizedCAs.class have been created. See Redirector Troubleshooting Checklist for details.

Common security problems

If you still experience problems with security or certificates after completing the security troubleshooting steps, the following are some problems that other customers have encountered and their solutions.

  1. COMM662, COMM663, or COMM666 error messages
  2. Editing the registry if drivers load when starting IKEYMAN
  3. Using two smartcard readers in IKEYMAN
  4. Adding a client certificate
  5. Creating self-signed certificates on Smartcards
  6. Certificate exported using z/OS utility gskkyman appears to be invalid or corrupted
  7. On the Windows platform, after Host On-Demand is upgraded to a new level, the Certificate Wizard panel does not appear when the Certificate Wizard is started
  8. UTF-8 support for Host On-Demand Version 7 server and Host On-Demand Versions 5 and 6 cached clients
  9. Certificate management utility limitation on AIX
  10. Using Host On-Demand with other security products
COMM662, COMM663, or COMM666 error messages

Refer to the following COMM error messages for more information:

Editing the registry if drivers load when starting IKEYMAN

When starting IKEYMAN on a Windows 2000 Host On-Demand Server, an error message occurs when loading slbck.dll during startup. A Schlumberger smart card reader must first be installed and then uninstalled. Some Schlumberger entries may remain in the registry. In order to get rid of this message, a user must clear all Schlumber entries out of the registry, or they must edit a file in Host On-Demand.

Host On-Demand Certificate Management uses the PKCS11 interface to access smartcard functions. This interface is used mostly for creating self-signed certificates in smartcards, or downloading a certificate in a .pfx or .p12 file to a smartcard.

Before the smartcard can be accessed, additional configuration may need to be done. When Host On-Demand is installed, it is programmed to determine if any smartcards are present in the system. Currently the only smartcards that are recognized are the IBM Security Card and the Schlumberger Reflex readers installed with the Cryptoflex Security Kit V3.0c.

IBM Certificate Management reads all its parameters from an initialization file named ikminit_hod.properties that is stored in the hostondemand\bin directory. If the IBM Security Card is recognized, the following line will appear in the properties file:

DEFAULT_CRYPTOGRAPHIC_MODULE=w32pk2ig.dll

This tells IBM Certificate Manager to load this dll when smartcard functions are needed.

If no IBM Security Card is detected, but a Schlumberger card is detected, the line will be similar to:

DEFAULT_CRYPTOGRAPHIC_MODULE=C:\\Program
Files\\Schlumberger\\Smart Cards and Terminals\\Common Files\\slbck.dll

These are the only security devices that have been tested with IBM Certificate Management. If there is another security device that implements the PKCS11 interface through a dll, the security device can be tested by changing the name and location of the dll in the ikminit_hod.properties file.

If the security device is ever removed from the system, IBM Certificate Management will report the following error at startup:

Cryptographic token initialization failed.

To prevent this error, remove the DEFAULT_CRYPTOGRAPHIC_MODULE statement from the ikminit_hod.properties file.

Using two smartcard readers in IKEYMAN

Installing more than one smartcard on the same computer may cause Host On-Demand smartcard support to function incorrectly.

For instance, if the IBM Security Card cannot be opened by Host On-Demand Certificate Manager and a Schlumberger smartcard was previously installed on your computer, there may be values left in your registry that may cause the IBM Security Card drivers to function incorrectly.

To remedy this problem, make a backup of your registry and carefully delete any of the following keys that remain after you have uninstalled the Schlumberger card:

Adding a client certificate

When the Host On-Demand client contacts an SSL server that requests a client certificate, such as Communications Server for Windows NT, Communications Server for AIX, or Communications Server for OS/390 in client authentication mode, the Host On-Demand client may invoke the MSCAPI interface to request all available client certificates. MSCAPI will return all registered certificates, whether they are stored completely in the MSCAPI database, or are associated through MSCAPI with some security device, such as a smartcard or thumbprint reader. The list of certificates that are currently registered in a MSCAPI database can be displayed in the following way:

  1. Start the Internet Explorer 5.x browser.
  2. From the menu bar, choose Tools then Internet Options.
  3. Across the top of the Internet Options panel, choose the Content tab.
  4. On the Content panel, click the Certificates button.
  5. Across the top of the Certificates panel, choose the Personal tab; if it is not already chosen. These are the certificates that will appear in the drop down list on the Host On-Demand session configuration panel and the Server Requesting Certificate panel. If the certificate is not in this list, it cannot be used by Host On-Demand for client authentication.

Any smartcard or security device that is recognized by MSIE can be used by Host On-Demand for client authentication. Certificates are usually obtained by visiting a Web page with the MSIE browser, filling out a form on the Web page, and then storing the new certificate in either the browser's database or a security device.

An example of this can be seen by loading http://freecerts.entrust.com/webcerts/ag_browser_req.htm into the MSIE browser [1] . Fill out the information requested, press Proceed to Step 2 and then Proceed to Step 3. At the bottom of this page is a drop down list that lets you specify where to put the certificate.

Choosing Microsoft Base Cryptographic Provider 1.0 will put the certificate into the MSCAPI database. No extra hardware will be needed to access it.

Choosing Schlumberger Cryptographic Service Provider or Gemplus GemSAFE Card CSP v1.0 will put the certificate into a smartcard. If you choose this destination, the name of the certificate will appear in the MSIE Certificates panel; just like a certificate that has been put into the MSCAPI database.However, the certificate will only be accessible if you have plugged in the smartcard by which the certificate was downloaded to.

The certificate obtained from freecerts.entrust.com should be used for testing purposes only. After downloading the certificate, go to the the MSIE Certificates panel, as shown above, and choose the Trusted Root Certification Authorities tab. Scroll down the list until you find a certificate issued to Entrust PKI Demonstration Certificates. Highlight this certificate and export it to a file. Then add the exported file to the trusted list of your client authenticating SSL server. With this configuration, the SSL server should trust the Entrust certificate if it is returned by the Host On-Demand client. This exercise should only be used for testing purposes, and the Entrust PKI Demonstration Certificate should be removed from any production server.

[1]Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

Creating self-signed certificates on Smartcards

Host On-Demand Certificate Management uses the PKCS11 interface to access smartcard functions. This interface is used mostly for creating self-signed certificates in smartcards, or downloading a certificate in a .pfx or .p12 file to a smartcard. (Note: The IBM Security Card supports the creation of a self-signed certificate, but not the downloading of an existing certificate in a .pfx or .p12 file.)

Before the smartcard can be accessed, additional configuration may need to be done. When Host On-Demand is installed, it tries to determine if any smartcards are present in the system. Currently the only smartcards that are recognized are the IBM Security Card and the Schlumberger Reflex readers installed with the Cryptoflex Security Kit V3.0c.

IBM Certificate Management reads all its parameters from an initialization file named ikminit_hod.properties that is stored in the hostondemand\bin directory. If the IBM Security Card was recognized, the following line will appear in the properties file:

    DEFAULT_CRYPTOGRAPHIC_MODULE=w32pk2ig.dll

This tells IBM Certificate Manager to load this dll when smartcard functions are needed.

If no IBM Security Card was detected, but a Schlumberger card was, the line will be similar to

    DEFAULT_CRYPTOGRAPHIC_MODULE=C:\\Program Files\\Schlumberger\\Smart Cards and Terminals\\Common Files\\slbck.dll

These are the only security devices that have been tested with IBM Certificate Management. If you have another security device that implements the PKCS11 interface through a dll, it can be tested by changing the name and location of the dll in the ikminit_hod.properties file. If the smartcards are ever removed from the system, these lines should be removed from ikminit_hod.properties.

With this configuration, a self-signed certificate can be created in the smartcard with the following steps:

  1. Start IBM Certificate Management
  2. On the menu bar, select Cryptographic Token
  3. On the Cryptographic Token panel, type in the PIN number of the smartcard, and clear the external database and press OK
  4. The Cryptographic Token (smartcard) is now open.

Both the IBM Security Card and Schlumberger cards can create self-signed certificates. The Schlumberger card can also have a certificate in a .p12 or .pfx file imported to the card.

If self-signed certificates are created, then the public portion of the certificates must be extracted (not exported) and added to the trusted list of the SSL server that will request the certificate.

If a self-signed certificate is created in the IBM Security Card, it must be registered with MSCAPI. To do this, start the GemSAFE Card Details Tool. It will check the card, see that the certificate in the card has not been registered with MSCAPI, and ask if you want to register it.

In our testing, not all readers supported all operations on all platforms. Here is a table of what readers were tested on which platforms.

  Entrust Self-signed Add .p12 Windows 98/NT Windows 2000
IBM Security Card PCMCIA Reader X X   X  
IBM Security Card Serial Reader X     X  
Schlumberger Reflex 20 Reader X X X X X
Schlumberger Reflex 72 Reader X X X X  
Schlumberger Reflex Lite X X X   X

Certificate exported using z/OS utility gskkyman appears to be invalid or corrupted

Host On-Demand and its utilities will not read PKCS12 files exported using the z/OS utility gskkyman. The problem is that gskkyman uses PFX v1 format for PKCS12 files, whereas Host On-Demand and its utilities use PFX v3 format for PKCS12 files.

Here is an example of a failing scenario:

  1. On a z/OS server, the system administrator:
    1. Uses gskkyman to create a self-signed certificate to serve as a personal certificate for a Host On-Demand client.
    2. Uses gskkyman to export the key and the certificate to a PKCS12 file. (The function on the gskkyman Export Key Menu is 'Export keys to a PKCS12 file'.
    3. Transmits the PKCS12 file to the user.

  2. On the client, the user:
    1. Starts a 3270 Display session that requires that the client send a certificate to the host, and that specifies that the certificate source be a PKCS12 or PFX file.
    2. Provides, when prompted by the 3270 Display session startup program, the path of the PKCS12 file that was received from the system administrator.

  3. On the client, the 3270 Display session startup program displays the following error message:
    Certificate password was incorrect or certificate found at <path of PKCS12 file> was corrupted. (ECL0034)

Another failing scenario may be that the certificate cannot be read by any of the Host On-Demand certificate utilities.

The fix is to convert the PKCS12 file to PFX v3 format before sending the PKCS12 file to a user and before using the PKCS12 file with any Host On-Demand utility or session. To convert the format, take the following steps:

  1. Download the certificate to a workstation that has a Netscape 4.x or Netscape 6.x browser installed.
  2. Use Netscape to import the certificate from the PKCS12 file. Netscape can read the PFX v1 format. To import the file using Netscape 4.x:
    1. Click Communicator > Tools > Security Info.
    2. Under Certificates, click Yours.
    3. Click Import a Certificate... and follow the instructions to import the certificate from a PKCS12 file.
  3. Use Netscape to export the certificate to a PKCS12 file. Netscape will export the certificate in PFX v3 format. To export the file using Netscape 4.x:
    1. Click Communicator > Tools > Security Info.
    2. Under Certificates, click Yours.
    3. Click the imported certificate.
    4. Click Export a Certificate... and follow the instructions to export the certificate to a PKCS12 file.

On the Windows platform, after Host On-Demand is upgraded to a new level, the Certificate Wizard panel does not appear when the Certificate Wizard is started

On the Windows platform, after Host On-Demand is upgraded to a new level, the Certificate Wizard panel may not appear when the Certificate Wizard is started.

This problem is caused by the fact that program processes associated with the Certificate Wizard may have been left running during the upgrade. On Windows 2000 these program processes are:

As a temporary workaround, you can use the Certificate Manager.

To actually fix this problem using the Windows 2000 platform, follow these steps:

  1. Stop the processes if they are still running.
    1. Type Ctrl-Alt-Delete.
    2. Click Task Manager.
    3. On the Task Manager panel, click the Applications tab.
    4. On the Task Manager panel, click Host On-Demand Certificate Wizard, then click End Task. The entry for Host On-Demand Certificate Wizard should disappear.
    5. On the Task Manager panel, click File > Exit Task Manager.
  2. Uninstall Host On-Demand using the Windows 2000 Add/Remove Programs wizard.
  3. Reinstall Host On-Demand.

UTF-8 support for Host On-Demand Version 7 server and Host On-Demand Versions 5 and 6 cached clients

When using Host On-Demand Versions 5 and 6 cached clients, in order to support UTF-8 encoding with the self-signed certificates created with the Host On-Demand Version 7 server, download either the 6.0.4a or 5.0.8a Interim Service Build (ISB) from the IBM Software Internet Service Delivery site at http://www6.software.ibm.com/aim/home.html. You will need to register on this site if you have not already done so.

Certificate management utility limitation on AIX

The certificate management utility on AIX requires JRE 1.1.8. If you are running JVM 1.3, you will receive the following error message:
Exception in thread "main" java.lang.VerifyError

To use the certificate management utility on AIX with JRE 1.1.8, set the JAVA_HOME environment variable to point to the Java 1.1.8 installation before running the "CertificateManagement" script.

Using Host On-Demand with other security products

When using other vendors' security products that lock or overwrite files, such as Netscape's Mission Control, be aware that the edited client configuration files may cause problems when upgrading to a newer version of Host On-Demand.

For example, if the signed.db file is locked or overwritten, the prior version of Host On-Demand's signed certificate is presented. Consequently, because the incorrect version of the certificate continues to be presented, users are prompted to grant or deny access to the newer version's Host On-Demand applets each time they try to log in. Selecting the "Remember this decision" checkbox has no effect. Other symptoms include blank lines or undefined hexadecimal certificate information in Netscape's Java/Javascript Certificate list.

To resolve this, follow the security program's instructions on how to recapture the configuration to use the newer version of Host On-Demand's signed certificate before distributing to users.

Top of page Table of contents