Full-text search maintenance

Maintaining full-text search entails occasional tasks, such as repairing problematic records, purging the operation log files, optimizing the index, tuning Solr performance, and updating the configuration to accommodate changes in the schema.

Repairing problematic records

At times, when the record extractor is run in batch or update mode, it might fail to process records. When this occurs, the record extractor creates repair files corresponding to the failed records. The repair files are written to the directory where the ClearQuest search properties file, CQ-db_name-dbset_name.xml resides, which is typically the Solr home directory. Monitor this directory to check for instances of failed processing and use the record extractor in repair mode to process the problematic records. See the cqtsdbcrawler reference help for usage information.

Purging operation logs

By default, no operation log (oplog) purging occurs in the user database. When you implement an oplog scrubbing policy to minimize wasted space, use caution to avoid removing oplogs prematurely. Oplogs should exist long enough for the update-mode record extractor to process. If the update-mode record extractor does not run for a long period of time, when it resumes, it starts by processing the oldest update oplogs that have not been processed.

If you are running ClearQuest MultiSite, your full-text search oplog scrubbing policy should not be more aggressive than the existing ClearQuest MultiSite oplog scrubbing policy.

See the ClearQuest Multisite help on oplog scrubbing for details.

back to top

Optimizing the index

Over time, as new ClearQuest records are added, modified, and deleted, the Lucene index becomes fragmented, which can degrade performance.

To optimize the index, complete the following steps:
  1. Change the directory to your ClearQuest home directory.
  2. At the command prompt, run the record extractor in optimize mode:
    cqperl cqtsdbcrawler.pl -optimize CQ-db_name-dbset_name.xml -site site_name

If you are not running ClearQuest MultiSite, you can omit the -site option. See the cqtsdbcrawler reference help for usage information.

back to top

Tuning Solr performance

The Solr search engine should not impede ClearQuest performance. However, if you suspect it does, first try optimizing the Lucene index, as discussed in the previous section. Also consider the following measures to optimize Solr performance:

  • Increase the memory allocation of the Java Virtual Machine (JVM). Instead of starting Solr with the default memory allocation, allocate Xms512M and Xmx1024M.

    Note that you may need to temporarily increase the JVM memory allocation when you run the record extractor in batch mode and then reset it.

  • Move the Solr home directory to a fast secondary hard drive. Search engines are hard drive, processor, and memory intensive, particularly if you have a large index. By moving the Solr home directory to a fast secondary hard drive with 10,000 RPM and a fast seek time, Solr performance will improve considerably compared to using a standard 54,000 RPM hard drive. Even using a partition separate from the operating system partition can improve performance.
  • Increase the physical memory of your server, the number of processors, and the clock speed.
  • Tune the performance settings in the Solr configuration file solrconfig.xml. See the Solr web site for details.
  • Reduce the number of record types and associated fields that you are indexing.
  • Install Solr on a dedicated server.
  • Enable the Solr master and slave configuration. This configuration consumes twice the disk space of the Lucene index default configuration, but provides reliable performance even when heavy demands are made on the index.
  • Consult the Solr web site for additional performance tuning tips.

back to top

Accommodating schema changes

Changes to the database schema often require corresponding changes to the search schema. Following are examples:
  • Adding a new entity. If the new entity is a submittable record type that you want to make searchable by using ClearQuest full-text search, you must update the ClearQuest search properties file, CQ-db_name-dbset_name.xml, and the Solr schema file, schema.xml, to account for the new record type. To do this, repeat the following steps, which you performed earlier:
    1. Create and edit the entity file.
    2. Create and edit the ClearQuest search properties file.
    3. Create the Solr search file.
    4. Configure which records to index by updating the Solr schema file schema.xml and restarting Solr.
    5. Run the installutil setmasterpropertyfromfile subcommand to upload the ClearQuest search properties file, CQ-db_name-dbset_name.xml, to the user database.

    The update-mode record extractor will use the updated ClearQuest search properties file the next time it indexes.

  • Adding a field to an entity. If you add a new field to an entity that is a submittable record type and you want the field to be indexed, follow the same procedure as just outlined for adding a new entity.
  • Renaming an entity. If you rename an entity in your ClearQuest schema, follow the same procedure as just outlined for adding a new entity. In addition, you need to perform these steps:
    1. Index the renamed record type again:
      1. Create a new ClearQuest search properties file that contains only the renamed record type.
      2. Run the record extractor in batch mode (cqperl cqtsdbcrawler.pl -batch).
    2. Delete all record types that belong to the old entity name by issuing the following Solr command:
      On Windows:
      1. cd %SOLR_HOME%\yourSchemaName\Solr\exampledocs\
      2. java -Ddata=args -jar post.jar "<delete><query>record_type:recordTypeName</query></delete>"
      On the UNIX system and Linux:
      1. cd $SOLR_HOME/yourSchemaName/Solr/exampledocs/
      2. java -Ddata=args -jar post.jar "<delete><query>record_type:record_type</query></delete>"
  • Renaming a field of an entity. If you rename a field of an entity in your ClearQuest schema, follow the same procedure as just outlined for renaming an entity.

back to top

Related tasks
Establishing the location of the index
Creating and configuring the entity file
Creating and configuring the ClearQuest search properties file
Creating the Solr search file
Configuring the fields for Solr to index
Related reference
cqtsdbcrawler.pl
setmasterpropertyfromfile
scruboplog

Feedback