cqtssetup.pl

Performs setup tasks for full-text search.

Synopsis

Generate a file of submittable entities:
cqperl cqtssetup.pl –d dbname –s dbset –u user –p password –o Entity
Generate a ClearQuest search properties file:
cqperl cqtssetup.pl –d dbname –s dbset –u user –p password –o CQ [ -f entity_file ]
Generate a Solr search schema file:
cqperl cqtssetup.pl –d dbname –s dbset –u user –p password –o Solr [ -f entity_file ]

Description

The cqtssetup.pl command performs setup tasks for the full-text search feature, according to the mode of operation that is specified by the -o option: Entity, CQ, or Solr. The output is a file that is named according to the mode of operation:
  • Entity-db_name-dbset_name.txt
  • CQ-db_name-dbset_name.xml
  • Solr-db_name-dbset_name.xml
The steps for using cqtssetup.pl are as follows:
  1. Generate a file of all submittable entities (-o Entity). These entities and their fields are the candidates for searching.
  2. Generate the ClearQuest search properties file (-o CQ) that is to be used by the record extractor (cqtsdbcrawler.pl) and by the search feature. Typically, you specify the edited output of cqtssetup.pl ... -o Entity as the argument to the -f option.
  3. Generate the Solr search schema file (-o Solr), which specifies how entities are indexed. Typically, you specify the output of cqtssetup.pl ... -o CQ as the argument to the -f option.

This command is installed in the default product installation directory.

Options and Arguments

–d dbname
Name of the database.
–s dbset
Name of the database set.
–u user
User name.
–p password
Password.
–o { Entity | CQ | Solr }
The mode of operation:
  • Entity creates the entity file.
  • CQ creates the search properties file.
  • Solr creates the Solr search schema file.
–f entity_file
The entity file. This option can be specified with the options CQ and Solr only. When this option is omitted, all submittable entity types (and their fields) that are found in the database are used as the command argument.

The entity file

The entity file is generated when -o Entity is specified as the operation mode. It contains the submittable entities found in the database. Because these entities are well understood by the developer of the ClearQuest schema, further explanation of the entity file is unnecessary.

The ClearQuest search properties file

The ClearQuest search properties file is generated when -o CQ is specified as the operation mode. It contains several sections: search engine, database crawler, database, and entity. If the ClearQuest database is replicated, then there are multiple instances of each section. Each instance corresponds to a replica and may require a configuration that is specific to the replica.

The search engine section

This section describes XML tags in the search-engine section of the file, CQ-db_name-dbset_name.xml. Following is an example of that section:
<!- Solr Search Engine default parameters -–>
<SearchEngine sitename="<local>" provider="Solr" version="1.2">
     <searchrequest retries="3"
          url=http://localhost:14080/solr/select/?q=%s&start=%d&rows=%d />
     <searchprop pagesize="25" cachesize="1000" />
</SearchEngine>
<SearchEngine>
Specifies the configuration that is used to communicate with the search engine. For a nonreplicated database, sitename="<local>".
When the database is replicated, there is an instance of this tag for each replica and the value of sitename specifies the site of that replica. The full-text search feature reads the value of sitename. If that value does not match the name of the site, then the search feature is disabled.
You do not need to modify this tag.
<searchrequest>
Specifies the search-engine server URL and how to communicate with that server. You do not need to modify this tag.
<searchprop>
Specifies the default page and cache sizes.
  • pagesize specifies the number of hits to return per page.
  • cachesize specifies the number of hits to read and cache.
You do not need to modify this tag.

The database crawler section

This section describes XML tags in the database crawler section of the file,CQ-db_name-dbset_name.xml. Following is an example of that section:

<!- ClearQuest Database Crawler default parameters -->
<CQDbCrawler sitename="<local>">
     <batch size="250" delay="5" directory="./batch-output"
          url=http://localhost/solr/update/" />
     <update size="250" delay="300" directory=".update-output"
          url=http://localhost/solr/update/" />
</CQDbCrawler>
<CQDbcrawler>
The value of this tag is read by the record extractor in both batch and update modes of execution. It specifies the site configuration to use to communicate with the search engine. For a nonreplicated database, sitename="<local>".

For a replicated database, there is a separate <CQDbCrawler> tag for each replica, with each instance of sitename specifying the site of a replica. In the case of a replicated database, the record extractor looks for a sitename value that matches the name of the site at which it is running. If none is found, the record extractor fails to start.

<batch>, <update>
The values of these tags are read by the cqtsdbcrawler command in batch and update modes. You may want to modify their values to suit your ClearQuest environment.
  • size specifies the number of records to be read at a time before the records are sent to the search engine for indexing.
  • delay specifies the time, in seconds, that the record extractor waits between processing the groups of records of the size specified by size.
  • url specifies how to communicate with the search engine to add records to the index.
  • thread is applicable to batch mode only. It specifies the number of threads to create when executing queries. The more threads, the faster record extraction proceeds. The maximum number of threads that is allowed is 10. Values that are greater than 10 are truncated to 10.

The database section

This section describes XML tags in the database section of the file, CQ-db_name-dbset_name.xml. Following is an example of that section:

<!-- ClearQuest database and CM Server parameters -->
<CQDatabase sitename="<local>">
     <cmserver>http://localhost/TeamWeb/services/Team/</cmserver>
     <repository dbset="TextSearch" dbname="SAMPL"
          <username="admin" password="" keyfile=""></repository>
</CQDatabase>
<CQDatabase>
The record extractor reads the value of this tag, in both batch and update modes of execution, to learn the site configuration to use to communicate with the ClearQuest database and the Change Managment (CM) Server to extract records and monitor updates to the database. When the database is not replicated, sitename="<local>".

In the case of a replicated database, there is one instance of CQDatabase for each replica, with sitename specifying the site of the replica. The record extractor looks for a sitename that matches the site at which it is running; if none is found, the record extractor fails to start.

You do not need to modify this tag.

<cmserver>
Specifies the Change Management Server (CMServer) URL. Modify the value of this tag as appropriate.
<repository>
Specifies the database connection profile. Modify the following values according to your profile:
  • dbset specifies the name of the database set.
  • dbname specifies the name of the database.
  • username specifies the identity that is assumed by the record extractor to log in to ClearQuest.
  • password specifies the encrypted password that is generated by cqtsdbcrawler.pl.
  • keyfile specifies the full path name of the key file that is used to decrypt the password.
Attention: You must specify a valid value for the username, password, and keyfile, else errors will result when you run the command cqdbtscrawler.pl to perform indexing.

The entity section

This section describes XML tags in the entity section of the file,CQ-db_name-dbset_name.xml. Following is an example of that section:

<!- ClearQuest entity parameters -->
<CQEntity name="Customer" index="true">
     <field name="schema_mastership" disp_order="0" index="true" />
     <field name="customer_name" disp_order="1" index="true" />
     <field name="phone" disp_order="0" index="true" />
     <field name="company_name" disp_order="0" index="true" />
          .
          .
          .
</CQEntity>
<CQEntity>
The value of this tag is read by both the record extractor and the search feature. When the ClearQuest database is replicated, instances of CQEntity must be consistent across all sites, else indexing (and hence search results) will be inconsistent from replica to replica.
<field>
The record extractor uses the values of this tag to determine whether or not a field is indexed and searchable, as follows:
  • name specifies the name of the field.
  • index specifies whether or not the field is index. When index="<true>", the specified entity is indexed and searchable.
  • disp_order indicates whether or not the field is displayed in search results, as follows:
    • The value 0 means that the field is not displayed.
    • The value 1 means that the field is displayed. Only one field in an entity is displayable.
    Attention: All record types that are indexed by using the ClearQuest full-text search feature must have only one field specified with a disp_order value of 1. If more than one field has a disp_order value of 1, no results display.

The Solr search schema file

The Solr search schema file is generated when -o Solr is specified as the operation mode. As is the case with -o CQ, this operation mode takes as its input the entity file generated by running this command in entity mode (-o Entity). You do not edit the Solr search schema file; instead, you copy sections of it into the Solr schema file, schema.xml (for information about schema.xml, see the Solr documentation).

The most important section of the Solr search schema file is the search engine section, which the following excerpt shows:
<Solr>
	<fields>
		<field name="Operator_Value" type="text" indexed="true" stored="false" required="false" multivalued="true"></field>
		<field name="misc_info" type="text" indexed="true" stored="false" required="false" multivalued="true"></field>
...
		<field name="Resolution" type="text" indexed="true" stored="false" required="false" multivalued="false"></field>

...
		<field name="comments" type="text" indexed="true" stored="false" required="false" multivalued="true"></field>

...
		<field name="RATL_CQ_UNIQUE_FIELD" type="string" indexed="true" stored="true" required="true" multivalued="false"></field>
...
		<field name="RATL_CQ_ALL_FIELDS" type="text" indexed="true" stored="false" required="false" multivalued="true"></field>
	</fields>
</Solr>
<uniqueKey>RATL_CQ_UNIQUE_FIELD</uniqueKey>
<defaultSearchField>RATL_CQ_ALL_FIELDS</defaultSearchField>
<copyField source="Operator_Value" dest="RATL_CQ_ALL_FIELDS"></copyField>
<copyField source="misc_info" dest="RATL_CQ_ALL_FIELDS"></copyField>
...
<copyField source="Resolution" dest="RATL_CQ_ALL_FIELDS"></copyField>
...
<copyField source="Version" dest="RATL_CQ_ALL_FIELDS"></copyField>
...
<copyField source="comments" dest="RATL_CQ_ALL_FIELDS"></copyField>
...
<copyField source="Fax" dest="RATL_CQ_ALL_FIELDS"></copyField>
<field>
Provides Solr with information about indexing a field:
  • name specifies the field name.
  • type specifies the data type.
  • indexed specifies whether or not the field is to be indexed.
  • stored specifies whether or not data in the field is to be stored in the Lucene index.
  • required specifies whether or not a document must contain the field (and its data) in order to be added to the index.
  • multivalued specifies whether or not there can be another <field> tag with this field name.
Do not modify this tag.
<unique_key>
Specifies the field in the Lucene document definition that is to be used as the document identifier. This field is read by Solr, Lucene, the record extractor, and the ClearQuest search feature to uniquely identify a ClearQuest record. Do not modify this tag.
<defaultSearchfield>
Specifies the field that is to be used when a search is executed without a field qualifier. Solr and Lucene use this field to search all fields in a record. Do not modify this tag.
<copyField>
Specifies that the content of a field is to be copied to another field, in this case <defaultSearchfield>. Do not modify this tag.

Examples

  1. Generate a file of all submittable entities in the SAMPL database:
    cqperl cqtssetup.pl -d SAMPL -s TextSearch -u admin -p "" -o Entity

    This command generates the file Entity-SAMPL-TextSearch.txt. The content of the file is similar to the following text:

    Email_Rule=ratl_mastership,ratl_keysite,record_type,dbid,is_active,version,lock_version,locked_by,name,…,…,…
    Customer=ratl_mastership,ratl_keysite,dbid,is_active,version,lock_version,locked_by,record_type,Name,Phone,Fax,Email,CallTrackingID,…,…,…
    Project=ratl_mastership,ratl_keysite,dbid,is_active,version,lock_version,locked_by,…,…,…
    Defect=ratl_mastership,record_type,dbid,is_active,id,state,version,lock_version,locked_by,isduplicate,unduplicate_state,Headline,Description,Priority,Severity,Submitter,…,…,…

    Edit the file to specify the record types and fields that are to be searchable. For example, if only records of the type Defect are to be searchable, then delete all lines in the file except the one that begins Defect=. If only some of the fields that are defined by this record type are to be searchable, preserve only those fields, deleting the rest of them. For example, if only Headline, Description, and Severity are to be searchable, then the edited file would read

    Defect=record_type,dbid,version,Headline,Description,Severity

    Note that the fields record_type, dbid, and version are preserved. It is a requirement of the implementation of full-text search that those three fields be indexed.

  2. Generate a search properties file, specifying the entity file, Entity-SAMPL-TextSearch.txt, which was generated by the previous command.
    cqperl cqtssetup.pl -d SAMPL -s TextSearch -u admin -p ""
    -o CQ -f Entity_SAMPLE_TextSearch.txt

    This command generates the file, CQ-SAMPL-TextSearch.xml, which is used both by the record extractor (cqtsdbcrawler.pl) and the search feature.

  3. Generate a Solr search schema file, specifying the entity file, Entity-SAMPL-TextSearch.txt.
    cqperl cqtssetup.pl -d SAMPL -s TextSearch -u admin -p ""
    -o Solr Entity-SAMPL-TextSearch.txt

    This command generates the file, Solr-SAMPL-TextSearch.xml. You do not edit this file: you copy sections of it into the Solr file, schema.xml.

  4. Update the Solr schema file schema.xml by replacing certain sections with corresponding sections of the file Solr-SAMPL-TextSearch.xml:
    1. Replace the <fields> section of schema.xml with the <fields> section of Solr-SAMPL-TextSearch.xml.
    2. Replace the <copyfield> section in the same manner.
    3. Replace the <uniqueKey> section in the same manner. This section should already exist if you started with an existing schema.xml file.
    4. Replace the <defaultSearchField> section in the same manner. This section should already exist if you started with an existing schema.xml file.
  5. Restart WAS to effect the changes that you have made.
Note: When you edit your own schema.xml, change the value of <field name= ... to the name of your database schema. If you use ClearQuest MultiSite, also specify the name of the local site; for example, <field name="Acme Defect Tracking at Madrid"... The Solr administration web page will reflect this change.

See also

cqtsdbcrawler.pl, setmasterpropertyfromfile


Feedback