QUESTION: Why isn't my notification mail being delivered to my users?

ANSWER



There are a variety of reasons that DDTS notification mail might not

reach the desired list of recipients. This FAQ is a compilation of

the knowledge gained from helping customers debug such situations,

discussions on the ClearDDTS User's Group alias, and examination of

the DDTS source code. By using the information in this FAQ you should

be able to determine why your mail is not being received, and you should

also be able to correct the problem yourself in the majority of cases.



This document is arranged with the most-specific information which

covers the most-common cases first, in a Question/Answer format.

This is followed by two sections "THE DETAILS OF THE LOG FILE"

and "THE DETAILS OF EMAIL ADDRESSING." Unless specifically

directed to do so by the Question/Answer text, or if you exhaust

the Question/Answer text and have not found a solution to your

problem, or if you are simply seeking information, there is probably

no reason to read the "THE DETAILS OF..." sections.





OVERVIEW



Problems that can prevent proper deliver of notification mail include:



1. Misunderstandings about how the facility works. Notification mail is

described in detail in the Administrator's Guide (Chapter 11,  pages

11-4 and 11-5 of the release 4.1 manual) and in the ClearDDTS User's

Guide (pages 2-6 through 2-8), but several aspects of this are not

clearly documented or are often overlooked. The bugmail manpage

('man bugmail') contains some useful information as well but that

resource is frequently overlooked. Most of the rest of this FAQ is

designed to correct those gaps and misunderstandings.



2. Errors in DDTS customization or configuration that cause DDTS to be

unable to appropriately build, address, or send the message.



3. Configuration errors in your email system which cause delivery of

the messages to fail.





TOP 10 REASONS FOR MAIL DELIVERY PROBLEMS



Here, in roughly decreasing order of occurrance, are the top reasons

that notification mail delivery doesn't occur as expected:





Issue #1: I'm trying to get it to send mail to myself as a test, and

it's not arriving; what happened?"



Answer: Email notification is never sent to the person who performed

the action. This is documented in the ClearDDTS User's Guide on page

2-8 but this note is frequently overlookeed. This characteristic is

is completely intentional; the assumption is that a person would

already be aware of an action that they themselves initiated, so why

inform them by email?



This means that if you are trying to test email notification by, for

example, assigning a defect to yourself, your test is flawed; you

would never be notified of such an event. To properly test notification,

you should log on as a different user and assign the defect to yourself,

or log on as yourself and assign the defect to someone else (and check

with them to see if they got the mail).



If you want more detail about how this works internally, see "THE

DETAILS OF EMAIL ADDRESSING" below.





Issue #2: I've created a new field called "My-mail" and I'm populating

it in my master.tmpl, but DDTS isn't sending notification mail to the

users listed there; what happened?



Answer: DDTS only sends mail to the users listed in the three special

defect record fields mentioned on pages 11-4 of the Admin Guide:

Submitter-mail, Engineer-mail, and Other-mail. Creating any other field

(or changing the spelling, punctuation, or capitalization of one of

*these* fields) will not cause DDTS to send notification mail to the

users on that list. You can *only* use those fields!!!





Issue #3: A user of mine did several state changes in one operation,

and nobody got notification mail.



Answer: Examine your DDTS LOG (see "THE DETAILS OF THE LOG"

below if you are not familiar with the LOG) for messages from bugmail

like:



   ERROR: BUGno12345: Can't go from state R to state O



If you got this message, then your DDTS administrator has inadvertently

caused this problem through inappropriate modification of the 'states'

file. The 'states' file determines the progression of states that a

defect can pass through, and bugmail needs to use this file to determine

which states the defect passed through in the current transaction.



DDTS allows one transaction to span multiple state changes. Your DDTS

Administrator may have observed this and tried to restrict it by

changing the 'states' file to have only "single-hop" transitions

(transitions from one state directly to the next state; removing all

complex "from-to-through" routes for mutiple-state transitions).



The problem is that removing these transition definitions from the

'states' file does not completely remove the user's ability to make

such transitions. xddts does not require the user to Commit after

each and every modification; therefore a user can take a defect

originally in a state (say New for example), and can transition it

to the next state (say "Assigned"). Then, once it's in Assigned they

will be allowed to transition to the next state (say Open). Then,

they can commit the compound transaction.



Thus, with no New->Open transition defined in 'states', the user has

accomplished a New->Open transition. xddts handles this fine, because

it looks at the transitions one-at-a-time; from the defect's current

state to one potential target state.



However, bugmail sends notification only once for the whole (compound)

transaction, so it needs to know the list of states the defect passed

through on its way from the original starting state to the ultimate

ending state.



With the 'states' file defined as described above, it cannot determine

that. To correct the problem, the administrator needs to define

state transitions for all possible "multi-hop" transitions; for example,

for the error shown above the missing line is probably:



  R O A



Translation: "from state R to state O by way of state A". See the DDTS

Administrator's Guide in chapter 8 for a description of how to modify

the states file.





Issue #4: A user expected to get two notifications because he's on the

project notification list and he is also the person who submitted the

defect. He only got one.



Answer: Correct. DDTS suppresses duplicates and generally only sends one

notification per person per transaction (there are some obscure

exceptions to this; see below).





Issue #5: Nobody is getting notification mail for anything; the system

seems to have completely stopped sending notification mail.



Answer: Check the bugmail spool (~ddts/spool/bugmail) and see if there

are files in it. If there are many, and some of them seem to be old,

then the bugmail daemon is probably "stuck" processing a specific

transaction and therefore will never get around to processing any of

the others unless you give it some help.



To further confirm that this is the case, check the DDTS LOG; if the

only messages you see from bugmail are ones indicating "Another

process (PID) is already running. Exiting.", *and* the indicated PID

exists (use "ps" to see), *and* there are no other progress messages

from that process on the LOG, then it is probably "stuck".



There are a few reasons that bugmail can get "stuck", the most common

of which is a problem accessing a user's home directory. For each

email recipient, bugmail tries to access that person's UNIX home

directory to see if they have some special settings in their .ddtsrc

file (we won't get into the specifics of that here). In large

installations with many UNIX workstations, particularly if some of

them may be down, the UNIX automounter will be unable to mount the

user's home directory. Unfortunately, under many circumstances a situation

like this will cause the UNIX automounter to hang (and thus bugmail,

which was waiting for the automounter to mount the home directory also

hangs). It does not return failure, it just hangs.



Another problem that can cause bugmail to hang is a "loop" in the

'states' file. As mentioned above under Issue #3, bugmail uses the

states file to determine which states a defect passed through during

the current transaction. If the states file is inapproprately set up

so there are "loops" in the state transition diagram, then bugmail

may loop. For example, a state transition like "S N S" (or in fact any

specification where the first and third columns are the same *and*

different from the middle column), would create such a loop.



The solution from DDTS's perspective is to kill the running bugmail

daemon. You can get its PID from the LOG message mentioned above, from

a 'ps' command, or from the file ~ddts/spool/bugmail.pid. Log on as

user 'ddts' and use the UNIX 'kill' command to kill the running bugmail

process. Watch the LOG file ('tail -f ~ddts/spool/LOG&' is a handy

command in this regard) and you should see a new instance of the

bugmail daemon start up and begin processing the backlog.



Eventually the new daemon will encounter the transaction that caused

the first daemon to hang. If the situation (automounter hang for example)

was transient and has cleared up, then bugmail will successfully process

 the transaction this time and will continue normally until its spool is empty.



 If the condition still exists, then bugmail will hang again. If this

 happens, kill it again. After three cycles, bugmail will recognize that

 it cannot process that transaction and will move it off to the

 spool/bugmail.lost/ directory and send the administrator a message about

 it.



 See the text of that email message, or contact DDTS Technical Support,

 if you need help determining the underlying cause or corrective steps

 for such a situation; for example, examine the states file and look for

 a "loop" as described above.



 In any event, getting the "bad" transaction out of the way will free

 up bugmail to process all the "good" transactions.





 Issue #6: Users are not getting notified about updates to defects owned

 on remote sites.



 Answer: The rules for notification mail when several DDTS sites are

 connected are not clearly spelled out in the manual but basically

 they are:



 - Notification mail to the submitter ($Submitter-mail) is done only

 on the DDTS site from which the defect was originally submitted. Thus,

 if a user used DDTS site AAAaa to submit a defect but now someone on

 the owning site (say BBBbb) has updated it, it is normal that site

 BBBbb would not send notification mail to the submitter... but when

 the defect-change transaction is propagated over to site AAAaa, then

 bugmail on *that* DDTS site should send the mail to the submitter.



 - Notification mail to the assigned engineer ($Engineer-mail) and the

 "other" list ($Other-mail) is only sent by the DDTS site that owns the

 project to which the defect belongs. Thus, in the connected-site example

 above (defect submitted from site AAAaa, updated on owning site BBBbb),

 it is normal that site AAAaa would not send mail to the assigned

 engineer or the addressees on the Other-mail list, since presumably the

 other site (BBBbb) has already done that.



 - Project-level notification (the proj.notify file) is separate by

 site:

   . For projects that are only connected and not subscribed, then

        all the notification is done on the owning site (there is no

        projects/PROJECTNAME/proj.notify file on the connected site;

        that's normal!

   . For subscribed projects, there is a proj.notify file on each

        site and they are completely independent. Each site will send

        to the list of recipients from its proj.notify file. If you

        put a user into both files, they will get two copies of the

        notification (one sent from each DDTS site).



 The rules are spelled out in more detail in "THE DETAILS OF EMAIL

 ADDRESSING" below.





 Issue #7: I'm not sure the messages are even being sent; how can I tell?



 Answer: To narrow the scope of the problem, look in the LOG to see if

 bugmail tried to send mail, and to get details about the precise manner

 in which it sent it; see "THE DETAILS OF THE LOG FILE" below for

 details. It is best to focus on a specific transaction: a specific

 defect updated in a particular way (state change? modify defect? add

 enclosure?) by a specific user (get the username) at a specific date

 and time. Look in the LOG for lines from the 'bugmail' daemon for that

 period of time (as described below) and look particularly for messages

 from the mailer, indicating that the message was sent.





 Issue #8: I looked in the LOG and there is a message being sent, but

 the user says they didn't get it. What now?



 Answer: If the LOG indicates a call to the mail program (typically

 /usr/bin/mailx) and the command and operands (particularly the

 recipient address) look correct but the message was not delivered,

 then DDTS is probably not at fault. You need to talk to the mail

 system administrators at your site and ask them why a message

 sent in the fashion shown in the LOG, from user 'ddts' on that host,

 to the recipient address as shown on the LOG, would not be deliverable.



 Try sending a test message by logging on as user 'ddts' to the DDTS

 server host, and sending the message in the manner shown on the LOG.

 If you get errors or the mail bounces, then you'll need to work with

 your administrators to find out why and to fix the problem.



 Issue #9: The messages on the LOG show a call to the mailer, but there

 is a problem with the command.



 Answer: There are two common cases here:



 a) The command name or operand syntax are wrong for your system. If

 this is the case, run the 'adminbug mins' command and when it gets to

 the prompt about the mailer, backspace over the existing entry and

 enter the correct response. The response should be the "skeleton"

 of a mail command; it should be identical to the syntax you would use

 from the UNIX command line to send a message, except that:



   - You can use '%s' in place of the actual mail subject string

        (remember to enclose it in quotes, since the string may

        contain multiple tokens).



   - You need not put the recipient address on the command line. This

        will be appended by bugmail before issuing the command.



   - You need not specify the source of the input mail body text. This

        will be piped to the command's stdin by bugmail.



 b) The other common problem is that the recipient address is

 improperly formatted or missing some needed qualification to make it

 deliverable. For example, suppose you see the message going to a

 simple unqualified userid like 'fred', but since Fred is a user on

 a different email system the email address needs to be adorned with

 some qualification (i.e. 'fred@mailhub.mycompany.com').



 DDTS gets the recipient addresses from the *-mail and *-notify fields

 as described in the Admin Guide and in more detail below under "THE

 DETAILS OF EMAIL ADDRESSING". If the address is wrong on the

 LOG, that means it is wrong in one of those fields. You need to

 determine which field it's coming from and implement a strategy to

 repair the value.



 For example, if the person whose address is wrong was the submitter of

 the defect, then you'll want to look in the Submitter-mail field

 derivation of your master.tmpl and verify that the field is being

 properly populated. There are a variety of strategies you can use and

 no single strategy is applicable to all customers; that is why DDTS is

 flexible enough to allow you to customize the derivation of values for

 fields (like Submitter-mail). If you need help customizing master.tmpl

 please see the Administrator's Guide (chapters 7 and 8), the sample

 software class master.tmpl supplied with DDTS, or contact DDTS

 Technical Support if none of the examples are applicable to your case.



 If you decide you need to change master.tmpl, that will work correctly

 for new defects but existing defects in your database still have wrong

 values. To correct that, you should use the 'findbug' utility to

 find affected defects and the 'batchbug' utility to fix the field

 values; for example:



   findbug Submitter-mail isequal fred |\

        batchbug -n -l 600 Submitter-mail fred@mailhub.mycompany.com



 See the manpages for findbug and batchbug (type "man findbug" or

 "man batchbug") for details.





 Issue #10: The LOG does not show any calls to the mailer. What now?



 Answer: If the LOG does not indicate any calls to the mailer program

 for that recipient for this transaction, then DDTS must have examined

 the list of possible recipients and deciding to reject each of them for

 various reasons. If your situation is not explained by any of the

 issues above (especially the one about DDTS not notifying the person

 performing the change) then read through "THE DETAILS OF EMAIL

 ADDRESSING" below and work through your specific case in detail.





 Issue #11: Users are getting notified several times for changes to

 the same defect.



 Answer: DDTS sends notification mail for *every* modification to a

 defect, not just state changes. This is stated in a couple of places,

 but it is easily overlooked because the prompts during project

 definition are misleading; for example:



   Enter mail addresses of those to notify upon the arrival

   of a new bug:



 A more accurate wording of this question would be:



   Enter mail addresses of those to notify when a new defect is

   submitted or a defect in the "New" state is updated (including

   modifying the defect fields, adding/editing an enclosure, or

   committing a CM transaction referencing this defect):



 The latter wording more accurately reflects the way DDTS works.



 Note: In release 4.1, DDTS has been changed so that CM transactions

 do not trigger notification mail.





 WHAT ELSE CAN GO WRONG????



 Answer: Well, a lot of things. If your case does not appear to be

 covered by any of the above, then go through the explanations offered

 below under "THE DEAILS OF THE LOG FILE" and "THE DETAILS

 OF EMAIL ADDRESSING" and see if anything there explains your situation.

 If not, then gather all the data discussed under "THE DETAILS OF

 EMAIL ADDRESSING" pertaining to one specific transaction that did not

 yield the expected results, and contact DDTS Tech Support.



 Here is a summary of the factors that influence bugmail; please gather

 all this data for one or more instances of the problem prior to

 contacting Support:



 1) For defect affected by this transaction, what are the values of:

    $Project

    $Submitter-id

    $Submitter-mail

    $Engineer

    $Engineer-mail

    $Other-mail

    $Updated-by



 2) If the current transaction was a modification of the defect while

    staying in the same state, then what is the $Status field of the

    defect?



    If the current transaction was a state transition then what state

    was the defect transitioning from, what state was it transitioning

    to (from $Status in the defect) and what states are between the

    starting and ending point (per your states file)?



    For each state so identified, for the project the defect is in, what

    are the lines from ~ddts/projects/$Project/proj.notify?



 3) What is the userid who you expected to recieve mail but did not?



 4) What are the lines on the ~ddts/spool/LOG from bugmail from the

    system's attempt to commit this change?





 THE DETAILS OF THE LOG FILE



 In finding out why notification mail is *not* sent, it is useful to

 understand what happens in the processing of a *correct* delivery.

 One information source to gain insight into the workings of the

 bugmail daemon (which handles notification mail delivery) is the

 DDTS "LOG".



 The LOG is a file (~ddts/spool/LOG) which is written-to by all the

 DDTS daemons. The format of lines in the LOG is:



 daemon_name YYMMDD HHMMSS Some text....



 Where 'daemon_name' is the name of the daemon we're interested in

 (bugmail in this case), YYMMDD and HHMMSS are the date and time the

 message was written, and "Some text....." would be the actual text

 of the message.



 A normal transaction from bugmail results in several lines on the LOG.

 When problems occur, it is useful to look at the LOG and try to see

 what is *not* there that we would expect to see. That can often tell

 us exactly what the problem is.



 For example here are some lines from my system (edited to remove 'bugmail

 date time' from the front of each line to improve the readability):



 Daemon Started - 2652

 Start executing spool/bugmail/FOUrb00399

 FOUrb00399: Transitions: R

 Subject of mail: FAQ FOUrb00399 was updated by jveilleu

   /usr/bin/mailx -s 'FAQ FOUrb00399 was updated by jveilleu' tech-sys

   /usr/bin/mailx -s 'FAQ FOUrb00399 was updated by jveilleu' vijayp@rational.co

m

 Finished executing spool/bugmail/FOUrb00399

 Daemon Terminated Normally - 2652



 This example shows bugmail handling a modification to defect FOUrb00399,

 in which user 'jveilleu' transitioned the defect to the R state. There

 are two users to be notified, one of which ('tech-sys') is an email

 alias and the other ('vijayp@rational.com') is an ordinary user.



 The number in the "Daemon Started" and "Daemon Terminated Normally"

 messages is the the process ID of the deamon.



 Between the "Daemon Started" and "Daemon Terminated Normally" messages

 there will be one or more sets of the "Start executing..." and "Finished

 executing..." lines. Each set pertains to one transaction file (i.e.,

 one defect update for which notification mail needs to be sent). In

 this example there is only one: file "spool/bugmail/FOUrb00399".



 The "...Transitions:..." line indicates the states the defect passed

 through in this transaction. There may be only one letter, as shown

 here, or there may be several if the user did a multiple-state

 transition all in one operation.



 For each transition state, bugmail uses the ~ddts/class/CLASSNAME/mail.subject

 template to derive the subject string for the mail message and shows it

 here on the LOG.



 Then, for each recipient who should receive this mail (see "THE

 DETAILS OF EMAIL ADDRESSING"), bugmail builds and submits a UNIX command

 to send the message. The actual command is shown here in the LOG (the two

 lines that start "/usr/bin/mailx....".



 Several things are noteworthy about this:



 1. It is normal for bugmail to process a transaction and send no mail

 (no "/usr/bin/mailx..." commands like the ones shown here) *if* there

 were no users who should receive notification of this event. See "THE

 DETAILS OF EMAIL ADDRESSING" for more detail.



 2. bugmail only ever sends a message to one recipient per call to the

 mailer (this is primarily to support older mailers that did not

 support multiple recipients on one command). If the message needs to

 go to multiple recipents, bugmail will issue multiple commands (which

 would each be shown here).



 3. You can discern a lot by what *isn't* here: for example, if you see

 lines similar to the first three in my example ("Daemon Started...",

 "Start executing...", "...Transitions...") but the next line is

 another "Daemon Started..." message, then since you do *not* see

 the expected "Subject...", "..mailx..", "Finished...", and "Daemon

 Terminated..." lines then it would be fair to assume that the bugmail

 daemon crashed. In such an event, look for a 'core' file in the DDTS

 home directory and contact DDTS Tech Support.



 THE DETAILS OF EMAIL ADDRESSING



 The high-level discussion about email addressing in the Administrator's

 Guide on pages 11-4 and 11-5 is technically correct but it leaves out

 some details that can be useful to debugging this kind of a problem. The

 text below attempts to fill that gap.



 Here is some additional detail about how the process works:



 1. bugmail starts by reading its transaction file (a file in spool/bugmail)

 and uses its contents to determine the identity of the defect for which

 notification mail is to be sent. The most important defect-record fields

 in this regard are:



 - The Identifier field, which it uses to find the pathname to the

   defect's allbugs file; see below.



 - The three email fields (Submitter-mail, Engineer-mail, Other-mail).



 - The Project field, which allows it to build the pathname to the

   proj.notify file. That file is used later to get the notification

   lists for the various state-changes the defect passed through

   during this transaction.



 Note a subtle point: the transaction file in bugmail's spool directory

 is the *old* copy of the bug file, before the current transaction was

 applied to it (basically a copy of the allbugs file *before* bugs.in

 committed the transaction). bugmail uses information in the transaction

 file (the old data) but also retrieves the defect's current allbugs file

 (which by this time has been updated to incorporate the current

 transaction) so that it can get old-and-new values for certain key

 fields. For example, this allows bugmail to determine the old and new

 state of the defect so it can determine the list of states that the

 defect passed through in the current transaction; it also allows bugmail

 to compute the "field change report" shown in the bottom of the message,

 to determine whether certain key fields like $Engineer have changed

 (see below), etc.



 2. For Submitter-mail, bugmail uses the following logic to determine

 whether this person should receive a message:



    If the first 5 characters of this defect's Identifier matches

       this DDTS site's site ID (i.e., submitted on this site)

    AND $Submitter-id is not null,

    AND $Submitter-mail is not null

    AND $Submitter-id is not equal to $Updated-by

    then add $Sumitter-mail to the list of recipients



    In other words: Send it if the defect was submitted on this site,

    if there is a value in Submitter-mail (someone to send to), and

    this person was not the person who committed this change.



 3. Likewise for $Engineer-mail:



    If this record's $Engineer field is not null,

    AND $Engineer-mail is not null

    AND $Engineer is not equal to $Updated-by

    AND the project indicated in $Project is owned on this DDTS site

    then add $Engineer-mail to the list of recipients



    In other words: Send it if the defect is in a project owned on

    this site, if there is a value in Engineer-mail (someone to

    send to), and this person was not the person who committed this

    change.



    NOTE: If the $Engineer has changed (meaning the defect has been

    reassigned from one person to another), both the old and the new

    values of $Engineer-mail are added to the recipient list.



 4. Similarly for Other-mail but there is no corresponding "-id" and

    "-mail":



    If the project indicated in $Project is owned on this DDTS site

    then

      for each token in $Other-mail

        if token != Updated-by

          add 'token' to the list of recipients



    In other words: If the project is owned on this site, send it to

    each person on this list who is *not* the person who committed the

    current transaction.



 5. Similarly for the project notification lists:



    - Determine the states that the defect transitioned through,

         using the 'states' file starting from the original state (from

         the old defect record) until reading the new state (from the

         transaction file).

    - For each state that the defect transitioned through on the

      current transaction, retrieve the X-notify (where "X" is the

         state letter of the state we're examining) from this project's

         proj.notify file.

    - For each token on each retrieved list, if it is not equal to

      $Updated-by then add that token to the list of recipients.



 All the notifications are actually built and queued in a big table, then

 sent at the very end. This process inherently drops duplicates since the

 key to the table is the mail address to which mail is being sent. So if

 the same user is listed as Submitter-mail and on the project notify list

 for example, they shouldn't receive two copies of the mail.



 This can sometimes get faked-out if the userid is specified differently

 in the two places, i.e. as a simple userid in one place and as a

 fully-qualified email address in the other.





 REFERENCES

 ----------

 ClearDDTS Administrator's Guide, Version 4.1, Chapter 11

 ClearDDTS User's Guide, pages 2-6 through 2-8

 The sample class/software/master.tmpl contains enlightening comments.

 man bugmail