You can have as many message definition files as you want within one message set. But you should limit your message sets to a few related message definition files that share the same physical formats. There are several reasons for this:
So for example, if you have an XML message and an unrelated CWF message in the same message set, then both XML and CWF physical format properties will be present for all objects. But the XML properties are of no interest to the CWF message and so will take default values in that message. (Similarly, the CWF properties are not of interest to the XML message.) This can result in unwanted task list warnings.
So if you are modeling XML messages that have a recursive structure, you must ensure that recursive XML messages do not share a message set with non-XML physical formats.