A more detailed diagram of the batch streaming architecture is included below. Two additional elements are of note here:
- Chunk Key
This table (called BatchChunkKey) is essentially used as a key server, allowing chunks to be "served" up to individual streams without creating contention on the chunk table itself. It is also worth noting that the value of the next key available can be examined to determine the progress of the batch program.
- Batch params
The details of the parameters passed into the Chunker batch process are stored to make them available to each stream without being re-entered. This table, called BatchProcess, also contains the total number of chunks written, along with some other information about the Batch Process.
Note: The batch streaming architecture also supports the dynamic addition of streams while the batch process is being run, subject to the appropriate hardware being available to execute them.
Figure 1. Streaming Architecture Details