Analytics Platform Mediation 

Installation and Configuration Guide for CSP Solutions

Contents
Introduction
Installation Guide
Prerequisites
RPM installation
Building Prerequisites
Building LookupManager
Building ITE
Building RawTDR
Vedia Supercollector
Host configuration
Parameters configuration
Output directory
Streams targets
Start / stop
Cleanup
Storage cleanup
Zookeeper setup
External Zookeeper
Streams main domain
Sizing considerations
Customization
Compile-time parameters
Data Loader Scripts customization
Administration Guide
Starting the applications
Stopping the applications
Checking the application status
Cleaning temporary files
Updating the provisioning
Consulting log files

This document provides information regarding installation, deployment, configuration and maintenance of the mediation layer of the Analytics Platform 2.0 system.
The business logic of Analytics Platform 2.0 is the same as Analytics Platform 1.0; the only difference is that the mediation is Streams-based instead of Vedia-based.
The mediation solutions includes the following components:
•	Super collector (Vedia-based)
•	Lookup Manager (Streams-based)
•	ITE Application to produce 5 minute aggregations (Streams-based)
•	Raw TDR processing Application (Streams-based)
•	DB Loader Application (Streams-based)
Please note that this document will not cover the installation of the Streams platform, which is a prerequisite for the application deployment.
Please also note that the Hadoop installation and Hadoop-based aggregation is out of scope of this document.

Introduction
This document describes steps needed to install and deploy the following Analytics Platform 2.0 interfaces:
Gn
LTE S1U / S11
Voice & SMS
It also describes how to install and configure all mediation components described in the Executive section above.
Installation Guide
This section describes how to install the mediation layer of Analytics Platform 2.0.
The system is composed of the following Streams applications:
•	ITE – Aggregates TDRs over 5 minutes. Several builds of the same application are provided, one for every supported interface.
•	RawTDR – Prepares raw TDRs for loading into the DB. Several builds of the same application are provided, one for every supported interface.
•	LookupManager – The controlling process for all ITE jobs, which provides access to shared provisioning data (Home and Visited MCC/MNC, Cells)
•	DBLoader – Loads aggregated records and raw TDRs to the DB
•	It is possible to install one or more interfaces in the same blade if needed.

Prerequisites
This installation assumes that:
•	You have done a sizing assessment (per interface) in terms of traffic throughput (Gbit/s), number of cells and locations and number of unique daily subscribers for each of the SourceWorks probes
•	IBM InfoSphere Streams 4.0.1 is installed
•	There is a biadmin user belonging to a biadmin group, and this user can run Streams commands
•	The ulimit settings for the user are compatible with Streams (see http://www-01.ibm.com/support/knowledgecenter/SSCRJU_4.0.1/com.ibm.streams.install.doc/doc/ibminfospherestreams-install-operating-system-settings.html?lang=en)
•	BigInsights and its dependencies (Hadoop and the Analytics Platform Data Loader) are correctly installed as per the Installation-And-Configuration-Guide.pdf located in the Analytics Platform framework release.
•	Network attached storage is set up and attached to all blades where both Streams and Vedia Supercollector are to be deployed. The shared storage has r/w permissions for the biadmin user. The disk capacity needs to be sufficient to contain at least an hour worth of gzipped feed from all SourceWorks probes.

RPM installation
The installation of mediation components is performed through RedHat's RPM system.
All components of Analytics Platform  are contained within a ZIP archive RapidAnalytics-xxxxxx
rapidanalytics
├───build
│   ├───ra-platform
│   │   └─bis-main-2.x.x.rpm
│   ├───streams_gn
│   │   └─ite_gn-2.x.x.x.x.compile.rpm
│   ├───streams_lm
│   │   └─lookup_manager-2.x.x.x.x.compile.rpm
│   ├───streams_loader
│   │   └─loader-2.x.x.x.x.compile.rpm
│   ├───streams_lte
│   │   └─ite_lte-2.x.x.x.x.compile.rpm
│   ├───streams_rawtdr_gn
│   │   └─rawtdr_gn-2.x.x.x.x.compile.rpm
│   ├───streams_rawtdr_lte
│   │   └─rawtdr_lte-2.x.x.x.x.compile.rpm
│   ├───streams_rawtdr_voice
│   │   └─rawtdr_voice-2.x.x.x.x.compile.rpm
│   ├───streams_voice
│   │   └─ite_voice-2.x.x.x.x.compile.rpm
│   ├───vedia_fileloader
│   │   └─file_loader-1.1.0.0.0.rpm
│   ├───vedia_streamsloader
│   │   └─streams_loader-1.x.x.x.x.0.rpm
│   ├───interface-adapter
│   │   └─interface-adapter-8.2.0.1.2.rpm
│   ├───stream-adapter
│   │   └─stream-adapter-8.2.1.rpm
│   └─revision.txt
└─ReadMe.txt
Building Prerequisites
The Streams-based RPMs are self compiling: installing the provided RPM generates a binary RPM which can then be deployed on all the blades. Because of this the blade where this compilation is performed must meet the following additional prerequisites:
•	The packages rpm-build and zlib-devel are installed
•	InfoSphere Streams 4.0.1.0 is sourced in the shell from which the commands are issued
•	Building DBLoader
•	The DBLoader RPM is:
•	rapidanalytics/build/streams_loader/loader-2.x.x.x.x.compile.rpm

A configuration file can be created to specify additional options; it must be named loader-2.x.x.x.x.compile.ini in the same directory as the RPM, as specified in section 4.5.1. The package can then be built by issuing the following command as root in the same directory as the RPM:
rpm -ivh loader-2.x.x.x.x.compile.rpm --prefix $PWD
The binary RPM will be created as loader-2.x.x.x.x.rpm; if the operation fails, the file will still be present, but unusable, and the package will have to be removed (see later).
After a successful completion, the binary RPM will be ready to  be deployed on all blades. To install the binary RPM execute:
rpm -ivh loader-2.x.x.x.x.rpm
After the binary RPM is successfully deployed, the compiling package must be removed from the  blade with: 
rpm -e ra_loader_compile
This operation will remove the generated binary RPM, so it should be backed up somewhere else if it's necessary to preserve it.
Building LookupManager

The LookupManager RPM is:
rapidanalytics/build/streams_lm/lookup_manager-2.x.x.x.x.compile.rpm
A configuration file can be created to specify additional options; it must be named lookup_manager-2.x.x.x.x.compile.ini in the same directory as the RPM, as specified in section 4.5.1. The package can then be built by issuing the following command as root in the same directory as the RPM:
rpm -ivh lookup_manager-2.x.x.x.x.compile.rpm --prefix $PWD
The binary RPM will be created as lookup_manager-2.x.x.x.x.rpm; if the operation fails, the file will still be present, but unusable, and the package will have to be removed (see later).
After a successful completion, the binary RPM will be ready to  be deployed on all blades. To install the binary RPM execute:
rpm -ivh lookup_manager-2.x.x.x.x.rpm
After the binary RPM is successfully deployed, the compiling package must be removed from the  blade with: 
rpm -e ra_lookup_manager_compile
This operation will remove the generated binary RPM, so it should be backed up somewhere else if it's necessary to preserve it.
Building ITE
The ITE RPMs are (one per interface):
rapidanalytics/build/streams_gn/ite_gn-2.x.x.x.x.compile.rpm
rapidanalytics/build/streams_lte/ite_lte-2.x.x.x.x.compile.rpm
rapidanalytics/build/streams_voice/ite_voice-2.x.x.x.x.compile.rpm
A configuration file can be created to specify additional options; it must be named ite_interface-2.x.x.x.x.compile.ini (where interface is the ITE interface to build) in the same directory as the RPM, as specified in section 4.5.1. The package can then be built by issuing the following command as root in the same directory as the RPM:
rpm -ivh ite_interface-2.x.x.x.x.compile.rpm --prefix $PWD
The binary RPM will be created as ite_interface-2.x.x.x.x.rpm; if the operation fails, the file will still be present, but unusable, and the package will have to be removed (see later).
After a successful completion, the binary RPM will be ready to  be deployed on all blades. To install the binary RPM execute:
rpm -ivh ite_interface-2.x.x.x.x.rpm
After the binary RPM is successfully deployed, the compiling package must be removed from the  blade with: 
rpm -e ra_ite_interface_compile
This operation will remove the generated binary RPM, so it should be backed up somewhere else if it's necessary to preserve it.
Building RawTDR
The RawTDR RPMs are (one per interface):
rapidanalytics/build/streams_rawtdr_gn/rawtdr_gn-2.x.x.x.x.compile.rpm
rapidanalytics/build/streams_rawtdr_lte/rawtdr_lte-2.x.x.x.x.compile.rpm
rapidanalytics/build/streams_rawtdr_voice/rawtdr_voice-2.x.x.x.x.compile.rpm
A configuration file can be created to specify additional options; it must be named rawtdr_interface-2.x.x.x.x.compile.ini (where interface is the ITE interface to build) in the same directory as the RPM, as specified in section 4.5.1. The package can then be built by issuing the following command as root in the same directory as the RPM:
rpm -ivh rawtdr_interface-2.x.x.x.x.compile.rpm --prefix $PWD
The binary RPM will be created as rawtdr_interface-2.x.x.x.x.rpm; if the operation fails, the file will still be present, but unusable, and the package will have to be removed (see later).
After a successful completion, the binary RPM will be ready to  be deployed on all blades. To install the binary RPM execute:
rpm -ivh rawtdr_interface-2.x.x.x.x.rpm
After the binary RPM is successfully deployed, the compiling package must be removed from the  blade with: 
rpm -e ra_rawtdr_interface_compile
This operation will remove the generated binary RPM, so it should be backed up somewhere else if it's necessary to preserve it.
Listing and removing packages
The installed Streams RPMs can be listed with:
rpm -qa | grep ra_ | sort
An example output (where the compile RPMs have not yet been uninstalled) is:
ra_lm-2.x.x.x.x-r509.x86_64
ra_lm_compile-2.x.x.x.x-509.x86_64
ra_loader-2.x.x.x.x-r509.x86_64
ra_loader_compile-2.x.x.x.x-509.x86_64
ra_ite_gn-2.x.x.x.x-r509.x86_64
ra_ite_gn_compile-2.x.x.x.x-509.x86_64
ra_ite_lte-2.x.x.x.x-r509.x86_64
ra_ite_lte_compile-2.x.x.x.x-509.x86_64
ra_ite_voice-2.x.x.x.x-r509.x86_64
ra_ite_voice_compile-2.x.x.x.x-509.x86_64
ra_rawtdr_gn-2.x.x.x.x-r509.x86_64
ra_rawtdr_gn_compile-2.x.x.x.x-509.x86_64
ra_rawtdr_lte-2.x.x.x.x-r509.x86_64
ra_rawtdr_lte_compile-2.x.x.x.x-509.x86_64
ra_rawtdr_voice-2.x.x.x.x-r509.x86_64
ra_rawtdr_voice_compile-2.x.x.x.x-509.x86_64
Note that the release number (509 in the example) is the SVN revision number of the source code.
In order to remove all packages, you can execute the following command as root:
rpm -qa | grep ra_ | xargs rpm -e
For Vedia Supercollector installation please refer to the next section.
Vedia Supercollector
Streams components have no ability to collect files from a remote host (ftp/sftp). The remote collection is done using a Vedia Supercollector component. It connects to a set of SourceWorks probes and downloads netro or pagebased feed files to the network attached shared storage (see the prerequisites section). The shared storage can be then accessed by all Streams hosts.
This section describes steps necessary to deploy the Vedia Supercollector. Ideally, it has to be deployed into a separate blade or even multiple blades.
Installation
Extract the following file from the Analytics Platform 2.0 ZIP archive: rapidanalytics/build/vedia_fileloader/file_loader-1.1.0.0.0.rpm. Then, as user root, perform the RPM installation in each host where you want to deploy Vedia Supercollector using the command:
rpm -ivh file_loader-1.1.0.0.0.rpm
Vedia Supercollector is now installed in /opt/tnf/file_loader/vedia/ as the user biadmin.
Host configuration
Change collectors and probes parameters in /opt/tnf/file_loader/vedia/cfg/VediaMain.xml. This is typically done in one host only and then all configuration files are copied to all hosts where Vedia Supercollector was installed.
First of all define as many Supercollector hosts in the <hosts> section as there are Supercollector blades. One host needs to be the main host.
<hosts>
    <host name="Collector1" address="blade01" main="1"/>
    <host name="Collector2" address="blade02"/>
</hosts>

Then define collection applications using <applications> sections. Each applications tag needs to have the following attribute:
probe_type: value can be netro, lte, pb, S1_S6_S13, SGs, VoiceSMS_A or VoiceSMS_IuCS
All collectors in this applications node group will download data from probes with this type of data only.
Then each <applications> tag has to contain one or more <application> tags that define a collector application (process):
  <applications name="CollectorNT" host="Collector" config="Collection" probe_type="netro">
    <application name="CollectionNT1" probe_name="NYCEM101" probe_ip="scw_gn_blade01"/>
    <application name="CollectionNT2" probe_name="DACEM101" probe_ip="scw_gn_blade02"/>
  </applications>

  <applications name="CollectorLTE" host="Collector" config="Collection" probe_type="lte">
    <application name="CollectionLTE1" probe_name="NYCEM103" probe_ip="scw_lte_blade03"/>
  </applications>

Each application needs to have the following attributes and all of them are mandatory:
name: unique logical name of this Collector
probe_ip: IP address or host name of the SCW probe host
probe_name: name of the SCW probe for storing in storage output directory (see section 4.3.3.2 for more details on the output directory structure). The name is arbitrary with two restrictions:
It needs to be unique within the whole deployment. The uniqueness also applies for probe names across different interfaces. The only exception is for Voice/SMS interface where there can be the same probe name for types VoiceSMS_A and VoiceSMS_IuCS
It needs to match to probe names as defined in Streams in configuration files probes.csv and probes-tdr.csv, see sections 4.5.2 and 4.5.3.
Parameters configuration
These parameters in /opt/tnf/file_loader/vedia/cfg/consts.xml can be configured:
SourceWorks probes
Parameter	Description	
probe_protocol	Remote collection protocol (sftp/ftp)	
probe_user	Username to probe (S)FTP Server	
probe_password	Password to probe (S)FTP Server	
probe_base_disk_dir	Remote directory of netro feed on probe (S)FTP Server	
probe_voicesms_base_dir	Remote directory for Voice/SMS collection	
probe_files_mask	Regular expression matching the file names; must include a timestamp between parentheses in the format below.	
probe_file_date_format	Timestamp format as found in the collected file names.	

By default, the Vedia Supercollector assumes standard feed directories and standard feed file masks in the SCW probe. These are defined in if probe_type="…" sections in consts.xml. If there is a different than standard location or file mask then you can set these different values for individual feed types in the corresponding if probe_type="…" section. In the example bellow different probe directory and file mask is set for the netro feed:
<if probe_type="netro">
  <const name="probe_dir" value="{probe_base_dir}netrofeedbin/" description="Ftp dir"/>
  <const name="probe_files_mask" value="netrobin_([0-9]{14})\.log\.gz" description="Ftp files mask"/>
</if>

Output directory
By default the output directory for all downloaded files is set to /opt/tnf/file_loader/vedia/storage/, but it can be changed:
Parameter	Description	
output_base_dir	Directory where downloaded files are stored (storage/). This has to be linked to the file system that is located in a network attached disk	

Files are being downloaded into the following directory hierarchy structure:
{output_base_dir}/{probe_name}|{probe_ip}/{probe_type}
E.g:
/opt/tnf/file_loader/vedia/P1/netro/netro_20120425000019.log.gz
Voice/SMS parameters
There are two types of Voice/SMS feed: Aif and IuCS. If a Supercollector is configured to collect Voice/SMS feed (using collection applications where probe_type=”VoiceSMS_A” or probe_type=”VoiceSMS_IuCS”), then the following can be configured in consts.xml in sections
<if probe_type="VoiceSMS_A">
   ...
</if>

and
<if probe_type="VoiceSMS_IuCS">
   ...
</if>

Parameter	Description	Default value for Aif / IuCS feed	
probe_dir	Ftp dir for the Voice/SMS Aif / IuCS feed	{probe_voicesms_base_dir}a/	
		{probe_voicesms_base_dir}iucs/	
probe_files_mask	File mask for the Voice/SMS Aif feed	Aif_ASN1_BER_[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}_([0-9]{8}\.[0-9]{4,6})	
		IuCS_ASN1_BER_[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}_([0-9]{8}\.[0-9]{4,6})	
adapted_files_mask	File mask for the adapted Voice/SMS Aif feed.
Note 1: This needs to be different than the probe_files_mask above.
Note 2: It needs to match to the Interface Adaptor output prefix and output options defined in datafeed.xml (see section 4.3.3.4)	Aif_([0-9]{8}\.[0-9]{6})_[0-9]{5}\.csv\.gz	
		IuCS_([0-9]{8}\.[0-9]{6})_[0-9]{5}\.csv\.gz	
probe_file_date_format	File date format for the Voice/SMS Aif / IuCS feed
Note: If call-interface-adaptor=no (see below) then it is assumed that the feed is already adapted. In that case this probe_file_date_format is used for this adapted feed and not the adapted_file_date_format (see below).	yyyyMMdd.HHmmss	
adapted_file_date_format	File date format for the adapted Voice/SMS Aif / IuCS feed	yyyyMMdd.HHmmss	
call-interface-adapter	Flag to enable call to the Voice/SMS feed adaptor (yes/no)	yes	
adapter-executable	Adapter script/executable to use for Voice/SMS feed	run-adapter-from-vedia-voice.sh	
adapter-data-feed-id	Feed ID of the Aif file to be adapted	polystar_a_v1	
		polystar_iucs_v1	
probe_user	Default Voice/SMS probe username	same as other feeds, can be changed	
probe_password	Default Voice/SMS probe password	same as other feeds, can be changed	

Voice/SMS Interface Adaptor installation
If call-interface-adaptor flag is set to yes then the installation of the Voice/SMS Interface Adaptor also needs to be performed. The adaptor converts Aif and IuCS feeds from different voice probe formats to a CSV format that can be parsed by Streams.
Extract the following file from the Analytics Platform 2.0 ZIP archive: rapidanalytics/build/interface-adapter/interface-adapter-8.2.0.1.2.rpm. Then, as user root, perform the RPM installation in each host where Vedia Supercollector was deployed already:
rpm -ivh interface-adapter-8.2.0.1.2.rpm
This will install the Interface Adapter into /opt/tnf/apps/interface-adapter as the user boss (ignore any warnings in case user boss does not exist). To modify the ownership of the adapters to biadmin, perform the following as root:
chown –R biadmin:biadmin /opt/tnf/apps/interface-adapter
The adaptor is invoked automatically from the Vedia Supercollector. 
Streams targets
Set the following variable depending on the number of Streams jobs that will collect each downloaded file from {output_base_dir}/{probe_name}|{probe_ip}/{probe_type} (i.e. as per the section above). 

Parameter	Description	
streams-target-list	Comma separated list of Streams-based mediation targets (symlinks will be created for these) – default: Streams-ITE,Streams-RawTDR	

If you want to load both aggregations (ITE application) and TDRs (RawTDR application) to Hadoop then there is no need to change the defaults. 
FYI, the purpose of these Streams targets is to create symlinks for each downloaded file in a location where Streams application expects them. E.g. for netro feed:
/opt/tnf/file_loader/vedia/storage/Streams-ITE/P1/in/data/netro/netro_20120425000019.log.gz
/opt/tnf/file_loader/vedia/storage/Streams-RawTDR/P1/in/data/netro/netro_20120425000019.log.gz

They both reference the same file: /opt/tnf/file_loader/vedia/P1/netro/netro_20120425000019.log.gz
Similarly for the pagebased feed
/opt/tnf/file_loader/vedia/storage/Streams-ITE/P1/in/data/page/pbqoe_20120425000019.log.gz
/opt/tnf/file_loader/vedia/storage/Streams-RawTDR/P1/in/data/page/pbqoe_20120425000019.log.gz

They both reference the same file: /opt/tnf/file_loader/vedia/P1/page/pbqoe_20120425000019.log.gz
It is expected that these directories (/opt/tnf/file_loader/vedia/storage/Streams-ITE/ and /opt/tnf/file_loader/vedia/storage/Streams-RawTDR/) are configured for individual Streams jobs in configuration files probes.csv (section 4.5.2) and probes-tdr.csv (section 4.5.3).
Please note that Vedia Supercollector creates the necessary directory hierarchy (required by Streams) automatically. I.e. the following is created automatically within the output directory:
./in/urgent/
./in/data/
./in/data/{feed_type}/
./in/inputDirectories.txt

Important note: It is expected that different file types (such as netro feed and pagebased feed) have unique file name. I.e. prefix of different file types is different.
Start / stop
To start, first start all “slave” Vedia Supercollector instances. Then start main instance.
To start a Vedia instance do this:
cd /opt/tnf/file_loader/vedia/
./vedia start

Stop has to be done in the reverse order to starting the Vedia Supercollector, i.e. first stop the master instance and then stop all “slave” instances.
To stop a Vedia instance do this:
cd /opt/tnf/file_loader/vedia/
./vedia stop

Cleanup
There are logs and temporary files that need to be cleaned periodically and also the downloaded files in the storage/ directory. Both tasks need to be scheduled in crontab in every host where Vedia Supercollector is deployed.
Logs cleanup
First add the following line to crontab:
# run pack every day
0 0 * * * cd /opt/tnf/file_loader/vedia; ./vedia pack

Secondly you need to decide what the retention period for keeping original (unzipped) and zipped log files is. This depends on the available storage. The following parameters can be set in the file /opt/tnf/file_loader/vedia/vedia:
logs_days_zip – Number of days to keep log files before they get zipped (default is 2 days)
logs_days_del – Number of days to keep zipped log files before they get deleted (default is 7 days)
tmp_days_del – Number of days to keep tmp files before they get cleaned (default is 2 days)
Storage cleanup
The deployment includes a script /opt/tnf/file_loader/vedia/disk-cleanup.sh for cleaning downloaded files in the storage output directories. All files older than 5 hours are cleaned (including symlinks for Streams based mediation). The period is configurable by setting the variable HOURS within the cleanup script.
Add the following line to crontab to run the script hourly:
# run storage cleanup every hour
0 * * * * /opt/tnf/file_loader/vedia/disk-cleanup.sh

Please note that the cleanup script uses the default output storage/ directory. If you changed this directory (output_base_dir value in consts.xml), need make the same change in this script:
LOCATION=/opt/tnf/file_loader/vedia/storage/
Streams domain and instance configuration
The intended setup of the system is to have one Streams instance per each interface per each blade. Also, ITE application and Raw TDR application should use different Streams instances. 
Zookeeper setup
There are two options for a Zookeeper:
Embedded (simpler)
External (potentially better performance)
Embedded Zookeeper
No installation is required to use the embedded Zookeeper. The embedded Zookeeper is already part of the Streams 4.0.1 installation. What is only needed is to specify the option --embeddedzk when creating a Streams domain using the mkdomain command (see section 4.4.2), i.e. 
unset STREAMS_ZKCONNECT
streamtool mkdomain -d rapidanalytics_domain --embeddedzk
Also, it is needed to increase the heap size using the following command before any Streams domain is created:
streamtool setbootproperty streams.zookeeper.jvmFlags=-Xmx4096m

External Zookeeper
An external Zookeeper can deliver potentially better performance. To install the external Apache Zookeeper please follow these instructions.
Ideally, the installation should be performed in a host where there is no other Streams domain. The recommended host is the host where Vedia Supercollector is installed (see section 4.3).
After untarring the Zookeeper package (zookeeper-3.4.6.tar) into /opt/tnf/rapid/zookeeper-3.4.6/ set the following parameters in the config file /opt/tnf/rapid/zookeeper-3.4.6/conf/zoo.cfg:
dataDir=/opt/tnf/rapid/zookeeper-3.4.6/data
clientPort=2181
maxClientCnxns=-1

Then the heap size of Zookeeper needs to be increased by putting a file java.env in the configuration directory (/opt/tnf/rapid/zookeeper-3.4.6/conf/java.env) with this command:
export JVMFLAGS="-Xmx4096m"
To start the Apache Zookeeper:
cd /opt/tnf/rapid/zookeeper-3.4.6/bin
./zkServer.sh start /opt/tnf/rapid/zookeeper-3.4.6/conf/zoo.cfg

To stop the Apache Zookeeper:
cd /opt/tnf/rapid/zookeeper-3.4.6/bin
./zkServer.sh stop /opt/tnf/rapid/zookeeper-3.4.6/conf/zoo.cfg

Next step is to define an environment variable $STREAMS_ZKCONNECT in all hosts where streams is going to be deployed. It must be set to the IP and port of the running Zookeeper instance, e.g.
export STREAMS_ZKCONNECT=blade01:2181
On each of the Streams blades, set the environment variable $STREAMS_ZKCONNECT to the same value (in a .profile or a .bash_profile file)
Streams main domain
In order to avoid any interference the blades should share the same Streams domain.
Provided that the environment variable $STREAMS_ZKCONNECT is correctly configured as per the previous chapter, the domain can be set up with the following commands (where rapidanalytics_domain is the domain to be created):
streamtool mkdomain -d rapidanalytics_domain
streamtool genkey -d rapidanalytics_domain
streamtool startdomain -d rapidanalytics_domain
The environment variable $STREAMS_DOMAIN_ID should be set to this Streams domain (in a .profile or a .bash_profile file) in every blade so that it is not necessary to specify it when running a Streams command every time.
Note that the -d rapidanalytics_domain in the commands above can be avoided if $STREAMS_DOMAIN_ID is set.
Streams instances per blade per interface
There has to be a different Streams instance per blade per interface per application type (ITE or Raw TDR processor).
Run the following commands to create the instance (where rapidanalytics_instance_blade02_ite_gn is the name of the interface to be created and blade02 is one of the blade's external IP addresses or a host name pointing to it):
streamtool mkinstance -i rapidanalytics_instance_blade02_ite_gn --hosts blade02
streamtool startinstance -i rapidanalytics_instance_blade02_ite_gn
As there must be one instance per interface on each blade, it might be sensible to choose a name that reflects this, e.g. a combination of blade name, application type and interface as per the example above.
In a deployment where only one Streams interface (i.e. only one blade, one interface) is configured that one Streams instance can be set to an environment variable $STREAMS_INSTANCE_ID so that it is not necessary to type it with every Streams command.
Note that the -i instance in the commands above can be avoided if $STREAMS_INSTANCE_ID is set.
Before switching off or rebooting the blade, it's recommended to manually stop the instance with
streamtool stopinstance -i rapidanalytics_instance_blade02_ite_gn
In case the blade was rebooted without stopping the instance, it can be stopped forcefully with
streamtool stopinstance -i rapidanalytics_instance_blade02_ite_gn --force
Afterwards it can be restarted with:
streamtool startinstance -i rapidanalytics_instance_blade02_ite_gn
You can also list all created instances with:
streamtool lsinstance 
Or list all instances for one domain by adding the -d rapidanalytics_domain option:
streamtool lsinstance -d rapidanalytics_domain
Sizing considerations
By default the ITE applications use 4 parallel collection chains and a maximum of 5 parallel aggregation contexts (configurable at compile-time as specified in section 4.5.1) divided in 3 groups for Gn and LTE (UP, UPS, CP+CPS) and 2 for Voice&SMS (VSMS, VSMSStatus); how the contexts are split between the groups is configurable as per section 4.5.6, but by default 3 contexts are used for UP, 4 for VSMS and 1 for the others.
With the above setup the tested throughput per probe is 4 million TDR/min on an HP blade and 6.5 million TDR/min on a Flex blade; performances degrade gradually when increasing the number of probes, and the maximum recommended amount of fully loaded probes is 3 on an HP blade (totaling 7.5 million TDR/min) and 4 on a Flex blade (totaling 16 million TDR/min).
If the probes' throughput is lower than 1 million TDR/min, the ITE applications can be configured with appSplitterEnabled=false in the constants file (see section 4.5.6), so that all aggregation is performed in the same context and more probes than listed above can be processed by the same blade.
The application's footprint can be further decreased by reducing the overall maximum number of contexts (NumContexts=1) and chain processors (NumChains=1), but this configuration is a compile-time parameter (see section 4.5.1).
Customization
The following is a list of customization files for all interfaces. Please review all of them and change the configuration as needed.
Unless otherwise stated, the files are located in /opt/tnf/rapid/cfg/.
Compile-time parameters
The Streams RPMs (see section 4.2) accept a configuration file at the path indicated by the --prefix option; it must be named as the RPM file but with suffix .ini, so e.g. loader-2.x.x.x.x.compile.rpm will use a configuration file named loader-2.x.x.x.x.compile.ini. The content must be a list of options in the form A=B, one per line, with blank lines and comments (starting with '#') allowed; the following table lists the available options.
Parameter	Description	
HadoopSegments	Used by DBLoader and ITE apps; it specifies how many segments are configured in Hadoop (default 3)	
NumChains	Used by ITE and RawTDR apps; it specifies how many parallel collection chains to use (default 4)	
NumContexts	Used by ITE apps; it specifies how many aggregation contexts to use (default 5)	
NumAggProbes	Used by LookupManager; it specifies how many probes can be configured at most per each interface in probes.csv (see next section, default 4)	
CellCacheSizeMB	Used by LookupManager; it specifies how much space to allocate for the Cell lookup, in megabytes	
MCCMNCCacheSizeMB	Used by LookupManager; it specifies how much space to allocate for the MCC-MNC lookup, in megabytes	
BNumberCacheSizeMB	Used by LookupManager; it specifies how much space to allocate for the BNumber lookup, in megabytes	
IMSIMSISDNCacheSizeMB	Used by LookupManager; it specifies how much space to allocate for the IMSI-MSISDN lookup, in megabytes	
StatusCacheSizeMB	Used by LookupManager; it specifies how much space to allocate for the Status code lookup, in megabytes	
minFields6013,maxFields6013,minFields6021,maxFields6021,minFileds6022,maxFields6022	Used only for Gn interface ITE app. Specify the minimum and maximum number of CSV fields, expected in input files of the given type. TDRs with less fields or more fields than expected are rejected.	
minFields7013,maxFields7013,minFields7021,maxFields7021,minFileds7022,maxFields7022	Used only for LTE interface ITE app. Specify the minimum and maximum number of CSV fields, expected in input files of the given type. TDRs with less fields or more fields than expected are rejected.	
minFieldsVoice,maxFieldsVoice	Used only for Voice/SMS interface ITE app. Specify the minimum and maximum number of CSV fields, expected in input file. TDRs with less fields or more fields than expected are rejected.	

probes.csv
This file lists all probes collected by the blade for processing by the ITE apps. The default content of the file is a guide on how to configure it. For each probe you must specify on one line, pipe separated, the following fields:
Parameter	Description	
instance	The name of the Streams instance that the probe will be processed by	
type	One of gn, lte or voice (note that it must be the same for all lines referring to the same instance)	
name	The probe name as it should appear in the output records	
directory	A directory path (see also section 4.3.3.2 with Vedia Supercollector output directory settings), inside which a directory must exist with the same name as the probe, and inside this directory the file in/inputDirectories.txt, containing the input directory names, one per line.
Default is /opt/tnf/file_loader/vedia/storage/Streams-ITE/	

Please note that you cannot configure more probes per interface than specified in the NumAggProbes option (see section 4.5.1).
As indicated in the description, an input directory for a probe has to have a specific directory hierarchy. The default hierarchy starts at the (default) path /opt/tnf/file_loader/vedia/storage/Streams-ITE/.
In the above example path the input TDRs are expected to appear in /opt/tnf/file_loader/vedia/storage/Streams-ITE/{probe_name}/in/data/{feed_type}/, where {probe_name} is the name of the probe as per the table above and {feed_type} corresponds to different file types (e.g. netro, pagebased, etc). Any data that appear in /opt/tnf/file_loader/vedia/storage/Streams-ITE/{probe_name}/in/urgent/ will be processed with a higher priority (required during recovery). A successfully processed file is moved to the /opt/tnf/file_loader/vedia/storage/Streams-ITE/{probe_name}/in/archive/ directory.
Please also note that when Vedia Supercollector is deployed then input files are actually symlinks, because one input file might be collected by multiple mediation components. When a symlink is moved to the archive/ directory then the original file is still available for the other mediation component.
probes-tdr.csv
This file lists all probes collected by the blade for processing by the RawTDR app. The default content of the file is a guide on how to configure it. For each probe you must specify on one line, pipe separated, the following fields:
Parameter	Description	
instance	The name of the Streams instance that the probe will be collected by	
type	One of: gn, lte or voice (note that it must be the same for all lines referring to the same instance)	
name	The probe name as it should appear in the output records	
directory	A directory path (see also section 4.3.3.2 with Vedia Supercollector output directory settings), inside which a directory must exist with the same name as the probe, and inside this directory the file in/inputDirectories.txt, containing the input directory names, one per line.
Default is /opt/tnf/file_loader/vedia/storage/Streams-RawTDR/	

The input directory contains sub-directories as per the previous section with the only exception that instead of Streams-ITE subdirectory the RawTDR application uses Streams-RawTDR as per the default directory in the parameter list above.

cells.csv
The cells provisioning file: it must contain the following fields, comma-separated (no comments allowed). Only TDRs with cells specified in this file may be processed (others will be rejected).
Parameter	Description	
lac/tac	The cell area code, LAC for Gn probes and TAC for LTE probes	
ci/eci	The cell identifier, CI for gn probes and ECI for lte probes	

mcc_mnc.csv
The country/network provisioning file: it must contain the following fields, comma-separated (no comments allowed):
Parameter	Description	
mcc	The mobile country code	
mnc	The mobile network code	
operator type	One of 0 (international), 1 (home) or 2 (probe)	
country	The name of the country	
prefix	The country's international prefix	
network	The name of the network provider	

consts_gn.ini, consts_lte.ini, consts_voice.ini
These two files have similar contents, but each is used by the specific interface; among the many options, the following are those of greater interest (defaults in bold):
Parameter	Description	
netroFileMask	Gn and LTE only: the file mask for netro input files*	
pagebasedFileMask	Gn and LTE only: the file mask for page based input files*	
aFileMask	Voice only: the file mask for Aif (voice-a) input files*	
iucsFileMask	Voice only: the file mask for IuCS (voice-iucs) input files*	
appSplitterEnabled	Enable/Disable aggregation segmentation (true – split among multiple contexts, false – do not split)	
numContextsUP	Gn and LTE only: specify how many contexts to split UP data in when appSplitterEnabled=true (3)	
numContextsUPS	Gn and LTE only: specify how many contexts to split UPS data in when appSplitterEnabled=true (1)	
numContextsCP	Gn and LTE only: specify how many contexts to split CP data in when appSplitterEnabled=true (1)	
numContextsVSMS	Voice only: specify how many contexts to split Voice&SMS data in when appSplitterEnabled=true (4)	
numContextsVSMSStatus	Voice only: specify how many contexts to split Voice&SMS Status data in when appSplitterEnabled=true (1)	
loadUnknownSubscribers	Enable/Disable the filter for unknown IMSIs in level 3 tables (true – load data for unknown IMSIs to DB, false – filter out unknown IMSIs)	
loadUnknownCells	Enable/Disable the filter for unknown cells in level 2 tables (true – load data for unknown cells to DB, false – filter out unknown cells)	
filterCPsuccessCodes	Enable/Disable control plane status code filters (true – filter out success codes, false – load all codes to DB)	
CPsuccessCodes	The comma-separated list of success codes for the interface	
ignorePageBased	Enable/Disable page based feed (true – ignore PB TDRs, false – process PB TDRs)	
roamersFilter	Specify roamers IN behaviour (subscriber – IMSI is used as aggregation key, MSISDN and IMEI fields are populated; operator – MCC/MNC extracted from the IMSI are used as an aggregation key and the rest of IMSI characters are replaced by zeros, MSISDN and IMEI fields are not populated; filter – records generated by roamers IN are filtered out and not used for aggregations)	
checkProbeCells	Specify whether to use the provisioned cell list (true – only provisioned cells are allowed, false – non-provisioned cells are replaced by zeroes)	
operator_prefix	Gn and LTE only: specify a list of operator prefixes; if set to a non-null value, only IMSIs starting with a value in the comma-separated list will be processed	
loaderDirectory	Specify where the output should be written (/opt/tnf/rapid/loaderDir)	

* The file masks must contain a timestamp in the format YYYYMMDD[.]hhmmss (where the square brackets indicate that the dot is optional) in order for the file ordering to correctly sort the input feed by file timestamp.
rawtdr.ini
The RawTDR app prepares raw TDRs for loading into the database. The submission time parameters are:
Parameter	Description	
fileMask	The file mask for collected files. Default is '^.+\\.gz$'. 
Files that don’t match the mask will not be processed and will be placed into the ‘invalid’ directory (in/invalid)	
waitMin	Define how long (in minutes) should the 1-hour partitions wait untouched until being sent for loading to hadoop. Default 60 minutes	
maxRecordsGn
	Max number of records that can be stored in 1h partitioned files (for Gn) before they get closed for loading. If set to 0 the file will grow indefinitely and will close after the waitMin time	
maxRecordsLTE	Max number of records that can be stored in 1h partitioned files (for LTE) before they get closed for loading. If set to 0 the file will grow indefinitely and will close after the waitMin time	
maxRecordsVoice	Max number of records that can be stored in 1h partitioned files (for Voice) before they get closed for loading. If set to 0 the file will grow indefinitely and will close after the waitMin time	
ignorePageBased	Define whether page based records (x013) should be ignored (applicable for gn and lte)
1: ignore page based, 0: don't ignore	
loaderDirectory	The absolute path to the directory where rawtdr will store output files 
for processing by the loader (/opt/tnf/rapid/loaderDir/raw/)	

loader.ini
The DBLoader collects aggregated and TDR files, converts them through an external script, then loads the records to the DB through yet another external script. The loader app accepts several submission time parameters which are automatically provided to the application by the rapid.sh script depending on the loader type so there are currently no configurable settings available in loader.ini
Data Loader Scripts customization
The Loader application invokes loader scripts which may require customizing. The scripts and the configuration are installed into /opt/tnf/rapid/etc/script/
The configuration file is: /opt/tnf/rapid/etc/script/loader.ini
Settings that may need modifying:
logFile – define where the script will write its log. Default: /opt/tnf/rapid/stat/loader_script/loader.log
dataLoaderLog – log file for the Analytics Platform data loader. Default: /opt/tnf/rapid/stat/loader_script/dataloader.log
The mapping section between the aggregation levels and table names is used in case of ITE applications (not RawTDR). While the Voice/SMS aggregation levels contain a dedicated set of mappings (VS/VSS), Gn and LTE data is stored in the same set of aggregation level dirs by the DBLoader so the mappings should be configured in case of Gn/LTE ITE applications, as follows:
For Gn (default values):
UP_subscriber=gnuserplane_3_min_5
UP_cell=gnuserplane_2_min_5
UP_network=gnuserplane_1_min_5
UPS_subscriber=gnuserp_st_2_min_5
# UPS_cell=gnuserp_st_2_min_5
UPS_network=gnuserp_st_1_min_5
CP_subscriber=gncontrolp_3_min_5
CP_cell=gncontrolp_2_min_5
CP_network=gncontrolp_1_min_5
CPS_subscriber=gncontrolpst_3_min_5
CPS_cell=gncontrolpst_2_min_5
CPS_network=gncontrolpst_1_min_5

For LTE:
UP_subscriber=ltes1u_3_min_5   
UP_cell=ltes1u_2_min_5          
UP_network=ltes1u_1_min_5       
UPS_subscriber=ltes1u_st_2_min_5
# UPS_cell=ltes1u_st_2_min_5      
UPS_network=ltes1u_st_1_min_5   
CP_subscriber=ltes11_3_min_5    
CP_cell=ltes11_2_min_5          
CP_network=ltes11_1_min_5       
CPS_subscriber=ltes11_st_3_min_5
CPS_cell=ltes11_st_2_min_5      
CPS_network=ltes11_st_1_min_5

For all other applications (ITE Voice/SMS, RawTDR) the default mappings can be left as is.
Other settings in this file would usually not require changes.

Administration Guide
In the following instance represents the instance name as configured in probes.csv or in probes-tdr.csv.
Starting the applications
Before starting the applications check that they are not already running as detailed in section 5.3.
In order to start the applications run:
/opt/tnf/rapid/rapid.sh -i instance start
Stopping the applications
To stop the applications run:
/opt/tnf/rapid/rapid.sh -i instance stop
Checking the application status
To check what applications are running, use:
streamtool lsjobs -i instance
To check the internal state of running applications, use:
/opt/tnf/rapid/rapid.sh -i instance status
Cleaning temporary files
To clean up temporary files run:
/opt/tnf/rapid/rapid.sh -i instance clean
To clean up logs and checkpoints in the probe directories run:
/opt/tnf/rapid/rapid.sh -i instance cleanprobes
To clean all the contents from the loader directories run:
/opt/tnf/rapid/rapid.sh -i instance cleanload
Updating the provisioning
The ITE applications load the provisioning CSV files in cfg at start time; if these files are changed while the applications are running it's possible to force them to to reload the provisioning without the need for a restart. In order to do so, issue the following command:
/opt/tnf/rapid/rapid.sh -i instance reprovision
Consulting log files
Collection
The files {probe_dir}/{probe_name}/out/statistics/{date}_ite.{i}_Statistics.txt, where {probe_dir} and {probe_name} are the values configured in probes.csv (see section 4.5.2), ite.{i} is the job name as shown in section 5.3, while {date} is in the format YYYYMMDD, contain the collection statistics for all chains (4 by default). The content of the files are a list of labelled file attributes, as in the following LTE example (one line per one file):
{filename="/opt/tnf/file_loader/vedia/storage/Streams-ITE/P1/in/data/netro_20120227080601.log.gz",fileType="netro",filesize=17346,urgent=false,reprocess=false,filetime=1330358761,processingStartedAt="2015-08-26 07:57:06",processingStoppedAt="2015-08-26 07:57:06",duration=0.025828,startTimestamp=(1440601026,719628000,0),sentRecords=238,nRecordsDecodedASN1=0,nBytesDecodedASN1=0,nBytesReceivedASN1=0,nBytesDroppedASN1=0,recordStatsASN1={},nBytesReceivedBIN=0,nBytesDroppedBIN=0,recordCountsBIN={},byteCountsBIN={},recordStatsBIN={},nLinesDroppedCSV=0,recordStatsCSV={"m7013":{"errors":0,"records":0},"m7021":{"errors":0,"records":232},"m7022":{"errors":0,"records":6}},warnings=[],errors=[],duplicate=false,invalidFile=validFile,rejectedInvalids=0,tableStats={},tableFiles=[],chainSequenceNumber=311,groupId="00:311,01:0,02:0,03:0,04:0",chainId="02",logFileDate="20150826"}
The fields of interest are:
filename: identifies the file the attributes refer to;
sentRecords: contains the total number of TDRs in the file sent to the aggregation contexts;
recordStatsCSV: this attribute is further divided in specific counters per TDR type;
invalidFile: can be used to trace rejected files;
rejectedInvalids: can be used to trace rejected TDRs;
groupId: this string lists the aggregation contexts (comma separated) and specifies how many of the processed TDRs were sent to each (colon separated) – note that this number can be higher than sentRecords in case TDRs are duplicated for the aggregation;
chainId: identifies the chain that processed the file.
The rejected TDRs will have a more detailed explanation listed in a corresponding file in /opt/tnf/rapid/stat/{interface}/probes/{probe_name}/rejected/, together with the line number in the file where the TDR was located and a numeric error code.
Lookup
The statistics for lookup tables are printed to /opt/tnf/rapid/stat/{interface}/{date}_LookupManagerStatistics.txt, where {interface} is the interface type and {date} is in the format YYYYMMDD. Example content from the file follows:
{filename="",command="SUCCESS request:submit",commandStartedAt="Wed Aug 26 07:56:10 2015",commandProcessedAt="",hostStatistics=[],logFileDate="20150826"}
{filename="/opt/tnf/rapid/control/lte/cmd/init_all.cmd",command="init",commandStartedAt="Wed Aug 26 07:56:15 2015",commandProcessedAt="Wed Aug 26 07:56:51 2015",hostStatistics=["Streams segments:{[sgmnt:mcc_mnc- table:mcc_mnc- reserved:1048576- free:525184-free %:50- processed:1637][sgmnt:status- table:status- reserved:2097152- free:1139760-free %:54- processed:1020][sgmnt:bnumber- table:bnumber- reserved:31457280- free:21855792-free %:69- processed:89088][sgmnt:cells- table:cells- reserved:10485760- free:4247616-free %:40- processed:81077][sgmnt:imsi_msisdn- table:imsi_msisdn- reserved:629145600- free:629145600-free %:100- processed:0]}"],logFileDate="20150826"}
{filename="appl.ctl.cmd",command="SUCCESS request:restart,ite.0",commandStartedAt="Wed Aug 26 07:58:16 2015",commandProcessedAt="",hostStatistics=[],logFileDate="20150826"}
In particular, the middle log refers to the tables being loaded: the attribute hostStatistics provides the number of records loaded (processed) and the remaining memory (free %) for each table. Please ensure that there is remaining memory for all tables; the cache size can be configured at compile-time as specified in section 4.5.1.
Aggregation
The files /opt/tnf/rapid/stat/{interface}/probes/{probe_name}/context_statistics_{i}.txt, where {interface} is the interface type, {probe_name} is the value configured in probes.csv while {i} represents the context number, contain the aggregation statistics. An example line from the file follows:
{message="file=netro_20120227080601.log.gz, chain=03, composite=Context00.ContextRecordRegionStream.CustomContext.CustomContext.UP_SubscriberAggregateIntermediate, toolate=208, totalCount={\"20120227075000\":3,\"20120227075500\":2,\"20120227080000\":200,\"20120227072000\":1,\"20120227080500\":24,\"20120227073000\":2}"}
In particular:
totalCount: can be used to trace the number of TDRs (colon separated) processed in each period (comma separated);
toolate: can be used to trace the amount of TDRs ignored in the period.
Database
The files in /opt/tnf/rapid/stat/loader_script/, contain eventual errors relating to the DB loader.
The variables logFile and dataLoaderLog in the file /opt/tnf/rapid/etc/script/loader.ini contain the path of the log files for the loader scripts (see section 4.5.9).
Performance
The file /opt/tnf/rapid/stat/{interface}/vmstat.stat, where {interface} is the interface type, contains the memory and CPU statistics measured during the applications' current life time, as measured by the vmstat command. The monitoring daemon's pid is stored in /opt/tnf/rapid/control/{interface}/vmstat.pid, and can be used to monitor its status. The daemon is killed during the stop command and both files are reset upon a full restart.
Recovery mechanisms
All applications are monitored every minute during the application's life time; if a Streams job fails or reports a status different from “run” (fully operational), the job is restarted. The monitoring daemon is started automatically by the “start” command, and is killed by the “stop” command; the process has the same name as the management script, rapid.sh, and its process ID is stored in /opt/tnf/rapid/control/{interface}/health.pid, which can be used to monitor its status, e.g. with
ps -fp `cat /opt/tnf/rapid/control/{interface}/health.pid`
If no process is listed in the command's output, the monitoring process might have been killed or terminated unexpectedly: the process might have to be killed manually to examine Streams logs that would be removed during a restart. In this case there is no embedded mechanism to restart it, and it's recommended to manually restart the applications to restore the functionality.
The file /opt/tnf/rapid/stat/{interface}/health.log contains the health monitor logs and can be used to detect instabilities in the system. Streams-related instabilities might have to be analysed using the individual job and PE logs (see related Streams documentation).
In the following example content:
08/26/15 07:56:37 === Started monitoring instance rapid_lte
08/26/15 07:57:48 DBLoader (job #16): status Running/yes/run
08/26/15 07:57:58 LookupManager (job #14): status Running/yes/run
08/26/15 07:57:58 ite.0 - P1 (job #-): status -/terminated
streamtool submitjob /opt/tnf/rapid/bin/ITE_LTE.sab -C data-directory=/opt/tnf/file_loader/vedia/storage/Streams-ITE/P1 -P probeId=P1 -P ite.jobName=ite.0  -P netroFileMask='netro_([0-9]{14})\\.log\\.gz'\
	 -P pagebasedFileMask='pbqoe_([0-9]{14})\\.log\\.gz'\
	 -P appSplitterEnabled=false\
	 -P numContextsUP=3\
	 -P numContextsUPS=1\
	 -P numContextsCPCPS=1\
	 -P loadUnknownSubscribers=false\
	 -P loadUnknownCells=false\
	 -P filterCPsuccessCodes=true\
	 -P CpsuccessCodes=16,17,18,19\
	 -P ignorePageBased=false\
	 -P roamersFilter=subscriber\
	 -P checkProbeCells=true -P global.applicationControlDirectory=/opt/tnf/rapid/control/lte
CDISC0079I The following number of applications were submitted to the rapid_lte instance: 1. The instance is in the rapid domain.
CDISC0080I The 17 job was submitted for the following application: ITE_LTE.sab. The job was submitted to the rapid_lte instance in the rapid domain.
CDISC0020I Submitted job IDs: 17
08/26/15 07:59:24 DBLoader (job #16): status Running/yes/run
08/26/15 07:59:32 LookupManager (job #14): status Running/yes/run
08/26/15 07:59:40 ite.0 - P1 (job #17): status Running/yes/run
We can see that the applications were started at 07:56, but at 07:57 the ITE LTE application had been detected as terminated; it was immediately restarted (the individual parameters depend on the content of the ITE configuration file), and at 07:59 all applications are healthy.
In case of a failure of an ITE job, as in the above example, the last 5 processed files are also resubmitted for collection; the ITE application will automatically identify TDRs that had been already aggregated and possibly written and will not output them again to prevent duplicates.
In case of a failure of the LookupManager a full restart is automatically performed.