RWFLOWAPPEND(8)

NAME
rwflowappend - Append incremental SiLK Flow files to hourly files

SYNOPSIS
  rwflowappend --incoming-directory=DIR_PATH --root-directory=DIR_PATH
        --error-directory=DIR_PATH [--archive-directory=DIR_PATH]
        [--post-command=COMMAND] [--flat-archive]
        [--hour-file-command=COMMAND] [--reject-hours-past=NUM]
        [--reject-hours-future=NUM] [--polling-interval=NUM]
        [--byte-order=ENDIAN] [--pad-header]
        [--compression-method=COMP_METHOD]
        [--site-config-file=FILENAME]
        { --log-destination=DESTINATION
          | --log-directory=DIR_PATH [--log-basename=BASENAME]
          | --log-pathname=FILE_PATH }
        [--log-level=LEVEL] [--log-sysfacility=NUMBER]
        [--pidfile=FILE_PATH] [--no-daemon]

  rwflowappend --help

  rwflowappend --version

DESCRIPTION
rwflowappend is a daemon that watches a directory for files that contain small numbers of SiLK Flow records---these files are called incremental files---as generated by rwflowpack(8) when it is run with --output-mode=sending. The rwflowappend will append these SiLK Flow records to the hourly files stored in a directory tree whose root is specified by the --root-directory switch.

Having the rwflowappend separate from rwflowpack is used when you wish to copy the packed SiLK Flow records from the machine doing the packing to multiple machines for use by analysts. Almost any network file transport protocol may be used to move the files from the packing machine to the destination machine where rwflowappend is running, though we have written the rwsender(8) and rwreceiver(8) to perform this task.

OPTIONS
Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

  General Configuration
The following switches are required:

--incoming-directory=DIR_PATH
    Watch this directory for new incremental files to append to the hourly files. rwflowappend ignores any files in this directory that are empty or whose names begin with a dot ("."). In addition, new files will only be considered when their size is constant for one polling-interval after they are first noticed.

--root-directory=DIR_PATH
    Append to existing hourly files and create new hourly files in the directory tree rooted at this location. The directory tree has the same subdirectory structure as that created by rwflowpack.

--error-directory=DIR_PATH
    Store in this directory incremental files that were NOT successfully appended to an hourly file.

The following switches are optional:

--archive-directory=DIR_PATH
    Move each incremental file to DIR_PATH or a subdirectory of it after rwflowappend has successfully appended the incremental file to an hourly file. If this switch is not provided, the incremental files are deleted once they are successfully appended to an hourly file. When the --flat-archive switch is also provided, incremental files are moved into the top of DIR_PATH; when --flat-archive is not given, each incremental file is moved to a subdirectory of DIR_PATH that mirrors the path of the hourly file to which the incremental file was appended. Removing files from the archive-directory is not the job of rwflowappend; the system administrator should implement a separate process to clean this directory. This switch is required when the --post-command switch is present.

--post-command=COMMAND
    Run COMMAND on each incremental file after rwflowappend has successfully appended it to an hourly file and moved it into the archive-directory. Each occurrence of the string %s in COMMAND will be replaced with the full path to the incremental file in the archive-directory. When using this feature, the --archive-directory must be specified.

--flat-archive
    When archiving incremental files via --archive-directory, move the files into the top of the archive-directory, not into subdirectories of it. This switch has no effect if --archive-directory is not also specified. This switch can be used to allow another process to watch for new files appearing in the archive-directory.

--hour-file-command=COMMAND
    Run COMMAND upon creation of a new hourly file. The string %s in COMMAND will be replaced with the full path to the hourly file.

--reject-hours-past=NUM
    Reject incremental files containing records whose starting hour occurs more than this number of hours in the past relative to the current hour. Incremental files that violate this value are moved into the error directory. Times are compared using the starting hour of the flow record and the current hour. For example, flow records that start at 18:02:56 and 18:58:04 are considered 1 hour in the past whether the current time is 19:01:47 or 19:59:33. When performing live data collection, it is not uncommon to get flows one to two hours in the past due to the flow generator's active timeout (often 30 minutes) and the time to transfer the flow records through the collection system. The default is to accept all incremental files.

--reject-hours-future=NUM
    Similar to --reject-hours-past, but reject incremental files containing records whose starting hour occurs more than this number of hours in the future relative to the current hour. Future dated flow records are rare, but can occur due to time drift at the sensor. The default is to accept all incremental files.

--polling-interval=NUM
    Configure rwflowappend to check the incoming directory for new files every NUM seconds. The default polling interval is 15 seconds.

--pad-header
    This switch is ignored. It is for backward compatibility only, since the header is always padded to a multiple of the record size.

--byte-order=ENDIAN
    Set the byte order for newly created SiLK Flow files. When appending records to an existing file, the byte order of the file is maintained. The argument is one of the following:

    "as-is"
        Maintain the byte order of the incremental files (i.e., the byte order specified to rwflowpack). This is the default.

    "native"
        Use the byte order of the machine where rwflowappend is running.

    "big"
        Use network byte order (big endian) for the flow files.

    "little"
        Write the flow files in little endian format.

--compression-method=COMP_METHOD
    Set the compression method for newly created hourly files to COMP_METHOD. When appending records to an existing file, the compression method of the file is maintained. If this switch is not provided, the compression method used by the incremental file (i.e., the compression method specified to rwflowpack) will be used for the new hourly file.

--site-config-file=FILENAME
    Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, the location specified by the SILK_CONFIG_FILE environment variable is used if that variable is not empty. The value of SILK_CONFIG_FILE should include the name of the file. Otherwise, the application looks for a file named silk.conf in the following directories: the directory specified by the --root-directory switch; the directories $SILK_PATH/share/silk/ and $SILK_PATH/share/; and the share/silk/ and share/ directories parallel to the application's directory.

  Logging and Daemon Configuration
One of the following mutually-exclusive switches is required:

--log-destination=DESTINATION
    Specify the destination where logging messages are written. When DESTINATION begins with a slash "/", it is treated as a file system path and all log messages are written to that file; there is no log rotation. When DESTINATION does not begin with "/", it must be one of the following strings:

    "none"
        Messages are not written anywhere.

    "stdout"
        Messages are written to the standard output.

    "stderr"
        Messages are written to the standard error.

    "syslog"
        Messages are written using the syslog(3) facility.

    "both"
        Messages are written to the syslog facility and to the standard error (this option is not available on all platforms).

--log-directory=DIR_PATH
    Use DIR_PATH as the directory where the log files are written. DIR_PATH must be a complete directory path. The log files have the form

      DIR_PATH/LOG_BASENAME-YYYYMMDD.log

    where YYYYMMDD is the current date and LOG_BASENAME is the application name or the value passed to the --log-basename switch when provided. The log files will be rotated: at midnight local time a new log will be opened and the previous day's log file will be compressed using gzip(1). (Old log files are not removed by rwflowappend; the administrator should use another tool to remove them.) When this switch is provided, a process-ID file (PID) will also be written in this directory unless the --pidfile switch is provided.

--log-pathname=FILE_PATH
    Use FILE_PATH as the complete path to the log file. The log file will not be rotated.

The following set of switches is optional:

--log-level=LEVEL
    Set the severity of messages that will be logged. The levels from most severe to least are: "emerg", "alert", "crit", "err", "warning", "notice", "info", "debug". The default is "info".

--log-sysfacility=NUMBER
    Set the facility that syslog(3) uses for logging messages. This switch takes a number as an argument. The default is a value that corresponds to "LOG_USER" on the system where rwflowappend is running. This switch produces an error unless --log-destination=syslog is specified.

--log-basename=LOG_BASENAME
    Use LOG_BASENAME in place of the application name for the files in the log directory. See the description of the --log-directory switch.

--pidfile=FILE_PATH
    Set the complete path to the file in which rwflowappend writes its process ID (PID) when it is running as a daemon. No PID file is written when --no-daemon is given. When this switch is not present, no PID file is written unless the --log-directory switch is specified, in which case the PID is written to LOGPATH/rwflowappend.pid.

--no-daemon
    Force rwflowappend to stay in the foreground---it does not become a daemon. Useful for debugging.

--help
    Print the available options and exit.

--version
    Print the version number and information about how SiLK was configured, then exit the application.

ENVIRONMENT

SILK_CONFIG_FILE
    This environment variable is used as the value for the --site-config-file when that switch is not provided.

SILK_PATH
    This environment variable gives the root of the install tree. As part of its search for the SiLK site configuration file, rwflowappend checks for a file named silk.conf in the directories $SILK_PATH/share/silk and $SILK_PATH/share.

SEE ALSO
The SiLK Installation Handbook, rwflowpack(8), rwreceiver(8), rwsender(8), syslog(3)