# salt.modules.saltcheck

A module for testing the logic of states and highstates on salt minions

codeauthor:

  William Cannon <william.cannon@gmail.com>

maturity:

  new

Saltcheck provides unittest like functionality requiring only the knowledge of salt module execution and yaml. Saltcheck uses salt modules to return data, then runs an assertion against that return. This allows for testing with all the features included in salt modules.

In order to run state and highstate saltcheck tests, a sub-folder in the state directory must be created and named `saltcheck-tests`. Tests for a state should be created in files ending in `*.tst` and placed in the `saltcheck-tests` folder. `tst` files are run through the salt rendering system, enabling tests to be written in yaml (or renderer of choice), and include jinja, as well as the usual grain and pillar information. Like states, multiple tests can be specified in a `tst` file. Multiple `tst` files can be created in the `saltcheck-tests` folder, and should be named the same as the associated state. The `id` of a test works in the same manner as in salt state files and should be unique and descriptive.

New in version 3000: The `saltcheck-tests` folder can be customized using the `saltcheck_test_location` minion configuration setting. This setting is a relative path from the formula's `salt://` path to the test files.

## Usage

Example Default file system layout:

    /srv/salt/apache/
        init.sls
        config.sls
        saltcheck-tests/
            init.tst
            config.tst
            deployment_validation.tst

Alternative example file system layout with custom saltcheck_test_location:

### Minion configuration:

    saltcheck_test_location: tests/integration/saltcheck

### Filesystem layout:

    /srv/salt/apache/
        init.sls
        config.sls
        tests/integration/saltcheck/
            init.tst
            config.tst
            deployment_validation.tst

Tests can be run for each state by name, for all `apache/saltcheck/*.tst` files, or for all states assigned to the minion in top.sls. Tests may also be created with no associated state. These tests will be run through the use of `saltcheck.run_state_tests`, but will not be automatically run by `saltcheck.run_highstate_tests`.

    salt '*' saltcheck.run_state_tests apache,apache.config
    salt '*' saltcheck.run_state_tests apache check_all=True
    salt '*' saltcheck.run_highstate_tests
    salt '*' saltcheck.run_state_tests apache.deployment_validation

## Saltcheck Keywords

module_and_function:

  (str) This is the salt module which will be run locally, the same as `salt-call --local <module>`. The `saltcheck.state_apply` module name is special as it bypasses the local option in order to resolve state names when run in a master/minion environment.

args:

  (list) Optional arguments passed to the salt module

kwargs:

  (dict) Optional keyword arguments to be passed to the salt module

assertion:

  (str) One of the supported assertions and required except for `saltcheck.state_apply` Tests which fail the assertion and expected_return, cause saltcheck to exit which a non-zero exit code.

expected_return:

  (str) Required except by `assertEmpty`, `assertNotEmpty`, `assertTrue`, `assertFalse`. The return of module_and_function is compared to this value in the assertion.

assertion_section:

  (str) Optional keyword used to parse the module_and_function return. If a salt module returns a dictionary as a result, the `assertion_section` value is used to lookup a specific value in that return for the assertion comparison.

assertion_section_delimiter:

  (str) Optional delimiter to use when splitting a nested structure. Defaults to ':'

print_result:

  (bool) Optional keyword to show results in the `assertEqual`, `assertNotEqual`, `assertIn`, and `assertNotIn` output. Defaults to True.

output_details:

  (bool) Optional keyword to display `module_and_function`, `args`, `assertion_section`, and assertion results text in the output. If print_result is False, assertion results will be hidden. This is a per test setting, but can be set globally for all tests by adding `saltcheck_output_details: True` in the minion configuration file. Defaults to False

pillar_data:

  (dict) Optional keyword for passing in pillar data. Intended for use in potential test setup or teardown with the `saltcheck.state_apply` function.

skip:

  (bool) Optional keyword to skip running the individual test

New in version 3000: Multiple assertions can be run against the output of a single `module_and_function` call. The `assertion`, `expected_return`, `assertion_section`, and `assertion_section_delimiter` keys can be placed in a list under an `assertions` key. See the multiple assertions example below.

## Sample Cases/Examples

### Basic Example

    echo_test_hello:
      module_and_function: test.echo
      args:
        - "hello"
      kwargs:
      assertion: assertEqual
      expected_return:  'hello'

### Example with jinja

    {% for package in ["apache2", "openssh"] %}
    {# or another example #}
    {# for package in salt['pillar.get']("packages") #}
    test_{{ package }}_latest:
      module_and_function: pkg.upgrade_available
      args:
        - {{ package }}
      assertion: assertFalse
    {% endfor %}

### Example with setup state including pillar

    setup_test_environment:
      module_and_function: saltcheck.state_apply
      args:
        - common
      pillar_data:
        data: value

    verify_vim:
      module_and_function: pkg.version
      args:
        - vim
      assertion: assertNotEmpty

### Example with jinja

    {% for package in ["apache2", "openssh"] %}
    {# or another example #}
    {# for package in salt['pillar.get']("packages") #}
    test_{{ package }}_latest:
      module_and_function: pkg.upgrade_available
      args:
        - {{ package }}
      assertion: assertFalse
    {% endfor %}

### Example with setup state including pillar

    setup_test_environment:
      module_and_function: saltcheck.state_apply
      args:
        - common
      pillar-data:
        data: value

    verify_vim:
      module_and_function: pkg.version
      args:
        - vim
      assertion: assertNotEmpty

### Example with skip

    package_latest:
      module_and_function: pkg.upgrade_available
      args:
        - apache2
      assertion: assertFalse
      skip: True

### Example with assertion_section

    validate_shell:
      module_and_function: user.info
      args:
        - root
      assertion: assertEqual
      expected_return: /bin/bash
      assertion_section: shell

### Example with a nested assertion_section

    validate_smb_signing:
      module_and_function: lgpo.get
      args:
        - 'Machine'
      kwargs:
        return_full_policy_names: True
      assertion: assertEqual
      expected_return: Enabled
      assertion_section: 'Computer Configuration|Microsoft network client: Digitally sign communications (always)'
      assertion_section_delimiter: '|'

### Example suppressing print results

    validate_env_nameNode:
      module_and_function: hadoop.dfs
      args:
        - text
        - /oozie/common/env.properties
      expected_return: nameNode = hdfs://nameservice2
      assertion: assertNotIn
      print_result: False

### Example with multiple assertions and output_details

    multiple_validations:
      module_and_function: network.netstat
      assertions:
        - assertion: assertEqual
          assertion_section: "0:program"
          expected_return: "systemd-resolve"
        - assertion: assertEqual
          assertion_section: "0:proto"
          expected_return: "udp"
      output_details: True

## Supported assertions

- assertEqual

- assertNotEqual

- assertTrue

- assertFalse

- assertIn

- assertNotIn

- assertGreater

- assertGreaterEqual

- assertLess

- assertLessEqual

- assertEmpty

- assertNotEmpty

Warning

The saltcheck.state_apply function is an alias for `state.apply`. If using the ACL system `saltcheck.*` might provide more capability than intended if only `saltcheck.run_state_tests` and `saltcheck.run_highstate_tests` are needed.

class salt.modules.saltcheck.SaltCheck(saltenv='base')

  This class validates and runs the saltchecks

  run_test(test_dict)

    Run a single saltcheck test

class salt.modules.saltcheck.StateTestLoader(saltenv='base')

  Class loads in test files for a state e.g. state_dir/saltcheck-tests/[1.tst, 2.tst, 3.tst]

  add_test_files_for_sls(sls_name, check_all=False)

    Detects states used, caches needed files, and adds to test list

  load_test_suite()

    Load tests either from one file, or a set of files

salt.modules.saltcheck.parallel_scheck(data)

  triggers salt-call in parallel

salt.modules.saltcheck.report_highstate_tests(saltenv=None)

  Report on tests for states assigned to the minion through highstate. Quits with the exit code for the number of missing tests.

  CLI Example:

      salt '*' saltcheck.report_highstate_tests

  New in version 3000.

salt.modules.saltcheck.run_highstate_tests(saltenv=None, only_fails=False)

  Execute all tests for states assigned to the minion through highstate and return results

  Parameters:

    - saltenv (str) -- optional saltenv. Defaults to base

    - only_fails (bool) -- boolean to only print failure results

  CLI Example:

      salt '*' saltcheck.run_highstate_tests

salt.modules.saltcheck.run_state_tests(state, saltenv=None, check_all=False, only_fails=False)

  Execute tests for a salt state and return results Nested states will also be tested

  Parameters:

    - state (str) -- state name for which to run associated .tst test files

    - saltenv (str) -- optional saltenv. Defaults to base

    - check_all (bool) -- boolean to run all tests in state/saltcheck-tests directory

    - only_fails (bool) -- boolean to only print failure results

  CLI Example:

      salt '*' saltcheck.run_state_tests postfix,common

  Tests will be run in parallel by adding "saltcheck_parallel: True" in minion config. When enabled, saltcheck will use up to the number of cores detected. This can be limited by setting the "saltcheck_processes" value to an integer to set the maximum number of parallel processes.

salt.modules.saltcheck.run_state_tests_ssh(state, saltenv=None, check_all=False, only_fails=False)

  This function is an alias of `run_state_tests`.

  > Execute tests for a salt state and return results Nested states will also be tested

  > param str state:

  >   state name for which to run associated .tst test files

  > param str saltenv:

  >   optional saltenv. Defaults to base

  > param bool check_all:

  >   boolean to run all tests in state/saltcheck-tests directory

  > param bool only_fails:

  >   boolean to only print failure results

  > CLI Example:

  >     salt '*' saltcheck.run_state_tests postfix,common

  > Tests will be run in parallel by adding "saltcheck_parallel: True" in minion config. When enabled, saltcheck will use up to the number of cores detected. This can be limited by setting the "saltcheck_processes" value to an integer to set the maximum number of parallel processes.

salt.modules.saltcheck.run_test(**kwargs)

  Execute one saltcheck test and return result

  Parameters:

    test (keyword arg) --

  CLI Example:

      salt '*' saltcheck.run_test
          test='{"module_and_function": "test.echo",
                 "assertion": "assertEqual",
                 "expected_return": "This works!",
                 "args":["This works!"] }'

salt.modules.saltcheck.state_apply(state_name, **kwargs)

  Runs `state.apply` with given options to set up test data. Intended to be used for optional test setup or teardown

  Reference the `state.apply` module documentation for arguments and usage options

  CLI Example:

      salt '*' saltcheck.state_apply postfix