# salt.modules.highstate_doc

This module renders highstate configuration into a more human readable format.

How it works:

highstate or lowstate data is parsed with a processor this defaults to highstate_doc.processor_markdown. The processed data is passed to a jinja template that builds up the document content.

configuration: Pillar

    # the following defaults can be overridden
    highstate_doc.config:

        # list of regex of state names to ignore in `highstate_doc.process_lowstates`
        filter_id_regex:
            - '.*!doc_skip$'

        # list of regex of state functions to ignore in `highstate_doc.process_lowstates`
        filter_state_function_regex:
            - 'file.accumulated'

        # dict of regex to replace text after `highstate_doc.render`. (remove passwords)
        text_replace_regex:
            'password:.*^': '[PASSWORD]'

        # limit size of files that can be included in doc (10000 bytes)
        max_render_file_size: 10000

        # advanced option to set a custom lowstate processor
        processor: highstate_doc.processor_markdown

State example

    {{sls}} note:
        highstate_doc.note:
            - name: example
            - order: 0
            - contents: |
                example `highstate_doc.note`
                ------------------
                This state does not do anything to the system! It is only used by a `processor`
                you can use `requisites` and `order` to move your docs around the rendered file.

    {{sls}} a file we don't want in the doc !doc_skip:
        file.managed:
            - name: /root/passwords
            - contents: 'password: sadefgq34y45h56q'
            # also could use `highstate_doc.config: text_replace_regex` to replace
            # password string. `password:.*^': '[PASSWORD]`

To create the help document build a State that uses highstate_doc.render. For performance it's advised to not included this state in your top.sls file.

    # example `salt://makereadme.sls`
    make helpfile:
        file.managed:
            - name: /root/README.md
            - contents: {{salt.highstate_doc.render()|json}}
            - show_diff: {{opts['test']}}
            - mode: '0640'
            - order: last

Run our makereadme.sls state to create /root/README.md.

    # first ensure `highstate` return without errors or changes
    salt-call state.highstate
    salt-call state.apply makereadme
    # or if you don't want the extra `make helpfile` state
    salt-call --out=newline_values_only salt.highstate_doc.render > /root/README.md ; chmod 0600 /root/README.md

## Creating a document collection

From the master we can run the following script to creates a collection of all your minion documents.

    salt '*' state.apply makereadme

    #!/bin/python
    import os
    import salt.client
    s = salt.client.LocalClient()
    # NOTE: because of issues with `cp.push` use `highstate_doc.read_file`
    o = s.cmd('*', 'highstate_doc.read_file', ['/root/README.md'])
    for m in o:
        d = o.get(m)
        if d and not d.endswith('is not available.'):
            # mkdir m
            #directory = os.path.dirname(file_path)
            if not os.path.exists(m):
                os.makedirs(m)
            with open(m + '/README.md','wb') as f:
                f.write(d)
            print('ADDED: ' + m + '/README.md')

Once the master has a collection of all the README files. You can use pandoc to create HTML versions of the markdown.

    # process all the readme.md files to readme.html
    if which pandoc; then echo "Found pandoc"; else echo "** Missing pandoc"; exit 1; fi
    if which gs; then echo "Found gs"; else echo "** Missing gs(ghostscript)"; exit 1; fi
    readme_files=$(find $dest -type f -path "*/README.md" -print)
    for f in $readme_files ; do
        ff=${f#$dest/}
        minion=${ff%%/*}
        echo "process: $dest/${minion}/$(basename $f)"
        cat $dest/${minion}/$(basename $f) |             pandoc --standalone --from markdown_github --to html             --include-in-header $dest/style.html             > $dest/${minion}/$(basename $f).html
    done

It is also nice to put the help files in source control.

> # git init git add -A git commit -am 'updated docs' git push -f

## Other hints

If you wish to customize the document format:

    # you could also create a new `processor` for perhaps reStructuredText
    # highstate_doc.config:
    #     processor: doc_custom.processor_rst

    # example `salt://makereadme.jinja`
    """
    {{opts['id']}}
    ==========================================

    {# lowstates is set from highstate_doc.render() #}
    {# if lowstates is missing use salt.highstate_doc.process_lowstates() #}
    {% for s in lowstates %}
    {{s.id}}
    -----------------------------------------------------------------
    {{s.function}}

    {{s.markdown.requisite}}
    {{s.markdown.details}}

    {%- endfor %}
    """

    # example `salt://makereadme.sls`
    {% import_text "makereadme.jinja" as makereadme %}
    {{sls}} or:
        file.managed:
            - name: /root/README_other.md
            - contents: {{salt.highstate_doc.render(jinja_template_text=makereadme)|json}}
            - mode: '0640'

Some replace_text_regex values that might be helpful:

    CERTS
    -----

    ``'-----BEGIN RSA PRIVATE KEY-----[\r\n\t\f\S]{0,2200}': 'XXXXXXX'``
    ``'-----BEGIN CERTIFICATE-----[\r\n\t\f\S]{0,2200}': 'XXXXXXX'``
    ``'-----BEGIN DH PARAMETERS-----[\r\n\t\f\S]{0,2200}': 'XXXXXXX'``
    ``'-----BEGIN PRIVATE KEY-----[\r\n\t\f\S]{0,2200}': 'XXXXXXX'``
    ``'-----BEGIN OPENSSH PRIVATE KEY-----[\r\n\t\f\S]{0,2200}': 'XXXXXXX'``
    ``'ssh-rsa .* ': 'ssh-rsa XXXXXXX '``
    ``'ssh-dss .* ': 'ssh-dss XXXXXXX '``

    DB
    --

    ``'DB_PASS.*': 'DB_PASS = XXXXXXX'``
    ``'5432:*:*:.*': '5432:*:XXXXXXX'``
    ``"'PASSWORD': .*": "'PASSWORD': 'XXXXXXX',"``
    ``" PASSWORD '.*'": " PASSWORD 'XXXXXXX'"``
    ``'PGPASSWORD=.* ': 'PGPASSWORD=XXXXXXX'``
    ``"_replication password '.*'":  "_replication password 'XXXXXXX'"``

    OTHER
    -----

    ``'EMAIL_HOST_PASSWORD =.*': 'EMAIL_HOST_PASSWORD =XXXXXXX'``
    ``"net ads join -U '.*@MFCFADS.MATH.EXAMPLE.CA.* ": "net ads join -U '.*@MFCFADS.MATH.EXAMPLE.CA%XXXXXXX "``
    ``"net ads join -U '.*@NEXUS.EXAMPLE.CA.* ": "net ads join -U '.*@NEXUS.EXAMPLE.CA%XXXXXXX "``
    ``'install-uptrack .* --autoinstall': 'install-uptrack XXXXXXX --autoinstall'``
    ``'accesskey = .*': 'accesskey = XXXXXXX'``
    ``'auth_pass .*': 'auth_pass XXXXXXX'``
    ``'PSK "0x.*': 'PSK "0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'``
    ``'SECRET_KEY.*': 'SECRET_KEY = XXXXXXX'``
    ``"password=.*": "password=XXXXXXX"``
    ``'<password>.*</password>': '<password>XXXXXXX</password>'``
    ``'<salt>.*</salt>': '<salt>XXXXXXX</salt>'``
    ``'application.secret = ".*"': 'application.secret = "XXXXXXX"'``
    ``'url = "postgres://.*"': 'url = "postgres://XXXXXXX"'``
    ``'PASS_.*_PASS': 'PASS_XXXXXXX_PASS'``

    HTACCESS
    --------

    ``':{PLAIN}.*': ':{PLAIN}XXXXXXX'``

salt.modules.highstate_doc.markdown_basic_jinja_template(**kwargs)

  Return text for a simple markdown jinja template

  This function can be used from the highstate_doc.render modules jinja_template_function option.

salt.modules.highstate_doc.markdown_default_jinja_template(**kwargs)

  Return text for a markdown jinja template that included a header

  This function can be used from the highstate_doc.render modules jinja_template_function option.

salt.modules.highstate_doc.markdown_full_jinja_template(**kwargs)

  Return text for an advanced markdown jinja template

  This function can be used from the highstate_doc.render modules jinja_template_function option.

salt.modules.highstate_doc.process_lowstates(**kwargs)

  return processed lowstate data that was not blacklisted

  render_module_function is used to provide your own. defaults to from_lowstate

salt.modules.highstate_doc.processor_markdown(lowstate_item, config, **kwargs)

  Takes low state data and returns a dict of processed data that is by default used in a jinja template when rendering a markdown highstate_doc.

  This lowstate_item_markdown given a lowstate item, returns a dict like:

      vars:       # the raw lowstate_item that was processed
      id:         # the 'id' of the state.
      id_full:    # combo of the state type and id "state: id"
      state:      # name of the salt state module
      function:   # name of the state function
      name:       # value of 'name:' passed to the salt state module
      state_function:    # the state name and function name
      markdown:          # text data to describe a state
          requisites:    # requisite like [watch_in, require_in]
          details:       # state name, parameters and other details like file contents

salt.modules.highstate_doc.read_file(name)

  output the contents of a file:

  this is a workaround if the cp.push module does not work. https://github.com/saltstack/salt/issues/37133

  help the master output the contents of a document that might be saved on the minions filesystem.

      #!/bin/python
      import os
      import salt.client
      s = salt.client.LocalClient()
      o = s.cmd('*', 'highstate_doc.read_file', ['/root/README.md'])
      for m in o:
          d = o.get(m)
          if d and not d.endswith('is not available.'):
              # mkdir m
              #directory = os.path.dirname(file_path)
              if not os.path.exists(m):
                  os.makedirs(m)
              with open(m + '/README.md','wb') as fin:
                  fin.write(d)
              print('ADDED: ' + m + '/README.md')

salt.modules.highstate_doc.render(jinja_template_text=None, jinja_template_function='highstate_doc.markdown_default_jinja_template', **kwargs)

  Render highstate to a text format (default Markdown)

  if jinja_template_text is not set, jinja_template_function is used.

  jinja_template_text: jinja text that the render uses to create the document. jinja_template_function: a salt module call that returns template text.

  Options:

    highstate_doc.markdown_basic_jinja_template highstate_doc.markdown_default_jinja_template highstate_doc.markdown_full_jinja_template