salt-ssh jinja TemplateNotFound

[chekolyn]

I believe I found an issue with salt-ssh when using jinja map template files.

Please take a look at https://github.com/chekolyn/cherokee-formula/blob/master/cherokee/templates/cherokee.conf.jinja

This formula renders fine when using the master node configuration but when I try to deploy it using salt-ssh I get the following error:

file_|-/etc/cherokee/cherokee.conf_|-/etc/cherokee/cherokee.conf_|-managed:
    ----------
    __run_num__:
        9
    changes:
        ----------
    comment:
        Traceback (most recent call last):
          File "/tmp/.salt/salt/utils/templates.py", line 74, in render_tmpl
            output = render_str(tmplstr, context, tmplpath)
          File "/tmp/.salt/salt/utils/templates.py", line 166, in render_jinja_tmpl
            output = jinja_env.from_string(tmplstr).render(**unicode_context)
          File "/tmp/.salt/jinja2/environment.py", line 969, in render
            return self.environment.handle_exception(exc_info, True)
          File "/tmp/.salt/jinja2/environment.py", line 742, in handle_exception
            reraise(exc_type, exc_value, tb)
          File "<template>", line 1, in top-level template code
          File "/tmp/.salt/salt/utils/jinja.py", line 103, in get_source
            raise TemplateNotFound(template)
        TemplateNotFound: cherokee/map.jinja

    name:
        /etc/cherokee/cherokee.conf
    result:
        False

[thatch45]

Ahh, seems I overlooked this case, all of the needed files are sent over to the minion in the state package, but unfortunately the jinja import references are not being included, this needs to be added.

[sandbender]

I came across this today while trying to prep a pull request for some salt-ssh fixes... workaround by hard-coding, but @thatch45 : if you have quick pointers on how/where to add support for this I'm willing to add it in, let me know...

[basepi]

I guess the question is whether the file is just not making it down to the minion, or if it's just not being put in a place such that the template can import it. It's going to be a change probably in salt/client/ssh/state.py or salt/client/ssh/wrapper/state.py, I think.

[sandbender]

Not making it at all... templates referenced from sls'es are included in the package sent to the minion, it's templates referenced from other templates that don't show up. Ie: if I have a template-based file being managed, and the template in question does a jinja-include of another template... the other template which is included via jinja is what doesn't show up. So... the 'root' template which would be referenced in the sls via source: salt://path/to/template.jinja is getting picked up via the file_refs stuff that happens... it's a template, referenced via a jinja include, etc... from within template.jinja in this example that's not being seen because no de-referencing of stuff in the template files themselves is happening.

[basepi]

Oh right, that's going to be quite a bit harder, because now we have to not only inspect the state files, but we're going to have to inspect the actual templates themselves.....hrm, I'm not sure how to go about this one.

[sandbender]

@basepi @thatch45

I have a fix for this, although not sure if it's acceptable or not upstream, hopefully this works though:

I've added support for an "extra_filerefs" keyword CLI argument to salt-ssh, ie:

salt-ssh ... state.sls your_module extra_filerefs=salt://path/to/included/file

extra_filerefs accepts a comma-separated list of salt-protocol paths to files and/or directories which should be included in the salt data package (ie: salt_state.tgz) shipped to a minion during a salt-ssh run.

There's precedent for this in a way, via the extra modules argument already supported by "salt-run thin.generate", ie: you can pass a comma separated list of extra modules to package up in the thintar.

There were a few other approaches I spent a LOT of time playing with/testing/etc, but both proved inadequate:

1. render all template content (ie: for file.managed, etc) on the master and replace 'source' and 'template' with a 'content' string in the relevant states in the lowstate data generated for the minion/client. Problem: templates potentially use grains dynamically, including to build dynamic includes, so this would require a non-trivial mod to get the target's grain data on the master first, then pass that down and into the template rendering. Possible, but seems messy and error-prone.

2. parse templates on master during lowstate_filerefs() call, in the context of the appropriate template engine(s), and locate include statements, etc, to convert to salt:// paths to be consumed by the caller of lowstate_filerefs(). Problem(s): similar to 1) above, this is difficult to do properly and statically, since there could be dynamic elements which contribute to include paths in the template, etc. Also, due to the location of the code that needs to ultimately receive the additional filerefs, the changes will be non-trivial and also error-prone.

"Error prone" above means because of the location of the theoretical mods, it may be rather difficult to ensure that the mods wouldn't have subtle impacts/generate errors elsewhere, etc, although that perception may just be my own lack of understanding of the codebase as a whole?

Anyways... the fix to support the extra_filerefs is simple, generically applicable, and ready to go, so assuming there's no issue with the mechanic itself, let me know and I can issue a pull request for it.

[basepi]

This sounds like the best immediate workaround to me, even though it's not as automatic -- at least it works.

Even if it's not the best long-term solution, since it's fairly easy to get in there, I would recommend putting in a pull request.

[thatch45]

I have been chewing on this one for a WHILE, trying to figure out the best approach, I am going to try a few things....

[thatch45]

I have another fix for this coming in a PR that should make it smoother as well.

[thatch45]

Given the additional PR I am going to close this out

[davidjb]

Was there another fix in the works for this, or is the accepted solution extra_filerefs?

[thatch45]

I have hammered on this one a lot but have been unable to fix it in a way that makes me happy. So for the time being we need to keep it where it is. We have a more elaborate solution planned but it will require some substantial development.

[davidjb]

@thatch45 Thanks for that. This is okay and workable for now. Is it possible to include the extra_filerefs in a Saltfile or anywhere else to avoid me having to include it on the command line each time?

[thatch45]

It is. I will add it to the roster or saltfile

[thatch45]

#16733

Can Haz @davidjb

[max-arnold]

Here is an experimental one-line patch: #31531 (comment)