molecule[ec2] documentation

[nemonik]

Building on #266, I would love the same.

Finding /molecule-plugins/test/ec2/scenarios/driver/ec2/molecule/default/molecule.yml I'm like woot, woot, until I realize the ../../../../../resources/playbooks/ec2/create.yml and ../../../../../resources/playbooks/ec2/destroy.yml playbooks aren't included in the repo.

[konstruktoid]

https://github.com/ansible-community/molecule-plugins/tree/9943538d487b25a03d24c886980f3d0a643aaaaa/src/molecule_plugins/ec2/cookiecutter/%7B%7Bcookiecutter.molecule_directory%7D%7D/%7B%7Bcookiecutter.scenario_name%7D%7D

and molecule init scenario -d ec2 myrepo should create it for you.

[...]
TASK [Expand templates] ********************************************************
changed: [localhost] => (item=molecule/myrepo/molecule.yml)
changed: [localhost] => (item=molecule/myrepo/destroy.yml)
changed: [localhost] => (item=molecule/myrepo/create.yml)
changed: [localhost] => (item=molecule/myrepo/converge.yml)

[nemonik]

Yeah, but these are partially populated stubs. I assume from this and the lack of docs I am to write ansible playbooks referencing keys declared in the molecule file.

[konstruktoid]

no, just write your standard ansible roles, generate the molecule skeleton, update the molecule.yml with platforms and such. after that you should add tests to verify.yml and then run molecule test.

in https://github.com/konstruktoid/ansible-role-template/tree/main/molecule, I got the default scenario that uses vagrant and an additional one using podman.

[nemonik]

I did this to start... the ec2 isn't created... let me loop back and simplify my molecule.yml to the bare minimum and try that again. The `create.yml` and `destroy.yml` created by the init using the ec2 driver will need to be deleted based in what you have said but these files existing stubbed out after the init had me thinking otherwise... the lack of docs didn't give anything to gut check against vice reading the code.

[konstruktoid]

I'm not defending the lack of documentation in any way, but I've never had to create or modifiy create.yml or destroy.yml

[apatard]

fwiw, for some plugins (I guess it's including ec2), there's an issue coming from molecule. Cookiecutter support has been removed (and I don't remember seeing some kind of replacement possible from -plugins point of view). The leads to the issue of molecule init bringing "useless" molecule file like create.yml and destroy.yml files intended (for delegated driver iirc?). Most of times, the solution is just to remove these two files. People using podman/docker plugins are not seeing that as molecule has been patched to not copy these files when they're used.

[nemonik]

When using the EC2 plugin? Okay. This good to know. The fault is likely with my molecule.yml and the stubbed out create.yml and destroy.yml being created by the init. I will revisit this morning.

[nemonik]

Hmm.

I'm on Ubuntu 22.04.4 LTS using pyenv and poetry to install molecule and its dependencies:

➜  molecule --version
molecule 24.8.0 using python 3.10
    ansible:2.16.10
    azure:23.5.3 from molecule_plugins
    containers:23.5.3 from molecule_plugins requiring collections: ansible.posix>=1.3.0 community.docker>=1.9.1 containers.podman>=1.8.1
    default:24.8.0 from molecule
    docker:23.5.3 from molecule_plugins requiring collections: community.docker>=3.4.11 ansible.posix>=1.4.0
    ec2:23.5.3 from molecule_plugins
    gce:23.5.3 from molecule_plugins requiring collections: google.cloud>=1.0.2 community.crypto>=1.8.0
    openstack:23.5.3 from molecule_plugins requiring collections: openstack.cloud>=2.1.0
    podman:23.5.3 from molecule_plugins requiring collections: containers.podman>=1.7.0 ansible.posix>=1.3.0
    vagrant:23.5.3 from molecule_plugins

My Python dependency tree is

molecule-plugins 23.5.3 Molecule Plugins
└── molecule >=6.0.0a1
    ├── ansible-compat >=24.6.1
    │   ├── ansible-core >=2.14
    │   │   ├── cryptography *
    │   │   │   └── cffi >=1.12
    │   │   │       └── pycparser *
    │   │   ├── jinja2 >=3.0.0
    │   │   │   └── markupsafe >=2.0
    │   │   ├── packaging *
    │   │   ├── pyyaml >=5.1
    │   │   └── resolvelib >=0.5.3,<1.1.0
    │   ├── jsonschema >=4.6.0
    │   │   ├── attrs >=22.2.0
    │   │   ├── jsonschema-specifications >=2023.03.6
    │   │   │   └── referencing >=0.31.0
    │   │   │       ├── attrs >=22.2.0 (circular dependency aborted here)
    │   │   │       └── rpds-py >=0.7.0
    │   │   ├── referencing >=0.28.4 (circular dependency aborted here)
    │   │   └── rpds-py >=0.7.1 (circular dependency aborted here)
    │   ├── packaging * (circular dependency aborted here)
    │   ├── pyyaml * (circular dependency aborted here)
    │   └── subprocess-tee >=0.4.1
    ├── ansible-core >=2.12.10 (circular dependency aborted here)
    ├── click >=8.0,<9
    │   └── colorama *
    ├── click-help-colors *
    │   └── click >=7.0,<9 (circular dependency aborted here)
    ├── enrich >=1.2.7
    │   └── rich >=9.5.1
    │       ├── markdown-it-py >=2.2.0
    │       │   └── mdurl >=0.1,<1.0
    │       └── pygments >=2.13.0,<3.0.0
    ├── jinja2 >=2.11.3 (circular dependency aborted here)
    ├── jsonschema >=4.9.1 (circular dependency aborted here)
    ├── packaging * (circular dependency aborted here)
    ├── pluggy >=0.7.1,<2.0
    ├── pyyaml >=5.1 (circular dependency aborted here)
    ├── rich >=9.5.1 (circular dependency aborted here)
    └── wcmatch >=8.1.2
        └── bracex >=2.1.1

I'm authenticated into the aws cli and verified I can pull the contents from s3 via aws s3 ls.

My molecule folder only holds {{project path}}/molecule/grr/molecule.yml and nothing else

The contents for {{project path}}/molecule/grr/molecule.yml are (some of which I had to guess at referring to the defaults https://github.com/ansible-community/molecule-plugins/blob/main/src/molecule_plugins/ec2/cookiecutter/%7B%7Bcookiecutter.molecule_directory%7D%7D/%7B%7Bcookiecutter.scenario_name%7D%7D/create.yml#L19) -- I later found and verified against https://github.com/ansible-community/molecule-plugins/blob/main/doc/ec2/platforms.rst:

---
driver:
  name: ec2
platforms:
  - name: instance
    image: ami-0ce8fc041db68907c
    instance_type: t2.micro
    vpc_subnet_id: subnet-06543a923334a6b41
    security_groups::
     - sg-0f93b0e3a1d7626dc
     - sg-0924b78790d0d9590
     - sg-0bb3f716a64b69244
     - sg-0737a1bcac58b9a2d
    region: us-gov-west-1
    assign_public_ip: false
    key_name: MY_RSA
    private_key_path: ~/.ssh/MY_RSA
    ssh_user: ec2-user
provisioner:
  name: ansible
  env:
    ANSIBLE_COLLECTIONS_PATH: ../../ansible_collections
  playbooks:
    converge: ../../linux.yml
verifier:
  name: ansible

Executing molecule --debug -v create --scenario-name grr --driver-name ec2 doesn't create the instance

DEBUG    Validating schema /home/nemonik/Development/workspace/playbook_prometheus_node_exporter/molecule/grr/molecule.yml.
WARNING  Driver ec2 does not provide a schema.
INFO     grr scenario test matrix: dependency, create, prepare
INFO     Performing prerun with role_name_check=0...
INFO     Running grr > dependency
WARNING  Skipping, missing the requirements file.
WARNING  Skipping, missing the requirements file.
INFO     Running grr > create
WARNING  Skipping, instances already created.
INFO     Running grr > prepare
WARNING  Skipping, prepare playbook not configured.

Well, it sez it does

➜  molecule list
WARNING  Driver ec2 does not provide a schema.
WARNING  Driver ec2 does not provide a schema.
WARNING  Driver ec2 does not provide a schema.
INFO     Running grr > list
                                           ╷             ╷                  ╷               ╷         ╷
  Instance Name                            │ Driver Name │ Provisioner Name │ Scenario Name │ Created │ Converged
╶──────────────────────────────────────────┼─────────────┼──────────────────┼───────────────┼─────────┼───────────╴
  molecule_grr                             │ ec2         │ ansible          │ grr           │ true    │ false

But it doesn't as there no instance to ansible:

➜  molecule --debug -v converge --scenario-name grr
DEBUG    Validating schema /home/nemonik/Development/workspace/playbook_prometheus_node_exporter/molecule/grr/molecule.yml.
WARNING  Driver ec2 does not provide a schema.
INFO     grr scenario test matrix: dependency, create, prepare, converge
INFO     Performing prerun with role_name_check=0...
INFO     Running grr > dependency
WARNING  Skipping, missing the requirements file.
WARNING  Skipping, missing the requirements file.
INFO     Running grr > create
WARNING  Skipping, instances already created.
INFO     Running grr > prepare
WARNING  Skipping, prepare playbook not configured.
INFO     Running grr > converge
DEBUG: ANSIBLE ENVIRONMENT:
ANSIBLE_COLLECTIONS_PATH: ../../ansible_collections
ANSIBLE_CONFIG: /home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/ansible.cfg
ANSIBLE_FILTER_PLUGINS: /home/nemonik/.cache/pypoetry/virtualenvs/playbook-prometheus-node-exporter-TYBnf8te-py3.10/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/filter:/home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/plugins/filter:/home/nemonik/Development/workspace/playbook_prometheus_node_exporter/plugins/filter:/home/nemonik/.ansible/plugins/filter:/usr/share/ansible/plugins/filter
ANSIBLE_FORCE_COLOR: '1'
ANSIBLE_LIBRARY: /home/nemonik/.cache/pypoetry/virtualenvs/playbook-prometheus-node-exporter-TYBnf8te-py3.10/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/modules:/home/nemonik/.cache/pypoetry/virtualenvs/playbook-prometheus-node-exporter-TYBnf8te-py3.10/lib/python3.10/site-packages/molecule_plugins/vagrant/modules:/home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/library:/home/nemonik/Development/workspace/playbook_prometheus_node_exporter/library:/home/nemonik/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
ANSIBLE_ROLES_PATH: /home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/roles:/home/nemonik/Development/workspace:/home/nemonik/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
ANSIBLE_VERBOSITY: '1'

DEBUG: MOLECULE ENVIRONMENT:
MOLECULE_DEBUG: 'True'
MOLECULE_DEPENDENCY_NAME: galaxy
MOLECULE_DRIVER_NAME: ec2
MOLECULE_ENV_FILE: /home/nemonik/Development/workspace/playbook_prometheus_node_exporter/.env.yml
MOLECULE_EPHEMERAL_DIRECTORY: /home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr
MOLECULE_FILE: /home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/molecule.yml
MOLECULE_INSTANCE_CONFIG: /home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/instance_config.yml
MOLECULE_INVENTORY_FILE: /home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/inventory/ansible_inventory.yml
MOLECULE_PROJECT_DIRECTORY: /home/nemonik/Development/workspace/playbook_prometheus_node_exporter
MOLECULE_PROVISIONER_NAME: ansible
MOLECULE_SCENARIO_DIRECTORY: /home/nemonik/Development/workspace/playbook_prometheus_node_exporter/molecule/grr
MOLECULE_SCENARIO_NAME: grr
MOLECULE_STATE_FILE: /home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/state.yml
MOLECULE_VERIFIER_NAME: ansible
MOLECULE_VERIFIER_TEST_DIRECTORY: /home/nemonik/Development/workspace/playbook_prometheus_node_exporter/molecule/grr/tests

DEBUG: SHELL REPLAY:
ANSIBLE_COLLECTIONS_PATH=../../ansible_collections ANSIBLE_CONFIG=/home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/ansible.cfg ANSIBLE_FILTER_PLUGINS=/home/nemonik/.cache/pypoetry/virtualenvs/playbook-prometheus-node-exporter-TYBnf8te-py3.10/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/filter:/home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/plugins/filter:/home/nemonik/Development/workspace/playbook_prometheus_node_exporter/plugins/filter:/home/nemonik/.ansible/plugins/filter:/usr/share/ansible/plugins/filter ANSIBLE_FORCE_COLOR=1 ANSIBLE_LIBRARY=/home/nemonik/.cache/pypoetry/virtualenvs/playbook-prometheus-node-exporter-TYBnf8te-py3.10/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/modules:/home/nemonik/.cache/pypoetry/virtualenvs/playbook-prometheus-node-exporter-TYBnf8te-py3.10/lib/python3.10/site-packages/molecule_plugins/vagrant/modules:/home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/library:/home/nemonik/Development/workspace/playbook_prometheus_node_exporter/library:/home/nemonik/.ansible/plugins/modules:/usr/share/ansible/plugins/modules ANSIBLE_ROLES_PATH=/home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/roles:/home/nemonik/Development/workspace:/home/nemonik/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles ANSIBLE_VERBOSITY=1 MOLECULE_DEBUG=True MOLECULE_DEPENDENCY_NAME=galaxy MOLECULE_DRIVER_NAME=ec2 MOLECULE_ENV_FILE=/home/nemonik/Development/workspace/playbook_prometheus_node_exporter/.env.yml MOLECULE_EPHEMERAL_DIRECTORY=/home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr MOLECULE_FILE=/home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/molecule.yml MOLECULE_INSTANCE_CONFIG=/home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/instance_config.yml MOLECULE_INVENTORY_FILE=/home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/inventory/ansible_inventory.yml MOLECULE_PROJECT_DIRECTORY=/home/nemonik/Development/workspace/playbook_prometheus_node_exporter MOLECULE_PROVISIONER_NAME=ansible MOLECULE_SCENARIO_DIRECTORY=/home/nemonik/Development/workspace/playbook_prometheus_node_exporter/molecule/grr MOLECULE_SCENARIO_NAME=grr MOLECULE_STATE_FILE=/home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/state.yml MOLECULE_VERIFIER_NAME=ansible MOLECULE_VERIFIER_TEST_DIRECTORY=/home/nemonik/Development/workspace/playbook_prometheus_node_exporter/molecule/grr/tests

Using /home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/ansible.cfg as config file

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
fatal: [molecule_grr]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname molecule_grr: Temporary failure in name resolution", "unreachable": true}

PLAY RECAP *********************************************************************
molecule_grr               : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

CRITICAL Ansible return code was 4, command was: ansible-playbook --diff --inventory /home/nemonik/.cache/molecule/playbook_prometheus_node_exporter/grr/inventory --skip-tags molecule-notest,notest /home/nemonik/Development/workspace/playbook_prometheus_node_exporter/linux.yml

Adding back in the create.yml and destroy.yml created by molecule init configured to use the ec2 drive have no impact when rerunning the create.

My organization requires MFA, so I am setting these environment variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_SESSION_TOKEN
• AWS_CA_BUNDLE
• AWS_PROFILE

@ssbarnea thoughts?

[nemonik]

To get the create to work

I had to go to v23.5.3's

https://github.com/ansible-community/molecule-plugins/blob/v23.5.3/src/molecule_plugins/ec2/cookiecutter/%7B%7Bcookiecutter.molecule_directory%7D%7D/%7B%7Bcookiecutter.scenario_name%7D%7D/create.yml

and copy the content from between {% raw -%} and {%- endraw %} lines into my scenerio's create.yml... I suspect I'll need to do that for destroy.yml too...

[nemonik]

And yep... do the same with destory.yml... and molecule works.

I guess this is cuz cookiecutter support was removed or something?