Let the sensitive parts of credentials to be defined externally as variables (#121)

* let the sensitive parts of credentials to be defined externally as variables

* change from the command line to an example playbook

* add changelog fragments

* get rid off the command line

Author: ivarmu

changelogs/fragments/filetree_create_externalize_credentials.yml

@@ -0,0 +1,4 @@
+---
+minor_changes:
+  - Let the sensitive data of the credentials and users to be defined externally through a well know variables
+...

roles/filetree_create/README.md

@@ -25,6 +25,8 @@ The following variables are required for that role to work properly:
 | `output_path` | `/tmp/filetree_output` | yes | str | The path to the output directory where all the generated `yaml` files with the corresponding Objects as code will be written to. |
 | `input_tag` | `['all']` | no | List of Strings | The tags which are applied to the 'sub-roles'. If 'all' is in the list (the default value) then all roles will be called.  Valid tags can be found under `vars/valid_tags` |
 | `flatten_output` | N/A | no | bool | Whether to flatten the output in single files per each object type instead of the normal exportation structure |
+| `secrets_as_variables` | N/A | no | bool | Whether to export the secrets as variables that can be populated from existing variables/files. An example: `vaulted_eda_credentials_my_eda_credential_password`, that follows the syntax: `<secrets_as_variables_prefix>_<object_type>_<object_name>_<field_name>` |
+| `secrets_as_variables_prefix` | vaulted | no | str | The prefix to use for the variables defined by `secrets_as_variables` feature. |
 | `show_encrypted` | N/A | no | bool | Whether to remove the string '\$encrypted\$' in credentials output (not the actual credential value) |
 | `omit_id` | N/A | no | bool | Whether to create output files without objects id.|
 | `organization`| N/A | no | str | Default organization for all objects that have not been set in the source controller.|
@@ -247,6 +249,95 @@ This example will export all object but some with modifications:
 ...
 ```
 
+## Usage example for the `secrets_as_variables` feature
+
+To let the credentials and the users to be exported and imported 'as is', without any modification, the sensitive data (that can't be exported through the API) can be abstracted to extra vars (or variable's file) and vaulted. Those variables can be referenced at the original objects' code, so they can be imported without any manual modification. To clarify the described scenario, the following output shows the exported object for a gateway user, using the `secrets_as_variable` feature:
+
+Sample playbook:
+
+```yaml
+---
+- name: Filetree Create Test
+  hosts: all
+  connection: local
+  gather_facts: false
+  vars:
+    aap_username: "{{ vault_aap_username | default(lookup('env', 'CONTROLLER_USERNAME')) }}"
+    aap_password: "{{ vault_aap_password | default(lookup('env', 'CONTROLLER_PASSWORD')) }}"
+    aap_hostname: "{{ vault_aap_hostname | default(lookup('env', 'CONTROLLER_HOST')) }}"
+    aap_validate_certs: "{{ vault_aap_validate_certs | default(lookup('env', 'CONTROLLER_VERIFY_SSL')) }}"
+    output_path: /tmp/filetree_output_25
+    # Let the secrets to be defined externally (and vaulted) through well known variables
+    secrets_as_variables: true
+
+  pre_tasks:
+    - name: "Setup authentication (block)"
+      no_log: "{{ controller_configuration_filetree_create_secure_logging }}"
+      when: aap_oauthtoken is not defined
+      tags:
+        - always
+      block:
+        - name: "Get the Authentication Token for the future requests"
+          ansible.builtin.uri:
+            url: "https://{{ aap_hostname }}/api/gateway/v1/tokens/"
+            user: "{{ aap_username }}"
+            password: "{{ aap_password }}"
+            method: POST
+            force_basic_auth: true
+            validate_certs: "{{ aap_validate_certs }}"
+            status_code: 201
+          register: authtoken_res
+
+        - name: "Set the oauth token to be used since now"
+          ansible.builtin.set_fact:
+            aap_oauthtoken: "{{ authtoken_res.json.token }}"
+            aap_oauthtoken_url: "{{ authtoken_res.json.url }}"
+
+  roles:
+    - infra.aap_configuration_extended.filetree_create
+
+  post_tasks:
+    - name: "Delete the Authentication Token used"
+      ansible.builtin.uri:
+        url: "https://{{ aap_hostname }}{{ aap_oauthtoken_url }}"
+        user: "{{ aap_username }}"
+        password: "{{ aap_password }}"
+        method: DELETE
+        force_basic_auth: true
+        validate_certs: "{{ aap_validate_certs }}"
+        status_code: 204
+      when: aap_oauthtoken_url is defined
+...
+```
+
+Generated file: `/tmp/filetree_output_25/gateway_users.yaml`
+
+```yaml
+---
+aap_user_accounts:
+  - username: "test_user"
+    email: ""
+    first_name: ""
+    last_name: ""
+    password: "{{ vaulted_gateway_users_test_user_password }}"
+    is_superuser: "False"
+    authenticators: []
+    authenticator_uid: ""
+...
+```
+
+The variable `vaulted_gateway_users_test_user_password` can be defined in a third file:
+
+`~/vaulted_credentials.yaml`:
+
+```yaml
+vaulted_gateway_users_test_user_password: "SuperSecretPassword"
+```
+
+That file can be encrypted using `ansible-vault`.
+
+The import process can be executed directly, using that file with the extra_vars option: `ansible-playbook -e@~/vaulted_credentials.yaml`.
+
 ## License
 
 GPLv3+

roles/filetree_create/defaults/main.yml

@@ -34,4 +34,6 @@ eda_configuration_filetree_create_secure_logging: "{{ controller_configuration_s
 input_tag:
   - all
 organization: 'ORGANIZATIONLESS'
+
+secrets_as_variables_prefix: "vaulted"
 ...

roles/filetree_create/templates/controller_credentials.j2

@@ -15,13 +15,17 @@ controller_credentials:
 {% if template_overrides_resources.credential[current_credentials_asset_value.name].inputs is defined
       or (current_credentials_asset_value.inputs is defined and current_credentials_asset_value.inputs is not match('{}')) %}
     inputs:
-{% if show_encrypted is defined and show_encrypted %}
-{{ current_credentials_asset_value.inputs | to_nice_yaml(indent=2, sort_keys=false) | indent(width=6, first=true) }}
-{% else %}
-{{ template_overrides_resources.credential[current_credentials_asset_value.name].inputs
-   | default(current_credentials_asset_value.inputs)
-   | to_nice_yaml(indent=2, sort_keys=false) | indent(width=6, first=true) | replace("$encrypted$", "\'\'") }}
-{% endif %}
+{%   for input in (template_overrides_resources.credential[current_credentials_asset_value.name].inputs |
+                   default(current_credentials_asset_value.inputs)) | dict2items %}
+{%     if show_encrypted is defined and show_encrypted %}
+      {{ input.key }}: {{ input.value }}
+{%     elif secrets_as_variables is defined and secrets_as_variables %}
+      {{ input.key }}: {{ input.value | replace("$encrypted$", "\"{{ " + secrets_as_variables_prefix + "_controller_credentials_" + (current_credentials_asset_value.name | replace(' ', '_') | lower) + "_" + (input.key | replace(' ', '_') | lower) + " }}\"") }}
+{%     else %}
+      {{ input.key }}: {{ input.value | replace("$encrypted$", "\'\'") }}
+{%     endif %}
+{%   endfor %}
+
 {% endif %}
 {% if last_credential | default(true) | bool %}
 ...

roles/filetree_create/templates/controller_users.j2

@@ -3,7 +3,13 @@
 aap_user_accounts:
 {% endif %}
 - username: "{{ current_users_asset_value.username }}"
+{%  if show_encrypted is defined and show_encrypted %}
+  password: "{{ current_users_asset_value.password }}"
+{%  elif secrets_as_variables is defined and secrets_as_variables %}
+  password: "{{ current_users_asset_value.password | replace("$encrypted$", "{{ " + secrets_as_variables_prefix + "controller_users_" + (current_users_asset_value.username | replace(' ', '_') | lower) + "_password }}") }}"
+{%  else %}
   password: "INITIAL"
+{%  endif %}
   email: "{{ template_overrides_resources.user[current_users_asset_value.username].email
              | default(template_overrides_global.user.email)
              | default(current_users_asset_value.email) }}"

roles/filetree_create/templates/eda_credentials.j2

@@ -18,8 +18,16 @@ eda_credentials:
 {% endif %}
 {% if eda_credential.inputs is defined %}
     inputs:
-{{ template_overrides_resources.eda_credential[eda_credential.name].inputs
-   | default(eda_credential.inputs) | to_nice_yaml(indent=2,sort_keys=False) | indent(width=6, first=True) }}
+{%   for input in (template_overrides_resources.eda_credential[eda_credential.name].inputs |
+                   default(eda_credential.inputs)) | dict2items %}
+{%     if show_encrypted is defined and show_encrypted %}
+      {{ input.key }}: {{ input.value }}
+{%     elif secrets_as_variables is defined and secrets_as_variables %}
+      {{ input.key }}: {{ input.value | replace("$encrypted$", "\"{{ " + secrets_as_variables_prefix + "_eda_credentials_" + (eda_credential.name | replace(' ', '_') | lower) + "_" + (input.key | replace(' ', '_') | lower) + " }}\"") }}
+{%     else %}
+      {{ input.key }}: {{ input.value | replace("$encrypted$", "\'\'") }}
+{%     endif %}
+{%   endfor %}
 {%- endif %}
 {% endfor %}
 ...

roles/filetree_create/templates/gateway_users.j2

@@ -1,9 +1,10 @@
 ---
 aap_user_accounts:
 {% for current_user in current_gateway_users_asset_value %}
-  - username: "{{ template_overrides_resources.gateway_user[current_user.username].username
+{%   set _username = template_overrides_resources.gateway_user[current_user.username].username
                      | default(template_overrides_global.gateway_user.username)
-                     | default(current_user.username) }}"
+                     | default(current_user.username) %}
+  - username: "{{ _username }}"
     email: "{{ template_overrides_resources.gateway_user[current_user.name].email
                      | default(template_overrides_global.gateway_user.email)
                      | default(current_user.email) }}"
@@ -13,9 +14,19 @@ aap_user_accounts:
     last_name: "{{ template_overrides_resources.gateway_user[current_user.name].last_name
                      | default(template_overrides_global.gateway_user.last_name)
                      | default(current_user.last_name) }}"
+{%  if show_encrypted is defined and show_encrypted %}
     password: "{{ template_overrides_resources.gateway_user[current_user.name].password
                      | default(template_overrides_global.gateway_user.password)
-                     | default(current_user.password) | replace("$encrypted$", "\'\'") }}"
+                     | default(current_user.password) }}"
+{%  elif secrets_as_variables is defined and secrets_as_variables %}
+    password: "{{ template_overrides_resources.gateway_user[current_user.name].password
+                     | default(template_overrides_global.gateway_user.password)
+                     | default(current_user.password) | replace("$encrypted$", "{{ " + secrets_as_variables_prefix + "_gateway_users_" + (_username | replace(' ', '_') | lower) + "_password }}") }}"
+{%  else %}
+    password: "{{ template_overrides_resources.gateway_user[current_user.name].password
+                     | default(template_overrides_global.gateway_user.password)
+                     | default(current_user.password) | replace("$encrypted$", "") }}"
+{%  endif %}
     is_superuser: "{{ template_overrides_resources.gateway_user[current_user.name].is_superuser
                      | default(template_overrides_global.gateway_user.is_superuser)
                      | default(current_user.is_superuser) }}"

roles/filetree_create/vars/main.yml

@@ -24,6 +24,12 @@ valid_tags:
   - controller_schedules
   - controller_gateway_authenticator
   - controller_gateway_authenticator_maps
+  - eda_credentials
+  - eda_credential_types
+  - eda_decision_environments
+  - eda_event_streams
+  - eda_projects
+  - eda_rulebook_activations
   - gateway_applications
   - gateway_authenticator_maps
   - gateway_authenticators