Fix the Demo environment SNS SSB config

[ccostino]

The Demo environment currently has the incorrect AWS region set on line 74 in terraform/demo/main.tf. It's currently set to us-east-1, but needs to be us-west-2 to properly match the AWS region that the SSB service is actually running out of.

(see https://github.com/GSA/usnotify-ssb?tab=readme-ov-file#aws-accounts-and-regions-in-use for details)

However, if you just change and update it Terraform will fail with this error:

│ Error: Instance notify-api-sns-demo failed 0b28b133-9f7b-49f1-9f34-a179cbb2549f, reason: [Job (90b9f2e1-f82c-4368-8af1-e889cddda878) failed: update could not be completed: Service broker error: attempt to update parameter that may result in service instance re-creation and data loss]
│ 
│   with module.sns_sms.cloudfoundry_service_instance.sns,
│   on ../shared/sns/main.tf line 18, in resource "cloudfoundry_service_instance" "sns":
│   18: resource "cloudfoundry_service_instance" "sns" {

We probably just need to delete the service and let Terraform recreate, but I'm not 100% sure on that, so we need to figure out the right way to update this.

Implementation Sketch and Acceptance Criteria

• Figure out how to properly modify the service instance associated with the SNS SSB, or if we just need to delete it first
• Apply the AWS region update
• Deploy the change

Security Considerations

• We want to make sure our service instance are connected to the right AWS accounts and regions so that our app functions properly in all environments.

[ccostino]

The first two parts of this are figured out and complete! We just needed to delete the existing service with a cf delete-service notify-api-sns-demo (very similar to what we had to do in the staging environment recently) and modify the region in the demo environment Terraform to be us-west-2 to match all the other environments.

The next step should have just been simply to let Terraform take care of the rest on the next deploy to the demo environment, which it tries to do, but we're running into a really weird situation that seems to be a race condition of sorts that is causing a permissions issue.

During deploy, Terraform is correctly seeing that it needs to recreate the SNS service, like so:

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.sns_sms.cloudfoundry_service_instance.sns will be created
  + resource "cloudfoundry_service_instance" "sns" {
      + id                             = (known after apply)
      + json_params                    = jsonencode(
            {
              + monthly_spend_limit = 25
              + region              = "us-west-2"
            }
        )
      + name                           = "notify-api-sns-demo"
      + replace_on_params_change       = false
      + replace_on_service_plan_change = false
      + service_plan                   = "<REDACTED>"
      + space                          = "<REDACTED>"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

It'll start the creation process, but it ultimately fails with this error:

Error: Service Instance notify-api-sns-demo failed <REDACTED>, reason: [Job (<REDACTED>) failed: create managed could not be completed: provision failed: Error: error setting SNS SMS Preferences: AuthorizationError: User: arn:aws:iam::<REDACTED>:user/ssb-sms-broker is not authorized to perform: iam:PassRole on resource: arn:aws:iam::<REDACTED>:role/sns-<REDACTED>-SuccessFeedback status code: 403, request id: <REDACTED> on main.tf line 42, in resource "aws_sns_sms_preferences" "sms_settings": 42: resource "aws_sns_sms_preferences" "sms_settings" { exit status 1]

    with module.sns_sms.cloudfoundry_service_instance.sns,
    on ../shared/sns/main.tf line 18, in resource "cloudfoundry_service_instance" "sns":
    18: resource "cloudfoundry_service_instance" "sns" {

There's no clear indication of what's going wrong here. When removing the old service, all associated resources in Cloud Foundry and AWS are cleaned up properly, and Terraform is running through the steps to instantiate a new instances of the service via the ttsnotify-brokerpak-sms. This includes creating an IAM role associated with the instance that can be assumed by the ssb-sms-broker IAM user, which it does in every other environment we have running. Yet for the demo environment, a 403 is being returned when this is attempted during the setup despite all of the permissions and policies being set correctly (see here for details of what that should look like).

The only plausible thing that makes sense at the moment is that the SuccessFeedback role is not being created at the right time and may not be available when the other parts of the Terraform plan are run.

We're not sure what else to do at this point, but it's not worth spending more time on this. The demo environment is not in the path to production, and we're no longer in a state where we're demoing the product to other prospective customers in its current form.

This issue is being returned to the backlog for the time being in case we have a need to refer back to the work for something.