4472 - Reduce duplication code for compactions and bulk export

[ab295382]

Issue

• My PR addresses the following issues and references them in the PR title. For example, "Issue 1234 - My Sleeper
  PR"
  • Resolves Reduce duplication between starting bulk export and compaction tasks #4472

Tests

• My PR adds the following tests OR does not need testing for this extremely good reason:
  • MyUnitTest

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it, or I have linked to a
  separate issue for that below.
• If I have added or removed any dependencies from the project, I have updated the NOTICES file.

[patchwork01]

I think the only important thing left is the argument count checks in the main methods. Other than that this looks great. Have you tested it in AWS, for bulk export & compaction?

[ca61688]

I think the only important thing left is the argument count checks in the main methods. Other than that this looks great. Have you tested it in AWS, for bulk export & compaction?

Haven't tested in AWS but happy to give that a go, might need some direction on doing that though

[patchwork01]

Looks good, thanks. I've left a couple more minor comments. It would be good to test it in AWS before we merge it.

For compaction there are some acceptance system tests that cover this:

• CompactionST
• CompactionOnEC2ST

We don't currently have a system test for bulk export, so you'd need to use the deployAll manual testing setup for that. You'll need to get some data into a table (which the deployAll setup does for you by default), and then trigger a bulk export by sending a message to the queue. Here's the documentation for that:

https://github.com/gchq/sleeper/blob/develop/docs/usage/export.md

Here's the documentation for those two types of system testing (acceptance tests vs manual testing):

https://github.com/gchq/sleeper/blob/develop/docs/development/system-tests.md

[patchwork01]

I think it's a bit easier to read with the arguments brought out like this, since you can compare it against the usage and count check just above here more directly. Can we keep this as it was before? The same applies to BulkExportTaskHostScaler.

[ca61688]

Looks good, thanks. I've left a couple more minor comments. It would be good to test it in AWS before we merge it.

For compaction there are some acceptance system tests that cover this:

• CompactionST
• CompactionOnEC2ST

We don't currently have a system test for bulk export, so you'd need to use the deployAll manual testing setup for that. You'll need to get some data into a table (which the deployAll setup does for you by default), and then trigger a bulk export by sending a message to the queue. Here's the documentation for that:

https://github.com/gchq/sleeper/blob/develop/docs/usage/export.md

Here's the documentation for those two types of system testing (acceptance tests vs manual testing):

https://github.com/gchq/sleeper/blob/develop/docs/development/system-tests.md

Okay I've now tested it on aws and the system tests ran fine along with a bulk export manual check. I think this is ready to merge now