[BUG] Fails if encountering ThrottlingException from AWS for DescribeLogGroups request

[edruid]

Describe the bug
If the DescribeLogGroups call encounters a ThrottlingException from AWS the logger will throw a CloudWatchLogsException
400 Bad Request
{"__type":"ThrottlingException","message":"Rate exceeded"}

Expected behavior
If exception is thrown, sleep 1 second and retry (at least once).

Please provide the steps to reproduce and if possible a minimal demo of the problem
Run 6 parallel processes all trying to initialize the CloudWatch handler within the same second

Please tell about your environment:

• PHP Version: 7.2
• Operating system (distro): AWS Linux
• Application mode:
  • Web app
  • CLI app
  • Daemon worker

trace:
"/var/app/current/carrot/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:100",
"/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:203",
"/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:174",
"/var/app/current/carrot/vendor/guzzlehttp/promises/src/RejectedPromise.php:40",
"/var/app/current/carrot/vendor/guzzlehttp/promises/src/TaskQueue.php:47",
"/var/app/current/carrot/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:104",
"/var/app/current/carrot/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:131",
"/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:246",
"/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:223",
"/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:267",
"/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:225",
"/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:267",
"/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:225",
"/var/app/current/carrot/vendor/guzzlehttp/promises/src/Promise.php:62",
"/var/app/current/carrot/vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58",
"/var/app/current/carrot/vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86",
"/var/app/current/carrot/vendor/maxbanton/cwh/src/Handler/CloudWatch.php:285",
"/var/app/current/carrot/vendor/maxbanton/cwh/src/Handler/CloudWatch.php:248",
"/var/app/current/carrot/vendor/maxbanton/cwh/src/Handler/CloudWatch.php:172",
"/var/app/current/carrot/vendor/maxbanton/cwh/src/Handler/CloudWatch.php:150",
"/var/app/current/carrot/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:39",
"/var/app/current/carrot/vendor/monolog/monolog/src/Monolog/Logger.php:344",
"/var/app/current/carrot/vendor/monolog/monolog/src/Monolog/Logger.php:637",
...

[agheorghe-careplanner]

Hi.

We're experiencing this issue as well when a large number of parallel processes try to create CloudWatch logs at the same time. Would I be correct in thinking the following would work as a fix for it? From what I can see the throttling is handled in a similar way as described above through that.

diff --git a/src/Handler/CloudWatch.php b/src/Handler/CloudWatch.php
index 41c12f7..29c6492 100755
--- a/src/Handler/CloudWatch.php
+++ b/src/Handler/CloudWatch.php
@@ -299,6 +299,7 @@ class CloudWatch extends AbstractProcessingHandler
 
     private function initializeGroup(): void
     {
+        $this->checkThrottle();
         // fetch existing groups
         $existingGroups =
             $this

Thanks

[dmelskyi]

I have the same problem. Also, regarding the fix, I think it should be not only checked throttling inside of the function, but it should also be additional retries as well.

[hasibskrijelj]

Hello,

I have the same issue on the latest version 2.0.2. Will there be a fix in the near future ?

[ihr-it-projekt]

Have a look to https://github.com/maxbanton/cwh/blob/master/src/Handler/CloudWatch.php#L130 . You do not need to check normally that the log group is created and create one. This is insufficient. So use here false instead.