Bug: Thread Safety Issue: Static Scope Dictionary Causes Concurrent Access Exceptions

[nCubed]

Expected Behaviour

The Logger should be thread-safe when multiple concurrent tasks perform logging operations and context key manipulations (AppendKey, RemoveKey, GetAllKeys) simultaneously.

Current Behaviour

The static Scope dictionary in Logger.Scope.cs (line 15) is not thread-safe, causing exceptions when accessed concurrently:

1. InvalidOperationException: "Collection was modified; enumeration operation may not execute" - occurs during foreach enumeration of GetAllKeys()
2. ArgumentException: "Destination array is not long enough" - occurs when calling GetAllKeys().ToArray()

These exceptions occur in production Lambda environments when concurrent tasks use the logger (e.g., Task.WhenAll scenarios).

Real-World Stack Trace

AggregateException: An error occurred while writing to logger(s). (Collection was modified; enumeration operation may not execute.)
   at Microsoft.Extensions.Logging.Logger.ThrowLoggingError(List`1 exceptions)
   at AWS.Lambda.Powertools.Logging.Internal.PowertoolsLogger.GetLogEntry(...)
     → Line 229: foreach (var (key, value) in this.GetAllKeys())

Root Cause

Logger.Scope.cs line 15:

private static IDictionary<string, object> Scope { get; } = new Dictionary<string, object>(StringComparer.Ordinal);

Race Condition:

• Thread A: Enumerates Scope via GetAllKeys() (called during logging at PowertoolsLogger.GetLogEntry() line 229)
• Thread B: Modifies Scope via AppendKey(), RemoveKey(), or RemoveKeys()
• Result: Dictionary<TKey, TValue> is not thread-safe for concurrent read/write operations

Impact

• Production Lambda failures when concurrent operations use the logger
• Intermittent, difficult to debug due to race condition timing
• Affects any Lambda using concurrent tasks (common pattern for staying under API Gateway 29s timeout)

Code snippet

### Minimal Reproduction Test

[TestMethod]
public async Task ConcurrentAccess_ForeachOnGetAllKeys_ThrowsInvalidOperationException()
{
    Logger.RemoveKeys(Logger.GetAllKeys()?.Select(x => x.Key).ToArray() ?? []);

    var tasks = new List<Task>
    {
        // Thread 1: Enumerate (mimics GetLogEntry line 229)
        Task.Run(() => {
            for (int i = 0; i < 100; i++)
                foreach (var kvp in Logger.GetAllKeys()) { }
        }),

        // Thread 2: Log (also enumerates internally)
        Task.Run(() => {
            for (int i = 0; i < 100; i++)
                Logger.LogInformation($"Iteration {i}");
        }),

        // Thread 3: Modify keys
        Task.Run(() => {
            for (int i = 0; i < 100; i++) {
                Logger.AppendKey($"key_{i % 10}", i);
                Logger.RemoveKey($"key_{(i-1) % 10}");
            }
        })
    };

    await Task.WhenAll(tasks);
}

Possible Solution

1. Use ConcurrentDictionary<string, object> instead of Dictionary<string, object> for the static Scope property
2. Add locking around all Scope access (reads and writes) in Logger.Scope.cs
3. Use ThreadLocal<Dictionary<string, object>> if context should be thread-specific rather than shared

Steps to Reproduce

1. Create a Lambda function or test that uses Task.WhenAll() to run concurrent operations
2. Have multiple tasks simultaneously:
   • Call logging methods (LogInformation, LogDebug, etc.)
   • Call AppendKey() or RemoveKey() to manipulate context
3. Run the test repeatedly (race condition may not trigger every time)
4. Observe InvalidOperationException or ArgumentException

Powertools for AWS Lambda (.NET) version

latest (3.0.2)

AWS Lambda function runtime

dotnet8

Debugging logs

[hjgraca]

Hi @nCubed thanks for opening the issue, we are addressing this for support of LMI #1076 , should be released soon

[nCubed]

Hi @hjgraca - The LMI support may not apply to this scenario. Here we have a single lambda process running multiple threads for the same request. Whereas LMI is handling "multiple concurrent requests". In other words, this issue is about a single request, not multiple concurrent requests.

[hjgraca]

@nCubed I believe the fix overlaps.
I will be releasing today, if you can let me know after update if it fixed this issue that would be perfect.
Thank you

[nCubed]

@hjgraca Thanks for the update. Wondering if you've successfully run the code sample I provided?