negative interaction with audio_session

[lukepighetti]

ryanheise/audio_session#138

[alnitak]

Hey @lukepighetti,

I read ryanheise reply. Unfortunately, I don't have a Mac to try. What I can say is that loadAsset() does nothing else than read using rootbundle or give the file directly to the native if it is already cached.

In the flutter_soloud version you are using, an Isolate is used to perform the native call. This could be the only reason that come to my mind that could cause something out of the ordinary. The main branch got rid of the Isolate and maybe worth to try it. It's not yet been published, so you should link this repo in your pubspec.yaml like:

  flutter_soloud:
    git:
      url: https://github.com/alnitak/flutter_soloud.git
      # Add this if you want to try with the incoming web support.
      # path: web

Also, have you tried to load the asset before calling _session.setActive(true)?

[sbauly]

I’m also experiencing issues with SoLoud’s interaction with audio_session, although I haven’t experienced the same issue as @lukepighetti.

It seems that the way SoLoud is handling interruptions internally could be interfering with attempts to use audio_session.

I made a quick example app that highlights the issue on iOS. The app uses one AudioSession instance with the exact same implementation for just_audio and SoLoud.

RPReplay_Final1721036954.mov

You can see from the video:

1. When an interruption occurs in SoLoud, the audio will not resume. You have to call SoLoud.instance.init() again and reload the asset before you can call play() successfully. This is not ideal, as it adds a lot of code and also means that the audio must be restarted again from the very beginning of the track after an interruption, losing the position of the audio that was being played.

2. When an interruption occurs in just_audio, the session correctly loses and regains focus as you would expect, maintaining the position of the audio.

Ideally, SoLoud would leave all interruption logic to be handled in audio_session, which would allow for one centralized place for session logic and handling of more complex scenarios. I’ve been using the main branch as suggested by @alnitak, and also tried using a local fork of SoLoud, commenting out some of the interruption logic I could find like ma_device_notification_type_interruption_began - but I haven’t been able to find a fix yet. This only appears to be a problem on iOS.

[alnitak]

Hi @sbauly, thank you very much for your findings and your time.

This seems related to #86. When I tried to look at this, the only notifications I saw were the start and stop. These notifications are listened here if you are curious.

I'll try to look at this issue again even if I cannot try it on iOS. Do you mind sharing your "quick example"?
Thanks again.

[sbauly]

Thanks for your speedy reply @alnitak!

Regarding #86 I’ve found that mixing sounds is possible by using the audio_session package and setting AVAudioSessionCategoryOptions.mixWithOthers when you configure the audio session, I’ve been able to accomplish that successfully with SoLoud.

The problem I’ve found is pausing/resuming SoLoud when an interruption event occurs. Android allows multiple audio sessions so it isn’t causing problems there, but on iOS I suspect that some of the AVAudioSession code in flutter_soloud is conflicting with audio_session in some way (possibly from miniaudio.h?). Thanks for your suggestion - I tried commenting out the start and stop methods but haven’t managed to find success yet, other than re-initializing SoLoud after each interruption as a temporary workaround.

I have published the example app here, hopefully it can be helpful

[ABespolov]

you just need to set AVAudioSessionCategoryAmbient

 if (![pAudioSession setCategory:AVAudioSessionCategoryAmbient withOptions:options error:nil]) {
                return MA_INVALID_OPERATION;  /* Failed to set session category to Ambient. */
         }

        if (!pConfig->coreaudio.noAudioSessionActivate) {
            if (![pAudioSession setActive:true error:nil]) {
                ma_log_postf(ma_context_get_log(pContext), MA_LOG_LEVEL_ERROR, "Failed to activate audio session.");
                return MA_FAILED_TO_INIT_BACKEND;
            }
        }

[sbauly]

Hi @ABespolov , thanks for the suggestion. Could you elaborate on this issue?

I've tried setting the avAudioSessionCategory to AVAudioSessionCategory.ambient in my AudioSessionConfiguration, but it has still had no effect on the issue of SoLoud being unable to resume playback after an interruption.

The only solution I have found so far is to re-initialize SoLoud after each interruption event. In the example app I published, the just_audio and SoLoud implementations use the exact same audio_session configuration, yet only SoLoud has an issue, which has lead me to believe this is a SoLoud specific issue, rather than a problem with the audio_session configuration I have been using.

[alnitak]

Feel free to reopen if the problem still exists or if there are new info.

[sbauly]

Hi @lukepighetti, did you manage to find a solution to your issues regarding SoLoud's interactions with audio_session?

Unfortunately I've still been experiencing similar problems.

[sbauly]

Hi @alnitak, I think this issue should be reopened. I've taken another look at it and been attempting to debug, but with little success so far.

I wonder if the issue stems from -(void)handle_interruption inside miniaudio.h:

Code: handle_interruption implementation

-(void)handle_interruption:(NSNotification*)pNotification
{
    NSInteger type = [[[pNotification userInfo] objectForKey:AVAudioSessionInterruptionTypeKey] integerValue];
    switch (type)
    {
        case AVAudioSessionInterruptionTypeBegan:
        {
            ma_log_postf(ma_device_get_log(m_pDevice), MA_LOG_LEVEL_INFO, "[Core Audio] Interruption: AVAudioSessionInterruptionTypeBegan\n");

            /*
            Core Audio will have stopped the internal device automatically, but we need explicitly
            stop it at a higher level to ensure miniaudio-specific state is updated for consistency.
            */
            ma_device_stop(m_pDevice);

            /*
            Fire the notification after the device has been stopped to ensure it's in the correct
            state when the notification handler is invoked.
            */
            ma_device__on_notification_interruption_began(m_pDevice);
        } break;

        case AVAudioSessionInterruptionTypeEnded:
        {
            ma_log_postf(ma_device_get_log(m_pDevice), MA_LOG_LEVEL_INFO, "[Core Audio] Interruption: AVAudioSessionInterruptionTypeEnded\n");
            ma_device__on_notification_interruption_ended(m_pDevice);
        } break;
    }
}

Core Audio appears to be stopping the device automatically, as well as ma_device_stop() being called.

SoLoud then seems to be completely deinitialized after this happens.

On the Flutter side, I've tried every different AVAudioSession configuration available, and none have made a difference.

Re-initializing SoLoud after an interruption is a temporary workaround, but it means having to load all the AudioSources again, as well as losing the position of the audio that was playing before the interruption event occurred.

This note in the documentation is also interesting:

The interruption notifications are used on mobile platforms for detecting when audio is interrupted
due to things like an incoming phone call. Currently this is only implemented on iOS. None of the
Android backends will report this notification.
*/
typedef void (* ma_device_notification_proc)(const ma_device_notification* pNotification);
/*

Since this only appears to be a problem on iOS (interruptions work great on Android), this suggests this could be an issue with the way notifications are being handled internally.

[alnitak]

Hi @sbauly,

thanks for your interest! I could take another look at this, but IIRC the SoLoud lib uses miniaudio just as a backend. The engine is SoLoud. I am not sure the engine uses handle_interruption but maybe something can be done.

IIRC only the stop and start notifications are triggered.

I reopen this issue and I will be immensely happy for any help!

[sbauly]

Thank you for the prompt response and for reopening, @alnitak!

Perhaps I am looking in the wrong place... I've found this issue, which seems to match the issue that I have been facing exactly, but looking at SoLoud's miniaudio.h file, it appears we already have the changes that fixed this.

If not miniaudio, where would you suggest to look? I have also been investigating the soloud_miniaudio.cpp code.

[alnitak]

Hi @sbauly,

lately, I have been investigating this issue.

You were right about the relevant code in miniaudio.h. When an interruption occurs, CoreAudio automatically stops, and consequently, miniaudio must also stop and update its state.

To test on iOS, I re-enabled the stateChangedEvents, which is called from on_notification in C. Now, ma_device_start() is invoked when receiving the finished state from an interruption.

Regarding device initialization, I added some parameters here. In the future, users will be able to include these parameters in their initialization settings.

It seems to be working well on iOS and macOS. I tested the reroute feature on macOS: for example, when I connect headphones, I receive the appropriate state. On the simulator, I added code to simulate an interruption, as I do not have an iPhone to test with.

I would greatly appreciate your help in testing this on an iPhone and a real MacOS if you are available! If anyone else is willing to help, it would be truly welcome as well.

For Android, I added a method channel for AudioFocus in Java (I am not familiar with Kotlin). I was able to test this, and all interruptions seem to work correctly: gain, gain transient, gain transient may duck, etc.

The handling of interruptions can be toggled on or off, allowing users to choose whether to use audio_session if they prefer.

You can find everything in the interruptions branch. I wrote audio_focus.dart for testing purposes.

Thank you in advance for your support!