Feat: Type-safe configureAVAudioSession function that ensures compatibility with AVAudioSessionModes and other customizations

[ChristopherGabba]

@hirbod This is my first time contributing to open-source... like ever. And I should also add that this is my first time coding in Obj-C.... So please feel free to crush my soul on how I did all of this lmfao...

The gist of it is, after reading all the docs: https://developer.apple.com/documentation/avfaudio/avaudiosession, it became clear that each AVAudioSessionCategory could only be used with specific AVAudioSessionModes. I was pretty thorough in making sure that I have the compatible modes with each category.

In order to make this type-safe, I needed to combine them into one function. I also used this to change the types of AVAudioSessionMode and AVAudioSessionCategory to an enum type so that could properly get good developer experience tool tips when mousing over each category. For example:

Additionally, you'll note here that the setCategory function had multiple different parameters

I took my best shot at implementing this level of customization.

An example of this new function would be:

await VolumeManager.configureAVAudioSession({
   category: AVAudioSessionCategory.PlayAndRecord,
   mode: AVAudioSessionMode.Record,
   policy: AVAudioSessionRoutSharingPolicy.LongFormAudio, // Options: https://developer.apple.com/documentation/avfaudio/avaudiosession/routesharingpolicy-swift.enum
   options: [AVAudioSessionCategoryOptions.MixWithOthers, AVAudioSessionCategoryOptions.AllowAirPlay], // Options: https://developer.apple.com/documentation/avfaudio/avaudiosession/categoryoptions-swift.struct 
   prefersNoInterruptionsFromSystemAlerts: true,
   prefersInterruptionOnRouteDisconnect: true,
   allowHapticsAndSystemSoundsDuringRecording: true
})

I also marked the setCategory function and the setMode function as deprecated using this new API.

Among other changes, it would appear that the previously had Alarm category is no longer listed, and the AudioProcessing category is listed as deprecated. https://developer.apple.com/documentation/avfaudio/avaudiosession/category-swift.struct

So I removed the Alarm option and flagged the AudioProcessing as deprecated. (Note that you actually did not have it in your native code anyway...).

One last comment... I noticed while studying react-native-vision-camera Audio Session setup he has this code:

      if #available(iOS 14.5, *) {
        // prevents the audio session from being interrupted by a phone call
        try audioSession.setPrefersNoInterruptionsFromSystemAlerts(true)
      }

      if #available(iOS 13.0, *) {
        // allow system sounds (notifications, calls, music) to play while recording
        try audioSession.setAllowHapticsAndSystemSoundsDuringRecording(true)
      }

So I added this plus one other option I ran across in the docs.

So... the kicker... I compiled the app and did a little bit of testing, I worked through a few compile errors I missed (small things) and everything seems to be not crashing and the function appears to be calling correctly on iOS.

That being said.... You may want to review my code and test it somehow... Sorry for not knowing a lot, I just really wanted to help out with what I was trying to accomplish...

[ChristopherGabba]

So of the 4 commits:

• The first commit 1ebd892 was my first attempt at the new API I wanted, but using the same Objective C functions you already had written
• The second commit 8a73b49 was my attempt to create a new objective-C function that allowed further customization
• The third commit was me updating the docs with my new custom function
• The fourth commit I forgot to remove the old code from the example app

For the record, I'm also quite stupid at git as well... sorry for the mess...

[ChristopherGabba]

@hirbod I got a little addicted to the cause and I spent all morning adding another method called getAVAudioSessionStatus(). That did what I was requesting in my feature request #51. I also rewrote the readme with a little better formatting, examples, and descriptions in the API section.

I also improved some of the hover-over notes for the original new function I created.

[hirbod]

Amazing work, keep going! :D
It will take a few weeks until I have time to review, test, and merge it, but I really appreciate your efforts here. A lot of these things have been on my mind for a long time already—definitely a very good direction. I was also thinking of fully migrating to Swift and Kotlin too. Honestly, we don’t have to deprecate old methods; we can just wipe them and release a new major instead.

[hirbod]

We don’t have to support anything before iOS 15.1. I usually follow Expo’s support path, and I don’t see any reason why we’d need to support anything this old either.

[hirbod]

I am really unexperienced here, but maybe we should dig a bit more and give Android some love too :D

[hirbod]

whats this plugin

[ChristopherGabba]

Awesome, I'll review all these comments and correct them. I can remove all the "AV"s and I'll clear the deprecations out entirely, good call on the next release. I'll also do some studying up on Android and its Audio sessions.

I'll also remove that eslint plugin, sorry when I was first trying to build the example app, I got that error saying "eslint-plugin-xxx" is missing", install it, so I did lmfao. I think I was just making a mistake and not running yarn prepare first. I'll clear that eslint plugin out, forgot it was in there.

[hirbod]

Actually I removed that AV comment but seems like you already read the email lol :D

[ChristopherGabba]

@hirbod I agree with removing AV if we want to merge functionalities with Android somehow.

It seems like Android has an equivalent to AVAudioSession called AudioManager.

It seems like AudioSession has some similarities like DoNotMix and MixWithOthers. I mean this API is.... massive...

I am sure we could come up with an equivalent configureAudioSession function for android as well, but some things would not overlap.

So keeping the future in mind, how would you like to proceed?

Option 1 (Two separate functions)

  configureAudioSessionIOS()
  configureAudioSessionAndroid()

Option 2 (Almost like Platform.select)

configureAudioSession({
   ios: {},
   android: {},
})

Option 3 (Merge functionalities - produces less customization)

// These props are made up as an example
 configureAudioSession({
  allowRecording: true,
  allowBluetooth: true,
  mixWithOthers: false,
})

I guess my vote as a package user is to allow as much customization as possible, given that is what this library is all about. The way I see it is expo-audio serves as a basic package and this could be a very advanced version of the library. I think of it sort of as expo-camera vs. react-native-vision-camera.

To your other point about converting to swift and kotlin, I think it's a great idea, however, I should not be the one to perform this. I tried the swift conversion this morning and just fed the whole file to GPT to convert and finally got it to build right but there was a bunch of synthax and @MainActor errors that I didn't understand, and I don't want to do something incorrectly.

[ChristopherGabba]

Sorry bro, I'm in some sort of endless commit loop I'm losing my mind lmfao....

[hirbod]

You can’t really use this package together with expo-video at the same time, as Expo overwrites the audio session. If you manipulate it, the video will pause, and if you start a video, it will overwrite the session. Expo-video has mixWithOthers implemented as prop.

But this is not a package bug or limitation; it’s how the audio session in iOS works. Each AV session is responsible for its own handling. expo-video would need a “headless mode” where it trusts that another package has already configured the audio session, but that’s too complicated for most people.

As soon as a video is running, it already has a session. You won’t be able to change expo-video’s audio session or parameters, as they are set in its own code.

    const player = useVideoPlayer(uri, (player) => {
      player.audioMixingMode = 'doNotMix'

      player.bufferOptions = {
        waitsToMinimizeStalling: false,
        minBufferForPlayback: 1,
        preferredForwardBufferDuration: 2,
      }
      player.loop = true
    })

What I do though is to register AppState listeners to inform the OS that the Audio session is "free again", which allows paused music to continue. We added this part to allow Apple Music or Spotify to continue when the App was backgrounded.

const onAppStateChange = (state: AppStateStatus) => {
  if (isWeb) return
  if (state === 'active') {
    VolumeManager.enable(true)
  }

  if (state === 'background') {
    VolumeManager.enable(false)
  }
}

And right before I initialize expo-video, I call

  useEffect(() => {
    // Set the audio session to ambient
    // Expo-Video is too slow to switch the audio session, so we use VolumeManager to set the audio session to ambient
    // this allows for seamless audio switching
    VolumeManager.setCategory('Ambient', false)
    return () => {
      // when we unmount this screen, we want to disable the Audio Session to allow the system to resume the audio
      VolumeManager.enable(false)
      VolumeManager.setActive(false)
    }
  }, [])

[ChristopherGabba]

Okay so just so I understand, I went into expo-video's VideoManager.swift file and performed this patch

  // MARK: - Audio Session Management

  internal func setAppropriateAudioSessionOrWarn() {
    // let audioSession = AVAudioSession.sharedInstance()
    // let audioMixingMode = findAudioMixingMode()
    // var audioSessionCategoryOptions: AVAudioSession.CategoryOptions = audioSession.categoryOptions

    // let isOutputtingAudio = videoPlayers.allObjects.contains { player in
    //   player.isPlaying && !player.isMuted
    // }
    // let anyPlayerShowsNotification = videoPlayers.allObjects.contains { player in
    //   player.showNowPlayingNotification
    // }

    // let shouldMixOverride = audioMixingMode == .mixWithOthers
    // let doNotMixOverride = audioMixingMode == .doNotMix
    // let shouldDuckOthers = audioMixingMode == .duckOthers && isOutputtingAudio

    // // The now playing notification won't be shown if we allow the audio to mix with others
    // let autoShouldMix = !isOutputtingAudio && !anyPlayerShowsNotification
    // let shouldMixWithOthers = shouldMixOverride || autoShouldMix

    // if shouldMixWithOthers && !shouldDuckOthers && !doNotMixOverride {
    //   audioSessionCategoryOptions.insert(.mixWithOthers)
    // } else {
    //   audioSessionCategoryOptions.remove(.mixWithOthers)
    // }

    // if shouldDuckOthers && !doNotMixOverride {
    //   audioSessionCategoryOptions.insert(.duckOthers)
    // } else {
    //   audioSessionCategoryOptions.remove(.duckOthers)
    // }

    // if audioSession.categoryOptions != audioSessionCategoryOptions {
    //   log.warn("Set audio session to", audioSessionCategoryOptions)
    //   do {
    //     try audioSession.setCategory(.playback, mode: .moviePlayback, options: audioSessionCategoryOptions)
    //   } catch {
    //     log.warn("Failed to set audio session category. This might cause issues with audio playback and Picture in Picture. \(error.localizedDescription)")
    //   }
    }

    // Make sure audio session is active if any video is playing
    // if isOutputtingAudio || doNotMixOverride {
      // do {
      //   try audioSession.setActive(true)
      // } catch {
      //   log.warn("Failed to activate the audio session. This might cause issues with audio playback. \(error.localizedDescription)")
      // }
    // }

    // The now playing notification requires correct audio session category, notify the manager of about the change.
    // NowPlayingManager.shared.refreshNowPlaying()
  // }

Would this not strip expo-video's ability to do anything to the audio session and leave it all to the user of the react-native-volume-manager?

[hirbod]

In theory, yes. You’d need to set an appropriate audio session before the video plays. You can see that this function prepares the category and calls .setActive() at the end.

[hirbod]

Just dropping this here to study—the audio session manager part:
https://github.com/TheWidlarzGroup/react-native-video/pull/4466/files

My library is definitely hard to use with others that already manipulate or overwrite the audio session. The main idea was to control volume, the silent switch, and hide the volume UI. To do this, you need a running audio session. I’m not saying this library doesn’t have its place—it’s definitely nice for controlling sessions when backgrounding/foregrounding the app—but chances are high that other libraries already have listeners for this (like the old expo-av, which was really bad).

Unless libraries like expo-video or react-native-video provide “audio session-free modes,” it’s going to be really hard to make them work together nicely. But yeah, even for volume listening, you need access to the audio session.

[ChristopherGabba]

So I have this really weird use case in my app... My app will display a video and record a user on camera at the same time. This means that expo-video and react-native-vision-camera both trigger audio sessions and SOMETIMES they collide with each other and cause a recording failure. I solve this problem by stripping those of their audio sessions and using this library... So far, I'm using my PR and it seems to be working well.

[hirbod]

Yeah but this was only possible because you patched both libs, right?

[ChristopherGabba]

That is unfortunately correct lol....

[hirbod]

We need to pitch this to @behenate and @lukmccall from Expo, and @mrousavy for Vision Camera, that a “no audio session” mode would be very helpful, where the user is responsible for activating/managing the audio session. Ideally, one could pass a flag to those libs (also React Native Video—CC @KrzysztofMoch).

Could you guys consider a prop or method that completely avoids audio handling in your libraries? I’ll call this the “professional mode.” This would allow users with specialized audio handling needs to control the entire flow. Right now, it’s basically impossible since most methods in all the libs either activate, enable, or disable the audio session or change the category. I know this is beneficial in most cases for the average user, and the ask is not to remove it but rather to introduce a flag that disables audio management on your end and leaves it to the user.

[ChristopherGabba]

I agree.
Maybe just a prop that says:

// Defaults to false (for obvious reasons)
<Camera
   disableAudioSessionManagement={true} // Allows developers to manage custom sessions on their own
/>
 
<Video
   disableAudioSessionManagement={true}  // Allows developers to manage custom sessions on their own
/>

I submitted this about a year ago:
mrousavy/react-native-vision-camera#2581

Some related issues:
mrousavy/react-native-vision-camera#2350

Honestly should be fairly simply PR / additions. Basically just an if statement.

[ChristopherGabba]

@hirbod I was thinking a nice feature adjustment to this new API I've been working on would be like some basic configurations pre-built?

Like:

configureAudioSession(PremadeConfig.IgnoreSilentSwitch)
configureAudioSession(PremadeConfig.DuckBackgroundAudio)

[KrzysztofMoch]

Could you guys consider a prop or method that completely avoids audio handling in your libraries? I’ll call this the “professional mode.” This would allow users with specialized audio handling needs to control the entire flow.

Yeah this should be simple to do. It should affect all video components, so I am thinking about flag DISABLE_AUDIO_SESSION_MANAGMENT in Pod/gradle, to allow users to disable it for all components on installation step. WDYT?

Prop approach is also fine for me

Also I think we should add asserts in (video/vision-camera) in places where we require specific config, to avoid bugs and misconceptions

[mrousavy]

Without reading any of the comments yet (i'm on vacation in japan), an ideal scenario would be if this would be a Nitro Module, and the audio session is a Hybrid Object.

Then we can use it as a common type and VisionCamera can interop with it seamlessly - same as other libs.

[mrousavy]

Disabling audio session management by VisionCamera/RNVideo is also okay I guess - but what would be really cool and most powerful is definitely just passing around a configured Audio Session that's gon be used.

I guess approacg #1 is simplest tho

[mrousavy]

@KrzysztofMoch don't use build flags for that

[hirbod]

While I can see why a Hybrid Object might be tempting, I’d look for something more generic and easier for now. Having a mode offered by RNV, Expo Video (+Audio), RNVC, and possibly the next big one—React Native Track Player—where the audio session is “skipped” seems relatively easy to implement for all parties without much overhead. By default, all libraries continue managing audio sessions, as that’s likely still the best approach for the ecosystem. However, having a professional mode for more sophisticated routing, where an “audio-less” mode is available, would be amazing.

Very happy to see first positive signals here by @mrousavy (enjoy your vacation man!) and @KrzysztofMoch. I'll try to chat with @behenate via Slack to get a signal / how they think about.