Implement audio chat provider #227
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
julien-nc
force-pushed
the
enh/noid/voice2voice
branch
from
d285a78 to
c9fa7c1
Compare
lukasdotcom
reviewed
Jul 2, 2025
julien-nc
force-pushed
the
enh/noid/voice2voice
branch
from
a8ac1af to
264930e
Compare
lukasdotcom
reviewed
Jul 3, 2025
julien-nc
force-pushed
the
enh/noid/voice2voice
branch
3 times, most recently
from
9adb87e to
720d619
Compare
|
A few new things:
|
lukasdotcom
requested changes
Jul 9, 2025
julien-nc
force-pushed
the
enh/noid/voice2voice
branch
from
720d619 to
688d951
Compare
kyteinsky
approved these changes
Jul 10, 2025
| throw new RuntimeException($serviceName . ' text to speech generation failed with: ' . $e->getMessage()); | ||
| } | ||
| } else { | ||
| $output = base64_decode($message['audio']['data']); |
There was a problem hiding this comment.
this audio data expires at a certain time expires_at. Should we check that and fallback to the above TTS approach to get the new audio and audio id?
Signed-off-by: Julien Veyssier <[email protected]>
Signed-off-by: Julien Veyssier <[email protected]>
Signed-off-by: Julien Veyssier <[email protected]>
Signed-off-by: Julien Veyssier <[email protected]>
Signed-off-by: Julien Veyssier <[email protected]>
…input if using chat endpoint Signed-off-by: Julien Veyssier <[email protected]>
Signed-off-by: Julien Veyssier <[email protected]>
…he server task type is available, handle the case when the chat endpoint does not return audio Signed-off-by: Julien Veyssier <[email protected]>
Signed-off-by: Julien Veyssier <[email protected]>
… multimodal model Signed-off-by: Julien Veyssier <[email protected]>
Signed-off-by: Julien Veyssier <[email protected]>
julien-nc
force-pushed
the
enh/noid/voice2voice
branch
from
6a48831 to
3ea9b77
Compare
lukasdotcom
approved these changes
Jul 10, 2025
|
@kyteinsky @lukasdotcom The expiration is checked where the history array is built. Currently the only place where we do that is when the assistant schedules an audio chat task, in https://github.com/nextcloud/assistant/pull/292/files#diff-bacc1ea613cd8effdacf1a14dd4d819266ed4315d83dc81e94cad93b8d071ff4R591-R592 In integration_openai, we won't receive the expiration date in the history input, we can't check it here. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Requires nextcloud/assistant#291 (for the mp3 recording).
Related with nextcloud/server#53759
If connected to OpenAI, we can use the chat completion endpoint with the
gpt-4o-audio-previewmodel which accepts audio messages as input.If not connected to OpenAI, it's safer to split the process in: STT -> LLM -> TTS.