78 lines
2.1 KiB
Markdown
78 lines
2.1 KiB
Markdown
# Local ASR Sidecar Protocol
|
|
|
|
Mastermind local-first mode expects speech-to-text to run as an external local
|
|
streaming service. The app connects to the service over WebSocket and sends
|
|
16 kHz mono PCM audio.
|
|
|
|
## Default Endpoint
|
|
|
|
```text
|
|
ws://127.0.0.1:8765/v1/asr/stream
|
|
```
|
|
|
|
The endpoint is configurable in the Local AI settings.
|
|
|
|
## Client Start Message
|
|
|
|
After the WebSocket opens, the app sends a JSON start frame:
|
|
|
|
```json
|
|
{
|
|
"type": "start",
|
|
"sampleRate": 16000,
|
|
"channels": 1,
|
|
"encoding": "pcm_s16le",
|
|
"language": "en-US"
|
|
}
|
|
```
|
|
|
|
After that, the app sends binary frames containing raw little-endian signed
|
|
16-bit PCM audio at 16 kHz.
|
|
|
|
## Sidecar Events
|
|
|
|
The sidecar should send JSON text frames:
|
|
|
|
```json
|
|
{ "type": "ready" }
|
|
```
|
|
|
|
```json
|
|
{ "type": "partial", "text": "intermediate transcript" }
|
|
```
|
|
|
|
```json
|
|
{ "type": "final", "text": "final transcript" }
|
|
```
|
|
|
|
```json
|
|
{ "type": "error", "error": "human-readable error" }
|
|
```
|
|
|
|
Only `final` transcript events enter local semantic reduction. `partial` events
|
|
are transient status and must not be persisted.
|
|
|
|
## v1 Scope
|
|
|
|
- Required language modes: English (`en-US`), Russian (`ru-RU`), and automatic
|
|
or mixed Russian-English recognition (`auto` or an implementation-equivalent
|
|
mode).
|
|
- Sidecar implementation is external to this repository.
|
|
- The app does not require a specific Nemotron, NeMo, Riva, or ONNX runtime as
|
|
long as the WebSocket protocol above is implemented.
|
|
- The endpoint must be loopback-only. LAN and internet ASR endpoints are not
|
|
Local Providers.
|
|
|
|
## Native macOS Client Compatibility
|
|
|
|
The Swift-native app uses the same WebSocket protocol as the Electron client.
|
|
Audio frames are raw 16 kHz mono signed 16-bit little-endian PCM.
|
|
|
|
Microphone and system audio use independent client connections so the sidecar
|
|
does not mix channel roles. The sidecar does not need to know whether a client
|
|
is Electron or Swift; Mastermind associates each connection with its Source.
|
|
|
|
Full transcripts are ephemeral sensitive input. The Swift client consumes a
|
|
`final` event, derives structured local context, and then releases transcript
|
|
content instead of storing it in Assistant History or the Context Graph.
|