native first
This commit is contained in:
Илья Глазунов
@@ -0,0 +1,60 @@
|
||||
# Local ASR Sidecar Protocol
|
||||
|
||||
Mastermind local-first mode expects speech-to-text to run as an external local
|
||||
streaming service. The app connects to the service over WebSocket and sends
|
||||
16 kHz mono PCM audio.
|
||||
|
||||
## Default Endpoint
|
||||
|
||||
```text
|
||||
ws://127.0.0.1:8765/v1/asr/stream
|
||||
```
|
||||
|
||||
The endpoint is configurable in the Local AI settings.
|
||||
|
||||
## Client Start Message
|
||||
|
||||
After the WebSocket opens, the app sends a JSON start frame:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "start",
|
||||
"sampleRate": 16000,
|
||||
"channels": 1,
|
||||
"encoding": "pcm_s16le",
|
||||
"language": "en-US"
|
||||
}
|
||||
```
|
||||
|
||||
After that, the app sends binary frames containing raw little-endian signed
|
||||
16-bit PCM audio at 16 kHz.
|
||||
|
||||
## Sidecar Events
|
||||
|
||||
The sidecar should send JSON text frames:
|
||||
|
||||
```json
|
||||
{ "type": "ready" }
|
||||
```
|
||||
|
||||
```json
|
||||
{ "type": "partial", "text": "intermediate transcript" }
|
||||
```
|
||||
|
||||
```json
|
||||
{ "type": "final", "text": "final transcript" }
|
||||
```
|
||||
|
||||
```json
|
||||
{ "type": "error", "error": "human-readable error" }
|
||||
```
|
||||
|
||||
Only `final` transcript events are sent to the local LLM. `partial` events are
|
||||
shown as status text.
|
||||
|
||||
## v1 Scope
|
||||
|
||||
- STT target: English, `en-US`.
|
||||
- Sidecar implementation is external to this repository.
|
||||
- The app does not require a specific Nemotron, NeMo, Riva, or ONNX runtime as
|
||||
long as the WebSocket protocol above is implemented.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Native macOS POC Results
|
||||
|
||||
## Implemented
|
||||
|
||||
- SwiftPM-based macOS POC under `native/MastermindPOC`.
|
||||
- AppKit lifecycle with a menu bar status item.
|
||||
- Transparent non-activating overlay window with hide/show and click-through controls.
|
||||
- Overlay window uses `NSWindowSharingType.none` and ScreenCaptureKit filtering excludes the current process windows from app-owned capture.
|
||||
- ScreenCaptureKit screen stream proof with frame counters.
|
||||
- ScreenCaptureKit system audio proof with best-effort PCM extraction to 16 kHz mono signed 16-bit little-endian frames.
|
||||
- AVAudioEngine microphone proof with 16 kHz mono signed 16-bit little-endian frames.
|
||||
- Separate microphone and system audio counters in the overlay.
|
||||
- Compact HUD controls for Settings, Hide, and Quit.
|
||||
- Draggable overlay title area.
|
||||
- Settings panel with persisted overlay background opacity.
|
||||
- Swift unit tests for status labels, PCM frame metadata, PCM S16LE encoding, and separate audio counters.
|
||||
- App bundle build script at `native/MastermindPOC/scripts/build-app.sh`.
|
||||
|
||||
## Verified Automatically
|
||||
|
||||
- `swift test` passes for `MastermindPOCCore`.
|
||||
- `swift build --product MastermindPOC` compiles the native executable.
|
||||
- `native/MastermindPOC/scripts/build-app.sh` produces `native/MastermindPOC/build/MastermindPOC.app`.
|
||||
- Existing Electron local provider tests pass when run outside the managed sandbox restrictions.
|
||||
|
||||
## Manual Verification Needed
|
||||
|
||||
- Launch `native/MastermindPOC/build/MastermindPOC.app`.
|
||||
- Confirm the menu bar item is visible while the app runs.
|
||||
- Confirm the overlay appears locally and can be hidden.
|
||||
- Confirm the overlay can be dragged by its top title bar.
|
||||
- Confirm the overlay Settings button opens the settings panel.
|
||||
- Confirm the opacity slider changes only the HUD background opacity and persists across relaunches.
|
||||
- Confirm the overlay Quit button exits the app.
|
||||
- Confirm click-through lets pointer events pass to apps underneath.
|
||||
- Grant Screen Recording permission when macOS prompts, then restart the POC if needed.
|
||||
- Start `Screen + System Audio` and confirm screen frame and system PCM counters increase.
|
||||
- Start `Microphone` and confirm microphone PCM counters increase while speaking.
|
||||
- Confirm `Pause All Capture` stops screen, system audio, and microphone updates.
|
||||
- Confirm app-owned screen capture does not include the Mastermind overlay.
|
||||
|
||||
## Boundary
|
||||
|
||||
This POC is not a stealth or anti-detection system. It shows a menu bar item while running, requests normal macOS permissions, and only excludes its own overlay from the context it captures for itself.
|
||||
@@ -0,0 +1,825 @@
|
||||
# macOS Native AI Companion Reconstruction Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Reconstruct Mastermind from an Electron interview-helper-shaped app into a native macOS AI companion with reliable overlay, screen context, system audio, voice interaction, and explicit trust indicators.
|
||||
|
||||
**Architecture:** Build a Swift/AppKit-first macOS host that owns windows, capture, audio, permissions, menu bar status, and privacy boundaries. Keep AI providers and local sidecars behind protocol boundaries so the rewrite can progress incrementally instead of becoming a risky full replacement.
|
||||
|
||||
**Tech Stack:** Swift, AppKit, SwiftUI where appropriate, ScreenCaptureKit, AVAudioEngine, CoreAudio, Keychain, Application Support storage, URLSession streaming, WebSocket, local ASR sidecar protocol, OpenAI-compatible HTTP/SSE, optional Electron bridge during migration.
|
||||
|
||||
---
|
||||
|
||||
## Product Thesis
|
||||
|
||||
Mastermind should become an **AI companion for your Mac**, not a hidden interview helper.
|
||||
|
||||
The assistant is a personal, visible-to-the-user HUD and agent system. It can help during calls, presentations, coding, writing, research, and everyday computer use. It may stay out of the content the user is presenting or sharing, just like presenter notes, a timer, captions, or a local command palette.
|
||||
|
||||
The product should not implement stealth, anti-detection, process hiding, monitoring bypass, proctoring bypass, or policy evasion. If a workplace, school, exam, or managed system prohibits AI assistants, Mastermind should not help users hide that it is running.
|
||||
|
||||
## Core Boundary
|
||||
|
||||
**Allowed:** The assistant window does not appear in the user's own screen context, screenshots, screen-share content, or presentation output when technically possible.
|
||||
|
||||
**Not allowed:** Hiding the app process, bundle identifier, permissions, network use, Accessibility use, Screen Recording use, microphone use, or audio capture use from operating-system tools or managed environments.
|
||||
|
||||
Practical phrasing:
|
||||
|
||||
> Mastermind does not hide from the user or the system. It only avoids contaminating the content the user intentionally shares or asks the assistant to analyze.
|
||||
|
||||
## Target User Experience
|
||||
|
||||
### Normal Companion Mode
|
||||
|
||||
The assistant is available through voice, keyboard, and a small native overlay. It can answer questions, remember context, summarize active work, and trigger actions.
|
||||
|
||||
### Meeting Mode
|
||||
|
||||
The assistant can listen to microphone and system audio with clear status indicators. It can summarize discussion, draft follow-ups, keep agenda state, and help the user stay oriented.
|
||||
|
||||
### Presentation Mode
|
||||
|
||||
The user can share slides, a browser, or an app window while seeing local prompts, timing, plan notes, likely objections, and speaker guidance in Mastermind's HUD. The shared audience does not need to see the HUD.
|
||||
|
||||
### Screen Context Mode
|
||||
|
||||
Screen context is opt-in. The user explicitly asks the assistant to look at the screen, capture a snapshot, or follow a bounded live stream. Mastermind excludes its own UI from that context.
|
||||
|
||||
### Idle Mode
|
||||
|
||||
When not actively listening, viewing, or processing, Mastermind stays quiet and visibly idle in the menu bar.
|
||||
|
||||
## Trust Model
|
||||
|
||||
Trust is a first-class feature, not a settings afterthought.
|
||||
|
||||
Required visible signals:
|
||||
|
||||
- Menu bar icon is always present while the app runs.
|
||||
- Status reflects actual state: idle, listening, reading screen, processing, paused, permission problem.
|
||||
- Mic capture, system audio capture, and screen context have separate indicators.
|
||||
- The user can pause capture immediately.
|
||||
- The app exposes a local activity log showing recent context use: microphone, system audio, screen snapshot, screen stream, provider request.
|
||||
- The app does not provide stealth labels, anti-detection switches, or hidden-running modes.
|
||||
|
||||
Recommended status vocabulary:
|
||||
|
||||
- `Idle`
|
||||
- `Listening`
|
||||
- `System Audio`
|
||||
- `Screen Snapshot`
|
||||
- `Screen Stream`
|
||||
- `Agent Working`
|
||||
- `Paused`
|
||||
- `Permission Needed`
|
||||
- `Error`
|
||||
|
||||
## Current Project Context
|
||||
|
||||
Current repository shape:
|
||||
|
||||
- Electron Forge app with JavaScript entry point at `src/index.js`.
|
||||
- Window management and global shortcuts in `src/utils/window.js`.
|
||||
- AI provider/session logic in `src/utils/gemini.js`, `src/utils/localai.js`, and `src/utils/localProviders.js`.
|
||||
- Local ASR sidecar protocol already documented in `docs/local-sidecar-protocol.md`.
|
||||
- JSON storage in `src/storage.js`.
|
||||
- Lit-based UI under `src/components`.
|
||||
|
||||
Important existing behavior to preserve:
|
||||
|
||||
- Always-on-top assistant window.
|
||||
- Global keyboard shortcuts.
|
||||
- Click-through toggle.
|
||||
- Hide/show assistant.
|
||||
- Session history.
|
||||
- Local mode with ASR sidecar.
|
||||
- BYOK provider support.
|
||||
- OpenAI-compatible provider support.
|
||||
- Groq/Gemini provider support.
|
||||
- Configurable prompt/profile/language.
|
||||
|
||||
Important behavior to replace:
|
||||
|
||||
- Electron-owned transparent window quirks.
|
||||
- Chromium `getDisplayMedia` screen capture dependency.
|
||||
- Fragile loopback/system-audio capture.
|
||||
- Renderer-local storage access from main process.
|
||||
- `nodeIntegration: true` and `contextIsolation: false`.
|
||||
- Hidden coupling between capture, transcription, provider routing, and renderer events.
|
||||
|
||||
## Reconstruction Strategy
|
||||
|
||||
Recommended path: **Swift-native host first, full app migration second**.
|
||||
|
||||
Do not start with a full SwiftUI rewrite of every screen. The riskiest parts are windowing, screen capture, audio capture, permissions, and trust indicators. Prove those first in a native macOS host, then migrate provider and UI surfaces incrementally.
|
||||
|
||||
Migration shape:
|
||||
|
||||
```text
|
||||
Phase 1: Native capability proof
|
||||
Swift/AppKit overlay
|
||||
menu bar status item
|
||||
ScreenCaptureKit screen context
|
||||
ScreenCaptureKit system audio
|
||||
AVAudioEngine microphone audio
|
||||
|
||||
Phase 2: Native companion shell
|
||||
state machine
|
||||
permissions
|
||||
trust indicators
|
||||
local sidecar bridge
|
||||
provider boundary
|
||||
|
||||
Phase 3: Agentic assistant
|
||||
modes
|
||||
memory
|
||||
tools
|
||||
activity log
|
||||
meeting and presentation flows
|
||||
|
||||
Phase 4: Retire Electron
|
||||
migrate settings/history
|
||||
package/notarize Swift app
|
||||
deprecate Electron runtime
|
||||
```
|
||||
|
||||
## Proposed Native Architecture
|
||||
|
||||
```text
|
||||
Mastermind.app
|
||||
AppCoordinator
|
||||
owns lifecycle, mode, permissions, menu bar status
|
||||
|
||||
OverlayWindowController
|
||||
owns transparent HUD, click-through, Spaces behavior, focus behavior
|
||||
|
||||
CaptureCoordinator
|
||||
owns screen context and system audio through ScreenCaptureKit
|
||||
|
||||
MicrophoneCaptureEngine
|
||||
owns microphone stream through AVAudioEngine
|
||||
|
||||
AudioPipeline
|
||||
owns channel separation, resampling, VAD, PCM frame output
|
||||
|
||||
AssistantSession
|
||||
owns active conversation, transcript, memory references, provider routing
|
||||
|
||||
ProviderClients
|
||||
Gemini, Groq, OpenAI-compatible, local LLM, local ASR sidecar
|
||||
|
||||
TrustStatusStore
|
||||
owns visible state, activity log, pause/resume, capture indicators
|
||||
|
||||
Storage
|
||||
Application Support for non-secret app data
|
||||
Keychain for secrets
|
||||
```
|
||||
|
||||
## Agentic System Direction
|
||||
|
||||
The future assistant should be built around modes and tools, not a single chat box.
|
||||
|
||||
Core modes:
|
||||
|
||||
- General Companion
|
||||
- Meeting Assistant
|
||||
- Presentation Coach
|
||||
- Coding Assistant
|
||||
- Research Assistant
|
||||
- Focus Assistant
|
||||
|
||||
Core agent abilities:
|
||||
|
||||
- Listen and summarize.
|
||||
- Answer conversationally.
|
||||
- See screen only when requested.
|
||||
- Track agenda or presentation plan.
|
||||
- Draft follow-up notes.
|
||||
- Remember user preferences.
|
||||
- Use local tools after explicit permission.
|
||||
- Explain what context it used.
|
||||
|
||||
Agent boundaries:
|
||||
|
||||
- No autonomous destructive actions.
|
||||
- No hidden capture.
|
||||
- No silent screen streaming.
|
||||
- No policy bypass tooling.
|
||||
- No stealth process behavior.
|
||||
|
||||
## File Structure Target
|
||||
|
||||
Future native app structure:
|
||||
|
||||
```text
|
||||
native/Mastermind/
|
||||
Mastermind.xcodeproj
|
||||
Mastermind/
|
||||
App/
|
||||
MastermindApp.swift
|
||||
AppCoordinator.swift
|
||||
AppMode.swift
|
||||
PermissionState.swift
|
||||
Status/
|
||||
MenuBarController.swift
|
||||
TrustStatus.swift
|
||||
ActivityLog.swift
|
||||
Overlay/
|
||||
OverlayWindowController.swift
|
||||
OverlayRootView.swift
|
||||
OverlayViewModel.swift
|
||||
Capture/
|
||||
CaptureCoordinator.swift
|
||||
ScreenContextCapture.swift
|
||||
SystemAudioCapture.swift
|
||||
CaptureExclusionPolicy.swift
|
||||
Audio/
|
||||
MicrophoneCaptureEngine.swift
|
||||
AudioPipeline.swift
|
||||
PCMFrame.swift
|
||||
VoiceActivityDetector.swift
|
||||
Assistant/
|
||||
AssistantSession.swift
|
||||
AssistantMode.swift
|
||||
AssistantEvent.swift
|
||||
AssistantMemory.swift
|
||||
Providers/
|
||||
ProviderClient.swift
|
||||
LocalAsrSidecarClient.swift
|
||||
OpenAICompatibleClient.swift
|
||||
GeminiClient.swift
|
||||
GroqClient.swift
|
||||
Storage/
|
||||
AppStorageStore.swift
|
||||
KeychainStore.swift
|
||||
MigrationStore.swift
|
||||
Settings/
|
||||
SettingsWindowController.swift
|
||||
SettingsRootView.swift
|
||||
Presentation/
|
||||
PresentationPlan.swift
|
||||
PresentationCoach.swift
|
||||
```
|
||||
|
||||
Existing Electron files remain read-only during the proof phase except for bridge points explicitly required by a migration task.
|
||||
|
||||
## Task 1: Write Product Reconstruction Charter
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `docs/product/macos-native-ai-companion-charter.md`
|
||||
|
||||
- [ ] **Step 1: Create the product charter**
|
||||
|
||||
Write a concise charter with these sections:
|
||||
|
||||
```markdown
|
||||
# macOS Native AI Companion Charter
|
||||
|
||||
## Positioning
|
||||
|
||||
Mastermind is a native macOS AI companion for everyday computer work, meetings, presentations, research, and focused execution.
|
||||
|
||||
## What It Is
|
||||
|
||||
- A visible personal assistant for the user.
|
||||
- A voice-first and context-aware companion.
|
||||
- A local HUD for notes, guidance, and agent status.
|
||||
- A privacy-conscious screen and audio context tool.
|
||||
|
||||
## What It Is Not
|
||||
|
||||
- A hidden interview helper.
|
||||
- A proctoring bypass tool.
|
||||
- An anti-detection tool.
|
||||
- A process-hiding tool.
|
||||
- A tool for hiding AI use from managed systems that prohibit it.
|
||||
|
||||
## Core Promise
|
||||
|
||||
Mastermind does not hide from the user or the system. It only avoids contaminating the content the user intentionally shares or asks the assistant to analyze.
|
||||
|
||||
## Trust Requirements
|
||||
|
||||
- Show a menu bar status item whenever running.
|
||||
- Show when microphone, system audio, or screen context is active.
|
||||
- Allow immediate pause.
|
||||
- Keep a local activity log of context use.
|
||||
- Store secrets in Keychain.
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Review the charter language**
|
||||
|
||||
Confirm the document includes the phrases `AI companion for your Mac`, `does not hide from the user or the system`, and `visible personal assistant`.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/product/macos-native-ai-companion-charter.md
|
||||
git commit -m "docs: define macos ai companion direction"
|
||||
```
|
||||
|
||||
## Task 2: Build Native Capability Proof Project
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind.xcodeproj`
|
||||
- Create: `native/Mastermind/Mastermind/App/MastermindApp.swift`
|
||||
- Create: `native/Mastermind/Mastermind/App/AppCoordinator.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Status/MenuBarController.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Overlay/OverlayWindowController.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Capture/CaptureCoordinator.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Audio/MicrophoneCaptureEngine.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Audio/AudioPipeline.swift`
|
||||
|
||||
- [ ] **Step 1: Create a minimal native macOS app**
|
||||
|
||||
Create a macOS app target named `Mastermind`. Use Swift, AppKit lifecycle, and SwiftUI only for simple views.
|
||||
|
||||
- [ ] **Step 2: Add a menu bar status item**
|
||||
|
||||
Implement a menu bar item that always appears while the app runs. The menu must include:
|
||||
|
||||
```text
|
||||
Mastermind: Idle
|
||||
Pause All Capture
|
||||
Show Assistant
|
||||
Hide Assistant
|
||||
Settings
|
||||
Quit Mastermind
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Add a transparent overlay window**
|
||||
|
||||
Create an AppKit-controlled floating overlay window with these properties:
|
||||
|
||||
```text
|
||||
borderless
|
||||
transparent background
|
||||
always on top
|
||||
visible across Spaces
|
||||
does not steal focus when shown
|
||||
can become click-through
|
||||
can be hidden and restored
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add screen context proof**
|
||||
|
||||
Use ScreenCaptureKit to capture the main display. The proof succeeds only when captured frames exclude the assistant overlay.
|
||||
|
||||
- [ ] **Step 5: Add system audio proof**
|
||||
|
||||
Use ScreenCaptureKit audio output to receive system audio buffers.
|
||||
|
||||
- [ ] **Step 6: Add microphone proof**
|
||||
|
||||
Use AVAudioEngine to receive microphone PCM buffers.
|
||||
|
||||
- [ ] **Step 7: Add 16 kHz PCM output proof**
|
||||
|
||||
Convert microphone and system audio streams into 16 kHz signed 16-bit PCM frames.
|
||||
|
||||
- [ ] **Step 8: Manual verification**
|
||||
|
||||
Run the native app and verify:
|
||||
|
||||
```text
|
||||
menu bar item is visible
|
||||
overlay is visible locally
|
||||
overlay can become click-through
|
||||
overlay does not appear in app-owned captured frames
|
||||
system audio buffers arrive when another app plays audio
|
||||
microphone buffers arrive when speaking
|
||||
capture can be paused immediately
|
||||
```
|
||||
|
||||
- [ ] **Step 9: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind
|
||||
git commit -m "feat: add native macos capture proof"
|
||||
```
|
||||
|
||||
## Task 3: Define Trust State Machine
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind/Status/TrustStatus.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Status/ActivityLog.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/Status/MenuBarController.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/App/AppCoordinator.swift`
|
||||
|
||||
- [ ] **Step 1: Define trust states**
|
||||
|
||||
Use these states exactly:
|
||||
|
||||
```swift
|
||||
enum TrustStatus: Equatable {
|
||||
case idle
|
||||
case listening
|
||||
case systemAudio
|
||||
case screenSnapshot
|
||||
case screenStream
|
||||
case agentWorking
|
||||
case paused
|
||||
case permissionNeeded(String)
|
||||
case error(String)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Define context activity events**
|
||||
|
||||
Use these events exactly:
|
||||
|
||||
```swift
|
||||
enum ActivityEventKind: String, Codable {
|
||||
case microphoneStarted
|
||||
case microphoneStopped
|
||||
case systemAudioStarted
|
||||
case systemAudioStopped
|
||||
case screenSnapshotCaptured
|
||||
case screenStreamStarted
|
||||
case screenStreamStopped
|
||||
case providerRequestStarted
|
||||
case providerRequestFinished
|
||||
case capturePaused
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Wire status to menu bar copy**
|
||||
|
||||
Map trust states to visible labels:
|
||||
|
||||
```text
|
||||
idle -> Mastermind: Idle
|
||||
listening -> Mastermind: Listening
|
||||
systemAudio -> Mastermind: System Audio
|
||||
screenSnapshot -> Mastermind: Screen Snapshot
|
||||
screenStream -> Mastermind: Screen Stream
|
||||
agentWorking -> Mastermind: Agent Working
|
||||
paused -> Mastermind: Paused
|
||||
permissionNeeded -> Mastermind: Permission Needed
|
||||
error -> Mastermind: Error
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify state changes manually**
|
||||
|
||||
Trigger each capture path and confirm the menu bar label changes before any capture data leaves the machine.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind/Mastermind/Status native/Mastermind/Mastermind/App
|
||||
git commit -m "feat: add native trust status model"
|
||||
```
|
||||
|
||||
## Task 4: Preserve Local ASR Sidecar Boundary
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind/Providers/ProviderClient.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Providers/LocalAsrSidecarClient.swift`
|
||||
- Modify: `docs/local-sidecar-protocol.md`
|
||||
|
||||
- [ ] **Step 1: Define provider boundary**
|
||||
|
||||
Use this protocol as the native client boundary:
|
||||
|
||||
```swift
|
||||
protocol ProviderClient {
|
||||
associatedtype Event
|
||||
|
||||
func start() async throws
|
||||
func stop() async
|
||||
var events: AsyncStream<Event> { get }
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Mirror the current ASR sidecar protocol**
|
||||
|
||||
Implement the native client against the existing WebSocket flow:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "start",
|
||||
"sampleRate": 16000,
|
||||
"channels": 1,
|
||||
"encoding": "pcm_s16le",
|
||||
"language": "en-US"
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Preserve event semantics**
|
||||
|
||||
Support these sidecar events:
|
||||
|
||||
```text
|
||||
ready
|
||||
partial
|
||||
final
|
||||
error
|
||||
```
|
||||
|
||||
Only `final` transcript events should enter the assistant response pipeline by default.
|
||||
|
||||
- [ ] **Step 4: Document native compatibility**
|
||||
|
||||
Add a section to `docs/local-sidecar-protocol.md`:
|
||||
|
||||
```markdown
|
||||
## Native macOS Client Compatibility
|
||||
|
||||
The Swift-native app uses the same WebSocket protocol as the Electron app. Audio frames are sent as raw 16 kHz mono signed 16-bit little-endian PCM. The sidecar does not need to know whether the client is Electron or Swift.
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind/Mastermind/Providers docs/local-sidecar-protocol.md
|
||||
git commit -m "feat: add native local asr sidecar client"
|
||||
```
|
||||
|
||||
## Task 5: Design Native Audio Pipeline
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind/Audio/PCMFrame.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Audio/AudioPipeline.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Audio/VoiceActivityDetector.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/Audio/MicrophoneCaptureEngine.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/Capture/CaptureCoordinator.swift`
|
||||
|
||||
- [ ] **Step 1: Define PCM frame format**
|
||||
|
||||
Use this data model:
|
||||
|
||||
```swift
|
||||
struct PCMFrame: Equatable {
|
||||
let source: AudioSource
|
||||
let sampleRate: Int
|
||||
let channels: Int
|
||||
let pcmS16LE: Data
|
||||
let timestamp: Date
|
||||
}
|
||||
|
||||
enum AudioSource: String {
|
||||
case microphone
|
||||
case system
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Keep microphone and system audio separate**
|
||||
|
||||
The pipeline should not mix microphone and system audio before transcription. Separate streams preserve future diarization and meeting-context quality.
|
||||
|
||||
- [ ] **Step 3: Resample to 16 kHz**
|
||||
|
||||
Every frame sent to local ASR must be:
|
||||
|
||||
```text
|
||||
sample rate: 16000
|
||||
channels: 1
|
||||
encoding: signed 16-bit little-endian PCM
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add VAD boundary**
|
||||
|
||||
Voice activity detection should emit speech segments instead of forcing every buffer into transcription.
|
||||
|
||||
- [ ] **Step 5: Manual verification**
|
||||
|
||||
Verify:
|
||||
|
||||
```text
|
||||
mic frames continue when system audio is silent
|
||||
system frames continue when mic is silent
|
||||
both streams can be paused together
|
||||
either stream can be disabled independently
|
||||
sidecar receives valid 16 kHz PCM frames
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind/Mastermind/Audio native/Mastermind/Mastermind/Capture
|
||||
git commit -m "feat: add native audio pipeline model"
|
||||
```
|
||||
|
||||
## Task 6: Migrate Secrets and Storage Boundaries
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind/Storage/KeychainStore.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Storage/AppStorageStore.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Storage/MigrationStore.swift`
|
||||
- Reference: `src/storage.js`
|
||||
|
||||
- [ ] **Step 1: Map current storage**
|
||||
|
||||
Preserve these current logical groups:
|
||||
|
||||
```text
|
||||
config
|
||||
credentials
|
||||
preferences
|
||||
keybinds
|
||||
limits
|
||||
history
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Move secrets to Keychain**
|
||||
|
||||
Store these values in Keychain:
|
||||
|
||||
```text
|
||||
Gemini API key
|
||||
Groq API key
|
||||
OpenAI-compatible API key
|
||||
local LLM API key
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Store non-secret data in Application Support**
|
||||
|
||||
Store these values in Application Support:
|
||||
|
||||
```text
|
||||
preferences
|
||||
profiles
|
||||
keybinds
|
||||
history
|
||||
limits
|
||||
activity log
|
||||
presentation plans
|
||||
assistant memory references
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add one-way migration**
|
||||
|
||||
Read existing Electron JSON files from:
|
||||
|
||||
```text
|
||||
~/Library/Application Support/cheating-daddy-config
|
||||
```
|
||||
|
||||
Import values into native stores. Leave the old files untouched.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind/Mastermind/Storage
|
||||
git commit -m "feat: add native storage migration boundary"
|
||||
```
|
||||
|
||||
## Task 7: Build Presentation Coach Mode
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind/Presentation/PresentationPlan.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Presentation/PresentationCoach.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/Assistant/AssistantMode.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/Overlay/OverlayRootView.swift`
|
||||
|
||||
- [ ] **Step 1: Define presentation plan model**
|
||||
|
||||
Use this model:
|
||||
|
||||
```swift
|
||||
struct PresentationPlan: Codable, Equatable {
|
||||
var title: String
|
||||
var sections: [PresentationSection]
|
||||
}
|
||||
|
||||
struct PresentationSection: Codable, Equatable {
|
||||
var title: String
|
||||
var talkingPoints: [String]
|
||||
var expectedDurationSeconds: Int
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add presentation mode**
|
||||
|
||||
Add a mode named `presentationCoach`.
|
||||
|
||||
- [ ] **Step 3: Show local guidance in HUD**
|
||||
|
||||
HUD should show:
|
||||
|
||||
```text
|
||||
current section
|
||||
next talking point
|
||||
elapsed time
|
||||
suggested transition
|
||||
likely audience question
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Keep guidance out of shared content**
|
||||
|
||||
Use the same overlay exclusion policy as Screen Context Mode. The local HUD should be visible to the user and absent from app-owned captures.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind/Mastermind/Presentation native/Mastermind/Mastermind/Assistant native/Mastermind/Mastermind/Overlay
|
||||
git commit -m "feat: add presentation coach mode"
|
||||
```
|
||||
|
||||
## Task 8: Define Electron Retirement Gates
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `docs/migration/electron-retirement-gates.md`
|
||||
|
||||
- [ ] **Step 1: Create retirement gate checklist**
|
||||
|
||||
Write this checklist:
|
||||
|
||||
```markdown
|
||||
# Electron Retirement Gates
|
||||
|
||||
- [ ] Native overlay matches or exceeds Electron window behavior.
|
||||
- [ ] Native menu bar status item is always visible while running.
|
||||
- [ ] Native screen context excludes Mastermind UI.
|
||||
- [ ] Native system audio capture works without third-party loopback drivers.
|
||||
- [ ] Native microphone capture works independently from system audio.
|
||||
- [ ] Native local ASR sidecar client can transcribe 16 kHz PCM.
|
||||
- [ ] Native provider client can stream OpenAI-compatible responses.
|
||||
- [ ] Native settings can import current Electron preferences.
|
||||
- [ ] Native Keychain storage replaces JSON credential storage.
|
||||
- [ ] Native package can be signed and notarized.
|
||||
- [ ] Electron app remains available as fallback until native app covers core workflows.
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/migration/electron-retirement-gates.md
|
||||
git commit -m "docs: add electron retirement gates"
|
||||
```
|
||||
|
||||
## Task 9: Package and Distribution Direction
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `docs/migration/native-distribution.md`
|
||||
|
||||
- [ ] **Step 1: Document distribution requirements**
|
||||
|
||||
Include:
|
||||
|
||||
```text
|
||||
Developer ID signing
|
||||
notarization
|
||||
Screen Recording permission messaging
|
||||
Microphone permission messaging
|
||||
optional Accessibility permission messaging
|
||||
Sparkle or equivalent update path
|
||||
DMG distribution
|
||||
crash reporting decision
|
||||
local log export
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Document permission copy**
|
||||
|
||||
Use transparent user-facing copy:
|
||||
|
||||
```text
|
||||
Mastermind needs Screen Recording permission only when you ask it to use screen context.
|
||||
Mastermind needs Microphone permission only when voice input or meeting listening is enabled.
|
||||
Mastermind shows a menu bar status item whenever it is running.
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/migration/native-distribution.md
|
||||
git commit -m "docs: define native distribution requirements"
|
||||
```
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
Before claiming the reconstruction direction is ready for implementation:
|
||||
|
||||
- [ ] The charter clearly says this is an AI companion, not a hidden helper.
|
||||
- [ ] The plan preserves local ASR sidecar compatibility.
|
||||
- [ ] The plan does not require a full rewrite before validating native capture.
|
||||
- [ ] The plan includes a visible menu bar status item.
|
||||
- [ ] The plan separates user-visible HUD behavior from stealth behavior.
|
||||
- [ ] The plan includes microphone, system audio, and screen context as separate states.
|
||||
- [ ] The plan includes Keychain for secrets.
|
||||
- [ ] The plan includes Electron retirement gates.
|
||||
- [ ] The plan keeps current Electron app functional during migration.
|
||||
|
||||
## Recommended First Milestone
|
||||
|
||||
The first milestone should be **Native Capability Proof**, not full product migration.
|
||||
|
||||
Success definition:
|
||||
|
||||
```text
|
||||
A Swift/AppKit app shows a local transparent overlay, displays an always-visible menu bar status item, captures screen frames without its own overlay, receives system audio, receives microphone audio, and emits 16 kHz PCM frames compatible with the existing local ASR sidecar protocol.
|
||||
```
|
||||
|
||||
If this milestone fails, keep improving the Electron app while reassessing native capture options. If it succeeds, move provider clients and agent modes into the native shell incrementally.
|
||||
|
||||
Reference in New Issue
Block a user