Files

T

2026-09-05 02:20:11 +03:00

23 KiB

Raw Blame History

macOS Native AI Companion Reconstruction Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Reconstruct Mastermind from an Electron interview-helper-shaped app into a native macOS AI companion with reliable overlay, screen context, system audio, voice interaction, and explicit trust indicators.

Architecture: Build a Swift/AppKit-first macOS host that owns windows, capture, audio, permissions, menu bar status, and privacy boundaries. Keep AI providers and local sidecars behind protocol boundaries so the rewrite can progress incrementally instead of becoming a risky full replacement.

Tech Stack: Swift, AppKit, SwiftUI where appropriate, ScreenCaptureKit, AVAudioEngine, CoreAudio, Keychain, Application Support storage, URLSession streaming, WebSocket, local ASR sidecar protocol, OpenAI-compatible HTTP/SSE, optional Electron bridge during migration.

Product Thesis

Mastermind should become an AI companion for your Mac, not a hidden interview helper.

The assistant is a personal, visible-to-the-user HUD and agent system. It can help during calls, presentations, coding, writing, research, and everyday computer use. It may stay out of the content the user is presenting or sharing, just like presenter notes, a timer, captions, or a local command palette.

The product should not implement stealth, anti-detection, process hiding, monitoring bypass, proctoring bypass, or policy evasion. If a workplace, school, exam, or managed system prohibits AI assistants, Mastermind should not help users hide that it is running.

Core Boundary

Allowed: The assistant window does not appear in the user's own screen context, screenshots, screen-share content, or presentation output when technically possible.

Not allowed: Hiding the app process, bundle identifier, permissions, network use, Accessibility use, Screen Recording use, microphone use, or audio capture use from operating-system tools or managed environments.

Practical phrasing:

Mastermind does not hide from the user or the system. It only avoids contaminating the content the user intentionally shares or asks the assistant to analyze.

Target User Experience

Normal Companion Mode

The assistant is available through voice, keyboard, and a small native overlay. It can answer questions, remember context, summarize active work, and trigger actions.

Meeting Mode

The assistant can listen to microphone and system audio with clear status indicators. It can summarize discussion, draft follow-ups, keep agenda state, and help the user stay oriented.

Presentation Mode

The user can share slides, a browser, or an app window while seeing local prompts, timing, plan notes, likely objections, and speaker guidance in Mastermind's HUD. The shared audience does not need to see the HUD.

Screen Context Mode

Screen context is opt-in. The user explicitly asks the assistant to look at the screen, capture a snapshot, or follow a bounded live stream. Mastermind excludes its own UI from that context.

Idle Mode

When not actively listening, viewing, or processing, Mastermind stays quiet and visibly idle in the menu bar.

Trust Model

Trust is a first-class feature, not a settings afterthought.

Required visible signals:

Menu bar icon is always present while the app runs.
Status reflects actual state: idle, listening, reading screen, processing, paused, permission problem.
Mic capture, system audio capture, and screen context have separate indicators.
The user can pause capture immediately.
The app exposes a local activity log showing recent context use: microphone, system audio, screen snapshot, screen stream, provider request.
The app does not provide stealth labels, anti-detection switches, or hidden-running modes.

Recommended status vocabulary:

Idle
Listening
System Audio
Screen Snapshot
Screen Stream
Agent Working
Paused
Permission Needed
Error

Current Project Context

Current repository shape:

Electron Forge app with JavaScript entry point at src/index.js.
Window management and global shortcuts in src/utils/window.js.
AI provider/session logic in src/utils/gemini.js, src/utils/localai.js, and src/utils/localProviders.js.
Local ASR sidecar protocol already documented in docs/local-sidecar-protocol.md.
JSON storage in src/storage.js.
Lit-based UI under src/components.

Important existing behavior to preserve:

Always-on-top assistant window.
Global keyboard shortcuts.
Click-through toggle.
Hide/show assistant.
Session history.
Local mode with ASR sidecar.
BYOK provider support.
OpenAI-compatible provider support.
Groq/Gemini provider support.
Configurable prompt/profile/language.

Important behavior to replace:

Electron-owned transparent window quirks.
Chromium getDisplayMedia screen capture dependency.
Fragile loopback/system-audio capture.
Renderer-local storage access from main process.
nodeIntegration: true and contextIsolation: false.
Hidden coupling between capture, transcription, provider routing, and renderer events.

Reconstruction Strategy

Recommended path: Swift-native host first, full app migration second.

Do not start with a full SwiftUI rewrite of every screen. The riskiest parts are windowing, screen capture, audio capture, permissions, and trust indicators. Prove those first in a native macOS host, then migrate provider and UI surfaces incrementally.

Migration shape:

Phase 1: Native capability proof
  Swift/AppKit overlay
  menu bar status item
  ScreenCaptureKit screen context
  ScreenCaptureKit system audio
  AVAudioEngine microphone audio

Phase 2: Native companion shell
  state machine
  permissions
  trust indicators
  local sidecar bridge
  provider boundary

Phase 3: Agentic assistant
  modes
  memory
  tools
  activity log
  meeting and presentation flows

Phase 4: Retire Electron
  migrate settings/history
  package/notarize Swift app
  deprecate Electron runtime

Proposed Native Architecture

Mastermind.app
  AppCoordinator
    owns lifecycle, mode, permissions, menu bar status

  OverlayWindowController
    owns transparent HUD, click-through, Spaces behavior, focus behavior

  CaptureCoordinator
    owns screen context and system audio through ScreenCaptureKit

  MicrophoneCaptureEngine
    owns microphone stream through AVAudioEngine

  AudioPipeline
    owns channel separation, resampling, VAD, PCM frame output

  AssistantSession
    owns active conversation, transcript, memory references, provider routing

  ProviderClients
    Gemini, Groq, OpenAI-compatible, local LLM, local ASR sidecar

  TrustStatusStore
    owns visible state, activity log, pause/resume, capture indicators

  Storage
    Application Support for non-secret app data
    Keychain for secrets

Agentic System Direction

The future assistant should be built around modes and tools, not a single chat box.

Core modes:

General Companion
Meeting Assistant
Presentation Coach
Coding Assistant
Research Assistant
Focus Assistant

Core agent abilities:

Listen and summarize.
Answer conversationally.
See screen only when requested.
Track agenda or presentation plan.
Draft follow-up notes.
Remember user preferences.
Use local tools after explicit permission.
Explain what context it used.

Agent boundaries:

No autonomous destructive actions.
No hidden capture.
No silent screen streaming.
No policy bypass tooling.
No stealth process behavior.

File Structure Target

Future native app structure:

native/Mastermind/
  Mastermind.xcodeproj
  Mastermind/
    App/
      MastermindApp.swift
      AppCoordinator.swift
      AppMode.swift
      PermissionState.swift
    Status/
      MenuBarController.swift
      TrustStatus.swift
      ActivityLog.swift
    Overlay/
      OverlayWindowController.swift
      OverlayRootView.swift
      OverlayViewModel.swift
    Capture/
      CaptureCoordinator.swift
      ScreenContextCapture.swift
      SystemAudioCapture.swift
      CaptureExclusionPolicy.swift
    Audio/
      MicrophoneCaptureEngine.swift
      AudioPipeline.swift
      PCMFrame.swift
      VoiceActivityDetector.swift
    Assistant/
      AssistantSession.swift
      AssistantMode.swift
      AssistantEvent.swift
      AssistantMemory.swift
    Providers/
      ProviderClient.swift
      LocalAsrSidecarClient.swift
      OpenAICompatibleClient.swift
      GeminiClient.swift
      GroqClient.swift
    Storage/
      AppStorageStore.swift
      KeychainStore.swift
      MigrationStore.swift
    Settings/
      SettingsWindowController.swift
      SettingsRootView.swift
    Presentation/
      PresentationPlan.swift
      PresentationCoach.swift

Existing Electron files remain read-only during the proof phase except for bridge points explicitly required by a migration task.

Task 1: Write Product Reconstruction Charter

Files:

Create: docs/product/macos-native-ai-companion-charter.md
Step 1: Create the product charter

Write a concise charter with these sections:

# macOS Native AI Companion Charter

## Positioning

Mastermind is a native macOS AI companion for everyday computer work, meetings, presentations, research, and focused execution.

## What It Is

- A visible personal assistant for the user.
- A voice-first and context-aware companion.
- A local HUD for notes, guidance, and agent status.
- A privacy-conscious screen and audio context tool.

## What It Is Not

- A hidden interview helper.
- A proctoring bypass tool.
- An anti-detection tool.
- A process-hiding tool.
- A tool for hiding AI use from managed systems that prohibit it.

## Core Promise

Mastermind does not hide from the user or the system. It only avoids contaminating the content the user intentionally shares or asks the assistant to analyze.

## Trust Requirements

- Show a menu bar status item whenever running.
- Show when microphone, system audio, or screen context is active.
- Allow immediate pause.
- Keep a local activity log of context use.
- Store secrets in Keychain.

Step 2: Review the charter language

Confirm the document includes the phrases AI companion for your Mac, does not hide from the user or the system, and visible personal assistant.

Step 3: Commit

git add docs/product/macos-native-ai-companion-charter.md
git commit -m "docs: define macos ai companion direction"

Task 2: Build Native Capability Proof Project

Files:

Create: native/Mastermind/Mastermind.xcodeproj
Create: native/Mastermind/Mastermind/App/MastermindApp.swift
Create: native/Mastermind/Mastermind/App/AppCoordinator.swift
Create: native/Mastermind/Mastermind/Status/MenuBarController.swift
Create: native/Mastermind/Mastermind/Overlay/OverlayWindowController.swift
Create: native/Mastermind/Mastermind/Capture/CaptureCoordinator.swift
Create: native/Mastermind/Mastermind/Audio/MicrophoneCaptureEngine.swift
Create: native/Mastermind/Mastermind/Audio/AudioPipeline.swift
Step 1: Create a minimal native macOS app

Create a macOS app target named Mastermind. Use Swift, AppKit lifecycle, and SwiftUI only for simple views.

Step 2: Add a menu bar status item

Implement a menu bar item that always appears while the app runs. The menu must include:

Mastermind: Idle
Pause All Capture
Show Assistant
Hide Assistant
Settings
Quit Mastermind

Step 3: Add a transparent overlay window

Create an AppKit-controlled floating overlay window with these properties:

borderless
transparent background
always on top
visible across Spaces
does not steal focus when shown
can become click-through
can be hidden and restored

Step 4: Add screen context proof

Use ScreenCaptureKit to capture the main display. The proof succeeds only when captured frames exclude the assistant overlay.

Step 5: Add system audio proof

Use ScreenCaptureKit audio output to receive system audio buffers.

Step 6: Add microphone proof

Use AVAudioEngine to receive microphone PCM buffers.

Step 7: Add 16 kHz PCM output proof

Convert microphone and system audio streams into 16 kHz signed 16-bit PCM frames.

Step 8: Manual verification

Run the native app and verify:

menu bar item is visible
overlay is visible locally
overlay can become click-through
overlay does not appear in app-owned captured frames
system audio buffers arrive when another app plays audio
microphone buffers arrive when speaking
capture can be paused immediately

Step 9: Commit

git add native/Mastermind
git commit -m "feat: add native macos capture proof"

Task 3: Define Trust State Machine

Files:

Create: native/Mastermind/Mastermind/Status/TrustStatus.swift
Create: native/Mastermind/Mastermind/Status/ActivityLog.swift
Modify: native/Mastermind/Mastermind/Status/MenuBarController.swift
Modify: native/Mastermind/Mastermind/App/AppCoordinator.swift
Step 1: Define trust states

Use these states exactly:

enum TrustStatus: Equatable {
    case idle
    case listening
    case systemAudio
    case screenSnapshot
    case screenStream
    case agentWorking
    case paused
    case permissionNeeded(String)
    case error(String)
}

Step 2: Define context activity events

Use these events exactly:

enum ActivityEventKind: String, Codable {
    case microphoneStarted
    case microphoneStopped
    case systemAudioStarted
    case systemAudioStopped
    case screenSnapshotCaptured
    case screenStreamStarted
    case screenStreamStopped
    case providerRequestStarted
    case providerRequestFinished
    case capturePaused
}

Step 3: Wire status to menu bar copy

Map trust states to visible labels:

idle -> Mastermind: Idle
listening -> Mastermind: Listening
systemAudio -> Mastermind: System Audio
screenSnapshot -> Mastermind: Screen Snapshot
screenStream -> Mastermind: Screen Stream
agentWorking -> Mastermind: Agent Working
paused -> Mastermind: Paused
permissionNeeded -> Mastermind: Permission Needed
error -> Mastermind: Error

Step 4: Verify state changes manually

Trigger each capture path and confirm the menu bar label changes before any capture data leaves the machine.

Step 5: Commit

git add native/Mastermind/Mastermind/Status native/Mastermind/Mastermind/App
git commit -m "feat: add native trust status model"

Task 4: Preserve Local ASR Sidecar Boundary

Files:

Create: native/Mastermind/Mastermind/Providers/ProviderClient.swift
Create: native/Mastermind/Mastermind/Providers/LocalAsrSidecarClient.swift
Modify: docs/local-sidecar-protocol.md
Step 1: Define provider boundary

Use this protocol as the native client boundary:

protocol ProviderClient {
    associatedtype Event

    func start() async throws
    func stop() async
    var events: AsyncStream<Event> { get }
}

Step 2: Mirror the current ASR sidecar protocol

Implement the native client against the existing WebSocket flow:

{
  "type": "start",
  "sampleRate": 16000,
  "channels": 1,
  "encoding": "pcm_s16le",
  "language": "en-US"
}

Step 3: Preserve event semantics

Support these sidecar events:

ready
partial
final
error

Only final transcript events should enter the assistant response pipeline by default.

Step 4: Document native compatibility

Add a section to docs/local-sidecar-protocol.md:

## Native macOS Client Compatibility

The Swift-native app uses the same WebSocket protocol as the Electron app. Audio frames are sent as raw 16 kHz mono signed 16-bit little-endian PCM. The sidecar does not need to know whether the client is Electron or Swift.

Step 5: Commit

git add native/Mastermind/Mastermind/Providers docs/local-sidecar-protocol.md
git commit -m "feat: add native local asr sidecar client"

Task 5: Design Native Audio Pipeline

Files:

Create: native/Mastermind/Mastermind/Audio/PCMFrame.swift
Create: native/Mastermind/Mastermind/Audio/AudioPipeline.swift
Create: native/Mastermind/Mastermind/Audio/VoiceActivityDetector.swift
Modify: native/Mastermind/Mastermind/Audio/MicrophoneCaptureEngine.swift
Modify: native/Mastermind/Mastermind/Capture/CaptureCoordinator.swift
Step 1: Define PCM frame format

Use this data model:

struct PCMFrame: Equatable {
    let source: AudioSource
    let sampleRate: Int
    let channels: Int
    let pcmS16LE: Data
    let timestamp: Date
}

enum AudioSource: String {
    case microphone
    case system
}

Step 2: Keep microphone and system audio separate

The pipeline should not mix microphone and system audio before transcription. Separate streams preserve future diarization and meeting-context quality.

Step 3: Resample to 16 kHz

Every frame sent to local ASR must be:

sample rate: 16000
channels: 1
encoding: signed 16-bit little-endian PCM

Step 4: Add VAD boundary

Voice activity detection should emit speech segments instead of forcing every buffer into transcription.

Step 5: Manual verification

Verify:

mic frames continue when system audio is silent
system frames continue when mic is silent
both streams can be paused together
either stream can be disabled independently
sidecar receives valid 16 kHz PCM frames

Step 6: Commit

git add native/Mastermind/Mastermind/Audio native/Mastermind/Mastermind/Capture
git commit -m "feat: add native audio pipeline model"

Task 6: Migrate Secrets and Storage Boundaries

Files:

Create: native/Mastermind/Mastermind/Storage/KeychainStore.swift
Create: native/Mastermind/Mastermind/Storage/AppStorageStore.swift
Create: native/Mastermind/Mastermind/Storage/MigrationStore.swift
Reference: src/storage.js
Step 1: Map current storage

Preserve these current logical groups:

config
credentials
preferences
keybinds
limits
history

Step 2: Move secrets to Keychain

Store these values in Keychain:

Gemini API key
Groq API key
OpenAI-compatible API key
local LLM API key

Step 3: Store non-secret data in Application Support

Store these values in Application Support:

preferences
profiles
keybinds
history
limits
activity log
presentation plans
assistant memory references

Step 4: Add one-way migration

Read existing Electron JSON files from:

~/Library/Application Support/cheating-daddy-config

Import values into native stores. Leave the old files untouched.

Step 5: Commit

git add native/Mastermind/Mastermind/Storage
git commit -m "feat: add native storage migration boundary"

Task 7: Build Presentation Coach Mode

Files:

Create: native/Mastermind/Mastermind/Presentation/PresentationPlan.swift
Create: native/Mastermind/Mastermind/Presentation/PresentationCoach.swift
Modify: native/Mastermind/Mastermind/Assistant/AssistantMode.swift
Modify: native/Mastermind/Mastermind/Overlay/OverlayRootView.swift
Step 1: Define presentation plan model

Use this model:

struct PresentationPlan: Codable, Equatable {
    var title: String
    var sections: [PresentationSection]
}

struct PresentationSection: Codable, Equatable {
    var title: String
    var talkingPoints: [String]
    var expectedDurationSeconds: Int
}

Step 2: Add presentation mode

Add a mode named presentationCoach.

Step 3: Show local guidance in HUD

HUD should show:

current section
next talking point
elapsed time
suggested transition
likely audience question

Step 4: Keep guidance out of shared content

Use the same overlay exclusion policy as Screen Context Mode. The local HUD should be visible to the user and absent from app-owned captures.

Step 5: Commit

git add native/Mastermind/Mastermind/Presentation native/Mastermind/Mastermind/Assistant native/Mastermind/Mastermind/Overlay
git commit -m "feat: add presentation coach mode"

Task 8: Define Electron Retirement Gates

Files:

Create: docs/migration/electron-retirement-gates.md
Step 1: Create retirement gate checklist

Write this checklist:

# Electron Retirement Gates

- [ ] Native overlay matches or exceeds Electron window behavior.
- [ ] Native menu bar status item is always visible while running.
- [ ] Native screen context excludes Mastermind UI.
- [ ] Native system audio capture works without third-party loopback drivers.
- [ ] Native microphone capture works independently from system audio.
- [ ] Native local ASR sidecar client can transcribe 16 kHz PCM.
- [ ] Native provider client can stream OpenAI-compatible responses.
- [ ] Native settings can import current Electron preferences.
- [ ] Native Keychain storage replaces JSON credential storage.
- [ ] Native package can be signed and notarized.
- [ ] Electron app remains available as fallback until native app covers core workflows.

Step 2: Commit

git add docs/migration/electron-retirement-gates.md
git commit -m "docs: add electron retirement gates"

Task 9: Package and Distribution Direction

Files:

Create: docs/migration/native-distribution.md
Step 1: Document distribution requirements

Include:

Developer ID signing
notarization
Screen Recording permission messaging
Microphone permission messaging
optional Accessibility permission messaging
Sparkle or equivalent update path
DMG distribution
crash reporting decision
local log export

Step 2: Document permission copy

Use transparent user-facing copy:

Mastermind needs Screen Recording permission only when you ask it to use screen context.
Mastermind needs Microphone permission only when voice input or meeting listening is enabled.
Mastermind shows a menu bar status item whenever it is running.

Step 3: Commit

git add docs/migration/native-distribution.md
git commit -m "docs: define native distribution requirements"

Verification Checklist

Before claiming the reconstruction direction is ready for implementation:

The charter clearly says this is an AI companion, not a hidden helper.
The plan preserves local ASR sidecar compatibility.
The plan does not require a full rewrite before validating native capture.
The plan includes a visible menu bar status item.
The plan separates user-visible HUD behavior from stealth behavior.
The plan includes microphone, system audio, and screen context as separate states.
The plan includes Keychain for secrets.
The plan includes Electron retirement gates.
The plan keeps current Electron app functional during migration.

Recommended First Milestone

The first milestone should be Native Capability Proof, not full product migration.

Success definition:

A Swift/AppKit app shows a local transparent overlay, displays an always-visible menu bar status item, captures screen frames without its own overlay, receives system audio, receives microphone audio, and emits 16 kHz PCM frames compatible with the existing local ASR sidecar protocol.

If this milestone fails, keep improving the Electron app while reassessing native capture options. If it succeeds, move provider clients and agent modes into the native shell incrementally.

23 KiB Raw Blame History

macOS Native AI Companion Reconstruction Plan

Product Thesis

Core Boundary

Target User Experience

Normal Companion Mode

Meeting Mode

Presentation Mode

Screen Context Mode

Idle Mode

Trust Model

Current Project Context

Reconstruction Strategy

Proposed Native Architecture

Agentic System Direction

File Structure Target

Task 1: Write Product Reconstruction Charter

Task 2: Build Native Capability Proof Project

Task 3: Define Trust State Machine

Task 4: Preserve Local ASR Sidecar Boundary

Task 5: Design Native Audio Pipeline

Task 6: Migrate Secrets and Storage Boundaries

Task 7: Build Presentation Coach Mode

Task 8: Define Electron Retirement Gates

Task 9: Package and Distribution Direction

Verification Checklist

Recommended First Milestone

23 KiB

Raw Blame History