app development context
This commit is contained in:
Илья Глазунов
@@ -0,0 +1,3 @@
|
||||
# Swift-native host with local sidecars
|
||||
|
||||
Mastermind's production runtime is a Swift/AppKit macOS host that owns UI, lifecycle, permissions, capture, policy, and storage. Local LLM and ASR processes stay behind loopback or Unix-socket protocols, while Electron is retained only as legacy reference; this keeps macOS capabilities native without forcing model runtimes into the application process.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Encrypted observation ledger
|
||||
|
||||
The Context Graph uses application-encrypted SQLite/SQLCipher as its source of truth. Collectors append immutable Observations, project versioned Assertions, and create Facts only through explicit user confirmation; this preserves Provenance and temporal conflicts while keeping full-text and vector indexes rebuildable instead of making a separate graph service authoritative.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Continuous Collectors with ephemeral input
|
||||
|
||||
Enabled screen, microphone, system-audio, workspace, calendar, and terminal Collectors run continuously and adapt to power and thermal pressure. Raw frames, audio, OCR, and full transcripts remain ephemeral while only structured local derivations persist; this trades recoverability for a strict local data-minimization boundary, so processor outages are recorded as Gaps rather than buffered raw content.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Defer Codex until a no-tools boundary exists
|
||||
|
||||
Codex integration is excluded from the MVP even though it remains a future Provider direction. The official app-server is an agent protocol and its read-only sandbox does not provide a documented no-tools guarantee, so Mastermind will not ship OAuth, app-server code, or a hidden experimental path until a supported draft-only boundary can enforce the product's no-action contract.
|
||||
@@ -0,0 +1,217 @@
|
||||
# Native Mastermind Architecture
|
||||
|
||||
## Scope
|
||||
|
||||
This document defines the production architecture for Mastermind. Domain language comes from [`../../CONTEXT.md`](../../CONTEXT.md), product scope from [`../product/mastermind-product-brief.md`](../product/mastermind-product-brief.md), and privacy invariants from [`../privacy/local-first-data-contract.md`](../privacy/local-first-data-contract.md).
|
||||
|
||||
The production application lives under `native/Mastermind`. `native/MastermindPOC` is a capability reference, not the production architecture. The Electron application is legacy source material and is not part of the production runtime.
|
||||
|
||||
## Runtime boundary
|
||||
|
||||
The application host is Swift-native:
|
||||
|
||||
- AppKit owns lifecycle, windows, menu bar behavior, permissions, and capture.
|
||||
- SwiftUI may render Companion Island content inside AppKit-controlled windows.
|
||||
- Swift concurrency isolates Collectors and processing from the main actor.
|
||||
- The host owns the encrypted Context Graph and all policy decisions.
|
||||
- Local model processes remain outside the host behind narrow protocols.
|
||||
- No Node or Electron process is required.
|
||||
|
||||
Permitted external local processes:
|
||||
|
||||
- an OpenAI-compatible Local Provider on loopback or a Unix socket;
|
||||
- a local ASR sidecar using the existing WebSocket protocol.
|
||||
|
||||
Mastermind owns a small multilingual embedding component because chat endpoints do not reliably expose embeddings.
|
||||
|
||||
## System shape
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph host [Swift native host]
|
||||
AppCoordinator --> MenuBar
|
||||
AppCoordinator --> CompanionIsland
|
||||
AppCoordinator --> CollectorSupervisor
|
||||
CollectorSupervisor --> ScreenCollector
|
||||
CollectorSupervisor --> AudioCollectors
|
||||
CollectorSupervisor --> WorkspaceCollector
|
||||
CollectorSupervisor --> CalendarCollector
|
||||
CollectorSupervisor --> TerminalCollector
|
||||
ScreenCollector --> LocalReduction
|
||||
AudioCollectors --> LocalReduction
|
||||
WorkspaceCollector --> LocalReduction
|
||||
CalendarCollector --> LocalReduction
|
||||
TerminalCollector --> LocalReduction
|
||||
LocalReduction --> ObservationLedger
|
||||
ObservationLedger --> KnowledgeProjector
|
||||
KnowledgeProjector --> ContextGraph
|
||||
ContextGraph --> Retriever
|
||||
Retriever --> AssistantSession
|
||||
AssistantSession --> ContextReceipt
|
||||
CollectorSupervisor --> ActivityLog
|
||||
AssistantSession --> ActivityLog
|
||||
end
|
||||
|
||||
AudioCollectors --> AsrSidecar
|
||||
AssistantSession --> LocalProvider
|
||||
EmbeddingModel --> KnowledgeProjector
|
||||
EmbeddingModel --> Retriever
|
||||
```
|
||||
|
||||
Every arrow carrying Source content stays on the Mac. Activity Log entries contain metadata, not Source content.
|
||||
|
||||
## Application components
|
||||
|
||||
### AppCoordinator
|
||||
|
||||
Owns lifecycle and composes the application. It restores the persisted paused state before starting Collectors, coordinates sleep/wake and Launch at Login, and never places heavy processing on the main actor.
|
||||
|
||||
### CompanionIslandController
|
||||
|
||||
Owns the hidden, revealed, and expanded window states on the primary display. The controller separates non-activating hover behavior from the key window used for text entry and management.
|
||||
|
||||
### MenuBarController
|
||||
|
||||
Owns the stable status glyph and detailed menu. Collector states are represented in menu content rather than by changing the glyph.
|
||||
|
||||
### CollectorSupervisor
|
||||
|
||||
Owns independent Collector lifecycles. It starts enabled Collectors, pauses them atomically, persists Pause All, applies power and thermal policy, and records Gaps. One Collector failure must not collapse the rest of the pipeline.
|
||||
|
||||
### ScreenCollector
|
||||
|
||||
Uses ScreenCaptureKit for pixels and Accessibility for application/window structure. The active display is the display containing the key window, falling back to pointer location and then the primary display. Changed-region detection, OCR, and deduplication reduce pixels locally before immediate disposal.
|
||||
|
||||
Mastermind windows are excluded from its own stream. Third-party exclusion is a best-effort window policy, not a security guarantee.
|
||||
|
||||
### AudioCollectors
|
||||
|
||||
Microphone and system audio are separate Collectors and remain separate. Both emit 16 kHz mono signed 16-bit PCM frames to VAD and local ASR. The pipeline labels channel role as User Speech or System Speech but does not infer a Person from voice.
|
||||
|
||||
### WorkspaceCollector
|
||||
|
||||
Observes explicitly connected roots. It respects `.gitignore`, configured excludes, binary detection, and size limits, and emits changes to Artifacts and repository metadata.
|
||||
|
||||
### CalendarCollector
|
||||
|
||||
Uses EventKit read-only for selected calendars and Reminders lists. External applications remain systems of record for Tasks and Events.
|
||||
|
||||
### TerminalCollector
|
||||
|
||||
Receives shell integration events for working directory, command, exit status, duration, and Git metadata. It does not persist stdout or stderr by default.
|
||||
|
||||
## Knowledge pipeline
|
||||
|
||||
### Observation ledger
|
||||
|
||||
Collectors append immutable Observations. Each record includes:
|
||||
|
||||
- stable identifier;
|
||||
- Source identifier and Collector kind;
|
||||
- observed time and source event time when available;
|
||||
- structured payload;
|
||||
- sensitivity metadata;
|
||||
- derivation version.
|
||||
|
||||
Observations are retained until age or size eviction. Corrections never mutate them.
|
||||
|
||||
### Knowledge projector
|
||||
|
||||
The projector derives versioned Assertions, entity relationships, short summaries, and Provenance. Multiple contradictory Assertions may coexist. A current view selects relevant Assertions without erasing history.
|
||||
|
||||
Only explicit user confirmation or correction promotes an Assertion to a Fact. A correction creates a new version and supersedes prior current views; it does not rewrite supporting Observations.
|
||||
|
||||
Entity resolution automatically links only high-confidence matches. Ambiguous entities remain separate and can be merged or split from a Context Receipt.
|
||||
|
||||
### Context store
|
||||
|
||||
The source of truth is encrypted SQLite/SQLCipher with explicit schema migrations. It stores:
|
||||
|
||||
- Sources and Collector configuration;
|
||||
- immutable Observations;
|
||||
- entities and relations;
|
||||
- versioned Assertions and Facts;
|
||||
- Provenance;
|
||||
- Assistant History;
|
||||
- Context Receipts;
|
||||
- Activity Log metadata;
|
||||
- retention and Provider policies.
|
||||
|
||||
Full-text and vector indexes are derived and rebuildable. They must not become unencrypted alternate stores.
|
||||
|
||||
The database key is generated locally and protected by Keychain. Provider secrets use separate Keychain entries. An encrypted, versioned archive supports manual export and import without Provider credentials.
|
||||
|
||||
### Retention
|
||||
|
||||
The default inferred-context budget is 90 days and 2 GB. Eviction considers `lastObservedAt` and utility. User-confirmed Facts are pinned. Assistant History has no automatic TTL and supports per-session and complete deletion.
|
||||
|
||||
Raw frames, audio, OCR, and full transcripts never enter the Context store. If local reduction cannot complete, the system records a Gap.
|
||||
|
||||
## Retrieval and assistance
|
||||
|
||||
The Retriever combines:
|
||||
|
||||
- structured entity and relationship queries;
|
||||
- temporal filtering;
|
||||
- full-text search;
|
||||
- multilingual vector similarity;
|
||||
- recency and confidence;
|
||||
- Provenance completeness.
|
||||
|
||||
Retrieved Source content is always untrusted data. It cannot alter system instructions or grant capabilities.
|
||||
|
||||
An Assistant Session binds to one Provider profile. The MVP has one active OpenAI-compatible Local Provider profile at a time and streams text responses. Failure leaves the query draft intact and preserves access to graph search, history, settings, and diagnostics.
|
||||
|
||||
Every answer produces a Context Receipt containing:
|
||||
|
||||
- Provider profile and locality;
|
||||
- supporting Sources, Observations, Assertions, and Facts;
|
||||
- time range and stale/conflict indicators;
|
||||
- any external-transfer record.
|
||||
|
||||
## State model
|
||||
|
||||
Capture and assistant state are orthogonal. Do not reproduce the POC's single `TrustStatus` enum, which loses information when screen, microphone, and system audio run simultaneously.
|
||||
|
||||
Model at least:
|
||||
|
||||
- one `CollectorState` per Collector: disabled, starting, running, paused, permission-needed, degraded, or failed;
|
||||
- an application pause state persisted across launches;
|
||||
- Companion Island presentation state;
|
||||
- Assistant Session state;
|
||||
- Local Provider and sidecar health;
|
||||
- Context Graph migration and maintenance state.
|
||||
|
||||
The menu and Activity Log project these states for the user.
|
||||
|
||||
## Failure rules
|
||||
|
||||
- Permission loss changes only the affected Collector to permission-needed.
|
||||
- Processor or sidecar failure creates Gaps; it never creates a raw-content backlog.
|
||||
- Database or encryption failure prevents new context writes and surfaces a blocking error.
|
||||
- Thermal and low-power pressure reduce sampling and local processing.
|
||||
- Quit stops all Collectors and flushes metadata.
|
||||
- Relaunch while paused remains paused.
|
||||
|
||||
## Future boundaries
|
||||
|
||||
### Cloud Provider
|
||||
|
||||
A future Cloud Provider must implement the Provider boundary and Provider Context Permission. It may receive only minimized, locally filtered context and must generate a Context Receipt for every request.
|
||||
|
||||
### Codex
|
||||
|
||||
Codex remains documentation-only until it exposes a supported no-tools integration. The MVP has no Codex code or UI.
|
||||
|
||||
### Agent actions
|
||||
|
||||
The MVP may emit an Action Proposal as text. Computer Control, Tool execution, approvals, and action auditing form a future bounded context and are not represented by executable stubs in the MVP.
|
||||
|
||||
## Migration direction
|
||||
|
||||
1. Treat `native/MastermindPOC` as evidence that the required macOS capture and window primitives are viable.
|
||||
2. Create the production application under `native/Mastermind`.
|
||||
3. Port validated capability code behind production state and protocol boundaries.
|
||||
4. Build the Context Graph vertical slice and primary acceptance scenario.
|
||||
5. Retire the POC after production parity checks pass.
|
||||
6. Keep Electron code as legacy reference only; do not migrate its data or runtime.
|
||||
@@ -49,12 +49,29 @@ The sidecar should send JSON text frames:
|
||||
{ "type": "error", "error": "human-readable error" }
|
||||
```
|
||||
|
||||
Only `final` transcript events are sent to the local LLM. `partial` events are
|
||||
shown as status text.
|
||||
Only `final` transcript events enter local semantic reduction. `partial` events
|
||||
are transient status and must not be persisted.
|
||||
|
||||
## v1 Scope
|
||||
|
||||
- STT target: English, `en-US`.
|
||||
- Required language modes: English (`en-US`), Russian (`ru-RU`), and automatic
|
||||
or mixed Russian-English recognition (`auto` or an implementation-equivalent
|
||||
mode).
|
||||
- Sidecar implementation is external to this repository.
|
||||
- The app does not require a specific Nemotron, NeMo, Riva, or ONNX runtime as
|
||||
long as the WebSocket protocol above is implemented.
|
||||
- The endpoint must be loopback-only. LAN and internet ASR endpoints are not
|
||||
Local Providers.
|
||||
|
||||
## Native macOS Client Compatibility
|
||||
|
||||
The Swift-native app uses the same WebSocket protocol as the Electron client.
|
||||
Audio frames are raw 16 kHz mono signed 16-bit little-endian PCM.
|
||||
|
||||
Microphone and system audio use independent client connections so the sidecar
|
||||
does not mix channel roles. The sidecar does not need to know whether a client
|
||||
is Electron or Swift; Mastermind associates each connection with its Source.
|
||||
|
||||
Full transcripts are ephemeral sensitive input. The Swift client consumes a
|
||||
`final` event, derives structured local context, and then releases transcript
|
||||
content instead of storing it in Assistant History or the Context Graph.
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
# Native macOS POC Results
|
||||
|
||||
> This document records capability evidence, not the production product specification.
|
||||
> Canonical direction is defined by [`../../CONTEXT.md`](../../CONTEXT.md),
|
||||
> [`../product/mastermind-product-brief.md`](../product/mastermind-product-brief.md), and
|
||||
> [`../architecture/native-mastermind.md`](../architecture/native-mastermind.md).
|
||||
|
||||
## Implemented
|
||||
|
||||
- SwiftPM-based macOS POC under `native/MastermindPOC`.
|
||||
@@ -42,3 +47,19 @@
|
||||
## Boundary
|
||||
|
||||
This POC is not a stealth or anti-detection system. It shows a menu bar item while running, requests normal macOS permissions, and only excludes its own overlay from the context it captures for itself.
|
||||
|
||||
## Role in production migration
|
||||
|
||||
`native/MastermindPOC` proves that AppKit windowing, ScreenCaptureKit screen and system-audio capture, AVAudioEngine microphone capture, current-process exclusion, and 16 kHz PCM conversion are viable.
|
||||
|
||||
Production development moves to `native/Mastermind`. Validated capability code may be ported behind production boundaries, but the POC is not renamed into the product and is retired after parity checks pass.
|
||||
|
||||
The production design intentionally differs from the POC:
|
||||
|
||||
- the Companion Island is hidden at launch and anchored to the primary display's top center;
|
||||
- enabled Collectors start continuously after onboarding instead of through manual start menu items;
|
||||
- the Menu Bar Item uses a stable glyph with detailed states inside its menu;
|
||||
- screen, microphone, and system-audio states are simultaneous per-Collector states rather than one `TrustStatus`;
|
||||
- settings and management live inside the expanded Companion Island;
|
||||
- screen context follows the active display rather than always selecting the main display;
|
||||
- the Context Graph, Local Provider, local ASR client, retention, encryption, Context Receipts, and Activity Log are production requirements not implemented by the POC.
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
# Local-First Data Contract
|
||||
|
||||
This document is normative. Product and implementation work must preserve these boundaries unless a superseding ADR explicitly changes them.
|
||||
|
||||
## Local-first definition
|
||||
|
||||
Mastermind's Local Profile, Source configuration, captured context, Context Graph, Assistant History, Activity Log, embeddings, and inference remain on the user's Mac in the MVP.
|
||||
|
||||
A Provider is local only when it is reachable through loopback or a Unix socket on the same Mac. LAN and internet endpoints are external and are not supported by the MVP.
|
||||
|
||||
The application must remain useful without internet access. It may not silently fall back from local processing to a Cloud Provider.
|
||||
|
||||
## Data classes
|
||||
|
||||
### Ephemeral sensitive input
|
||||
|
||||
- screen frames and changed image regions;
|
||||
- microphone and system-audio PCM;
|
||||
- OCR output;
|
||||
- complete ASR transcripts;
|
||||
- transient prompts assembled for local derivation.
|
||||
|
||||
These values exist only in bounded processing buffers and must be destroyed after local reduction. They must not enter logs, crash reports, archives, fixtures, or the Context store.
|
||||
|
||||
### Persisted context
|
||||
|
||||
- structured Observations;
|
||||
- Assertions, Facts, summaries, and entity relationships;
|
||||
- Provenance references;
|
||||
- Source and Collector configuration;
|
||||
- Assistant History and Context Receipts;
|
||||
- Activity Log metadata;
|
||||
- retention and policy settings.
|
||||
|
||||
Persisted context is encrypted at the application layer.
|
||||
|
||||
### Secrets
|
||||
|
||||
- Context-store encryption key;
|
||||
- Local Provider credentials, when required;
|
||||
- future Cloud Provider credentials.
|
||||
|
||||
Secrets are stored in Keychain and excluded from Context Graph exports and diagnostics.
|
||||
|
||||
## Continuous collection
|
||||
|
||||
After guided onboarding, every enabled Collector starts with the app unless the user previously selected Pause All.
|
||||
|
||||
Screen, microphone, and system-audio collection are continuous. macOS permission and capture indicators must remain visible and unmodified. The stable Menu Bar Item glyph does not replace those indicators; its open menu shows actual Collector state.
|
||||
|
||||
Pause All:
|
||||
|
||||
- stops every Collector immediately;
|
||||
- leaves existing context, Assistant History, and local search available;
|
||||
- persists across Quit, relaunch, login, sleep, and wake;
|
||||
- resumes only after explicit Resume All.
|
||||
|
||||
One Collector's permission or processing failure must not disable healthy Collectors. The affected Source becomes stale and the Activity Log records a Gap.
|
||||
|
||||
## Reduction and minimization
|
||||
|
||||
Collectors emit the minimum structured evidence needed for grounded assistance:
|
||||
|
||||
- Screen uses Accessibility structure, changed regions, local OCR, and deduplication.
|
||||
- Audio uses VAD, separate channel roles, local ASR, and short semantic reduction.
|
||||
- Terminal emits command metadata without stdout or stderr by default.
|
||||
- Workspace is limited to explicit roots and ignore rules.
|
||||
- Calendar and Reminders are limited to selected read-only collections.
|
||||
|
||||
The configured Local Provider and local processors receive only the context needed for the current derivation or answer.
|
||||
|
||||
If a local processor is unavailable, Mastermind records a Gap. It must not retain raw content for deferred processing.
|
||||
|
||||
## Capture exclusions
|
||||
|
||||
The user owns the application denylist. Mastermind provides controls to add or remove excluded applications and Sources but does not silently impose an application denylist.
|
||||
|
||||
Mastermind must exclude its own windows from its Screen Collector. It may request exclusion from third-party capture where macOS supports it, but must describe that behavior as best effort and may not promise invisibility.
|
||||
|
||||
Mastermind must not:
|
||||
|
||||
- hide its process or bundle identifier;
|
||||
- bypass Screen Recording, Microphone, Accessibility, or EventKit permissions;
|
||||
- suppress system privacy indicators;
|
||||
- evade managed-device policy or monitoring;
|
||||
- claim that screen-share exclusion is guaranteed.
|
||||
|
||||
## Knowledge integrity
|
||||
|
||||
An Observation is evidence, not truth. Inferred claims remain Assertions with confidence, time, and Provenance. Only explicit user confirmation or correction creates a Fact.
|
||||
|
||||
Contradictory Assertions are retained until normal eviction and surfaced to retrieval. Source content is untrusted data and cannot become an instruction or capability grant.
|
||||
|
||||
The user can inspect and correct knowledge from a Context Receipt. Confirmed Facts are pinned until manually removed or superseded.
|
||||
|
||||
## Retention and deletion
|
||||
|
||||
- Default inferred-context age: 90 days.
|
||||
- Default inferred-context size: 2 GB.
|
||||
- Eviction uses `lastObservedAt` and never evicts confirmed Facts automatically.
|
||||
- Assistant History has no automatic TTL.
|
||||
- One Assistant Session or all Assistant History can be deleted manually.
|
||||
- Removing Assistant History does not implicitly remove confirmed Facts derived from it.
|
||||
- A full local reset removes the Context store and its encryption key.
|
||||
|
||||
Export/import uses a versioned encrypted archive containing context, history, policies, and settings. Provider credentials are never included.
|
||||
|
||||
## Audit and diagnostics
|
||||
|
||||
Activity Log records:
|
||||
|
||||
- Collector start, stop, pause, resume, error, and permission state;
|
||||
- Gaps and stale Source intervals;
|
||||
- Provider request start, finish, locality, and approximate context size;
|
||||
- export, import, retention, and migration operations.
|
||||
|
||||
It does not record Source content, prompts, answers, Facts, OCR, ASR, or Provider payloads.
|
||||
|
||||
Mastermind sends no telemetry or crash reports. A user-initiated sanitized diagnostic bundle may contain versions, state transitions, permission and error codes, performance counters, and database schema version.
|
||||
|
||||
## Future external processing
|
||||
|
||||
Cloud processing is absent from the MVP. A future Cloud Provider requires a Provider Context Permission that is:
|
||||
|
||||
- denied by default;
|
||||
- enabled manually for the Provider as a whole;
|
||||
- revocable for future requests;
|
||||
- unable to retract data already sent.
|
||||
|
||||
Permission is an upper bound, not permission to send everything. Before each request, Mastermind must minimize context and filter recognized secrets locally. Every request must create a Context Receipt naming the Provider, supporting Sources, and transferred context categories.
|
||||
|
||||
Codex remains disabled until a supported no-tools boundary exists.
|
||||
|
||||
## Onboarding disclosure
|
||||
|
||||
Before enabling continuous Collectors, onboarding must explain:
|
||||
|
||||
- which Source each permission exposes;
|
||||
- that screen and both audio channels run continuously while enabled;
|
||||
- that raw media and full transcripts are not retained;
|
||||
- where derived knowledge is stored;
|
||||
- how Pause All and the denylist work;
|
||||
- that other people may be represented in locally derived context;
|
||||
- that capture exclusion is best effort;
|
||||
- that the MVP performs no cloud context transfer.
|
||||
@@ -0,0 +1,87 @@
|
||||
# Companion Island
|
||||
|
||||
## Purpose
|
||||
|
||||
The Companion Island is Mastermind's only primary interaction surface. It keeps the assistant absent from the desktop until requested while making it available at a stable physical location.
|
||||
|
||||
It is inspired by the expansion behavior of Dynamic Island, but it is a Mastermind concept and must not be described as an Apple system feature.
|
||||
|
||||
## Placement
|
||||
|
||||
- The Companion Island belongs to the primary display only.
|
||||
- Its activation area is centered on the display's top edge.
|
||||
- On a notched display, the activation area follows the camera housing.
|
||||
- On a display without a notch, the same area behaves as a virtual camera housing.
|
||||
- Changing the primary display relocates the Companion Island.
|
||||
|
||||
## Interaction states
|
||||
|
||||
### Hidden
|
||||
|
||||
No Companion Island content is visible. Moving the pointer into the activation area begins the reveal transition.
|
||||
|
||||
### Revealed
|
||||
|
||||
A compact capsule grows from the camera area without taking keyboard focus. Leaving the activation region without clicking collapses it after a short grace period.
|
||||
|
||||
### Expanded
|
||||
|
||||
Clicking the revealed capsule expands it downward into an interactive panel and focuses text input. Expansion must feel spatially connected to the camera area rather than like an unrelated window appearing.
|
||||
|
||||
The expanded surface contains navigation for:
|
||||
|
||||
- Assistant;
|
||||
- Assistant History;
|
||||
- Sources;
|
||||
- Local Provider;
|
||||
- Context Graph and Context Receipts;
|
||||
- Activity Log;
|
||||
- Privacy, storage, export/import, and diagnostics.
|
||||
|
||||
Escape, an explicit close action, or clicking outside the panel returns it to Hidden. Long-running local work continues after collapse and is visible when the panel is reopened.
|
||||
|
||||
### Paused
|
||||
|
||||
Paused is a Collector condition, not a separate window mode. The user can still open the Companion Island, search existing context, and use Assistant History while all Collectors remain stopped.
|
||||
|
||||
## Menu Bar Item
|
||||
|
||||
The Menu Bar Item is visible whenever Mastermind runs. Its glyph does not change with capture state.
|
||||
|
||||
Its menu must expose:
|
||||
|
||||
- current state of every enabled Collector;
|
||||
- permission and processing errors;
|
||||
- Pause All or Resume All;
|
||||
- Show Mastermind;
|
||||
- Launch at Login state;
|
||||
- Quit Mastermind.
|
||||
|
||||
Show Mastermind expands the Companion Island and is the fallback when pointer activation is unavailable. The MVP has no global hotkey.
|
||||
|
||||
## Focus and accessibility
|
||||
|
||||
- Hidden and Revealed do not steal focus.
|
||||
- Expanded accepts keyboard focus and text input.
|
||||
- Pointer activation must not create a dead strip that prevents access to the macOS menu bar.
|
||||
- Animation respects Reduce Motion.
|
||||
- The interface remains keyboard-navigable after it is expanded.
|
||||
- Collector state and errors are conveyed with text, not color alone.
|
||||
|
||||
## Capture behavior
|
||||
|
||||
Mastermind excludes its own windows from the Screen Collector. It also requests exclusion from third-party capture where supported by macOS.
|
||||
|
||||
Capture exclusion is best effort. The UI must not promise that the Companion Island is invisible to every screen-sharing or recording application. Mastermind never hides its process or system permission indicators.
|
||||
|
||||
## Acceptance checks
|
||||
|
||||
- At launch, only the Menu Bar Item is visible.
|
||||
- Hovering the primary display's camera area reveals the capsule.
|
||||
- A notchless primary display gets the same top-center interaction.
|
||||
- A click expands the panel and focuses text input.
|
||||
- Show Mastermind works when hover activation cannot be used.
|
||||
- Escape and click-away collapse the panel.
|
||||
- Pause All does not prevent access to existing knowledge or history.
|
||||
- The Companion Island is absent from Mastermind's own captured frames.
|
||||
- Changing the primary display relocates the activation area.
|
||||
@@ -0,0 +1,134 @@
|
||||
# Mastermind Product Brief
|
||||
|
||||
## Product thesis
|
||||
|
||||
Mastermind is a personal, local-first assistant for macOS. It continuously builds an inspectable understanding of the user's work and answers grounded questions about what happened, what matters now, and what may come next.
|
||||
|
||||
Mastermind is not an interview helper, a hidden proctoring tool, or a generic shell around a cloud agent. The first product is a personal/internal Swift application for one macOS user.
|
||||
|
||||
## Core promise
|
||||
|
||||
The user's machine context remains on the Mac. Screen, audio, files, calendar, reminders, and terminal activity are reduced to useful knowledge locally. Mastermind does not transmit that context to a cloud service in the MVP.
|
||||
|
||||
The assistant distinguishes observed evidence, inferred claims, and user-confirmed facts. Every grounded answer can show what evidence it used and where that evidence came from.
|
||||
|
||||
## Target platform
|
||||
|
||||
- Apple Silicon Mac.
|
||||
- macOS 14 or newer.
|
||||
- One Local Profile for the current macOS user.
|
||||
- No Mastermind account, backend, telemetry, or synchronization in the MVP.
|
||||
- The domain keeps a future cloud identity separate from the Local Profile.
|
||||
- Personal/internal distribution; Mac App Store constraints are out of scope.
|
||||
|
||||
## Primary experience
|
||||
|
||||
Mastermind runs as a menu bar application without a Dock presence. Its Menu Bar Item is always present while the app runs and uses a stable glyph. Opening its menu reveals Collector states, permission failures, Pause All or Resume All, Show Mastermind, and Quit.
|
||||
|
||||
The Companion Island is hidden by default. Hovering the top-center camera area of the primary display reveals a compact capsule with a smooth animation. Clicking expands the capsule into the complete Mastermind interface. A virtual top-center activation area provides the same behavior when the primary display has no physical notch. Show Mastermind in the menu is the fallback; the MVP has no global shortcut or voice invocation.
|
||||
|
||||
The expanded Companion Island contains:
|
||||
|
||||
- text input and streamed answers;
|
||||
- Assistant History;
|
||||
- Context Receipts and Fact correction;
|
||||
- Source and Collector management;
|
||||
- Local Provider configuration;
|
||||
- Context Graph limits and encrypted export/import;
|
||||
- Activity Log and diagnostics.
|
||||
|
||||
Mastermind is on-demand, not proactive. Background collection may update status and knowledge, but the assistant does not interrupt the user with unsolicited advice.
|
||||
|
||||
## MVP sources
|
||||
|
||||
Enabled Collectors start automatically with the application unless Pause All was previously selected:
|
||||
|
||||
- Screen: the active display, defined by the frontmost key window, with pointer display and primary display as fallbacks.
|
||||
- Audio: separate microphone and system-audio channels.
|
||||
- Workspace: explicitly connected directories and Git repositories.
|
||||
- Calendar and Reminders: user-selected calendars and lists, read-only.
|
||||
- Terminal: shell integration metadata including working directory, command, exit status, duration, and Git metadata; terminal output is not retained by default.
|
||||
|
||||
Workspace indexing respects `.gitignore`, binary and size limits, and user-configured exclusions. Mastermind does not index the entire home directory.
|
||||
|
||||
The user configures the local capture denylist. Mastermind does not silently add application-level exclusions, but Pause All is always available.
|
||||
|
||||
## Context behavior
|
||||
|
||||
Collectors run continuously and adapt their work to meaningful changes, voice activity, duplication, Low Power Mode, and thermal pressure. A failed Collector degrades independently while the others continue.
|
||||
|
||||
Screen processing combines Accessibility metadata, changed-region detection, and local OCR. Audio uses separate 16 kHz PCM streams and local ASR; it distinguishes User Speech from System Speech but does not identify people by voice. Meeting inference may combine calendar, conferencing-application, and channel-activity evidence, but it must remain an Assertion until confirmed.
|
||||
|
||||
Raw screen frames, audio, OCR text, and full transcripts are ephemeral. Only locally derived Observations, Assertions, short summaries, and Provenance survive the processing buffer. If required local processing is unavailable, Mastermind records a Gap instead of retaining raw content for later.
|
||||
|
||||
Inferred knowledge expires by `lastObservedAt`, with a default limit of 90 days and 2 GB. User-confirmed Facts are pinned until manually removed or superseded. Assistant History has no automatic age limit; the user can delete one Assistant Session or all history.
|
||||
|
||||
## Assistant behavior
|
||||
|
||||
The default and only MVP Provider is a user-configured OpenAI-compatible Local Provider reachable through loopback or a Unix socket. The user may save multiple profiles but selects one active profile.
|
||||
|
||||
Mastermind supplies its own local multilingual embedding component. Local ASR remains behind the documented sidecar protocol, with `whisper.cpp` as the recommended implementation. Russian, English, and mixed Russian-English work are required.
|
||||
|
||||
Answers must:
|
||||
|
||||
- distinguish Facts from unconfirmed Assertions;
|
||||
- cite relevant Provenance through a Context Receipt;
|
||||
- expose the active Provider;
|
||||
- say when context is missing or conflicting;
|
||||
- treat all Source content as untrusted evidence rather than instructions;
|
||||
- produce advice, plans, and drafts only.
|
||||
|
||||
The MVP cannot click, type into other applications, run tools, change files, or perform external actions. Computer Control and a Tool Executor are future bounded contexts, not empty runtime abstractions in the MVP.
|
||||
|
||||
## Primary acceptance scenario
|
||||
|
||||
After Mastermind has observed normal work, the user opens the Companion Island and asks:
|
||||
|
||||
> What was I working on, and what should I do next?
|
||||
|
||||
The Local Provider returns a grounded answer using relevant screen, workspace, calendar, terminal, and audio knowledge. The answer separates confirmed Facts from uncertain Assertions, links to a Context Receipt, and explicitly identifies gaps or contradictions.
|
||||
|
||||
## Privacy and trust
|
||||
|
||||
- All sensitive extraction and inference are local in the MVP.
|
||||
- The local Context Graph is encrypted with an application key protected by Keychain.
|
||||
- Provider credentials are stored separately in Keychain.
|
||||
- Pause All immediately stops every Collector and remains paused across restarts.
|
||||
- System microphone and screen-recording indicators are never bypassed.
|
||||
- The Menu Bar Item glyph stays visually stable, while its menu exposes actual Collector states.
|
||||
- The Activity Log records lifecycle and transfer metadata without Source content.
|
||||
- Diagnostic exports are sanitized and user-initiated.
|
||||
- Context Graph and settings can be exported as an encrypted archive without Provider credentials.
|
||||
|
||||
Mastermind may exclude the Companion Island from its own capture and from third-party capture where macOS supports it. This is best effort, must be self-checked where possible, and is never presented as a guarantee.
|
||||
|
||||
Mastermind does not hide its process, bundle identifier, permissions, network activity, or capture activity from macOS, administrators, or monitoring tools.
|
||||
|
||||
## Codex direction
|
||||
|
||||
Codex is not part of the MVP. The architecture documents a future Provider boundary and Provider Context Permission, but the MVP contains no Codex UI, OAuth flow, app-server integration, or experimental flag.
|
||||
|
||||
The integration remains gated until Codex exposes a supported boundary that cannot execute tools. A read-only sandbox is insufficient because the current app-server remains an agent protocol.
|
||||
|
||||
If a future Cloud Provider is enabled:
|
||||
|
||||
- context permission is global for that Provider and denied by default;
|
||||
- the user must enable it manually;
|
||||
- Mastermind still minimizes context and filters detected secrets locally;
|
||||
- every request produces a Context Receipt;
|
||||
- revocation blocks future requests but cannot retract already transmitted data.
|
||||
|
||||
## Explicit non-goals
|
||||
|
||||
- Cloud inference or context transfer in the MVP.
|
||||
- Autonomous actions, Computer Control, or tool execution.
|
||||
- Proactive suggestions or scheduled briefings.
|
||||
- Wake word, push-to-talk, global shortcut, or voice queries.
|
||||
- Browser, mail, messages, clipboard, or full home-directory indexing.
|
||||
- Voice identity and speaker attribution to a Person.
|
||||
- Owning or synchronizing the user's tasks and calendar.
|
||||
- Multi-display Companion Island behavior.
|
||||
- A visual whole-graph explorer.
|
||||
- Electron feature development or an Electron bridge.
|
||||
- Intel Mac, Windows, Linux, or Mac App Store support.
|
||||
- Stealth, anti-detection, permission bypass, or guaranteed screen-share invisibility.
|
||||
@@ -0,0 +1,87 @@
|
||||
# MVP Acceptance
|
||||
|
||||
The MVP is complete only when the following scenarios work together in the production Swift application. Passing the current native capability proof alone is not sufficient.
|
||||
|
||||
## Product scenario
|
||||
|
||||
Given that enabled Collectors have observed normal work, when the user opens the Companion Island and asks, "What was I working on, and what should I do next?", then Mastermind:
|
||||
|
||||
- responds through the configured Local Provider;
|
||||
- uses relevant knowledge from screen, workspace, calendar, terminal, and audio Sources;
|
||||
- distinguishes user-confirmed Facts from inferred Assertions;
|
||||
- identifies conflicting or stale knowledge;
|
||||
- states when evidence is insufficient;
|
||||
- attaches an inspectable Context Receipt.
|
||||
|
||||
## Presence
|
||||
|
||||
- Mastermind launches as a menu bar application without a Dock icon.
|
||||
- Only the stable Menu Bar Item is visible at rest.
|
||||
- Hover reveals the Companion Island on the primary display.
|
||||
- Click expands it and focuses text input.
|
||||
- Show Mastermind in the menu provides a reliable fallback.
|
||||
- The expanded surface contains assistant, history, source, provider, graph, audit, and privacy controls.
|
||||
|
||||
## Continuous collection
|
||||
|
||||
- Enabled Collectors start after launch unless paused state was persisted.
|
||||
- Screen collection follows the display containing the key window and processes meaningful changes rather than every full frame.
|
||||
- Microphone and system audio remain separate through local ASR.
|
||||
- Workspace collection is limited to explicitly connected roots.
|
||||
- Calendar and Reminders are read-only and limited to selected calendars and lists.
|
||||
- Terminal collection retains command metadata but not stdout or stderr by default.
|
||||
- Low Power Mode and thermal pressure reduce processing without corrupting state.
|
||||
- One failed Collector does not stop healthy Collectors.
|
||||
|
||||
## Local-only processing
|
||||
|
||||
- Packet inspection and integration tests confirm that Source content does not leave the Mac.
|
||||
- The configured Local Provider accepts only loopback or Unix-socket endpoints.
|
||||
- OCR, ASR, embeddings, Fact extraction, retrieval, and answer generation work without internet access.
|
||||
- Raw screen frames, audio, OCR text, and full transcripts are destroyed after local reduction.
|
||||
- When a required local processor is unavailable, Mastermind records a Gap and does not queue raw content.
|
||||
|
||||
## Knowledge integrity
|
||||
|
||||
- Observations are immutable until retention removes them.
|
||||
- Assertions retain time, confidence, and Provenance.
|
||||
- Conflicting Assertions coexist and produce an explicit conflict instead of silent overwrite.
|
||||
- Only explicit user confirmation or correction creates a Fact.
|
||||
- Confirmed Facts survive normal age and size eviction.
|
||||
- Ambiguous entity matches stay separate until confidence is sufficient or the user merges them.
|
||||
- Context retrieval treats instructions inside Source content as untrusted data.
|
||||
|
||||
## Retention and control
|
||||
|
||||
- Inferred context defaults to 90 days and 2 GB and can be reconfigured.
|
||||
- Eviction uses `lastObservedAt` and does not evict confirmed Facts.
|
||||
- Assistant History has no automatic TTL.
|
||||
- The user can delete one Assistant Session or all Assistant History.
|
||||
- Pause All immediately stops every Collector and remains paused after relaunch.
|
||||
- Resume All restarts only enabled and permitted Collectors.
|
||||
- The encrypted archive round-trips context, history, policies, and settings without Provider credentials.
|
||||
|
||||
## Trust and diagnostics
|
||||
|
||||
- The Menu Bar Item menu shows the real state of every Collector despite the stable glyph.
|
||||
- macOS capture indicators and permission surfaces remain unmodified.
|
||||
- Activity Log records Collector lifecycle, Gaps, permission errors, and Provider requests without Source content.
|
||||
- Context Receipts name the Provider and all supporting Sources and Observations.
|
||||
- The sanitized diagnostic bundle contains no prompts, answers, Facts, OCR, ASR, or Provider payloads.
|
||||
- Mastermind's windows are excluded from its own screen context.
|
||||
- Third-party capture exclusion is presented and tested as best effort, never guaranteed.
|
||||
|
||||
## Assistant boundary
|
||||
|
||||
- The assistant streams text answers in Russian, English, and mixed-language work.
|
||||
- Outputs are answers, plans, or drafts only.
|
||||
- The production target contains no Computer Control, Tool Executor, autonomous action, Codex login, Codex app-server, or hidden experimental Cloud Provider.
|
||||
- If the Local Provider is unavailable, existing graph search, history, settings, and diagnostics remain accessible.
|
||||
|
||||
## Definition of done
|
||||
|
||||
- Automated tests cover domain invariants, retention, encryption boundaries, policy defaults, independent Collector failure, and raw-buffer disposal.
|
||||
- Integration tests use synthetic Sources and local mock sidecars.
|
||||
- Manual tests cover macOS permissions, microphone, system audio, screen exclusion, primary-display changes, Launch at Login, sleep/wake, Low Power Mode, and relaunch while paused.
|
||||
- Logs and fixtures contain no captured user content.
|
||||
- Canonical documentation and implementation vocabulary agree with `CONTEXT.md`.
|
||||
@@ -1,825 +0,0 @@
|
||||
# macOS Native AI Companion Reconstruction Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Reconstruct Mastermind from an Electron interview-helper-shaped app into a native macOS AI companion with reliable overlay, screen context, system audio, voice interaction, and explicit trust indicators.
|
||||
|
||||
**Architecture:** Build a Swift/AppKit-first macOS host that owns windows, capture, audio, permissions, menu bar status, and privacy boundaries. Keep AI providers and local sidecars behind protocol boundaries so the rewrite can progress incrementally instead of becoming a risky full replacement.
|
||||
|
||||
**Tech Stack:** Swift, AppKit, SwiftUI where appropriate, ScreenCaptureKit, AVAudioEngine, CoreAudio, Keychain, Application Support storage, URLSession streaming, WebSocket, local ASR sidecar protocol, OpenAI-compatible HTTP/SSE, optional Electron bridge during migration.
|
||||
|
||||
---
|
||||
|
||||
## Product Thesis
|
||||
|
||||
Mastermind should become an **AI companion for your Mac**, not a hidden interview helper.
|
||||
|
||||
The assistant is a personal, visible-to-the-user HUD and agent system. It can help during calls, presentations, coding, writing, research, and everyday computer use. It may stay out of the content the user is presenting or sharing, just like presenter notes, a timer, captions, or a local command palette.
|
||||
|
||||
The product should not implement stealth, anti-detection, process hiding, monitoring bypass, proctoring bypass, or policy evasion. If a workplace, school, exam, or managed system prohibits AI assistants, Mastermind should not help users hide that it is running.
|
||||
|
||||
## Core Boundary
|
||||
|
||||
**Allowed:** The assistant window does not appear in the user's own screen context, screenshots, screen-share content, or presentation output when technically possible.
|
||||
|
||||
**Not allowed:** Hiding the app process, bundle identifier, permissions, network use, Accessibility use, Screen Recording use, microphone use, or audio capture use from operating-system tools or managed environments.
|
||||
|
||||
Practical phrasing:
|
||||
|
||||
> Mastermind does not hide from the user or the system. It only avoids contaminating the content the user intentionally shares or asks the assistant to analyze.
|
||||
|
||||
## Target User Experience
|
||||
|
||||
### Normal Companion Mode
|
||||
|
||||
The assistant is available through voice, keyboard, and a small native overlay. It can answer questions, remember context, summarize active work, and trigger actions.
|
||||
|
||||
### Meeting Mode
|
||||
|
||||
The assistant can listen to microphone and system audio with clear status indicators. It can summarize discussion, draft follow-ups, keep agenda state, and help the user stay oriented.
|
||||
|
||||
### Presentation Mode
|
||||
|
||||
The user can share slides, a browser, or an app window while seeing local prompts, timing, plan notes, likely objections, and speaker guidance in Mastermind's HUD. The shared audience does not need to see the HUD.
|
||||
|
||||
### Screen Context Mode
|
||||
|
||||
Screen context is opt-in. The user explicitly asks the assistant to look at the screen, capture a snapshot, or follow a bounded live stream. Mastermind excludes its own UI from that context.
|
||||
|
||||
### Idle Mode
|
||||
|
||||
When not actively listening, viewing, or processing, Mastermind stays quiet and visibly idle in the menu bar.
|
||||
|
||||
## Trust Model
|
||||
|
||||
Trust is a first-class feature, not a settings afterthought.
|
||||
|
||||
Required visible signals:
|
||||
|
||||
- Menu bar icon is always present while the app runs.
|
||||
- Status reflects actual state: idle, listening, reading screen, processing, paused, permission problem.
|
||||
- Mic capture, system audio capture, and screen context have separate indicators.
|
||||
- The user can pause capture immediately.
|
||||
- The app exposes a local activity log showing recent context use: microphone, system audio, screen snapshot, screen stream, provider request.
|
||||
- The app does not provide stealth labels, anti-detection switches, or hidden-running modes.
|
||||
|
||||
Recommended status vocabulary:
|
||||
|
||||
- `Idle`
|
||||
- `Listening`
|
||||
- `System Audio`
|
||||
- `Screen Snapshot`
|
||||
- `Screen Stream`
|
||||
- `Agent Working`
|
||||
- `Paused`
|
||||
- `Permission Needed`
|
||||
- `Error`
|
||||
|
||||
## Current Project Context
|
||||
|
||||
Current repository shape:
|
||||
|
||||
- Electron Forge app with JavaScript entry point at `src/index.js`.
|
||||
- Window management and global shortcuts in `src/utils/window.js`.
|
||||
- AI provider/session logic in `src/utils/gemini.js`, `src/utils/localai.js`, and `src/utils/localProviders.js`.
|
||||
- Local ASR sidecar protocol already documented in `docs/local-sidecar-protocol.md`.
|
||||
- JSON storage in `src/storage.js`.
|
||||
- Lit-based UI under `src/components`.
|
||||
|
||||
Important existing behavior to preserve:
|
||||
|
||||
- Always-on-top assistant window.
|
||||
- Global keyboard shortcuts.
|
||||
- Click-through toggle.
|
||||
- Hide/show assistant.
|
||||
- Session history.
|
||||
- Local mode with ASR sidecar.
|
||||
- BYOK provider support.
|
||||
- OpenAI-compatible provider support.
|
||||
- Groq/Gemini provider support.
|
||||
- Configurable prompt/profile/language.
|
||||
|
||||
Important behavior to replace:
|
||||
|
||||
- Electron-owned transparent window quirks.
|
||||
- Chromium `getDisplayMedia` screen capture dependency.
|
||||
- Fragile loopback/system-audio capture.
|
||||
- Renderer-local storage access from main process.
|
||||
- `nodeIntegration: true` and `contextIsolation: false`.
|
||||
- Hidden coupling between capture, transcription, provider routing, and renderer events.
|
||||
|
||||
## Reconstruction Strategy
|
||||
|
||||
Recommended path: **Swift-native host first, full app migration second**.
|
||||
|
||||
Do not start with a full SwiftUI rewrite of every screen. The riskiest parts are windowing, screen capture, audio capture, permissions, and trust indicators. Prove those first in a native macOS host, then migrate provider and UI surfaces incrementally.
|
||||
|
||||
Migration shape:
|
||||
|
||||
```text
|
||||
Phase 1: Native capability proof
|
||||
Swift/AppKit overlay
|
||||
menu bar status item
|
||||
ScreenCaptureKit screen context
|
||||
ScreenCaptureKit system audio
|
||||
AVAudioEngine microphone audio
|
||||
|
||||
Phase 2: Native companion shell
|
||||
state machine
|
||||
permissions
|
||||
trust indicators
|
||||
local sidecar bridge
|
||||
provider boundary
|
||||
|
||||
Phase 3: Agentic assistant
|
||||
modes
|
||||
memory
|
||||
tools
|
||||
activity log
|
||||
meeting and presentation flows
|
||||
|
||||
Phase 4: Retire Electron
|
||||
migrate settings/history
|
||||
package/notarize Swift app
|
||||
deprecate Electron runtime
|
||||
```
|
||||
|
||||
## Proposed Native Architecture
|
||||
|
||||
```text
|
||||
Mastermind.app
|
||||
AppCoordinator
|
||||
owns lifecycle, mode, permissions, menu bar status
|
||||
|
||||
OverlayWindowController
|
||||
owns transparent HUD, click-through, Spaces behavior, focus behavior
|
||||
|
||||
CaptureCoordinator
|
||||
owns screen context and system audio through ScreenCaptureKit
|
||||
|
||||
MicrophoneCaptureEngine
|
||||
owns microphone stream through AVAudioEngine
|
||||
|
||||
AudioPipeline
|
||||
owns channel separation, resampling, VAD, PCM frame output
|
||||
|
||||
AssistantSession
|
||||
owns active conversation, transcript, memory references, provider routing
|
||||
|
||||
ProviderClients
|
||||
Gemini, Groq, OpenAI-compatible, local LLM, local ASR sidecar
|
||||
|
||||
TrustStatusStore
|
||||
owns visible state, activity log, pause/resume, capture indicators
|
||||
|
||||
Storage
|
||||
Application Support for non-secret app data
|
||||
Keychain for secrets
|
||||
```
|
||||
|
||||
## Agentic System Direction
|
||||
|
||||
The future assistant should be built around modes and tools, not a single chat box.
|
||||
|
||||
Core modes:
|
||||
|
||||
- General Companion
|
||||
- Meeting Assistant
|
||||
- Presentation Coach
|
||||
- Coding Assistant
|
||||
- Research Assistant
|
||||
- Focus Assistant
|
||||
|
||||
Core agent abilities:
|
||||
|
||||
- Listen and summarize.
|
||||
- Answer conversationally.
|
||||
- See screen only when requested.
|
||||
- Track agenda or presentation plan.
|
||||
- Draft follow-up notes.
|
||||
- Remember user preferences.
|
||||
- Use local tools after explicit permission.
|
||||
- Explain what context it used.
|
||||
|
||||
Agent boundaries:
|
||||
|
||||
- No autonomous destructive actions.
|
||||
- No hidden capture.
|
||||
- No silent screen streaming.
|
||||
- No policy bypass tooling.
|
||||
- No stealth process behavior.
|
||||
|
||||
## File Structure Target
|
||||
|
||||
Future native app structure:
|
||||
|
||||
```text
|
||||
native/Mastermind/
|
||||
Mastermind.xcodeproj
|
||||
Mastermind/
|
||||
App/
|
||||
MastermindApp.swift
|
||||
AppCoordinator.swift
|
||||
AppMode.swift
|
||||
PermissionState.swift
|
||||
Status/
|
||||
MenuBarController.swift
|
||||
TrustStatus.swift
|
||||
ActivityLog.swift
|
||||
Overlay/
|
||||
OverlayWindowController.swift
|
||||
OverlayRootView.swift
|
||||
OverlayViewModel.swift
|
||||
Capture/
|
||||
CaptureCoordinator.swift
|
||||
ScreenContextCapture.swift
|
||||
SystemAudioCapture.swift
|
||||
CaptureExclusionPolicy.swift
|
||||
Audio/
|
||||
MicrophoneCaptureEngine.swift
|
||||
AudioPipeline.swift
|
||||
PCMFrame.swift
|
||||
VoiceActivityDetector.swift
|
||||
Assistant/
|
||||
AssistantSession.swift
|
||||
AssistantMode.swift
|
||||
AssistantEvent.swift
|
||||
AssistantMemory.swift
|
||||
Providers/
|
||||
ProviderClient.swift
|
||||
LocalAsrSidecarClient.swift
|
||||
OpenAICompatibleClient.swift
|
||||
GeminiClient.swift
|
||||
GroqClient.swift
|
||||
Storage/
|
||||
AppStorageStore.swift
|
||||
KeychainStore.swift
|
||||
MigrationStore.swift
|
||||
Settings/
|
||||
SettingsWindowController.swift
|
||||
SettingsRootView.swift
|
||||
Presentation/
|
||||
PresentationPlan.swift
|
||||
PresentationCoach.swift
|
||||
```
|
||||
|
||||
Existing Electron files remain read-only during the proof phase except for bridge points explicitly required by a migration task.
|
||||
|
||||
## Task 1: Write Product Reconstruction Charter
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `docs/product/macos-native-ai-companion-charter.md`
|
||||
|
||||
- [ ] **Step 1: Create the product charter**
|
||||
|
||||
Write a concise charter with these sections:
|
||||
|
||||
```markdown
|
||||
# macOS Native AI Companion Charter
|
||||
|
||||
## Positioning
|
||||
|
||||
Mastermind is a native macOS AI companion for everyday computer work, meetings, presentations, research, and focused execution.
|
||||
|
||||
## What It Is
|
||||
|
||||
- A visible personal assistant for the user.
|
||||
- A voice-first and context-aware companion.
|
||||
- A local HUD for notes, guidance, and agent status.
|
||||
- A privacy-conscious screen and audio context tool.
|
||||
|
||||
## What It Is Not
|
||||
|
||||
- A hidden interview helper.
|
||||
- A proctoring bypass tool.
|
||||
- An anti-detection tool.
|
||||
- A process-hiding tool.
|
||||
- A tool for hiding AI use from managed systems that prohibit it.
|
||||
|
||||
## Core Promise
|
||||
|
||||
Mastermind does not hide from the user or the system. It only avoids contaminating the content the user intentionally shares or asks the assistant to analyze.
|
||||
|
||||
## Trust Requirements
|
||||
|
||||
- Show a menu bar status item whenever running.
|
||||
- Show when microphone, system audio, or screen context is active.
|
||||
- Allow immediate pause.
|
||||
- Keep a local activity log of context use.
|
||||
- Store secrets in Keychain.
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Review the charter language**
|
||||
|
||||
Confirm the document includes the phrases `AI companion for your Mac`, `does not hide from the user or the system`, and `visible personal assistant`.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/product/macos-native-ai-companion-charter.md
|
||||
git commit -m "docs: define macos ai companion direction"
|
||||
```
|
||||
|
||||
## Task 2: Build Native Capability Proof Project
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind.xcodeproj`
|
||||
- Create: `native/Mastermind/Mastermind/App/MastermindApp.swift`
|
||||
- Create: `native/Mastermind/Mastermind/App/AppCoordinator.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Status/MenuBarController.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Overlay/OverlayWindowController.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Capture/CaptureCoordinator.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Audio/MicrophoneCaptureEngine.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Audio/AudioPipeline.swift`
|
||||
|
||||
- [ ] **Step 1: Create a minimal native macOS app**
|
||||
|
||||
Create a macOS app target named `Mastermind`. Use Swift, AppKit lifecycle, and SwiftUI only for simple views.
|
||||
|
||||
- [ ] **Step 2: Add a menu bar status item**
|
||||
|
||||
Implement a menu bar item that always appears while the app runs. The menu must include:
|
||||
|
||||
```text
|
||||
Mastermind: Idle
|
||||
Pause All Capture
|
||||
Show Assistant
|
||||
Hide Assistant
|
||||
Settings
|
||||
Quit Mastermind
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Add a transparent overlay window**
|
||||
|
||||
Create an AppKit-controlled floating overlay window with these properties:
|
||||
|
||||
```text
|
||||
borderless
|
||||
transparent background
|
||||
always on top
|
||||
visible across Spaces
|
||||
does not steal focus when shown
|
||||
can become click-through
|
||||
can be hidden and restored
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add screen context proof**
|
||||
|
||||
Use ScreenCaptureKit to capture the main display. The proof succeeds only when captured frames exclude the assistant overlay.
|
||||
|
||||
- [ ] **Step 5: Add system audio proof**
|
||||
|
||||
Use ScreenCaptureKit audio output to receive system audio buffers.
|
||||
|
||||
- [ ] **Step 6: Add microphone proof**
|
||||
|
||||
Use AVAudioEngine to receive microphone PCM buffers.
|
||||
|
||||
- [ ] **Step 7: Add 16 kHz PCM output proof**
|
||||
|
||||
Convert microphone and system audio streams into 16 kHz signed 16-bit PCM frames.
|
||||
|
||||
- [ ] **Step 8: Manual verification**
|
||||
|
||||
Run the native app and verify:
|
||||
|
||||
```text
|
||||
menu bar item is visible
|
||||
overlay is visible locally
|
||||
overlay can become click-through
|
||||
overlay does not appear in app-owned captured frames
|
||||
system audio buffers arrive when another app plays audio
|
||||
microphone buffers arrive when speaking
|
||||
capture can be paused immediately
|
||||
```
|
||||
|
||||
- [ ] **Step 9: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind
|
||||
git commit -m "feat: add native macos capture proof"
|
||||
```
|
||||
|
||||
## Task 3: Define Trust State Machine
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind/Status/TrustStatus.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Status/ActivityLog.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/Status/MenuBarController.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/App/AppCoordinator.swift`
|
||||
|
||||
- [ ] **Step 1: Define trust states**
|
||||
|
||||
Use these states exactly:
|
||||
|
||||
```swift
|
||||
enum TrustStatus: Equatable {
|
||||
case idle
|
||||
case listening
|
||||
case systemAudio
|
||||
case screenSnapshot
|
||||
case screenStream
|
||||
case agentWorking
|
||||
case paused
|
||||
case permissionNeeded(String)
|
||||
case error(String)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Define context activity events**
|
||||
|
||||
Use these events exactly:
|
||||
|
||||
```swift
|
||||
enum ActivityEventKind: String, Codable {
|
||||
case microphoneStarted
|
||||
case microphoneStopped
|
||||
case systemAudioStarted
|
||||
case systemAudioStopped
|
||||
case screenSnapshotCaptured
|
||||
case screenStreamStarted
|
||||
case screenStreamStopped
|
||||
case providerRequestStarted
|
||||
case providerRequestFinished
|
||||
case capturePaused
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Wire status to menu bar copy**
|
||||
|
||||
Map trust states to visible labels:
|
||||
|
||||
```text
|
||||
idle -> Mastermind: Idle
|
||||
listening -> Mastermind: Listening
|
||||
systemAudio -> Mastermind: System Audio
|
||||
screenSnapshot -> Mastermind: Screen Snapshot
|
||||
screenStream -> Mastermind: Screen Stream
|
||||
agentWorking -> Mastermind: Agent Working
|
||||
paused -> Mastermind: Paused
|
||||
permissionNeeded -> Mastermind: Permission Needed
|
||||
error -> Mastermind: Error
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify state changes manually**
|
||||
|
||||
Trigger each capture path and confirm the menu bar label changes before any capture data leaves the machine.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind/Mastermind/Status native/Mastermind/Mastermind/App
|
||||
git commit -m "feat: add native trust status model"
|
||||
```
|
||||
|
||||
## Task 4: Preserve Local ASR Sidecar Boundary
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind/Providers/ProviderClient.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Providers/LocalAsrSidecarClient.swift`
|
||||
- Modify: `docs/local-sidecar-protocol.md`
|
||||
|
||||
- [ ] **Step 1: Define provider boundary**
|
||||
|
||||
Use this protocol as the native client boundary:
|
||||
|
||||
```swift
|
||||
protocol ProviderClient {
|
||||
associatedtype Event
|
||||
|
||||
func start() async throws
|
||||
func stop() async
|
||||
var events: AsyncStream<Event> { get }
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Mirror the current ASR sidecar protocol**
|
||||
|
||||
Implement the native client against the existing WebSocket flow:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "start",
|
||||
"sampleRate": 16000,
|
||||
"channels": 1,
|
||||
"encoding": "pcm_s16le",
|
||||
"language": "en-US"
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Preserve event semantics**
|
||||
|
||||
Support these sidecar events:
|
||||
|
||||
```text
|
||||
ready
|
||||
partial
|
||||
final
|
||||
error
|
||||
```
|
||||
|
||||
Only `final` transcript events should enter the assistant response pipeline by default.
|
||||
|
||||
- [ ] **Step 4: Document native compatibility**
|
||||
|
||||
Add a section to `docs/local-sidecar-protocol.md`:
|
||||
|
||||
```markdown
|
||||
## Native macOS Client Compatibility
|
||||
|
||||
The Swift-native app uses the same WebSocket protocol as the Electron app. Audio frames are sent as raw 16 kHz mono signed 16-bit little-endian PCM. The sidecar does not need to know whether the client is Electron or Swift.
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind/Mastermind/Providers docs/local-sidecar-protocol.md
|
||||
git commit -m "feat: add native local asr sidecar client"
|
||||
```
|
||||
|
||||
## Task 5: Design Native Audio Pipeline
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind/Audio/PCMFrame.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Audio/AudioPipeline.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Audio/VoiceActivityDetector.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/Audio/MicrophoneCaptureEngine.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/Capture/CaptureCoordinator.swift`
|
||||
|
||||
- [ ] **Step 1: Define PCM frame format**
|
||||
|
||||
Use this data model:
|
||||
|
||||
```swift
|
||||
struct PCMFrame: Equatable {
|
||||
let source: AudioSource
|
||||
let sampleRate: Int
|
||||
let channels: Int
|
||||
let pcmS16LE: Data
|
||||
let timestamp: Date
|
||||
}
|
||||
|
||||
enum AudioSource: String {
|
||||
case microphone
|
||||
case system
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Keep microphone and system audio separate**
|
||||
|
||||
The pipeline should not mix microphone and system audio before transcription. Separate streams preserve future diarization and meeting-context quality.
|
||||
|
||||
- [ ] **Step 3: Resample to 16 kHz**
|
||||
|
||||
Every frame sent to local ASR must be:
|
||||
|
||||
```text
|
||||
sample rate: 16000
|
||||
channels: 1
|
||||
encoding: signed 16-bit little-endian PCM
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add VAD boundary**
|
||||
|
||||
Voice activity detection should emit speech segments instead of forcing every buffer into transcription.
|
||||
|
||||
- [ ] **Step 5: Manual verification**
|
||||
|
||||
Verify:
|
||||
|
||||
```text
|
||||
mic frames continue when system audio is silent
|
||||
system frames continue when mic is silent
|
||||
both streams can be paused together
|
||||
either stream can be disabled independently
|
||||
sidecar receives valid 16 kHz PCM frames
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind/Mastermind/Audio native/Mastermind/Mastermind/Capture
|
||||
git commit -m "feat: add native audio pipeline model"
|
||||
```
|
||||
|
||||
## Task 6: Migrate Secrets and Storage Boundaries
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind/Storage/KeychainStore.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Storage/AppStorageStore.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Storage/MigrationStore.swift`
|
||||
- Reference: `src/storage.js`
|
||||
|
||||
- [ ] **Step 1: Map current storage**
|
||||
|
||||
Preserve these current logical groups:
|
||||
|
||||
```text
|
||||
config
|
||||
credentials
|
||||
preferences
|
||||
keybinds
|
||||
limits
|
||||
history
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Move secrets to Keychain**
|
||||
|
||||
Store these values in Keychain:
|
||||
|
||||
```text
|
||||
Gemini API key
|
||||
Groq API key
|
||||
OpenAI-compatible API key
|
||||
local LLM API key
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Store non-secret data in Application Support**
|
||||
|
||||
Store these values in Application Support:
|
||||
|
||||
```text
|
||||
preferences
|
||||
profiles
|
||||
keybinds
|
||||
history
|
||||
limits
|
||||
activity log
|
||||
presentation plans
|
||||
assistant memory references
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add one-way migration**
|
||||
|
||||
Read existing Electron JSON files from:
|
||||
|
||||
```text
|
||||
~/Library/Application Support/cheating-daddy-config
|
||||
```
|
||||
|
||||
Import values into native stores. Leave the old files untouched.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind/Mastermind/Storage
|
||||
git commit -m "feat: add native storage migration boundary"
|
||||
```
|
||||
|
||||
## Task 7: Build Presentation Coach Mode
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `native/Mastermind/Mastermind/Presentation/PresentationPlan.swift`
|
||||
- Create: `native/Mastermind/Mastermind/Presentation/PresentationCoach.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/Assistant/AssistantMode.swift`
|
||||
- Modify: `native/Mastermind/Mastermind/Overlay/OverlayRootView.swift`
|
||||
|
||||
- [ ] **Step 1: Define presentation plan model**
|
||||
|
||||
Use this model:
|
||||
|
||||
```swift
|
||||
struct PresentationPlan: Codable, Equatable {
|
||||
var title: String
|
||||
var sections: [PresentationSection]
|
||||
}
|
||||
|
||||
struct PresentationSection: Codable, Equatable {
|
||||
var title: String
|
||||
var talkingPoints: [String]
|
||||
var expectedDurationSeconds: Int
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add presentation mode**
|
||||
|
||||
Add a mode named `presentationCoach`.
|
||||
|
||||
- [ ] **Step 3: Show local guidance in HUD**
|
||||
|
||||
HUD should show:
|
||||
|
||||
```text
|
||||
current section
|
||||
next talking point
|
||||
elapsed time
|
||||
suggested transition
|
||||
likely audience question
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Keep guidance out of shared content**
|
||||
|
||||
Use the same overlay exclusion policy as Screen Context Mode. The local HUD should be visible to the user and absent from app-owned captures.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add native/Mastermind/Mastermind/Presentation native/Mastermind/Mastermind/Assistant native/Mastermind/Mastermind/Overlay
|
||||
git commit -m "feat: add presentation coach mode"
|
||||
```
|
||||
|
||||
## Task 8: Define Electron Retirement Gates
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `docs/migration/electron-retirement-gates.md`
|
||||
|
||||
- [ ] **Step 1: Create retirement gate checklist**
|
||||
|
||||
Write this checklist:
|
||||
|
||||
```markdown
|
||||
# Electron Retirement Gates
|
||||
|
||||
- [ ] Native overlay matches or exceeds Electron window behavior.
|
||||
- [ ] Native menu bar status item is always visible while running.
|
||||
- [ ] Native screen context excludes Mastermind UI.
|
||||
- [ ] Native system audio capture works without third-party loopback drivers.
|
||||
- [ ] Native microphone capture works independently from system audio.
|
||||
- [ ] Native local ASR sidecar client can transcribe 16 kHz PCM.
|
||||
- [ ] Native provider client can stream OpenAI-compatible responses.
|
||||
- [ ] Native settings can import current Electron preferences.
|
||||
- [ ] Native Keychain storage replaces JSON credential storage.
|
||||
- [ ] Native package can be signed and notarized.
|
||||
- [ ] Electron app remains available as fallback until native app covers core workflows.
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/migration/electron-retirement-gates.md
|
||||
git commit -m "docs: add electron retirement gates"
|
||||
```
|
||||
|
||||
## Task 9: Package and Distribution Direction
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `docs/migration/native-distribution.md`
|
||||
|
||||
- [ ] **Step 1: Document distribution requirements**
|
||||
|
||||
Include:
|
||||
|
||||
```text
|
||||
Developer ID signing
|
||||
notarization
|
||||
Screen Recording permission messaging
|
||||
Microphone permission messaging
|
||||
optional Accessibility permission messaging
|
||||
Sparkle or equivalent update path
|
||||
DMG distribution
|
||||
crash reporting decision
|
||||
local log export
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Document permission copy**
|
||||
|
||||
Use transparent user-facing copy:
|
||||
|
||||
```text
|
||||
Mastermind needs Screen Recording permission only when you ask it to use screen context.
|
||||
Mastermind needs Microphone permission only when voice input or meeting listening is enabled.
|
||||
Mastermind shows a menu bar status item whenever it is running.
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/migration/native-distribution.md
|
||||
git commit -m "docs: define native distribution requirements"
|
||||
```
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
Before claiming the reconstruction direction is ready for implementation:
|
||||
|
||||
- [ ] The charter clearly says this is an AI companion, not a hidden helper.
|
||||
- [ ] The plan preserves local ASR sidecar compatibility.
|
||||
- [ ] The plan does not require a full rewrite before validating native capture.
|
||||
- [ ] The plan includes a visible menu bar status item.
|
||||
- [ ] The plan separates user-visible HUD behavior from stealth behavior.
|
||||
- [ ] The plan includes microphone, system audio, and screen context as separate states.
|
||||
- [ ] The plan includes Keychain for secrets.
|
||||
- [ ] The plan includes Electron retirement gates.
|
||||
- [ ] The plan keeps current Electron app functional during migration.
|
||||
|
||||
## Recommended First Milestone
|
||||
|
||||
The first milestone should be **Native Capability Proof**, not full product migration.
|
||||
|
||||
Success definition:
|
||||
|
||||
```text
|
||||
A Swift/AppKit app shows a local transparent overlay, displays an always-visible menu bar status item, captures screen frames without its own overlay, receives system audio, receives microphone audio, and emits 16 kHz PCM frames compatible with the existing local ASR sidecar protocol.
|
||||
```
|
||||
|
||||
If this milestone fails, keep improving the Electron app while reassessing native capture options. If it succeeds, move provider clients and agent modes into the native shell incrementally.
|
||||
|
||||
Reference in New Issue
Block a user