app development context
This commit is contained in:
Илья Глазунов
@@ -0,0 +1,87 @@
|
||||
# Companion Island
|
||||
|
||||
## Purpose
|
||||
|
||||
The Companion Island is Mastermind's only primary interaction surface. It keeps the assistant absent from the desktop until requested while making it available at a stable physical location.
|
||||
|
||||
It is inspired by the expansion behavior of Dynamic Island, but it is a Mastermind concept and must not be described as an Apple system feature.
|
||||
|
||||
## Placement
|
||||
|
||||
- The Companion Island belongs to the primary display only.
|
||||
- Its activation area is centered on the display's top edge.
|
||||
- On a notched display, the activation area follows the camera housing.
|
||||
- On a display without a notch, the same area behaves as a virtual camera housing.
|
||||
- Changing the primary display relocates the Companion Island.
|
||||
|
||||
## Interaction states
|
||||
|
||||
### Hidden
|
||||
|
||||
No Companion Island content is visible. Moving the pointer into the activation area begins the reveal transition.
|
||||
|
||||
### Revealed
|
||||
|
||||
A compact capsule grows from the camera area without taking keyboard focus. Leaving the activation region without clicking collapses it after a short grace period.
|
||||
|
||||
### Expanded
|
||||
|
||||
Clicking the revealed capsule expands it downward into an interactive panel and focuses text input. Expansion must feel spatially connected to the camera area rather than like an unrelated window appearing.
|
||||
|
||||
The expanded surface contains navigation for:
|
||||
|
||||
- Assistant;
|
||||
- Assistant History;
|
||||
- Sources;
|
||||
- Local Provider;
|
||||
- Context Graph and Context Receipts;
|
||||
- Activity Log;
|
||||
- Privacy, storage, export/import, and diagnostics.
|
||||
|
||||
Escape, an explicit close action, or clicking outside the panel returns it to Hidden. Long-running local work continues after collapse and is visible when the panel is reopened.
|
||||
|
||||
### Paused
|
||||
|
||||
Paused is a Collector condition, not a separate window mode. The user can still open the Companion Island, search existing context, and use Assistant History while all Collectors remain stopped.
|
||||
|
||||
## Menu Bar Item
|
||||
|
||||
The Menu Bar Item is visible whenever Mastermind runs. Its glyph does not change with capture state.
|
||||
|
||||
Its menu must expose:
|
||||
|
||||
- current state of every enabled Collector;
|
||||
- permission and processing errors;
|
||||
- Pause All or Resume All;
|
||||
- Show Mastermind;
|
||||
- Launch at Login state;
|
||||
- Quit Mastermind.
|
||||
|
||||
Show Mastermind expands the Companion Island and is the fallback when pointer activation is unavailable. The MVP has no global hotkey.
|
||||
|
||||
## Focus and accessibility
|
||||
|
||||
- Hidden and Revealed do not steal focus.
|
||||
- Expanded accepts keyboard focus and text input.
|
||||
- Pointer activation must not create a dead strip that prevents access to the macOS menu bar.
|
||||
- Animation respects Reduce Motion.
|
||||
- The interface remains keyboard-navigable after it is expanded.
|
||||
- Collector state and errors are conveyed with text, not color alone.
|
||||
|
||||
## Capture behavior
|
||||
|
||||
Mastermind excludes its own windows from the Screen Collector. It also requests exclusion from third-party capture where supported by macOS.
|
||||
|
||||
Capture exclusion is best effort. The UI must not promise that the Companion Island is invisible to every screen-sharing or recording application. Mastermind never hides its process or system permission indicators.
|
||||
|
||||
## Acceptance checks
|
||||
|
||||
- At launch, only the Menu Bar Item is visible.
|
||||
- Hovering the primary display's camera area reveals the capsule.
|
||||
- A notchless primary display gets the same top-center interaction.
|
||||
- A click expands the panel and focuses text input.
|
||||
- Show Mastermind works when hover activation cannot be used.
|
||||
- Escape and click-away collapse the panel.
|
||||
- Pause All does not prevent access to existing knowledge or history.
|
||||
- The Companion Island is absent from Mastermind's own captured frames.
|
||||
- Changing the primary display relocates the activation area.
|
||||
@@ -0,0 +1,134 @@
|
||||
# Mastermind Product Brief
|
||||
|
||||
## Product thesis
|
||||
|
||||
Mastermind is a personal, local-first assistant for macOS. It continuously builds an inspectable understanding of the user's work and answers grounded questions about what happened, what matters now, and what may come next.
|
||||
|
||||
Mastermind is not an interview helper, a hidden proctoring tool, or a generic shell around a cloud agent. The first product is a personal/internal Swift application for one macOS user.
|
||||
|
||||
## Core promise
|
||||
|
||||
The user's machine context remains on the Mac. Screen, audio, files, calendar, reminders, and terminal activity are reduced to useful knowledge locally. Mastermind does not transmit that context to a cloud service in the MVP.
|
||||
|
||||
The assistant distinguishes observed evidence, inferred claims, and user-confirmed facts. Every grounded answer can show what evidence it used and where that evidence came from.
|
||||
|
||||
## Target platform
|
||||
|
||||
- Apple Silicon Mac.
|
||||
- macOS 14 or newer.
|
||||
- One Local Profile for the current macOS user.
|
||||
- No Mastermind account, backend, telemetry, or synchronization in the MVP.
|
||||
- The domain keeps a future cloud identity separate from the Local Profile.
|
||||
- Personal/internal distribution; Mac App Store constraints are out of scope.
|
||||
|
||||
## Primary experience
|
||||
|
||||
Mastermind runs as a menu bar application without a Dock presence. Its Menu Bar Item is always present while the app runs and uses a stable glyph. Opening its menu reveals Collector states, permission failures, Pause All or Resume All, Show Mastermind, and Quit.
|
||||
|
||||
The Companion Island is hidden by default. Hovering the top-center camera area of the primary display reveals a compact capsule with a smooth animation. Clicking expands the capsule into the complete Mastermind interface. A virtual top-center activation area provides the same behavior when the primary display has no physical notch. Show Mastermind in the menu is the fallback; the MVP has no global shortcut or voice invocation.
|
||||
|
||||
The expanded Companion Island contains:
|
||||
|
||||
- text input and streamed answers;
|
||||
- Assistant History;
|
||||
- Context Receipts and Fact correction;
|
||||
- Source and Collector management;
|
||||
- Local Provider configuration;
|
||||
- Context Graph limits and encrypted export/import;
|
||||
- Activity Log and diagnostics.
|
||||
|
||||
Mastermind is on-demand, not proactive. Background collection may update status and knowledge, but the assistant does not interrupt the user with unsolicited advice.
|
||||
|
||||
## MVP sources
|
||||
|
||||
Enabled Collectors start automatically with the application unless Pause All was previously selected:
|
||||
|
||||
- Screen: the active display, defined by the frontmost key window, with pointer display and primary display as fallbacks.
|
||||
- Audio: separate microphone and system-audio channels.
|
||||
- Workspace: explicitly connected directories and Git repositories.
|
||||
- Calendar and Reminders: user-selected calendars and lists, read-only.
|
||||
- Terminal: shell integration metadata including working directory, command, exit status, duration, and Git metadata; terminal output is not retained by default.
|
||||
|
||||
Workspace indexing respects `.gitignore`, binary and size limits, and user-configured exclusions. Mastermind does not index the entire home directory.
|
||||
|
||||
The user configures the local capture denylist. Mastermind does not silently add application-level exclusions, but Pause All is always available.
|
||||
|
||||
## Context behavior
|
||||
|
||||
Collectors run continuously and adapt their work to meaningful changes, voice activity, duplication, Low Power Mode, and thermal pressure. A failed Collector degrades independently while the others continue.
|
||||
|
||||
Screen processing combines Accessibility metadata, changed-region detection, and local OCR. Audio uses separate 16 kHz PCM streams and local ASR; it distinguishes User Speech from System Speech but does not identify people by voice. Meeting inference may combine calendar, conferencing-application, and channel-activity evidence, but it must remain an Assertion until confirmed.
|
||||
|
||||
Raw screen frames, audio, OCR text, and full transcripts are ephemeral. Only locally derived Observations, Assertions, short summaries, and Provenance survive the processing buffer. If required local processing is unavailable, Mastermind records a Gap instead of retaining raw content for later.
|
||||
|
||||
Inferred knowledge expires by `lastObservedAt`, with a default limit of 90 days and 2 GB. User-confirmed Facts are pinned until manually removed or superseded. Assistant History has no automatic age limit; the user can delete one Assistant Session or all history.
|
||||
|
||||
## Assistant behavior
|
||||
|
||||
The default and only MVP Provider is a user-configured OpenAI-compatible Local Provider reachable through loopback or a Unix socket. The user may save multiple profiles but selects one active profile.
|
||||
|
||||
Mastermind supplies its own local multilingual embedding component. Local ASR remains behind the documented sidecar protocol, with `whisper.cpp` as the recommended implementation. Russian, English, and mixed Russian-English work are required.
|
||||
|
||||
Answers must:
|
||||
|
||||
- distinguish Facts from unconfirmed Assertions;
|
||||
- cite relevant Provenance through a Context Receipt;
|
||||
- expose the active Provider;
|
||||
- say when context is missing or conflicting;
|
||||
- treat all Source content as untrusted evidence rather than instructions;
|
||||
- produce advice, plans, and drafts only.
|
||||
|
||||
The MVP cannot click, type into other applications, run tools, change files, or perform external actions. Computer Control and a Tool Executor are future bounded contexts, not empty runtime abstractions in the MVP.
|
||||
|
||||
## Primary acceptance scenario
|
||||
|
||||
After Mastermind has observed normal work, the user opens the Companion Island and asks:
|
||||
|
||||
> What was I working on, and what should I do next?
|
||||
|
||||
The Local Provider returns a grounded answer using relevant screen, workspace, calendar, terminal, and audio knowledge. The answer separates confirmed Facts from uncertain Assertions, links to a Context Receipt, and explicitly identifies gaps or contradictions.
|
||||
|
||||
## Privacy and trust
|
||||
|
||||
- All sensitive extraction and inference are local in the MVP.
|
||||
- The local Context Graph is encrypted with an application key protected by Keychain.
|
||||
- Provider credentials are stored separately in Keychain.
|
||||
- Pause All immediately stops every Collector and remains paused across restarts.
|
||||
- System microphone and screen-recording indicators are never bypassed.
|
||||
- The Menu Bar Item glyph stays visually stable, while its menu exposes actual Collector states.
|
||||
- The Activity Log records lifecycle and transfer metadata without Source content.
|
||||
- Diagnostic exports are sanitized and user-initiated.
|
||||
- Context Graph and settings can be exported as an encrypted archive without Provider credentials.
|
||||
|
||||
Mastermind may exclude the Companion Island from its own capture and from third-party capture where macOS supports it. This is best effort, must be self-checked where possible, and is never presented as a guarantee.
|
||||
|
||||
Mastermind does not hide its process, bundle identifier, permissions, network activity, or capture activity from macOS, administrators, or monitoring tools.
|
||||
|
||||
## Codex direction
|
||||
|
||||
Codex is not part of the MVP. The architecture documents a future Provider boundary and Provider Context Permission, but the MVP contains no Codex UI, OAuth flow, app-server integration, or experimental flag.
|
||||
|
||||
The integration remains gated until Codex exposes a supported boundary that cannot execute tools. A read-only sandbox is insufficient because the current app-server remains an agent protocol.
|
||||
|
||||
If a future Cloud Provider is enabled:
|
||||
|
||||
- context permission is global for that Provider and denied by default;
|
||||
- the user must enable it manually;
|
||||
- Mastermind still minimizes context and filters detected secrets locally;
|
||||
- every request produces a Context Receipt;
|
||||
- revocation blocks future requests but cannot retract already transmitted data.
|
||||
|
||||
## Explicit non-goals
|
||||
|
||||
- Cloud inference or context transfer in the MVP.
|
||||
- Autonomous actions, Computer Control, or tool execution.
|
||||
- Proactive suggestions or scheduled briefings.
|
||||
- Wake word, push-to-talk, global shortcut, or voice queries.
|
||||
- Browser, mail, messages, clipboard, or full home-directory indexing.
|
||||
- Voice identity and speaker attribution to a Person.
|
||||
- Owning or synchronizing the user's tasks and calendar.
|
||||
- Multi-display Companion Island behavior.
|
||||
- A visual whole-graph explorer.
|
||||
- Electron feature development or an Electron bridge.
|
||||
- Intel Mac, Windows, Linux, or Mac App Store support.
|
||||
- Stealth, anti-detection, permission bypass, or guaranteed screen-share invisibility.
|
||||
@@ -0,0 +1,87 @@
|
||||
# MVP Acceptance
|
||||
|
||||
The MVP is complete only when the following scenarios work together in the production Swift application. Passing the current native capability proof alone is not sufficient.
|
||||
|
||||
## Product scenario
|
||||
|
||||
Given that enabled Collectors have observed normal work, when the user opens the Companion Island and asks, "What was I working on, and what should I do next?", then Mastermind:
|
||||
|
||||
- responds through the configured Local Provider;
|
||||
- uses relevant knowledge from screen, workspace, calendar, terminal, and audio Sources;
|
||||
- distinguishes user-confirmed Facts from inferred Assertions;
|
||||
- identifies conflicting or stale knowledge;
|
||||
- states when evidence is insufficient;
|
||||
- attaches an inspectable Context Receipt.
|
||||
|
||||
## Presence
|
||||
|
||||
- Mastermind launches as a menu bar application without a Dock icon.
|
||||
- Only the stable Menu Bar Item is visible at rest.
|
||||
- Hover reveals the Companion Island on the primary display.
|
||||
- Click expands it and focuses text input.
|
||||
- Show Mastermind in the menu provides a reliable fallback.
|
||||
- The expanded surface contains assistant, history, source, provider, graph, audit, and privacy controls.
|
||||
|
||||
## Continuous collection
|
||||
|
||||
- Enabled Collectors start after launch unless paused state was persisted.
|
||||
- Screen collection follows the display containing the key window and processes meaningful changes rather than every full frame.
|
||||
- Microphone and system audio remain separate through local ASR.
|
||||
- Workspace collection is limited to explicitly connected roots.
|
||||
- Calendar and Reminders are read-only and limited to selected calendars and lists.
|
||||
- Terminal collection retains command metadata but not stdout or stderr by default.
|
||||
- Low Power Mode and thermal pressure reduce processing without corrupting state.
|
||||
- One failed Collector does not stop healthy Collectors.
|
||||
|
||||
## Local-only processing
|
||||
|
||||
- Packet inspection and integration tests confirm that Source content does not leave the Mac.
|
||||
- The configured Local Provider accepts only loopback or Unix-socket endpoints.
|
||||
- OCR, ASR, embeddings, Fact extraction, retrieval, and answer generation work without internet access.
|
||||
- Raw screen frames, audio, OCR text, and full transcripts are destroyed after local reduction.
|
||||
- When a required local processor is unavailable, Mastermind records a Gap and does not queue raw content.
|
||||
|
||||
## Knowledge integrity
|
||||
|
||||
- Observations are immutable until retention removes them.
|
||||
- Assertions retain time, confidence, and Provenance.
|
||||
- Conflicting Assertions coexist and produce an explicit conflict instead of silent overwrite.
|
||||
- Only explicit user confirmation or correction creates a Fact.
|
||||
- Confirmed Facts survive normal age and size eviction.
|
||||
- Ambiguous entity matches stay separate until confidence is sufficient or the user merges them.
|
||||
- Context retrieval treats instructions inside Source content as untrusted data.
|
||||
|
||||
## Retention and control
|
||||
|
||||
- Inferred context defaults to 90 days and 2 GB and can be reconfigured.
|
||||
- Eviction uses `lastObservedAt` and does not evict confirmed Facts.
|
||||
- Assistant History has no automatic TTL.
|
||||
- The user can delete one Assistant Session or all Assistant History.
|
||||
- Pause All immediately stops every Collector and remains paused after relaunch.
|
||||
- Resume All restarts only enabled and permitted Collectors.
|
||||
- The encrypted archive round-trips context, history, policies, and settings without Provider credentials.
|
||||
|
||||
## Trust and diagnostics
|
||||
|
||||
- The Menu Bar Item menu shows the real state of every Collector despite the stable glyph.
|
||||
- macOS capture indicators and permission surfaces remain unmodified.
|
||||
- Activity Log records Collector lifecycle, Gaps, permission errors, and Provider requests without Source content.
|
||||
- Context Receipts name the Provider and all supporting Sources and Observations.
|
||||
- The sanitized diagnostic bundle contains no prompts, answers, Facts, OCR, ASR, or Provider payloads.
|
||||
- Mastermind's windows are excluded from its own screen context.
|
||||
- Third-party capture exclusion is presented and tested as best effort, never guaranteed.
|
||||
|
||||
## Assistant boundary
|
||||
|
||||
- The assistant streams text answers in Russian, English, and mixed-language work.
|
||||
- Outputs are answers, plans, or drafts only.
|
||||
- The production target contains no Computer Control, Tool Executor, autonomous action, Codex login, Codex app-server, or hidden experimental Cloud Provider.
|
||||
- If the Local Provider is unavailable, existing graph search, history, settings, and diagnostics remain accessible.
|
||||
|
||||
## Definition of done
|
||||
|
||||
- Automated tests cover domain invariants, retention, encryption boundaries, policy defaults, independent Collector failure, and raw-buffer disposal.
|
||||
- Integration tests use synthetic Sources and local mock sidecars.
|
||||
- Manual tests cover macOS permissions, microphone, system audio, screen exclusion, primary-display changes, Launch at Login, sleep/wake, Low Power Mode, and relaunch while paused.
|
||||
- Logs and fixtures contain no captured user content.
|
||||
- Canonical documentation and implementation vocabulary agree with `CONTEXT.md`.
|
||||
Reference in New Issue
Block a user