Reverse Babelfish — Implementation Prompt for LLMs

Specification for building a macOS background dictation and translation utility powered by MLX-Whisper on Apple Silicon.

1. What to Build

A macOS-only Python application that acts as a system-wide hold-to-dictate tool. The user holds the right Command key, speaks, and on release the spoken audio is transcribed (and optionally translated to English) by a local MLX-Whisper model. The result is pasted into whatever text field is currently focused. A small floating overlay pill shows status throughout, and a menu bar icon provides runtime controls.

Core User Flow

1. The app runs as a background process (no dock icon by default) with a menu bar icon.
2. The user presses and holds the right Command key (keyCode 54).
3. Microphone recording starts immediately. A floating overlay appears showing "Listening..." with a red dot.
4. Optionally, interim transcription results update the overlay in near-realtime while the user speaks.
5. The user releases the right Command key. Recording stops. The overlay switches to "Transcribing..." with a yellow dot.
6. MLX-Whisper runs full inference on the recorded audio.
7. The resulting text is placed on the clipboard and a synthetic Cmd+V is fired to paste into the active app.
8. The overlay briefly shows "Inserted" with a green dot, then auto-hides.

State Machine

The entire app lifecycle is governed by three states:

• IDLE — waiting for the hotkey.
• RECORDING — microphone is active, user is holding the key.
• PROCESSING — key released, ASR is running on a background thread.

Legal transitions: IDLE → RECORDING (key press), RECORDING → PROCESSING (key release with enough audio), RECORDING → IDLE (cancellation if recording is too short), PROCESSING → IDLE (transcription finished or failed). All other transitions are illegal and must raise an error.

---

2. Platform and Tooling

• macOS only. The app depends on Quartz CGEventTap for global key detection, AppKit for the overlay and menu bar, and NSPasteboard for clipboard access.
• Python >= 3.11 with type hints.
• Build system: Use hatchling as the build backend with a pyproject.toml and src/ layout.
• CLI entrypoint: A console script that invokes the main bootstrap function.
• Dependencies: mlx-whisper (speech recognition engine), numpy (audio array manipulation), sounddevice (microphone capture), pyobjc (macOS framework bindings for Quartz, AppKit, Cocoa).
• Testing: pytest for unit tests.

---

3. macOS Permissions

The app requires three macOS permissions. Detect when each is missing and emit clear, actionable error messages pointing to the relevant System Settings path.

• Input Monitoring — required for the CGEventTap that detects global key events. If CGEventTapCreate returns None, this permission is missing.
• Microphone — required for audio recording via sounddevice.
• Accessibility — required for posting synthetic keyboard events (Cmd+V paste, Cmd+Z undo for live preview). Without this, paste injection silently fails.

---

4. Module Architecture

Organize the code into clearly separated modules with single responsibilities. The boundaries are:

State Machine

Implement a thread-safe finite state machine with the three states and transition rules described above. Use an enum for states. Protect state access with a threading.Lock. Accept an optional on_change(old_state, new_state) callback that fires outside the lock after every successful transition (to prevent deadlocks if the callback reads state). Raise a custom error on illegal transitions. Provide convenience predicates (is_idle(), is_recording(), is_processing()).

Configuration

Define a dataclass with all configurable fields and sensible defaults:

Field	Type	Default	Purpose
model	str	A large MLX Whisper model repo id	Local path or HF repo id for the Whisper model
translate	bool	True	Translate to English (True) vs. transcribe only (False)
language	str or None	None	ISO-639-1 source language code; None means auto-detect
sample_rate	int	16000	Audio sample rate (Whisper's native rate)
min_duration	float	0.3	Reject recordings shorter than this (seconds)
interim_interval	float	1.0	Minimum seconds between interim decode runs
interim_preview	bool	False	Show interim transcription in the overlay while recording
in_field_preview	bool	False	Paste interim text directly into the focused field
restore_clipboard	bool	True	Restore previous clipboard contents after paste
log_level	str	INFO	Logging level (DEBUG, INFO, WARNING, ERROR)
foreground	bool	False	Run as a foreground app with dock icon instead of background

Parsing precedence (lowest to highest priority):

1. Dataclass defaults
2. Saved preferences file (JSON at ~/Library/Application Support/reverse-babelfish/preferences.json)
3. Environment variables (prefixed RB_, e.g. RB_MODEL, RB_TRANSLATE)
4. Explicit CLI arguments

CLI arguments must default to None internally so "not specified" is distinguishable from "explicitly set" — only override lower layers when the user explicitly passed a flag.

Boolean environment variable parsing: "0", "false", "no" (case-insensitive) map to False; everything else maps to True.

Provide a function to persist user preferences (translate, language) to the JSON file — called when the user changes settings via the status bar menu so they survive restarts.

CLI flags to support: --model, --translate/--no-translate, --language, --min-duration, --interim-interval, --interim-preview/--no-interim-preview, --in-field-preview/--no-in-field-preview, --no-restore-clipboard, --foreground, --log-level.

Global Hotkey Listener

Detect the right Command key (keyCode 54) globally using a Quartz CGEventTap.

Critical macOS detail: Right Command is a modifier key. It does NOT fire kCGEventKeyDown/kCGEventKeyUp. It ONLY fires kCGEventFlagsChanged. The detection logic must:

1. Listen for kCGEventFlagsChanged events only.
2. Read kCGKeyboardEventKeycode from the event and verify it equals 54 (right Command).
3. Check kCGEventFlagMaskCommand in the event flags to determine whether the key went down (flag set, key not previously held) or up (flag cleared, key was held).
4. Track a _held boolean to detect edges and ignore auto-repeat.

Run the CGEventTap on a private CFRunLoop in a daemon thread so it never blocks the main thread. Accept on_press and on_release callbacks. Always return the event from the tap callback — failing to return blocks all keyboard input system-wide. Wrap the callback body in try/except so errors never kill the tap.

Audio Capture

Use sounddevice.InputStream for non-blocking microphone capture. Record mono float32 at 16 kHz with a blocksize of ~1024 samples (~64ms). Accumulate audio chunks in a list protected by a threading.Lock. Provide methods to start, stop, get the concatenated audio as a 1-D numpy array, and query the buffered duration. Accept an optional on_chunk callback for interim decode — invoke it from the sounddevice callback thread, wrapped in try/except.

MLX-Whisper Pipeline

Wrap mlx_whisper in a pipeline class that handles model resolution, lazy loading, and concurrency.

Model resolution logic:

• If the model string is a local directory path (expand ~), use it directly.
• If it contains / (looks like a Hugging Face repo id), resolve from the local HF cache only using huggingface_hub.snapshot_download(repo_id=..., local_files_only=True). Raise a clear error if the model is not cached — do not download.
• Otherwise, pass through as-is and let mlx_whisper handle it.

Lazy loading: Import mlx_whisper only when loading the model (not at module import time) to keep startup fast. Provide a load() method called once at startup.

Inference methods: transcribe(audio) for final full-utterance decode after key release, and transcribe_chunk(audio) for interim partial decode. Both call mlx_whisper.transcribe() under the hood.

Runtime option updates: Provide set_options(task, language) to change translate/transcribe mode and source language at runtime (called when the user toggles settings in the menu bar).

Concurrency guard: MLX-Whisper is NOT safe for concurrent inference calls. Use a threading.Lock (_infer_lock) around ALL mlx_whisper.transcribe() calls. Interim and final decodes must be serialized.

Text Injection

Implement clipboard-based text pasting into the active app:

1. Optionally save the current clipboard contents.
2. Place the new text on the clipboard via NSPasteboard.
3. Sleep briefly (~50ms) so the target app sees the clipboard change.
4. Simulate Cmd+V by creating CGEvent key-down and key-up events (keycode 9 for 'v') with kCGEventFlagMaskCommand flags, posting both via CGEventPost.
5. If restoring, sleep ~150ms then put the original contents back.

Live Preview Injector: Implement a separate class for in-field preview during recording. It tracks whether interim text has been pasted and uses Cmd+Z to undo the previous interim paste before inserting updated text. Methods: update(text) to replace interim text, finalize(text) to undo interim and inject final text, cancel() to undo interim and clean up. Use a lock for thread safety.

Overlay UI

Build a floating status pill using PyObjC/AppKit:

• Use an NSPanel with borderless + non-activating style mask.
• The panel must be always-on-top (NSFloatingWindowLevel + 1), non-activating (must NEVER steal keyboard focus), mouse-ignoring (setIgnoresMouseEvents_(True)), and visible on all Spaces/desktops.
• Position at bottom-center of the primary screen, slightly above the bottom edge.
• Content: a rounded dark translucent pill containing a colored dot and a status label.
• Three visual states with distinct dot colors: red for recording ("Listening..."), yellow for processing ("Transcribing..."), green for done ("Inserted").
• When interim preview is enabled, update the label with a compact preview of the transcription (truncated with ellipsis if too long).
• Auto-hide after showing "Inserted" or error messages (~1.5 seconds).
• Error state: yellow dot with a custom error message.
• All public methods must be thread-safe — schedule UI work on the main thread via NSOperationQueue.mainQueue().
• Lazy-build the panel on first use.

Status Bar (Menu Bar) Controller

Create a macOS menu bar item with runtime controls:

• Use NSStatusBar.systemStatusBar() to create a status item.
• Use SF Symbols for the icon when available: "mic" for idle, "mic.fill" for recording, "waveform" for processing. Fall back to text titles if SF Symbols are unavailable.
• Build an NSMenu with these items:
  • Translate to English — checkbox toggle.
  • Interim Preview — checkbox toggle.
  • In-Field Preview — checkbox toggle.
  • Separator.
  • Input Language submenu: Auto Detect, English, German, Spanish, French — with checkmark on the selected item.
  • Separator.
  • Quit (with Cmd+Q key equivalent).
• Use a helper NSObject subclass for menu action targets (required by PyObjC's Objective-C runtime).
• Accept callbacks for each toggle/selection change and for quit.
• Expose methods to update the icon state and menu checkmarks from outside.
• All UI mutations must run on the main thread.

Logging

Provide a simple setup_logging(level) function that configures the root logger with a stderr handler, a timestamp + level + module name format, and clears existing handlers on re-call. All modules use logging.getLogger(__name__).

---

5. Application Bootstrap and Orchestration

The main module wires all subsystems together in an App class.

Background Launch

By default, the CLI spawns a detached child process and the parent exits immediately:

• Use an environment variable guard (e.g. RB_BACKGROUND_CHILD=1) to detect the child. If the variable is set, this is the child — proceed with normal app startup.
• If --foreground is passed, skip detach and run in the calling terminal.
• The child is spawned via subprocess with start_new_session=True and stdin/stdout/stderr redirected to devnull for complete detachment.

App Initialization

The App class constructor creates all subsystems and wires their callbacks:

• State machine with an on_change callback that logs transitions and updates the status bar icon.
• Overlay, pipeline, audio capture, status bar, hotkey listener, and (conditionally) a live preview injector.
• Interim decode state: a buffer, lock, in-flight flag, last-decode timestamp, and a generation counter.

Event Loop

1. Create the shared NSApplication instance.
2. Set the activation policy: NSApplicationActivationPolicyAccessory for background mode or NSApplicationActivationPolicyRegular for foreground.
3. Call finishLaunching().
4. Start the status bar, load the Whisper model, start the hotkey listener.
5. Install a SIGINT handler for clean Ctrl+C shutdown.
6. Do NOT use NSApp.run(). It blocks Python's signal handling. Instead, manually pump events in a while loop using nextEventMatchingMask_untilDate_inMode_dequeue_ with a ~200ms timeout. Check a _running flag each iteration to support clean shutdown.

Hotkey Press Handler

When the right Command key is pressed:

1. Transition IDLE → RECORDING.
2. Reset all interim decode state (clear buffer, increment generation counter).
3. Start audio capture.
4. Show the overlay in recording state.

Hotkey Release Handler

When the right Command key is released:

1. Stop audio capture and cancel any in-flight interim decodes (increment generation counter).
2. Check the recorded duration against min_duration. If too short, transition RECORDING → IDLE (cancel), hide the overlay, cancel live preview if active, and return.
3. Transition RECORDING → PROCESSING.
4. Show the overlay in processing state.
5. Spawn a background thread for full ASR processing.

Interim Decode Flow

When interim_preview is enabled, the audio chunk callback drives throttled interim decodes:

• Accumulate chunks in a shared buffer (under lock).
• When enough time has elapsed since the last decode (respecting interim_interval, with a hard minimum floor of ~0.25s) and no interim decode is in-flight, snapshot the buffer and spawn a one-shot interim thread.
• The interim thread calls pipeline.transcribe_chunk(), then:
  • Checks the generation counter — if it no longer matches, discard the result (stale).
  • Updates the overlay with a compact preview (collapse whitespace, truncate with "..." if too long, prefix with "Listening: ").
  • Optionally updates the live preview injector (if in-field preview is enabled).

Full ASR Processing Thread

Runs after key release on a background thread:

1. Get the full concatenated audio from the capture buffer.
2. Call pipeline.transcribe(audio).
3. On success with text: finalize live preview (or direct inject), show overlay "Inserted".
4. On empty result: cancel live preview, hide overlay.
5. On exception: log error, cancel live preview, show overlay error.
6. Always transition PROCESSING → IDLE.

Runtime Option Changes (from Status Bar Callbacks)

• Translate toggle: Update config, call pipeline.set_options(), persist preferences.
• Language change: Update config, call pipeline.set_options(), persist preferences.
• Interim preview toggle: Update config. If disabling, also disable in-field preview and revert overlay text to "Listening...".
• In-field preview toggle: If enabling, also enable interim preview if it's off. If disabling, cancel any active live preview. Refresh the live preview injector instance.

In-field preview implies interim preview. Enabling in-field must also enable interim. Disabling interim must also disable in-field.

Clean Shutdown

The stop() method disables the hotkey listener, stops audio capture, stops the status bar, and sets _running = False to exit the event loop. It is triggered by Ctrl+C (SIGINT) or the Quit menu item.

---

6. Threading Model

The app uses multiple threads that must be carefully coordinated:

Thread	Role	Constraints
Main thread (AppKit)	Event dispatch, overlay UI, status bar	All UI mutations must happen here
Hotkey listener (daemon)	CFRunLoop for CGEventTap	Must not block — callbacks return quickly
sounddevice callback	Audio chunk delivery	Real-time audio thread — must not block or allocate
ASR worker	Full transcription after key release	One-shot per utterance
Interim ASR worker	Throttled partial decode during recording	One-shot, generation-gated

All locks and their purpose:

• State machine lock — protects state transitions.
• Audio capture lock — protects the chunk buffer and stream lifecycle.
• Pipeline inference lock — serializes all mlx_whisper.transcribe() calls (MLX is not thread-safe).
• Pipeline options lock — protects task/language for concurrent reads.
• Interim decode lock — protects the interim buffer, in-flight flag, and generation counter.
• Live preview injector lock — protects interim injection state.
• Overlay auto-hide timer — protected against concurrent schedule/cancel.

Never hold two locks simultaneously. Fire callbacks outside locks to prevent deadlocks.

---

7. Key Design Constraints and Pitfalls

1. Right Command is a modifier key. Never use keyDown/keyUp for it. Only kCGEventFlagsChanged fires for modifier keys on macOS.

2. Always return the event from the CGEventTap callback. Failing to return blocks all keyboard input system-wide.

3. Overlay must never steal focus. Use NSWindowStyleMaskNonactivatingPanel and setIgnoresMouseEvents_(True). If the overlay steals focus, the user's cursor leaves their text field and the paste target changes.

4. Do not use NSApp.run(). It blocks Python's signal handling. Manually pump events with nextEventMatchingMask_untilDate_inMode_dequeue_ in a loop.

5. MLX inference is not thread-safe. Always serialize with a lock. Concurrent interim + final transcribe calls will crash or corrupt.

6. Clipboard timing matters. After placing text on the clipboard, pause ~50ms before sending Cmd+V so the target app's event loop sees the new content. After pasting, pause ~150ms before restoring the original clipboard so the app finishes consuming it.

7. Stale interim results. Use a generation counter that increments on every new recording and on key release. Discard results from prior generations silently.

8. Config precedence. CLI must override env, env must override saved preferences, saved preferences must override defaults. CLI args that weren't passed must remain None so they don't clobber lower layers.

9. In-field preview implies interim preview. Enabling in-field must auto-enable interim. Disabling interim must auto-disable in-field.

10. Live preview uses Cmd+Z. The injector undoes previous interim pastes before inserting updated text. This works with standard Cocoa text fields but may not work in all apps — this is an accepted limitation.

11. Background detach. The default launch forks a detached child. The child uses an env var guard to prevent infinite spawn loops. --foreground bypasses this.

12. State machine callbacks fire outside the lock. This is deliberate — it prevents deadlocks when the callback needs to read or transition state.

13. Model path resolution is offline-only. If the model is a Hugging Face repo id, resolve from the local cache. Never download. Fail with a clear message if the model is missing.

---

8. Testing Strategy

Write tests using pytest. Keep tests deterministic, fast, and free of macOS framework dependencies — mock or avoid AppKit/Quartz in unit tests.

State Machine Tests

• Verify initial state is IDLE.
• Test the full happy-path cycle: IDLE → RECORDING → PROCESSING → IDLE.
• Test the cancel path: RECORDING → IDLE.
• Verify illegal transitions (IDLE → PROCESSING, PROCESSING → RECORDING) raise the custom error.
• Verify the on_change callback receives correct (old, new) pairs in order.
• Test the convenience predicates against each state.

Configuration Tests

• Test all default values are correct.
• Test each CLI flag individually overrides the correct field.
• Test each environment variable overrides the correct field.
• Test precedence: CLI > env > saved preferences > defaults.
• Test preference file persistence: saving writes correct JSON, loading restores values.
• Isolate the home directory (use tmp_path + monkeypatch) to avoid reading real preference files during tests.

Bootstrap Tests

• Test the background child detection logic: returns True by default, False when foreground is set, False when the env guard is already present.
• Test CLI argument preservation in the background spawn command.
• Test the preview text formatting helper: empty input falls back to default label, whitespace collapses, long strings truncate with ellipsis.

---

9. Development Workflow

Set up a virtual environment, install the package in editable mode, run tests, then launch:

python3 -m venv .venv
source .venv/bin/activate
pip install -e .
pytest
reverse-babelfish --foreground --log-level DEBUG

Use --foreground during development to keep the app attached to the terminal for log visibility. Use --log-level DEBUG to see state transitions and hotkey press/release events.

---

10. Implementation Order

Build incrementally so each layer can be tested before wiring the next:

1. Package scaffold — build config, version export, logging setup.
2. State machine with tests.
3. Configuration parsing with tests.
4. Audio capture — verify manually with a short recording script.
5. MLX-Whisper pipeline — verify with a hardcoded audio file.
6. Text injection — verify manually by pasting into a text editor.
7. Hotkey listener — verify with print-based callbacks.
8. Overlay — verify each visual state in isolation.
9. Main orchestrator — wire everything into the App class with the manual event loop.
10. Status bar — add the menu with toggle callbacks wired to the App.
11. Interim preview — chunk callback, throttled decode, generation tracking, overlay updates.
12. Live in-field preview — LivePreviewInjector with undo-based replacement.
13. Background launch with detach logic and env guard.
14. Polish — error states, preference persistence, bootstrap tests, README.