Small Model Hive Mind Assistant

This will be the last Nova interface release, please feel free to contribute, fork, make it your own. Nova plugins will continue to be added. I will be moving on to Omega, which will be able to use the Nova plugin system, but in the form of a streamlined desktop application, rather than a web interface, and focused on 20B+ models, as opposed to 4B+. I also plan to release a ligher weight Nova core, with most of the inbuilt conveniences carved out, meant purely as a plugin framework, so stay tuned and check out my AI suite.

Documentation

• Installation
• Plugin System Developer Guide
• API Reference
• Changelog

Plugins

• https://github.com/doctarock/Sprite-Plugin-for-Home-Assistant
• https://github.com/doctarock/Multilingual-Recognition-Plugin-for-Home-Assistant
• https://github.com/doctarock/Governance-Plugin-for-Home-Assistant
• https://github.com/doctarock/Code-Review-Plugin-for-Home-Assistant
• https://github.com/doctarock/Auto-plan-Plugin-for-Home-Assistant
• https://github.com/doctarock/Browser-Plugin-for-Home-Assistant-playwright-
• https://github.com/doctarock/Philosophy-Plugin-for-Home-Assistant
• https://github.com/doctarock/Wordpress-Bridge-Plugin-for-Home-Assistant
• https://github.com/doctarock/Finance-Plugin-for-Home-Assistant
• https://github.com/doctarock/Mail-Plugin-for-Home-Assistant
• https://github.com/doctarock/Calendar-Plugin-For-Home-Assistant
• https://github.com/doctarock/Project-Plugin-for-Home-Assistant

Support

• Buy me a coffee https://buymeacoffee.com/pixelforgestudio

Nova is a host-side AI orchestration application that combines:

• A web control panel
• Multi-brain LLM routing and execution across local and LAN endpoints
• A persistent queue-first task system
• Secure sandboxed tool execution in Docker
• Retrieval and document intelligence (Qdrant-backed)
• A plugin system for extending capabilities
• Home Assistant / IoT integration with secure instance management and device control tools
• Voice interface with trust-based command gating

It is designed to run continuously as an autonomous or semi-autonomous operator while still allowing direct user supervision.

2. Core User-Facing Features

2.1 Main UI and Control Surface

The web UI includes dedicated tabs for:

• Nova — identity, voice preferences, questions, trust records
• Queue — queued/in-progress/done/failed tasks, repairs, issues, schedules, history
• Brains — model status, endpoints, base brains, specialists, routing config
• Secrets — keychain handles for retrieval, mail, IoT, and custom secrets
• Capabilities — tool catalog, installed skills, capability request tracking
• Plugins — installed plugins, interfaces, operations
• System — gateway health, intake state, SSE logs, regression test runner

2.2 Conversational Intake and Routing

The system supports:

• Direct run (/api/agent/run)
• Triage-first route selection (/api/tasks/triage)
• Queue handoff for deeper worker execution
• Observer-native immediate responses for common requests (time/date/status-style prompts)
• Prompt rewrite assistance using idle helper brains
• Optional worker preflight for ambiguity checks and clarification
• Multi-specialist routing across local and remote Ollama endpoints
• Secure IoT / Home Assistant instance registration and worker tools for device discovery, state readout, and service calls

2.3 Queue and Task Lifecycle

Queue functionality includes:

• Enqueueing, dispatching, removing, aborting, and answering tasks
• Task event stream and per-task history retrieval
• Waiting-for-user task handling and resume flow
• Deduplication of recently queued tasks
• Repair/follow-up monitor views
• Reshape issue tracking and reset workflow
• Queue pause/resume controls

2.4 Voice Interface and Trust

Voice functionality includes:

• Browser speech synthesis and recognition integration
• Passive listening toggle flow
• Voice fingerprint capture and matching
• Threshold-based trust profile matching
• Command allow/block decisions using configured minimum voice trust level
• Persisted trust records that unify email and voice identity concepts
• Voice invitation flow for waiting tasks with yes/acknowledge acceptance and question time support

2.5 3D Avatar and Visual Stage

Avatar system includes:

• Three.js rendering with GLTF model support
• Emotion-to-animation mapping and talking clip rotation
• Stylization presets/effects (post-processing pipeline)
• Configurable room textures/backgrounds
• Configurable prop slots and model placement
• Scene addon extension points for custom visual effects and runtime integrations

3. Automation and Background Intelligence

3.1 Internal Periodic Jobs

The system maintains internal queue-backed recurring jobs for:

• Opportunity scanning
• Prompt memory question maintenance
• Mail-watch sweeps
• Recreation/free-time cycles

3.2 Opportunity and Maintenance Systems

Background logic includes:

• Idle-time workspace opportunity generation
• Helper-scout work-package generation
• Queue maintenance snapshots and reshape follow-up planning
• Automatic skip/backoff behavior based on activity, backlog, and lane capacity

3.3 Recreation Cycle

The recreation subsystem:

• Queues non-deliverable free-time reflection tasks
• Encourages self-directed browsing/thinking/writing
• Validates that personal-note output is actually persisted

4. Retrieval and Document Intelligence

Retrieval domain capabilities:

• Qdrant collection lifecycle and health usage
• Workspace document scanning and normalization
• Content chunking with overlap controls
• Embedding-based indexing and query
• Filtered search (workspace/root/document/source)
• Document overview/search summaries surfaced through observer-native tooling

5. Tooling, Skills, and Approvals

5.1 Tool Catalog and Governance

Tool governance includes:

• Unified catalog across intake and worker scopes
• Risk classification (normal, medium, high, approval)
• Per-tool autonomous approval flags
• Persistent tool registry state

5.2 Capability Request Tracking

The system records unmet capability demand:

• Missing tool requests
• Skill installation requests
• Aggregation and status tracking for open/resolved requests

5.3 Skill Library Integration

Skill features include:

• Search and inspect via clawhub inside sandbox context
• Install into sandbox workspace
• Approved-skill gating before operational usage
• Installed skill inventory and metadata

6. Plugin System

Plugins extend the observer at runtime without modifying core code:

• Dynamic plugin loading from the plugins directory
• Interface and operation registration
• Plugin inventory visible in the Plugins UI tab
• Runtime hook and lifecycle events for queue and worker execution telemetry
• Backward-compatible plugin tab mounting via the observer-compat browser host, supporting both native and legacy tab APIs
• See Plugin System Developer Guide for authoring details

7. Semantic Compression

Tool loop and shell output compression reduces context bloat during long task execution:

• Automatic summarization of large tool outputs
• Shell hook compression for verbose command output
• Configurable compression thresholds
• See Compression Quick Reference for details

8. Security and Isolation Model

8.1 Sandbox Model

Tool execution is isolated in a Docker container with:

• Read-only root filesystem
• Dropped Linux capabilities (--cap-drop ALL)
• no-new-privileges
• PID/memory/CPU limits
• Dedicated writable mounts only for allowed input/output/state paths

8.2 Secrets Management

Secrets are managed via OS keychain (keytar) with handles for:

• Mail agent passwords
• Retrieval/Qdrant API key
• IoT/Home Assistant long-lived access tokens
• Custom handles

8.3 Trust Controls

Trust system supports:

• Source trust levels (unknown, known, trusted)
• Email command minimum trust policy
• Voice command minimum trust policy
• Unified trust records with optional voice signature thresholds

9. API Surface Summary (Grouped)

See docs/API.md for the full endpoint-by-endpoint reference (methods, auth, request/response shapes).

Primary route groups:

• Runtime: /api/runtime/*, /events/*
• Intake/run: /api/agent/run, /api/tasks/triage, /api/prompts/review
• Queue/tasks: /api/tasks/*, /api/queue/control
• Cron/jobs: /api/cron/*
• Config/control: /api/app/config, /api/brains/config, /api/tools/config, /api/secrets/*
• Inspection/output: /api/inspect/*, /api/output/*
• Regressions: /api/regressions/*

10. Testing and Quality Features

Regression support includes:

• Built-in suite definitions for intake, planner, worker, and related flows
• UI-triggered regression execution from the System tab
• Latest persisted report retrieval
• Generated command-line helper for external run parity (run-regressions.js)

11. Runtime Dependencies and Deployment Shape

Expected runtime stack:

• Host Node.js observer process (node server.js)
• Ollama endpoint(s) for model execution
• Docker sandbox container for tool execution
• Qdrant for retrieval/search storage

12. Feature Positioning Snapshot

In practical terms, this application is:

• A local AI operations console
• A queue-first autonomous worker coordinator
• A safety-conscious sandboxed tool runtime
• A multi-brain routing layer across local and LAN Ollama endpoints

Handoff

This repo runs as a host-side Node observer with a Docker sandbox for LLM-controlled tools, plus Ollama for model execution. There is no gateway container.

Special note on security: the environment described is inherently secure, however, the interface is not currently suitable for open web-facing use.

The accuracy on the voice security is dubious. There is no spoofing protection on email trust. Use these features carefully and run this on a local environment.

Trust settings are under the Nova tab.

Current Setup

• Repo root: <your-repo-path>
• Observer app: nova-observer/
• Observer URL: http://127.0.0.1:3220/
• Observer server entry: nova-observer/server.js
• Observer config: nova-observer/observer.config.json
• Observer runtime root: nova-observer/.derpy-observer-runtime
• User-facing output folder: <your-repo-path>/observer-output

Runtime Shape

The observer process runs directly on the host:

• launch command: node server.js
• working directory: nova-observer/
• it owns:
  • the web UI
  • the scheduler
  • the queue
  • mail polling/sending
  • document indexing
  • task orchestration

The LLM does not get host shell access directly. Tool execution is isolated in Docker.

Docker Sandbox

The LLM tool sandbox is the important security boundary.

• container name: derpy-observer-sandbox
• image name: nova-safe
• named volume: derpy-observer-sandbox-state
• container home: /home/nova
• internal working workspace: /home/nova/.observer-sandbox/workspace
• host output export mount: /home/nova/observer-output

The observer creates this container automatically on startup if needed.

Tool install requirements

When adding or restoring a built-in tool for Nova, treat it as a runtime feature, not just a code change.

Required checks:

• if the tool shells out to a system command, that command must exist inside the nova-safe image, not just on the host
• if the tool depends on a language runtime, library, or binary, install that dependency in Dockerfile
• keep the tool name in code, prompts, and diagnostics aligned with the real callable name
• make sure the tool is present in the observer tool catalog so the worker prompt and approval s

This is a shortened preview. Read the full README at the source.