CyGuide

Security tool wrappers are typically glorified shell scripts with pretty formatting. CyGuide takes an entirely different architectural bet: instead of running isolated binaries and printing disjointed terminal text, it maintains a live, relational entity graph of everything discovered across an entire investigation session — so every scan enriches the last rather than starting from scratch.

Conceived, architected, and solo-engineered in Python 3.11+ and Textual, CyGuide unifies network reconnaissance, web fuzzing, OSINT lookups, and vulnerability checkers into an asynchronous, reactive ecosystem without coupling the core framework to any specific tool.

---

Core Architecture: The "Smart Adapter / Stupid Framework" Pattern

CyGuide decouples tool execution from state management through a strict unidirectional layered topology. The engine core understands only standardized findings, async process lifecycles, and relational entity persistence. All binary-specific knowledge (CLI flags, stdout streams, and error handling) is encapsulated in standalone adapters.

┌────────────────────────────────────────────────────────────────────────┐
│                        THE "STUPID" FRAMEWORK                          │
│                                                                        │
│   • Tool-Agnostic Core Engine (cyguide/engine)                         │
│   • Stateful Entity Graph & Audit Event Log (SQLite / aiosqlite)       │
│   • Asynchronous Subprocess Execution & Stream Draining (Executor)     │
│   • Single Programmatic Interface & Policy Gate (Facade & Gateway)     │
└───────────────────────────────────▲────────────────────────────────────┘
                                    │ Emits typed BaseFinding streams
┌───────────────────────────────────┴────────────────────────────────────┐
│                         THE "SMART" ADAPTERS                           │
│                                                                        │
│   • tools/nmap       -> Network host & service discovery               │
│   • tools/gobuster   -> Web directory & endpoint brute-forcing         │
│   • tools/nikto      -> Web server vulnerability scanning & CVE links  │
│   • tools/whois      -> Domain registration & netblock/ASN extraction  │
│   • tools/subfinder  -> Passive DNS subdomain discovery                │
│   • ShellAdapter (!) -> Direct arbitrary shell passthrough (User-only) │
└────────────────────────────────────────────────────────────────────────┘

Detailed Specification: Architecture & System Design Blueprint →

---

Data Model & Entity Graph Engine

CyGuide avoids heavy ORMs and client-server databases, implementing a custom graph engine directly on SQLite 3 with aiosqlite. Persistence is organized across a five-table relational schema:

• Container Hierarchy (workspaces, sessions): Scopes investigation boundaries with enforced foreign-key cascading deletes.
• Stateful Entity Graph (nodes, edges): Stores deduplicated target assets and their directed relationships. Nodes distinguish identity (pik_json) from attributes (data_json), allowing non-destructive merges on updates.
• Immutable Audit Trail (event_log): Records every executed ActionRequest, exit code, raw tool log, and emitted finding.

# Deterministic Node Identity (Primary Identity Key hashing)
def _generate_node_id(self, schema_type: str, pik: dict) -> str:
    pik_str = json.dumps(pik, sort_keys=True)
    return hashlib.sha256(f"{schema_type}:{pik_str}".encode()).hexdigest()

Graph Auto-Wiring & Edge Lifecycle

Tool adapters do not manage database foreign keys or graph edges directly; they simply emit typed BaseFinding objects declaring parent targets or semantic associations. During ingestion (GraphStore.upsert_finding): 1. Parent Relations: Auto-wires directed hierarchy edges (e.g. network.netblock $ o$ network.host $ o$ network.service $ o$ web.endpoint). Missing parent nodes are auto-created as deterministic stubs and seamlessly filled when their real scans complete. 2. Semantic Associations: Auto-wires cross-entity links that transcend tree hierarchy, such as DNS records resolving to hosts (resolves_to), WHOIS records mapping to domains (whois_data_for), or TLS certificates bound to services (has_cert).

Detailed Specification: Database Architecture & SQL Schema Specification →

---

Schema Vocabulary & Canonicalization

All findings in CyGuide conform to a 15-schema Pydantic v2 library across five security domains. Every schema defines its own Primary Identity Key (PIK) fields and strict validators:

Schema Domain	Schema Types	Description & Primary Keys
Network	network.host, network.service, network.netblock	Discovered machines (ip), listening ports (host_ip, port, protocol), and CIDR ownership ranges (cidr).
Web	web.endpoint, web.resource, web.vulnerability	HTTP/S endpoints (url), assets (url, resource_path), and application vulnerabilities (url, vuln_type, parameter).
DNS & OSINT	dns.record, domain.info	DNS resolution entries (domain, type, value) and WHOIS registry records (domain).
Credentials & Host	credential.found, tls.certificate, entity.alias, file.system_object, process.running, software.package	Tokens/passwords, SSL/TLS cert fingerprints, hostname aliases, and host internal processes/packages.
Vulnerability	vulnerability.found	Host/service-level vulnerabilities and CVE associations (host_ip, vuln_id).

Before hashing or storage, every string passes through a Canonicalization Boundary (canonicalize.py) that lowercases hostnames, cleans URLs (stripping redundant :80/:443 default ports), normalizes IPv4/IPv6 localhost representations, and coerces port types — ensuring identical entities generated by different tools resolve to the exact same hash.

Detailed Specification: Standard Schema Registry & PIK Definitions →

---

Two-Mode Operating Paradigm

CyGuide satisfies both security education and professional operational workflows through two dedicated TUI modes:

1. Learning Mode (Guided Mentorship)

Designed for students and newcomers to cybersecurity. * Curated Tool Browser: Explore bundled security tools, structured flags, and pre-configured command recipes. * Hybrid AI Explainer Engine: Real-time streaming first-principles breakdowns of executed commands and discovered vulnerabilities. Supports fully offline local inference via Ollama (gemma, llama3), cloud providers (OpenAI, Anthropic, OpenRouter) over Server-Sent Events (SSE), and a deterministic template fallback.

2. Power Mode (Operator Workbench)

Designed for penetration testers, security analysts, and CTF competitors. * Tabbed Job Workbench: Run multiple tool executions concurrently with independent log tabs, background job tracking, process termination (Ctrl+K), and live process counters. * Universal Context Variable Substitution: PowerWorkspaceFacade.get_context() resolves $TARGET, $PORT, $SERVICE, and $PROTOCOL from whichever node is selected in the entity tree, enabling rapid command execution without typing IP addresses. * Global Entity Graph & Job Queue (Ctrl+G): Workspace-wide merged topology tree and centralized background task monitor with instant cross-session jumping. * Docked Markdown Scratchpad (Ctrl+N): Live investigation notes auto-saved to disk.

Detailed Specification: CyGuide User Manual & Keyboard Reference →

---

Reactive AI Recommendation Engine

CyGuide eliminates manual post-scan correlation by acting as an event-driven subscriber to the entity graph. When newly confirmed findings enter SQLite, declarative recommendation rules evaluate the live topology and propose targeted next actions: * Confirmed open web ports (80, 443, 8080) $ o$ propose directory brute-forcing with gobuster. * Discovered web endpoints $ o$ propose web vulnerability scan with nikto. * Domain-like hostnames $ o$ propose DNS enumeration sweep with subfinder. * Public IP addresses $ o$ propose ASN/netblock lookup with whois.

Proposed findings are written with status='proposed' and surfaced in the interactive Suggestions Pane. Operators can approve suggestions with a single keypress (Enter pre-populates the command palette for instant review/execution) or dismiss them (d/delete) without corrupting verified state.

Detailed Specification: Recommendation Engine Implementation Walkthrough →

---

Zero-Copy Session Portability & Executive Reporting

• Standalone SQLite Archives: CyGuide uses SQLite's native ATTACH DATABASE transactional copying (cyg export <session_id> <path.db> / cyg import / Ctrl+E) to export self-contained session databases. Instructors, graders, and security teams can inspect full topology trees and command timelines with zero JSON serialization overhead.
• Automated Markdown Reporting: One-command executive report generation (--report <path.md>) dynamically synthesizes session metrics, categorizes findings into clean Markdown tables, generates Mermaid topology diagrams, and embeds collapsible chronological execution logs.

Detailed Specification: Session Export & Reporting Architecture Walkthrough →

---

Engineering Roadblocks & Technical Challenges

During the design and construction of the async engine and TUI, several performance bottlenecks and systems challenges were resolved:

• $O(N^2)$ Quadratic Markup Parsing: Textual's text widgets parse Rich markup by default. Appending lines and repainting accumulated logs re-parsed entire buffers with quadratic cost. Solution: Disabled markup on raw terminal stream widgets and used list-based $O(1)$ line accumulators.
• Async Event Loop Starvation: Tight synchronous reader and UI loops starved asyncio, freezing screen repaints and mouse clicks. Solution: Injected cooperative asyncio.sleep yields in process reader loops and delegated subprocesses to Textual @work background workers.
• OS Pipe 4KB Block-Buffering: Subprocesses redirected to pipes defaulted to 4KB block-buffering instead of line-buffering, causing output to appear in bursty stalls. Solution: Implemented concurrent non-blocking stdout/stderr stream drain loops and throttled UI repaints to a capped 20 FPS (every 50ms).
• Database I/O Contention: Opening and closing SQLite connections per port or finding stalled the event loop under rapid ingestion. Solution: Migrated GraphStore to persistent aiosqlite connection pooling with write-ahead locks.
• Headless Linux Clipboard Failures: Clipboard libraries raised unhandled exceptions on headless Linux environments. Solution: Defensively wrapped optional OS clipboard interactions in fallback handlers.

Detailed Post-Mortem: Developer Performance Notes & Post-Mortem Log →

---

Extensibility & Adding Adapters

Extending CyGuide with new security tools is strictly modular. Contributors create a folder under tools/<name>/ with two files: 1. manifest.toml: Declares binary dependencies, execution categories, accepted target schemas, produced finding schemas, and learning recipes. 2. adapter.py: Implements a ToolAdapter subclass defining validate_install(), build_command(), and the async generator parse_output().

class ToolAdapter(ABC):
    @abstractmethod
    def validate_install(self) -> bool: ...
    @abstractmethod
    def build_command(self, target: BaseFinding, params: dict) -> list[str]: ...
    @abstractmethod
    async def parse_output(self, raw_stdout: str, target: BaseFinding, context: dict = None) -> AsyncIterator[BaseFinding]: ...

Detailed Contributor Guide: Contributing & Adapter Authoring Guide →

---

Autonomous Agent Harness Vision (Future Scope)

CyGuide's single programmatic entry point (PowerWorkspaceFacade) and policy gate (ActionGateway) are architected to support autonomous AI agent execution loops directly on top of the deterministic core. In this model, autonomous agents act as external planners submitting standard ActionRequest objects governed by three planned operating tiers: 1. Strict Step-by-Step (Human-in-the-Loop): Every agent-proposed tool action requires explicit operator approval in the TUI. 2. Bounded Semi-Autonomous: Auto-executes read-only reconnaissance within strict target CIDR scopes, pausing on intrusive actions. 3. Full Autonomous Sandbox: End-to-end autonomous scan-analyze-exploit loops in sandboxed environments.

Detailed Design: Autonomous Agent Harness Architecture Specification → • Project Roadmap Tracker →

---

By the Numbers & Governance

• 15 Standard Finding Schemas in the shared vocabulary.
• 86 Automated Tests (100% Passing) across unit, parser fixtures, integration flows, and CLI commands.
• 5 Bundled Reference Adapters (nmap, gobuster, nikto, whois, subfinder) plus a direct ShellAdapter (!).
• Dual AI Explainer Backends: Fully local offline inference via Ollama + cloud API streaming (OpenAI / Anthropic / OpenRouter).
• 3-Tier Governance Architecture: Tier 1 (Frozen Engine/Store), Tier 2 (Review Required for Adapters/Executor), Tier 3 (Free-to-edit UI/Docs).