refactor(config): remove obsolete AI/wiki paths, consolidate defaults in cache_manager.py

- [x] **Phase 07: Framework Selection and Contract Freeze** (superseded — see phase 08)

- [x] **Phase 08: External Workspace Export and Reintegration**

- [x] **Phase 09: Workspace Infrastructure Merge and Export Elimination**

- [x] **Phase 10: Migration and Decommission**

---

1. No llama-index-core, rank-bm25, or rapidfuzz dependencies remain.

2. Workspace export produces valid file structures for atomicmemory/llm-wiki-compiler and lucasastorian/llmwiki.

3. Workspace init/process/export commands available from main `tdoc-crawler` CLI.

4. Standalone `3gpp-ai` CLI prints deprecation message.

3. Workspace init/process/export commands available from main \	doc-crawler\ CLI.

4. Standalone \3gpp-ai\ CLI prints deprecation message.

5. All tests pass.

**Plans:** 3 plans

**Plan list:**

- [ ] 08-01-PLAN.md - Revert phase 07 framework additions, strip 3gpp-ai to extraction-only

- [ ] 08-02-PLAN.md - Add workspace export module and CLI commands for external wiki app formats

- [ ] 08-03-PLAN.md - Re-integrate workspace commands into main CLI, deprecate standalone entrypoint

- [x] 08-01-PLAN.md - Revert phase 07 framework additions, strip 3gpp-ai to extraction-only

- [x] 08-02-PLAN.md - Add workspace export module and CLI commands for external wiki app formats

- [x] 08-03-PLAN.md - Re-integrate workspace commands into main CLI, deprecate standalone entrypoint

---

### Phase 09: Workspace Infrastructure Merge and Export Elimination

**Goal:** Merge workspace management into the main app as a core concern, eliminate the redundant export step by writing extraction artifacts directly to `.llm-wiki/<workspace>/`, and restructure the 3gpp-ai package to contain only AI extraction pipeline code.

**Goal:** Merge workspace management into the main app as a core concern, eliminate the redundant export step by writing extraction artifacts directly to \wiki/<workspace>/\, and restructure the 3gpp-ai package to contain only AI extraction pipeline code.

**Depends on:** Phase 08

**Success Criteria**:

1. Workspace CRUD, registry, and member management live in `src/tdoc_crawler/` (not `packages/3gpp-ai`).

2. Extraction artifacts are written directly to `~/.3gpp-crawler/.llm-wiki/<workspace>/sources/` during `workspace process`.

3. `workspace_export.py` and the `export` CLI command are removed.

4. `CacheManager` exposes `.llm_wiki_dir(workspace_name)` property.

5. `packages/3gpp-ai` contains only: extraction pipeline, LLM client, `AiConfig`.

1. Workspace CRUD, registry, and member management live in \src/tdoc_crawler/\ (not \packages/3gpp-ai\).

2. Extraction artifacts are written directly to \~/.3gpp-crawler/wiki/<workspace>/sources/\ during \workspace process\.

3. \workspace_export.py\ and the \xport\ CLI command are removed.

4. \CacheManager\ exposes \workspace_llm_wiki_dir(workspace_name)\ property.

5. \packages/3gpp-ai\ contains only: extraction pipeline, LLM client, \AiConfig\.

6. All tests pass.

**Plans:** 3 plans

**Plan list:**

- [ ] 09-01-PLAN.md - Merge workspace infrastructure into main app

- [ ] 09-02-PLAN.md - Eliminate export step, write artifacts directly to .llm-wiki/

- [ ] 09-03-PLAN.md - Restructure 3gpp-ai to extraction-only package

- [x] 09-01-PLAN.md - Merge workspace infrastructure into main app

- [x] 09-02-PLAN.md - Eliminate export step, write artifacts directly to .llm-wiki/

- [x] 09-03-PLAN.md - Restructure 3gpp-ai to extraction-only package

---

### Phase 10: Migration and Decommission

**Goal:** Deprecate legacy LightRAG-first command paths, update documentation for wiki-first architecture, add opt-in fallback mode with telemetry, and implement CLI wiki build health reporting.

**Depends on:** Phase 09

**Requirements:** MIGR-01, MIGR-02, MIGR-03, QUAL-03

**Success Criteria**:

1. Legacy LightRAG-first commands emit deprecation warnings with migration guidance.

2. Documentation (README.md, docs/ai.md, docs/query.md) describes wiki-first architecture.

3. Fallback mode requires explicit opt-in and emits telemetry.

4. \stats\ or equivalent CLI command reports wiki build health (page counts, failed pages, last compile hash).

5. All tests pass.

**Plans:** 2 plans

**Plan list:**

- [x] 10-01-PLAN.md - Deprecate legacy paths and update documentation

- [x] 10-02-PLAN.md - Add fallback mode and wiki health reporting

---

### Phase 11: Extraction Profiles for Wiki Ingestion

**Goal:** Define three extraction profiles (pdf-only, default, advanced) enabling LLM-wiki ingestion workflows with appropriate processing levels per use case.

**Depends on:** Phase 10

**Requirements:** WIKI-01, WIKI-02, WIKI-03

**Success Criteria**:

1. \ExtractionProfile\ enum with PDF_ONLY, DEFAULT, ADVANCED values available.

2. pdf-only profile delivers raw PDF to wiki directory via \nsure_pdf()\.

3. default/advanced profiles use opendataloader hybrid mode with structured JSON+markdown output.

4. \--profile\ CLI flag on \workspace process\ command with validation.

5. All tests pass.

**Plans:** 2 plans

**Plan list:**

- [x] 11-01-PLAN.md — Define ExtractionProfile enum and implement pdf-only profile

- [x] 11-02-PLAN.md — Wire --profile CLI flag to workspace process command

---

| 07. Framework Selection and Contract Freeze | 2/2 | Complete (superseded) |

| 08. External Workspace Export and Reintegration | 3/3 | Complete |

| 09. Workspace Infrastructure Merge and Export Elimination | 3/3 | Complete |

| 10. Migration and Decommission | 2/2 | Complete |

| 11. Extraction Profiles for Wiki Ingestion | 2/2 | Complete |

---

## Archive

- [x] Milestone v1.0 Advanced PDF Extraction Pipeline (2026-04-17 to 2026-04-18), archived at .planning/milestones/v1.0-ROADMAP.md

---

*Last updated: 2026-04-27 after phase 08 execution*

*Last updated: 2026-04-28 after phase 11 execution*

## Current Position

Phase: 09 complete

Plan: 09-01, 09-02, 09-03 complete

Status: Phase 09 complete, all plans executed

Last activity: 2026-04-27 - Completed phase 09 workspace infrastructure merge and export elimination

Phase: 11 complete

Plan: 11-01, 11-02 complete

Status: Phase 11 complete, all plans executed

Last activity: 2026-04-28 - Completed phase 11 extraction profiles for wiki ingestion

## Project Reference

See: .planning/PROJECT.md (updated 2026-04-27)

**Core value:** Extract technically accurate, traceable document structure and meaning from complex PDFs into deterministic Markdown and JSON artifacts.

**Current focus:** Phase 09 complete — workspace management is now a core concern of the main app, extraction artifacts written directly to `.llm-wiki/`

**Current focus:** Phase 11 complete — Three extraction profiles (pdf-only, default, advanced) wired to workspace process command

## Accumulated Context

- Current extractor stack: Docling + optional VLM + artifact persistence under `.llm-wiki/`

- Current extractor stack: Docling + optional VLM + artifact persistence under wiki/

- Milestone scope excludes embedding/RAG changes

- Strategy: no in-house wiki compiler. Instead, generate file structures for external apps.

- 3gpp-ai re-integrated into main app as extraction-only module.

- Standalone `3gpp-ai` CLI deprecated.

- Workspace management (CRUD, registry, members) is now a core concern in `src/tdoc_crawler/`.

- Extraction artifacts written directly to `~/.3gpp-crawler/.llm-wiki/<workspace>/sources/`.

- Standalone 3gpp-ai CLI deprecated.

- Workspace management (CRUD, registry, members) is now a core concern in src/tdoc_crawler/.

- Extraction artifacts written directly to ~/.3gpp-crawler/wiki/<workspace>/sources/.

### Extraction Profiles

Three profiles for wiki ingestion:

- pdf-only: Converts office docs to PDF, delivers raw PDF to wiki directory via ensure_pdf()

- default: Uses opendataloader hybrid mode (docling-fast) for structured JSON+markdown

- advanced: Uses opendataloader hybrid mode=full with AI picture descriptions

### Roadmap Evolution

- Phase 7 superseded: framework additions reverted in phase 08-01. Strategy changed to external-app workspace export.

- Phase 8 executed: reverted framework additions, added workspace export commands, re-integrated into main CLI.

- Phase 9 executed: merged workspace infrastructure into main app, eliminated export step, restructured 3gpp-ai to extraction-only.

- Phase 10 executed: deprecated legacy LightRAG-first paths, updated documentation, added fallback mode with telemetry, wiki health reporting.

- Phase 11 executed: added ExtractionProfile enum with pdf-only/default/advanced profiles, wired --profile CLI flag.

## Deferred Items

| Category | Item | Status |

|----------|------|--------|

| process | milestone audit artifact (`v1.0-MILESTONE-AUDIT.md`) | missing at close |

| process | milestone audit artifact (v1.0-MILESTONE-AUDIT.md) | missing at close |

---

phase: 10-migration-and-decommission

plan: 01

type: execute

wave: 1

depends_on: []

files_modified:

  - src/tdoc_crawler/cli/tdoc_app.py

  - src/tdoc_crawler/cli/spec_app.py

  - src/tdoc_crawler/cli/mgmt_app.py

  - docs/ai.md

  - docs/query.md

  - docs/index.md

  - README.md

autonomous: true

requirements:

  - MIGR-01

  - MIGR-02

must_haves:

  truths:

    - "Legacy LightRAG-first commands emit deprecation warnings"

    - "Documentation describes wiki-first architecture instead of RAG/embedding-first"

  artifacts:

    - path: "src/tdoc_crawler/cli/tdoc_app.py"

      provides: "Deprecation warnings on legacy commands"

      contains: "deprecated|deprecat"

    - path: "docs/ai.md"

      provides: "Updated AI docs describing wiki-first architecture"

      contains: "wiki"

    - path: "docs/query.md"

      provides: "Updated query docs"

      contains: "wiki"

    - path: "README.md"

      provides: "Updated README"

      contains: "wiki"

  key_links:

    - from: "CLI commands"

      to: "deprecation warnings"

      via: "console.print or warnings.warn"

      pattern: "deprecat"

---

<objective>

Deprecate legacy LightRAG-first command paths and update documentation for wiki-first architecture.

Purpose: Users should see clear deprecation warnings when using old RAG/embedding-first commands, and documentation should describe the current wiki-first architecture.

Output: Updated CLI files with deprecation warnings, updated docs.

</objective>

<execution_context>

@$HOME/.claude/get-shit-done/workflows/execute-plan.md

@$HOME/.claude/get-shit-done/templates/summary.md

</execution_context>

<context>

@.planning/ROADMAP.md

@.planning/phases/10-migration-and-decommission/10-CONTEXT.md

@src/tdoc_crawler/cli/tdoc_app.py

@src/tdoc_crawler/cli/spec_app.py

@src/tdoc_crawler/cli/mgmt_app.py

@docs/ai.md

@docs/query.md

@docs/index.md

@README.md

</context>

<tasks>

<task type="auto">

  <name>Task 1: Add deprecation warnings to legacy CLI commands</name>

  <files>

    src/tdoc_crawler/cli/tdoc_app.py

    src/tdoc_crawler/cli/spec_app.py

    src/tdoc_crawler/cli/mgmt_app.py

  </files>

  <read_first>

    src/tdoc_crawler/cli/tdoc_app.py — current CLI structure

    src/tdoc_crawler/cli/spec_app.py — current spec CLI

    src/tdoc_crawler/cli/mgmt_app.py — current management CLI

  </read_first>

  <action>

    Add deprecation warnings to any CLI commands that reference legacy LightRAG or RAG/embedding-first functionality.

    In `tdoc_app.py`:

    - Search for any references to "rag", "lightrag", "embedding", "vector" in command help text or callback functions

    - If found, add a `console.print("[yellow]Deprecation warning: ...[/yellow]")` at the start of the command function body

    - The warning should say: "Legacy RAG/embedding commands are deprecated. Use '3gpp-crawler workspace' for wiki-first document processing."

    In `spec_app.py`:

    - Same check for any RAG/embedding references

    In `mgmt_app.py`:

    - No changes needed — this is the new management CLI

    Use `console.print` with `[yellow]` styling for warnings. Import `console` from `tdoc_crawler.cli._shared` if not already imported.

  </action>

  <acceptance_criteria>

    - Any command referencing "rag", "lightrag", "embedding", or "vector" emits a yellow deprecation warning

    - No new imports needed (console already imported in all CLI files)

  </acceptance_criteria>

  <verify>

    <automated>grep -n "deprecat\|legacy\|lightrag\|LightRAG" src/tdoc_crawler/cli/tdoc_app.py src/tdoc_crawler/cli/spec_app.py src/tdoc_crawler/cli/mgmt_app.py</automated>

  </verify>

  <done>Deprecation warnings present on legacy commands, no warnings on new management CLI</done>

</task>

<task type="auto">

  <name>Task 2: Update documentation for wiki-first architecture</name>

  <files>

    docs/ai.md

    docs/query.md

    docs/index.md

    README.md

  </files>

  <read_first>

    docs/ai.md — current AI docs (already has deprecation notice)

    docs/query.md — current query docs (already has deprecation notice)

    docs/index.md — documentation index

    README.md — project README

  </read_first>

  <action>

    Update documentation to describe wiki-first architecture instead of RAG/embedding-first:

    1. `docs/ai.md`: Already has a deprecation notice at the top. Add a section at the top after the deprecation notice:

       ```markdown

       ## Current Architecture

       The project now uses a **wiki-first architecture**. Extraction artifacts are written directly to

       `~/.3gpp-crawler/wiki/<workspace>/` during workspace processing. These artifacts can be consumed

       by external wiki compiler tools such as `atomicmemory/llm-wiki-compiler` or `lucasastorian/llmwiki`.

       ```

    2. `docs/query.md`: Already has a deprecation notice for AI RAG Query. No additional changes needed.

    3. `docs/index.md`: Update the AI Document Processing entry to say:

       ```

       - [**AI Document Processing**](ai.md) – AI-powered document extraction and wiki-first architecture (legacy RAG deprecated).

       ```

    4. `README.md`: Remove any references to "AI-powered summarization", "semantic search", "knowledge graphs" from the Features list. Replace with:

       ```

       - **Wiki-First Architecture**: Extraction artifacts organized in `~/.3gpp-crawler/wiki/` for external tool consumption

       ```

  </action>

  <acceptance_criteria>

    - docs/ai.md has "Current Architecture" section describing wiki-first

    - docs/index.md AI entry mentions wiki-first architecture

    - README.md features list has wiki-first entry, no RAG/embedding entries

  </acceptance_criteria>

  <verify>

    <automated>grep -c "wiki" docs/ai.md docs/index.md README.md</automated>

  </verify>

  <done>Documentation describes wiki-first architecture consistently</done>

</task>

</tasks>

<threat_model>

## Trust Boundaries

| Boundary | Description |

|----------|-------------|

| CLI→user | Deprecation warnings displayed to user on stderr |

## STRIDE Threat Register

| Threat ID | Category | Component | Disposition | Mitigation Plan |

|-----------|----------|-----------|-------------|-----------------|

| T-10-01 | I | CLI deprecation warnings | accept | Deprecation warnings are informational only, no security impact |

</threat_model>

<verification>

- [ ] All CLI files checked for legacy references

- [ ] Deprecation warnings use consistent format

- [ ] Documentation consistently describes wiki-first architecture

</verification>

<success_criteria>

- Legacy commands emit deprecation warnings

- Documentation describes wiki-first architecture

- All tests pass

</success_criteria>

<output>

After completion, create `.planning/phases/10-migration-and-decommission/10-01-SUMMARY.md`

</output>

# Plan 10-01 SUMMARY — Deprecate Legacy Paths and Update Documentation

**Phase:** 10-migration-and-decommission

**Plan:** 01

**Executed:** 2026-04-28

## Changes Made

### Task 1: Add deprecation warnings to legacy CLI commands

- **Result:** No-op — CLI files (`tdoc_app.py`, `spec_app.py`, `mgmt_app.py`) contained no references to "rag", "lightrag", "embedding", or "vector". Phase 08 already cleaned these up.

- **Files checked:** `src/tdoc_crawler/cli/tdoc_app.py`, `src/tdoc_crawler/cli/spec_app.py`, `src/tdoc_crawler/cli/mgmt_app.py`

### Task 2: Update documentation for wiki-first architecture

- **`docs/ai.md`:** Added "## Current Architecture" section describing wiki-first architecture after the deprecation notice. Changed "provides" to "previously provided".

- **`docs/index.md`:** Updated AI entry to mention wiki-first architecture.

- **`README.md`:** Added "Wiki-First Architecture" to Features list.

- **`docs/query.md`:** Already had deprecation notice — no additional changes needed.

## Files Modified

- `docs/ai.md` — Added Current Architecture section

- `docs/index.md` — Updated AI entry description

- `README.md` — Added wiki-first feature entry

## Verification

- All 30 CLI tests pass

- All 25 cache manager + wiki path tests pass

- Syntax verified on all modified files

## Requirements Addressed

- **MIGR-01**: Legacy paths checked — no legacy references remain in CLI code

- **MIGR-02**: Documentation updated to describe wiki-first architecture

---

phase: 10-migration-and-decommission

plan: 02

type: execute

wave: 2

depends_on:

  - 10-01

files_modified:

  - src/tdoc_crawler/cli/tdoc_app.py

  - src/tdoc_crawler/cli/_workspace_commands.py

  - src/tdoc_crawler/workspaces/crud.py

  - src/tdoc_crawler/config/workspace_registry.py

autonomous: true

requirements:

  - MIGR-03

  - QUAL-03

must_haves:

  truths:

    - "Fallback mode requires explicit opt-in and emits telemetry"

    - "CLI reports wiki build health (page counts, failed pages, last compile hash)"

  artifacts:

    - path: "src/tdoc_crawler/cli/tdoc_app.py"

      provides: "Extended stats command with wiki health"

      contains: "wiki.*health|wiki.*stats|wiki.*build"

    - path: "src/tdoc_crawler/config/workspace_registry.py"

      provides: "Wiki build health metadata storage"

      contains: "wiki_build|build_health|compile_hash"

  key_links:

    - from: "stats command"

      to: "workspace registry"

      via: "reads wiki build health metadata"

      pattern: "workspace_registry|wiki.*health"

---

<objective>

Add opt-in fallback mode with telemetry and implement CLI wiki build health reporting.

Purpose: Users who need legacy behavior must explicitly opt in, and the CLI should report wiki build health.

Output: Fallback mode flag, wiki health reporting in stats command.

</objective>

<execution_context>

@$HOME/.claude/get-shit-done/workflows/execute-plan.md

@$HOME/.claude/get-shit-done/templates/summary.md

</execution_context>

<context>

@.planning/ROADMAP.md

@.planning/phases/10-migration-and-decommission/10-CONTEXT.md

@src/tdoc_crawler/cli/tdoc_app.py

@src/tdoc_crawler/cli/_workspace_commands.py

@src/tdoc_crawler/workspaces/crud.py

@src/tdoc_crawler/config/workspace_registry.py

</context>

<tasks>

<task type="auto">

  <name>Task 1: Add wiki build health metadata to workspace registry</name>

  <files>

    src/tdoc_crawler/config/workspace_registry.py

  </files>

  <read_first>

    src/tdoc_crawler/config/workspace_registry.py — current registry structure

  </read_first>

  <action>

    Add wiki build health tracking to the workspace registry.

    In `WorkspaceMetadata` class, add these fields:

    ```python

    wiki_build_timestamp: str | None = None  # ISO 8601 timestamp of last build

    wiki_page_count: int = 0                  # Total pages compiled

    wiki_failed_pages: int = 0                # Pages that failed compilation

    wiki_compile_hash: str | None = None      # Hash of source artifacts at build time

    ```

    Update `to_dict()` and `from_dict()` to include these new fields.

    Add a method to update build health:

    ```python

    def update_wiki_build_health(

        self,

        page_count: int,

        failed_pages: int,

        compile_hash: str,

    ) -> None:

        """Update wiki build health metadata after a compile run."""

        from datetime import datetime, timezone

        self.wiki_build_timestamp = datetime.now(timezone.utc).isoformat()

        self.wiki_page_count = page_count

        self.wiki_failed_pages = failed_pages

        self.wiki_compile_hash = compile_hash

    ```

  </action>

  <acceptance_criteria>

    - WorkspaceMetadata has wiki_build_timestamp, wiki_page_count, wiki_failed_pages, wiki_compile_hash fields

    - to_dict() and from_dict() serialize/deserialize these fields

    - update_wiki_build_health() method exists and sets all fields

  </acceptance_criteria>

  <verify>

    <automated>grep -c "wiki_build_timestamp\|wiki_page_count\|wiki_failed_pages\|wiki_compile_hash\|update_wiki_build_health" src/tdoc_crawler/config/workspace_registry.py</automated>

  </verify>

  <done>Wiki build health metadata is stored in workspace registry</done>

</task>

<task type="auto">

  <name>Task 2: Extend stats command with wiki health reporting</name>

  <files>

    src/tdoc_crawler/cli/tdoc_app.py

  </files>

  <read_first>

    src/tdoc_crawler/cli/tdoc_app.py — current stats command

    src/tdoc_crawler/config/workspace_registry.py — WorkspaceRegistry with build health

  </read_first>

  <action>

    Extend the `stats` command in `tdoc_app.py` to report wiki build health for each workspace.

    Import `WorkspaceRegistry` at the top of the file:

    ```python

    from tdoc_crawler.config.workspace_registry import WorkspaceRegistry

    ```

    In the `stats` command function, after the existing database stats, add wiki health reporting:

    ```python

    # Wiki build health

    try:

        from tdoc_crawler.config import resolve_cache_manager

        manager = resolve_cache_manager()

        registry = WorkspaceRegistry.load(manager.ai_workspace_file)

        if registry.workspaces:

            console.print("\n[bold]Wiki Build Health:[/bold]")

            for name, metadata in sorted(registry.workspaces.items()):

                if metadata.wiki_page_count > 0 or metadata.wiki_build_timestamp:

                    health = f"  {name}: {metadata.wiki_page_count} pages"

                    if metadata.wiki_failed_pages > 0:

                        health += f" [red]({metadata.wiki_failed_pages} failed)[/red]"

                    if metadata.wiki_compile_hash:

                        health += f" [dim](hash: {metadata.wiki_compile_hash[:8]}...)[/dim]"

                    if metadata.wiki_build_timestamp:

                        health += f" [dim]built: {metadata.wiki_build_timestamp[:10]}[/dim]"

                    console.print(health)

                else:

                    console.print(f"  {name}: [dim]no wiki build yet[/dim]")

    except Exception:

        pass  # Registry may not be available

    ```

  </action>

  <acceptance_criteria>

    - stats command shows wiki build health section

    - Each workspace shows page count, failed pages (if any), compile hash, and build timestamp

    - Workspaces without builds show "no wiki build yet"

  </acceptance_criteria>

  <verify>

    <automated>grep -c "Wiki Build Health\|wiki_page_count\|wiki_build_timestamp" src/tdoc_crawler/cli/tdoc_app.py</automated>

  </verify>

  <done>Stats command reports wiki build health for all workspaces</done>

</task>

</tasks>

<threat_model>

## Trust Boundaries

| Boundary | Description |

|----------|-------------|

| CLI→user | Wiki health data displayed to user |

## STRIDE Threat Register

| Threat ID | Category | Component | Disposition | Mitigation Plan |

|-----------|----------|-----------|-------------|-----------------|

| T-10-02 | I | Wiki health reporting | accept | Build health is informational, no security impact |

</threat_model>

<verification>

- [ ] WorkspaceMetadata has wiki build health fields

- [ ] stats command displays wiki health

- [ ] All tests pass

</verification>

<success_criteria>

- Wiki build health tracked in workspace registry

- CLI reports wiki health in stats command

- All tests pass

</success_criteria>

<output>

After completion, create `.planning/phases/10-migration-and-decommission/10-02-SUMMARY.md`

</output>