Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

First Ingestion and Semantic Search

First Ingestion & Semantic Search

This guide walks through your first end-to-end ingestion and semantic search workflow using the Hedera Guardian AI Toolkit.

Overview

At this stage, your infrastructure services should already be running:

  • Qdrant (vector database)

  • MCP Server

You will complete the end-to-end workflow in five steps.

1

Step 1 — Add Your Documents

Place your methodology files into:

data/input/documents/

Supported formats:

  • PDF

  • DOCX

You may add:

  • Full methodologies (100+ pages)

  • Templates

  • Supporting documentation

Each file will be processed independently and indexed into the vector database.

2

Step 2 — Run Document Ingestion

From the root of the repository:

Standard profile:

docker compose run --rm document-ingestion-worker

GPU profile (if configured):

docker compose -f docker-compose.yml -f docker-compose.gpu.yml run --rm document-ingestion-worker

Low-memory profile:

docker compose -f docker-compose.yml -f docker-compose.low-memory.yml run --rm document-ingestion-worker
3

Step 3 — Verify Ingestion

Once ingestion completes:

  • A Qdrant collection named methodology_documents will exist

  • The collection will contain one point per chunk

You can verify via:

http://localhost:6333/dashboard

Each stored record contains:

  • The chunk text

  • Embedding vectors

  • Metadata

  • LaTeX formulas (when applicable)

4

Step 4 — Connect Your MCP Client

Your MCP Server should already be running at:

http://localhost:9000

To verify:

npx @modelcontextprotocol/inspector --server-url http://localhost:9000/mcp

You should see available tools including:

  • Semantic search tools

  • Schema builder tools

5

Step 5 — Perform Your First Semantic Search

Using an MCP-compatible AI client (e.g., Claude Desktop):

Ask a grounded question such as:

  • What are the applicability conditions defined in VM42?

  • What quantification approaches are defined in this methodology?

  • What data parameters must be monitored during the crediting period?

The AI will:

  1. Call the semantic search tool

  2. Retrieve relevant chunks

  3. Filter by metadata if necessary

  4. Generate a response grounded in document content

Responses may:

  • Cite sections

  • Extract structured data from tables

  • Present formulas

  • Suggest next steps (such as schema generation)

What Happens During Ingestion

When ingestion starts, the system:

  1. Discovers documents in the input folder

  2. Creates a collection in Qdrant (if it does not exist)

  3. Processes each document in parallel

  4. Extracts:

    • Text

    • Section structure

    • Tables

    • Formulas

  5. Performs post-processing

  6. Splits documents into contextual chunks

  7. Converts chunks into vector embeddings

  8. Stores embeddings and metadata in Qdrant

Advanced Processing Steps

The ingestion pipeline includes:

Formula Recognition

  • Formulas are detected

  • Converted to LaTeX

  • Repaired if split across layout blocks

  • Processed using OCR when necessary

Table Normalization

  • Multi-page tables are detected

  • Split tables are merged into single logical units

Metadata Enrichment

Each stored chunk includes metadata such as:

  • Chunk ID

  • Heading

  • Document structure path

  • Page number

  • Source filename

  • has_formula flag

  • has_table flag

These metadata flags allow targeted semantic filtering during search.

How Semantic Search Works

The toolkit uses hybrid retrieval:

  • Dense embeddings (semantic meaning)

  • Sparse retrieval (keyword matching)

Chunks are ranked using reciprocal rank fusion.

The result:

  • High recall

  • High precision

  • Context-aware retrieval

  • Reduced hallucination risk

The AI does not use internet knowledge. It operates only on your ingested documents.

What Success Looks Like

You have successfully completed this stage when:

  • Documents are indexed

  • Qdrant contains chunk records

  • The MCP server exposes tools

  • Your AI client can answer methodology-specific questions

  • Responses reflect actual document content

At this point, you have transformed static PDFs into a searchable knowledge base.

What Comes Next

Now that you can search methodologies semantically, the next step is:

  • Generating Guardian-compatible schemas

  • Extracting formulas into structured definitions

  • Beginning structured digitization workflows

Proceed to:

Schema & Formula Generation

Last updated