Michele LaurelliBlog

Chunking

Chunking: Splitting documents into smaller segments for indexing, retrieval and insertion into model context. Size and overlap affect context preservation and retrieval specificity. Large chunks dilute signal; small chunks can lose definitions, tables or references. Examples include 500-token segments, Paragraph overlap, Chunking that respects headings.

In italiano: ChunkingLegge I · Embedding, feature e spazi vettoriali

Size and overlap affect context preservation and retrieval specificity. Large chunks dilute signal; small chunks can lose definitions, tables or references.

Examples

  • 500-token segments
  • Paragraph overlap
  • Chunking that respects headings