Michele LaurelliBlog

Prefix caching

Prefix caching: Reusing computation results for an identical prefix shared across requests. System prompts and initial documents can be processed once. Keys, invalidation and privacy must prevent reuse across incompatible contexts or tenants. Examples include Shared system prompt, Fixed document with different queries, KV-cache reuse.

In italiano: Prefix CachingLegge IV · Loss, gradiente e ottimizzatori

System prompts and initial documents can be processed once. Keys, invalidation and privacy must prevent reuse across incompatible contexts or tenants.

Examples

  • Shared system prompt
  • Fixed document with different queries
  • KV-cache reuse