Simulation run on this device. Scenario values are not sent anywhere — the engine is fully local.
ai studio · pro
RAG Chunking Workbench
Chunk a sample corpus three ways and grade retrieval by boundary quality.
Fixed, recursive, or semantic — where do boundaries hurt least?
Corpus: a five-paragraph note on KV caching (baked sample). Each strategy shows where its boundaries land.
Fixed (220 chars)
4 chunks · 2 bad boundaries · size spread 70
KV caches store one key tensor and one value tensor per layer, per attention head, for every token already processed. They exist so decoding doesn't recompute attention for prior tokens. Memory grows linearly with seque
nce length and batch size. A 7B model with 32 layers, 8 KV heads, and head dimension 128 stores about 128 KiB per token in 16-bit precision for a single sequence. Batching multiplies the requirement. Serving 32 concurre
nt sequences at 4K context can exceed the model weights themselves. PagedAttention (vLLM) treats the cache like virtual memory: fixed-size pages, shared prefixes, and eviction instead of one giant contiguous buffer per
request. Quantizing the cache to 8 bits halves the footprint with small quality cost for most workloads; 4-bit cache is riskier for long generations.
Recursive (target 220)
5 chunks · 0 bad boundaries · size spread 71
KV caches store one key tensor and one value tensor per layer, per attention head, for every token already processed. They exist so decoding doesn't recompute attention for prior tokens.
Memory grows linearly with sequence length and batch size. A 7B model with 32 layers, 8 KV heads, and head dimension 128 stores about 128 KiB per token in 16-bit precision for a single sequence.
Batching multiplies the requirement. Serving 32 concurrent sequences at 4K context can exceed the model weights themselves.
PagedAttention (vLLM) treats the cache like virtual memory: fixed-size pages, shared prefixes, and eviction instead of one giant contiguous buffer per request.
Quantizing the cache to 8 bits halves the footprint with small quality cost for most workloads; 4-bit cache is riskier for long generations.
Sentence-grouped
5 chunks · 0 bad boundaries · size spread 71
KV caches store one key tensor and one value tensor per layer, per attention head, for every token already processed. They exist so decoding doesn't recompute attention for prior tokens.
Memory grows linearly with sequence length and batch size. A 7B model with 32 layers, 8 KV heads, and head dimension 128 stores about 128 KiB per token in 16-bit precision for a single sequence.
Batching multiplies the requirement. Serving 32 concurrent sequences at 4K context can exceed the model weights themselves.
PagedAttention (vLLM) treats the cache like virtual memory: fixed-size pages, shared prefixes, and eviction instead of one giant contiguous buffer per request.
Quantizing the cache to 8 bits halves the footprint with small quality cost for most workloads; 4-bit cache is riskier for long generations.
Method
- Compares three boundary strategies on the same corpus: fixed character slicing, recursive paragraph→sentence packing, and sentence grouping.
- Reports chunk counts, size spread, and bad boundaries (mid-word cuts) so you can see why naive slicing loses context.