New KV cache compaction technique cuts LLM memory 50x without accuracy loss

MIT researchers developed Attention Matching, a KV cache compaction technique that compresses LLM memory by 50x in seconds — without the hours of GPU training that prior methods required.

Nvidia says it can shrink LLM memory 20x without changing model weights

Nvidia's KV Cache Transform Coding (KVTC) compresses LLM key-value cache by 20x without model changes, cutting GPU memory ...
WFXG

Breaking the 100M Token Limit: EverMind's MSA Architecture Achieves Efficient End-to-End Long-Term Memory for LLMs

The research introduces a novel memory architecture called MSA (Memory Sparse Attention). Through a combination of the Memory ...
Nature

Cache Performance and Memory Hierarchy Optimization

The dynamic interplay between processor speed and memory access times has rendered cache performance a critical determinant of computing efficiency. As modern systems increasingly rely on hierarchical ...
Semiconductor Engineering

Freeing Up Near-Memory Capacity For Cache Using Compression Techniques In A Flat Hybrid-Memory Architecture

A technical paper titled “HMComp: Extending Near-Memory Capacity using Compression in Hybrid Memory” was published by researchers at Chalmers University of Technology and ZeroPoint Technologies.
Electronic Design

Hardware Compression Works at the Memory Cache Level

How lossless data compression can reduce memory and power requirements. How ZeroPoint’s compression technology differs from the competition. One can never have enough memory, and one way to get more ...
Semiconductor Engineering

Dynamic KV Cache Scheduling in Heterogeneous Memory Systems for LLM Inference (Rensselaer Polytechnic Institute, IBM)

A new technical paper titled “Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System” was published by researchers at Rensselaer Polytechnic Institute and IBM. “Large ...
TMCnet

Penguin Solutions Introduces Industry's First Production-Ready CXL-Based KV Cache Server

Accelerating memory-dependent AI processes, Penguin's MemoryAI KV cache server increases memory capacity by integrating 3 TB ...
FullCleared on MSN

AMD's 9800X3D processor anchors this high-memory gaming build

The latest Area-51 desktop from Alienware centers around AMD’s Ryzen 7 9800X3D, an 8-core processor with 104MB of total cache ...