KV Cache La Gi?
KV Cache la bo nho dem chua cap Key-Value duoc tinh toan truoc do trong Transformer, giup model khong phai tinh lai tu dau moi lan sinh token moi. Khi ban chat voi ChatGPT, moi token duoc sinh ra deu dua tren toan bo token truoc do, va KV Cache chinh la cach model “nho” nhuung phep tinh nay de khong phai lam lai.
Noi cach khac, KV Cache giong nhu ban ghi chu khi giai toan. Ban khong phai tinh lai buoc 1, buoc 2, buoc 3 moi khi den buoc 4. Chi can xem lai ghi chu la xong.
Tai Sao KV Cache Quan Trong?
Trong co che Attention cua Transformer, moi token moi can tinh do lien ket voi tat ca token truoc do. Neu khong co cache, moi lan sinh token lai phai tinh lai toan bo phep Attention tu dau. Voi 1.000 token, dieu nay nghia la 1.000 lan tinh toan lap lai.
KV Cache giai quyet van de nay bang cach luu tru Key va Value cua tung token da qua. Khi token moi can tinh Attention, no chi can doc tu cache thay vi tinh lai toan bo. Ket qua la toc do sinh token (tokens/second) tang dot bien.
Trong thuc te, KV Cache co the tang toc do sinh token len 10-100 lan so voi khong cache, dac biet khi sinh van ban dai. Day la ly do ChatGPT co the tra loi nhanh ngay ca khi ban gui mot prompt rat dai.
Cau Tao Cua KV Cache
Trong Transformer, moi Attention Head tai moi layer se luu tru hai gia tri cho moi token:
- Key (K): Vector dai dien cho noi dung cua token, dung de tinh do lien ket
- Value (V): Vector chua thong tin cua token, dung de tinh dau ra cua Attention
Khi token moi duoc sinh, model chi can tinh Query (Q) cho token moi do, roi nhan voi tat ca Key trong cache de tinh diem Attention. Sau do nhan voi Value tuong ung de lay dau ra. Khong can tinh lai K va V cho cac token cu.
Van De Memory Cua KV Cache
Khong may, KV Cache an rat nhieu RAM. Luong memory can thoi tang tuyen tinh theo chieu dai context. Cu the, voi model nhu Llama 70B va context 32K token, KV Cache chiem khoang 8GB VRAM chi cho rieng viec luu tru.
Day la mot con so lon. Va chinh dieu nay da tro thanh tham chot co chan cua viec mo rong context window trong cac AI model hien nay.
Mot so ky thuat giam memory KV Cache
De giai quyet, cac nha nghien cuu da phat trien nhieu ky thuat:
- Quantization KV Cache: Luu Key-Value o dinh dang 8-bit hoac 4-bit thay vi 16-bit, giam memory 2-4 lan voi mat sat chat luong rat nho
- Paged Attention (vLLM): Chia KV Cache thanh cac page nho giong he dieu hanh quan ly RAM, giam phan vung memory khong duoc su dung
- Grouped Query Attention (GQA): Nhom cac Query Head chia se chung KV, giam so luong Key-Value can luu
- Sliding Window Attention: Chi luu KV cua token trong cua so cuon, bo qua token xa
KV Cache Khac Gi Voi Cache Thong Thuong?
Cache thong thuong (nhu Redis, Memcached) luu ket qua cua request de tai su dung sau. KV Cache cung tuong tu nhucung co mot diem khac biet quan trong: no luu tru trang thai tinh toan giua cac token, khong phai du lieu nguoi dung.
KV Cache la tai nguyen cua mot phien lam viec. Khi ban mo cuoc tro chuyen moi, cache cu bi xoa va bat dau lai. No khong the chia se giua cac nguoi dung vi moi nguoi co context khac nhau.
Yeu Cau Memory KV Cache Tinh Nhu The Nao?
Cong thuc tinh memory KV Cache phu thuoc 4 yeu to:
Memory = 2 x num_layers x num_heads x head_dim x seq_len x batch_size x dtype_size
Vi du voi GPT-3 175B:
- 96 layers, 96 heads, dim 128
- Seq len 2048, batch 1, FP16
- Ket qua: khoang 2.4 GB cho moi request
Khi tang len 32K token context, con so nay nhay len 38 GB. Khong nga nhien viec chay context dai can GPU đoi khi den 8 cap H100.
Vi Du Thuc Te
vLLM cua UC Berkeley da phat trien Paged Attention, giam memory KV Cache len den 60% va tang throughput len 2-4 lan. Day la ly do vLLM tro thanh inference engine pho bien nhat cho production.
Flash Attention cua Tri Dao cung toi uuu viec doc ghi KV Cache, giam memory bandwidth va tang toc do. No khong giam kich thuoc cache nhung lam viec truy cap cache nhanh hon nhieu.
Loi Va Nhuoc Diem Cua KV Cache
KV Cache khong phai hoan hao. No ton VRAM rat nhieu, dac biet voi model lon va context dai. Khi nhieu nguoi dung cung luc, moi phien can mot cache rieng, dan den viec GPU het memory nhanh chong.
Mot so ung dung phai gioi han so nguoi dung cung luc hoac chieu dai context chi vi KV Cache day RAM. Day la mot trong nhung thach thuc lon nhat cua AI inference hien nay.
Tuong Lai Cua KV Cache
Cac huong nghien cuu dang dien ra:
- Multi-Query Attention (MQA): Tat ca Query Head chia se chung mot cap KV, giam memory rat nhieu
- Hybrid cache: Luu token quan trong trong cache, bo token khong quan trong, giong cach bo nao cong cuong thong tin
- Persistent KV Cache: Luu cache giua cac request de tai su dung cho prompt giong nhau
Thuc te, cac model nhu GPT-5.6 va Claude da ap dung nhieu ky thuat nay de ho tro context window len den 1-2 trieu token ma khong het memory.
Ket Luan
KV Cache la mot trong nhung sang tao quan trong nhat khien Transformer co the chay nhanh trong thuc te. Khong co no, ChatGPT se cham chom va ton GPU den muc khong the trien khai.
Neu ban dang hoc ve AI inference hieu sau, hieu KV Cache la buoc khong the bo qua. No la cau noi giua ly thuyet Attention cua Transformer va hieu nang thuc te cua chatbot AI.
