摘要
Abstract
As the parameter scale of Large Language Models(LLMs)approaches the trillion-parameter level,memory capacity,communication bandwidth,and inference latency have emerged as major bottlenecks constraining intelligent computing systems.To systematically address these challenges,this paper establishes a full-stack hardware-software co-design framework spanning algorithmic optimization,training and inference systems,and underlying hardware architectures.At the algorithmic level,the framework covers the evolution of low-bit-width quantization,Input/Output(I/O)-aware attention mechanisms,and Key-Value(KV)cache compression.At the system level,it examines hybrid parallelism strategies,automatic parallelism search,and inference-stage techniques,including paged memory management,continuous batching,and speculative decoding.It further reviews specialized accelerator architectures and automatic generation techniques for high-performance operators at the hardware and compilation levels.Existing studies indicate that 4-bit weights can theoretically reduce weight-storage requirements by 75%relative to FP16 weights,while methods such as Accurate Post-Training Quantization for Generative Pre-trained Transformers(GPTQ)and Activation-Aware Weight Quantization(AWQ)have reported inference speedups of more than threefold under specific model,hardware,and implementation configurations.FlashAttention-2 reduces memory complexity from(O(N2))to(O(N))and achieves hardware utilization of 50%～73%.At the training-system level,three-dimensional hybrid parallelism achieves a Model Floating Point Operations(FLOPs)Utilization(MFU)of 54%～56%in training at the scale of thousands of Graphics Processing Unit(GPUs).At the inference-system level,PagedAttention and continuous batching deliver throughput improvements of 2～4×and 36.9×,respectively.At the hardware and compilation level,the H100 Transformer Engine achieves peak speedups of 9×for training and 30×for inference.Through cross-layer hardware-aware optimization and deep coordination,full-stack co-design overcomes the performance limitations of isolated single-layer optimization and provides a systematic technical roadmap for addressing the continued expansion of model scale and the increasing diversity of application scenarios.关键词
大语言模型/软硬件协同设计/分布式训练/推理优化/全栈编译环境Key words
Large Language Models(LLMs)/hardware-software co-design/distributed training/inference optimization/full-stack compilation environment分类
信息技术与安全科学
