大数据2026，Vol.12Issue(4)：97-116,20.DOI:10.11959/j.issn.2096-0271.2026004
基于动态权衡模型与SDAPC框架的Hadoop生态系统优化:跨层级存储与计算协同及可信调度策略
Optimization of the Hadoop ecosystem based on the dynamic trade-off model and SDAPC framework:cross-layer storage and computation synergy and trustworthy scheduling strategies
明龙 1胡建龙 1张心译 1王宇翔1
作者信息
- 1. 西北大学网络与数据中心,陕西 西安 710127
- 折叠
摘要
Abstract
To address the core bottlenecks of high latency and low utilization in Hadoop ecosystem resource scheduling for the Zettabyte(ZB)era,this study proposes a cross-layer collaborative optimization method driven by a dynamic trade-off model.A SDAPC collaborative optimization framework integrating quantum and classical algorithms is constructed.This framework first enhances data locality through storage-computation preprocessing to reduce problem complexity;its core innovation lies in mapping the YARN task allocation problem to an Ising model Hamiltonian minimization problem.Global optimization is achieved via D-Wave quantum annealing,while millisecond-level fine-grained adjustments are enabled by a graph attention network-driven deep reinforcement learning strategy.In TPC-DS benchmarks on a 256-node heterogeneous cluster,the method achieves ultra-low scheduling latency of 12.0±2.0ms and resource utilization of 87.3%±3.2%—reducing latency by 79.3%compared to conventional approaches,with GPU utilization fluctuation(σ)decreasing from 35%to 5%.关键词
Hadoop生态系统/Ising模型/D-Wave量子退火机/资源调度/深度强化学习Key words
Hadoop ecosystem/Ising machine/D-wave quantum annealing machine/resource scheduling/deep reinforcement learning分类
信息技术与安全科学引用本文复制引用
明龙,胡建龙,张心译,王宇翔..基于动态权衡模型与SDAPC框架的Hadoop生态系统优化:跨层级存储与计算协同及可信调度策略[J].大数据,2026,12(4):97-116,20.
