以支撑搜索服务为主的网络操作系统研制

秦伟俊

摘 要：随着信息技术和互联网的发展，搜索引擎作为支撑互联网服务的一个重要的基础设施，其显著的特点就是超大规模，主要体现在数据存储与数据处理两个方面。该课题以网络操作系统新型体系结构为目标，突破大规模分布式数据共享与管理、高效资源调度、虚拟资源管理、大规模分布式系统运维等关键技术，提出了高性能存储引擎与存储系统设计、MapReduce环境中基于虚拟分区的负载平衡策略、异构Hadoop集群环境中性能驱动的数据平衡方法、云计算资源计量方法、分布式计算引擎设计、数据密集型计算系统设计、基于历史信息比例因子动态调整的作业完成时间预测算法、空闲节点评估调度算法、MapReduce作业输入负载均衡策略、应用远程虚拟运营与应用开放平台设计、超大规模数据中心多层次网络互联设计、大规模机群系统监控管理技术、大规模机群系统软件远程部署技术、低功耗高密度服务器和机柜设计技术，以及机器与机群管理系统等多项技术成果。利用百度强大的互联网服务作为支撑，基于超大规模的云计算数据中心，设计并实现具有自主知识产权的新一代网络操作系统。该系统将以支持搜索业务为核心，以数据中心的集群平台为服务支撑，为搜索引擎服务提供超大规模数据存储和数据处理能力，同时为基于搜索的开放应用服务提供运营支撑。

关键词：网络操作系统 搜索服务 数据存储引擎 资源调度 系统监控 Hadoop MapReduce

Abstract： With the development of information technology and Internet， search engine plays an important role of internet service infrastructure with the feature of large scale dimensions， which focus on two aspects of data storage and data processing. With the aim of new architecture of network operating system， this project breaks through key technologies in large-scale distributed data sharing and management， efficient resource scheduling， virtual resource management， large-scale distributed system maintenance， and proposes several technical achievements including high performance storage engine and system design， load balance strategy based on virtual partition in MapReduce environment， performance-driven data balancing method in heterogeneous Hadoop cluster environment， resource measurement method in cloud computing， distributed computing engine design， system design of data intensive computing， job finishing time schedule algorithm based on historical information ratio factor dynamic adjustment， idle node evaluation schedule algorithm， MapReduce job injection load balance strategy， application remote virtual operation and platform design， multi-level network design for large-scale data center， large-scale cluster system monitoring and management， large-scale cluster system software remote deployment， low-energy， high-density server cabinet design， and clustering management system. With the support of internet search service from Baidu Inc.， the project design and implement new generation of network operating system with independent intellectual property rights based on large-scale cloud computing data center， which aims to provide technical supports for search services and clustering platform in data center， large-scale data storage and processing capacity for search engine and operational supports for upper open application services.

Key Words： Network Operating System； Search Service； Data Storage Engine； Resource Scheduling； System Monitoring； Hadoop； MapReduce

閱读全文链接（需实名注册）：http：//www.nstrs.cn/xiangxiBG.aspx？id=65523&flag=1