文献详情
Fast Failure Recovery in Vertex-Centric Distributed Graph Processing Systems
文献类型：期刊
通讯作者：Zhang, MH (reprint author), Beijing Inst Technol, Beijing 100081, Peoples R China.
期刊名称：IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING影响因子和分区
年：2019
卷：31
期：4
页码：733-746
ISSN：1041-4347
关键词：Distributed graph processing systems; failure recovery; checkpoint; log; compression; partition-based recovery
所属部门：信息学院
摘要：There is a growing need for distributed graph processing systems to have many more compute nodes processing graph-based Big Data applications, which, however, increases the chance of node failures. To address the issue, we propose a novel recovery scheme to accelerate the recovery process by parallelizing the recomputation. Once a failure occurs, all recomputations are confined to subgraphs that originally reside in the failed compute nodes. When the recovery starts, these subgraphs are reassign ...More
There is a growing need for distributed graph processing systems to have many more compute nodes processing graph-based Big Data applications, which, however, increases the chance of node failures. To address the issue, we propose a novel recovery scheme to accelerate the recovery process by parallelizing the recomputation. Once a failure occurs, all recomputations are confined to subgraphs that originally reside in the failed compute nodes. When the recovery starts, these subgraphs are reassigned to another set of compute nodes, where the recomputation over these subgraphs are conducted in parallel. To minimize the recovery latency, we also develop a reassignment strategy, from these subgraphs to the replaced compute nodes, by properly leveraging the computation and communication cost. We integrate the proposed recovery scheme into Giraph system, a widely used graph processing system. The experimental results over a variety of real graph datasets demonstrate that our proposed recovery scheme outperforms existing recovery methods by up to 30x on a cluster of 40 compute nodes. ...Hide

DOI：10.1109/TKDE.2018.2843361
百度学术：Fast Failure Recovery in Vertex-Centric Distributed Graph Processing Systems
语言：外文
人气指数：4
浏览次数：4
基金：National Key Research and Development Program of China [2018YFB1004400]; National Natural Science Foundation of ChinaNational Natural Science Foundation of China [61502504, U1711261, 61602297, 61732014]
作者其他论文



Reordering Transaction Execution to Boost High Frequency Trading Applications.Zhou, Ningnan, Zhou, Xuan, Zhang, Xiao, et al. .WEB AND BIG DATA, APWEB-WAIM 2017, PT II. 2017, 10367, 169-184.
An adaptive breadth-first search algorithm on integrated architectures.Zhang, Feng, Lin, Heng, Zhai, Jidong, et al. .JOURNAL OF SUPERCOMPUTING. 2018, 74(11,SI), 6135-6155.
Entity set expansion with semantic features of knowledge graphs.Chen, Jun, Chen, Yueguo, Zhang, Xiangling, et al. .JOURNAL OF WEB SEMANTICS. 2018, 52-53, 33-44.
epiCG: A GraphUnit Based Graph Processing Engine on epiC.Shen, Yanyan, Cai, Qingchao, Lu, Wei, et al. .BIG DATA RESEARCH. 2016, 4, 59-69.
Leveraging Fine-Grained Wikipedia Categories for Entity Search.Ma, Denghao, Chen, Yueguo, Chang, Kevin Chen-Chuan, et al. .WEB CONFERENCE 2018: PROCEEDINGS OF THE WORLD WIDE WEB CONFERENCE (WWW2018). 2018, 1623-1632.