摘要:作为流式大数据计算的主要平台之一，Storm在设计过程中由于缺乏节能的考虑，导致其存在高能耗与低效率的问题.传统的节能策略并未考虑Storm的性能约束，可能会对集群的实时性造成影响.针对这一问题，设计了资源约束模型、最优线程重分配模型以及数据迁移模型.进一步提出了Storm平台下的线程重分配与数据迁移节能策略（energy-efficient strategy based on executor reallocation and data migration in Storm，简称ERDM），包括资源约束算法与数据迁移算法.其中，资源约束算法根据集群各工作节点CPU、内存与网络带宽的资源占用率，判断集群是否允许数据的迁移.数据迁移算法根据资源约束模型与最优线程重分配模型，设计了数据迁移的最优化方法.此外，ERDM通过分配线程减少了节点间的通信开销，并根据大数据流式计算的性能与能效评估ERDM.实验结果表明，与现有研究相比，ERDM能够有效降低节点间通信开销与能耗，并提高集群的性能.



Abstract:As one of the most popular platforms in big data stream computing, Storm is suffering from the problem of high energy consumption and low energy efficiency due to the lack of consideration for energy saving strategy in the design process. Without taking the performance constraint of Storm into consideration, the traditional energy-efficient strategies may affect the real-time performance of cluster. Aiming at this issue, models of the resource constraint, the optimal executor reallocation, and the data migration are set up, and the energy-efficient strategy based on executor reallocation and data migration in Storm (ERDM) is further proposed, while ERDM is composed of resource constraint algorithm and data migration algorithm. The resource constraint algorithm estimates whether the cluster is appropriate for data migration according to the utilization of CPU, memory, and network bandwidth in each work node. The data migration algorithm designs optimal method to migrate data according to the resource constraint model and the optimal executor reallocation model. Moreover, the ERDM allocates the executors so as to reduce communication cost between nodes. The ERDM is evaluated by measuring the cluster performance as well as energy consumption efficiency in big data stream computing environment. The experimental results show that the proposed strategy can reduce communication cost and energy consequence efficiently while the cluster performance is improved compared with existing researches.



PDF全文下载地址:

http://jos.org.cn/jos/article/pdf/6074