A decentralized adaptation of model-free Q-learning for thermal-aware energy-efficient virtual machine placement in cloud data centers

The traditional method of saving energy in Virtual Machine Placement (VMP) is based on consolidating more virtual machines (VMs) in fewer servers and putting the rest in sleep mode, which may lead to the overheating of servers resulting in performance degradation and cooling cost. The lack of an accurate and computationally efficient model to describe the thermal condition of the data center environment makes it challenging to develop an effective and adaptive VMP mechanism. Although recently, data-driven approaches have acted successfully in model construction, the shortage of clean, adequate, and sufficient amounts of data put limits their generalizability. Moreover, any change in the data center configuration during operation, makes these models prone to error and forces them to repeat the learning process. Thus, researchers turn to applying model-free paradigms such as reinforcement learning. Due to the vast action-state space of real-world applications, scalability is one of the significant challenges in this area. In addition, the delayed feedback of environmental variables such as temperature give rise to exploration costs. In this paper, we present a decentralized implementation of reinforcement learning along with a novel state-action representation to perform the VMP in the data centers to optimize energy consumption and keep the host temperature as low as possible while satisfying Service Level Agreements (SLA). Our experimental results show more than 17% improvement in energy consumption and 12% in CPU temperature reduction compared to baseline algorithms. We also succeeded in accelerating optimal policy convergence after the occurrence of a configuration change.