A Review of Deep Multi-Objective Reinforcement Learning and Vision-Based Systems for Smart Cities

Saad Alharbi

Abstract


Smart cities leverage artificial intelligence to address urban challenges such as traffic congestion, environmental sustainability, public safety, and energy efficiency. Among AI techniques, the integration of multi-objective reinforcement learning (MORL) and computer vision (CV) offers adaptive, real-time decision-making capabilities while processing complex visual data streams. This paper presents a comprehensive review of the joint application of MORL and CV in smart city environments. A systematic search was conducted across six major databases (Scopus, Web of Science, IEEE, Springer, MDPI, and Elsevier) from 2019 to 2024, resulting in the selection of 90 relevant studies. The review follows a thematic analysis approach, categorizing the literature into smart mobility, infrastructure, environment, governance, and smart living. Key findings indicate that multi-agent MORL and CV are increasingly used in traffic signal control, autonomous vehicle navigation, energy management, surveillance, healthcare, and waste management. However, despite advancements in deep RL algorithms like DDPG, PPO, SAC, and advanced CV techniques such as semantic segmentation and multi-camera tracking, the direct integration of these technologies remains underexplored in many domains. The paper highlights current research gaps, including the lack of standardized frameworks for MORL-CV synergy, scalability limitations, ethical concerns, and insufficient quantitative benchmarking across studies. Additionally, trends such as federated learning, edge computing, and digital twins are identified as promising enablers for future MORL-CV solutions in urban contexts. This review serves as a resource for researchers and policymakers aiming to develop sustainable and intelligent urban systems by bridging perception (via CV) with adaptive control (via MORL) for real-time, multi-criteria decision support.


Full Text:

PDF

References


M. Herath H. ,. &. Mittal, “Adoption of artificial intelligence in smart cities: A comprehensive review,” Int. J. Inf. Manag. Data Insights 21 100076, 2022.

B. Song F. ,. Xing, H. ,. Wang, X. ,. Luo, S. ,. Dai, P. ,. Xiao, Z. ,. &. Zhao, “Evolutionary multi-objective reinforcement learning based trajectory control and task offloading in UAV-assisted mobile edge computing,” IEEE Trans. Mob. Comput. 2212 7387-7405, 2022.

H. Yang J. ,. Zhang, J. ,. &. Wang, “Urban traffic control in software defined internet of things via a multi-agent deep reinforcement learning approach,” IEEE Trans. Intell. Transp. Syst. 226 3742-3754, 2020.

Kothadiya, D., Chaudhari, A., Macwan, R., Patel, K., & Bhatt, C., “The convergence of deep learning and computer vision: smart city applications and research challenges.,” 3rd Int. Conf. Integr. Intell. Comput. Commun. Secur. ICIIC 2021, 2021.

U.-X. Lau B. P. L. ,. Marakkalage, S. H. ,. Zhou, Y. ,. Hassan, N. U. ,. Yuen, C. ,. Zhang, M. ,. &. Tan, “A survey of data fusion in smart city applications,” Inf. Fusion 52 357-374, 2019.

S. Mao J. ,. Sun, Q. ,. Wang, X. ,. Muthu, B. ,. &. Krishnamoorthy, “The importance of public support in the implementation of green transportation in the smart cities,” Comput. Intell. 401 E12326, 2024.

G. Paiva S. ,. Ahad, M. A. ,. Tripathi, G. ,. Feroz, N. ,. &. Casalino, “Enabling technologies for urban smart mobility: Recent trends, opportunities and challenges,” Sens. 216 2143, 2021.

A. Šurdonja S. ,. Giuffrè, T. ,. &. Deluka-Tibljaš, “Smart mobility solutions–necessary precondition for a well-functioning smart city,” Transp. Res. Procedia 45 604-611, 2020.

K. L. Ho G. T. S. ,. Tsang, Y. P. ,. Wu, C. H. ,. Wong, W. H. ,. &. Choy, “A computer vision-based roadside occupation surveillance system for intelligent transport in smart cities,” Sens. 198 1796, 2019.

S. I. Musa A. A. ,. Malami, S. I. ,. Alanazi, F. ,. Ounaies, W. ,. Alshammari, M. ,. &. Haruna, “Sustainable Traffic Management for Smart Cities Using Internet-of-Things-Oriented Intelligent Transportation Systems (ITS): Challenges and Recommendations,” Sustain. 1513 9859, 2023.

Y. Zhang T. ,. Cheng, J. ,. &. Zou, “Multimodal transportation routing optimization based on multi-objective Q-learning under time uncertainty,” Complex Intell. Syst. 102 3133-3152, 2024.

V. Upadhyay R. K. ,. Sharma, S. K. ,. &. Kumar, “Introduction to Intelligent Transportation System and Advanced Technology,” Intell. Transp. Syst. Adv. Technol. Pp, 2024.

H. Akram A. ,. Abbas, S. ,. Khan, M. ,. Athar, A. ,. Ghazal, T. ,. &. Al Hamadi, “Smart Energy Management System Using Machine Learning,” Comput. Mater. Contin. 781, 2024.

S. Bahaghighat M. ,. Abedini, F. ,. Xin, Q. ,. Zanjireh, M. M. ,. &. Mirjalili, “Using machine learning and computer vision to estimate the angular velocity of wind turbines in smart grids remotely,” Energy Rep. 7 8561-8576, 2021.

L. Chan H. C. ,. &. Chan, “Smart library and smart campus,” J. Serv. Sci. Manag. 116 543-564, 2018.

A. Dimitriadou E. ,. &. Lanitis, “A critical evaluation, challenges, and future perspectives of using artificial intelligence and emerging technologies in smart classrooms,” Smart Learn. Environ. 101 12, 2023.

J. Patil, “SMART VISION OF IOT TECHNOLOGY AND DIGITIZATION IN VARIOUS AREAS OF ACADEMICS IN CURRENT ERA,” Online J. Distance Educ. E-Learn. 111, 2023.

D. Mshali H. ,. Lemlouma, T. ,. Moloney, M. ,. &. Magoni, “A survey on health monitoring systems for health smart homes,” Int. J. Ind. Ergon. 66 26-56, 2018.

Z. Brommeyer M. ,. Whittaker, M. ,. Mackay, M. ,. Ng, F. ,. &. Liang, “Building health service management workforce capacity in the era of health informatics and digital health–A scoping review,” Int. J. Med. Inform. 169 104909, 2023.

N. Jaya Lakshmi N. ,. &. Jabalia, “Body Sensor Networks as Emerging Trends of Technology in Health Care System: Challenges and Future,” Effic. Data Handl. Massive Internet Med. Things Healthc. Data Anal. 129-157, 2021.

A. O. Okolo C. A. ,. Arowoogun, J. O. ,. Chidi, R. ,. &. Adeniyi, “Telemedicine’s role in transforming healthcare delivery in the pharmaceutical industry: A systematic review,” World J. Adv. Res. Rev. 212 1836-1856, 2024.

A. Jeddi Z. ,. &. Bohr, “Remote patient monitoring using artificial intelligence,” Artif. Intell. Healthc. Pp, 2020.

E. J. Peralta-Ochoa A. M. ,. Chaca-Asmal, P. A. ,. Guerrero-Vásquez, L. F. ,. Ordoñez-Ordoñez, J. O. ,. &. Coronel-González, “Smart healthcare applications over 5G networks: A systematic review,” Appl. Sci. 133 1469, 2023.

D. Z. Ribeiro D. A. ,. Melgarejo, D. C. ,. Saadi, M. ,. Rosa, R. L. ,. &. Rodríguez, “A novel deep deterministic policy gradient model applied to intelligent transportation system security problems in 5G and 6G network scenarios,” Phys. Commun. 56 101938, 2023.

Q. Zhang K. ,. Xu, H. ,. Pan, B. ,. &. Zheng, “Explicit coordinated signal control using soft actor–critic for cycle length determination,” IET Intell. Transp. Syst., 2024.

L. D. Nguyen K. K. ,. Duong, T. Q. ,. Vien, N. A. ,. Le-Khac, N. A. ,. &. Nguyen, “Distributed deep deterministic policy gradient for power allocation control in D2D-based V2V communications,” IEEE Access 7 164533-164543, 2019.

G. Chen X. ,. &. Liu, “Federated deep reinforcement learning-based task offloading and resource allocation for smart cities in a mobile edge network,” Sens. 2213 4738, 2022.

B. Qiu C. ,. Hu, Y. ,. Chen, Y. ,. &. Zeng, “Deep deterministic policy gradient (DDPG)-based energy harvesting wireless communications,” IEEE Internet Things J. 65 8577-8588, 2019.

R. Duhayyim M. A. ,. Elfadil Eisa, T. A. ,. Al-Wesabi, F. N. ,. Abdelmaboud, A. ,. Hamza, M. A. ,. Zamani, A. S. ,. Rizwanullah, M. ,. &. Marzouk, “Deep Reinforcement Learning Enabled Smart City Recycling Waste Object Classification,” Comput. Mater. Contin. 713, 2022.

W. Ryu K. ,. &. Kim, “Multi-objective optimization of energy saving and throughput in heterogeneous networks using deep reinforcement learning,” Sens. 2123 7925, 2021.

E. Vamvakas D. ,. Michailidis, P. ,. Korkas, C. ,. &. Kosmatopoulos, “Review and evaluation of reinforcement learning frameworks on smart grid applications,” Energ. 1614 5326, 2023.

M. Vijayalakshmi V. ,. &. Saravanan, “Reinforcement learning-based multi-objective energy-efficient task scheduling in fog-cloud industrial IoT-based systems,” Soft Comput. 2723 17473-17491, 2023.

J.-H. Yun W. J. ,. Park, S. ,. Kim, J. ,. Shin, M. ,. Jung, S. ,. Mohaisen, D. A. ,. &. Kim, “Cooperative multi-agent deep reinforcement learning for reliable surveillance via autonomous multi-UAV control,” IEEE Trans. Ind. Inform. 1810 7086-7096, 2022.

X. Li W. ,. &. Xu, “A hybrid evolutionary and machine learning approach for smart building: Sustainable building energy management design,” Sustain. Energy Technol. Assess. 65 103709, 2024.

S.-H. Quang Tran D. ,. &. Bae, “Proximal policy optimization through a deep reinforcement learning framework for multiple autonomous vehicles at a non-signalized intersection,” Appl. Sci. 1016 5722, 2020.

N. Mazzi Y. ,. Sassi, H. B. ,. Errahimi, F. ,. &. Es-Sbai, “A soft actor-critic reinforcement learning framework for optimal energy management in electric vehicles with hybrid storage,” J. Energy Storage 99 113344, 2024.

Emami, N., Pacheco, L., Di Maio, A., & Braun, T, “Rc-tl: Reinforcement convolutional transfer learning for large-scale trajectory prediction.,” NOMS 2022-2022 IEEEIFIP Netw. Oper. Manag. Symp., 2022.

R. Hafiz A. M. ,. Parah, S. A. ,. &. Bhat, “Reinforcement learning applied to machine vision: state of the art,” Int. J. Multimed. Inf. Retr. 102 71-82, 2021.

K. Diyan M. ,. Silva, B. N. ,. &. Han, “A multi-objective approach for optimal energy management in smart home using the reinforcement learning,” Sens. 2012 3450, 2020.

M. H. Abid M. S. ,. Apon, H. J. ,. Hossain, S. ,. Ahmed, A. ,. Ahshan, R. ,. &. Lipu, “A novel multi-objective optimization based multi-agent deep reinforcement learning approach for microgrid resources planning,” Appl. Energy 353 122029, 2024.

C. E. Leite G. M. C. ,. Jiménez-Fernández, S. ,. Salcedo-Sanz, S. ,. Marcelino, C. G. ,. &. Pedreira, “Solving an energy resource management problem with a novel multi-objective evolutionary reinforcement learning method,” Knowl.-Based Syst. 280 111027, 2023.

L. Prawiyogi A. G. ,. Purnama, S. ,. &. Meria, “Smart cities using machine learning and intelligent applications,” Int. Trans. Artif. Intell. 11 102-116, 2022.

R. Ullah Z. ,. Al-Turjman, F. ,. Mostarda, L. ,. &. Gagliardi, “Applications of artificial intelligence and machine learning in smart cities,” Comput. Commun. 154 313-323, 2020.

Z. Vázquez-Canteli J. R. ,. Ulyanin, S. ,. Kämpf, J. ,. &. Nagy, “Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities,” Sustain. Cities Soc. 45 243-257, 2019.

V. Zhang M. ,. Wang, X. ,. Sathishkumar, V. ,. &. Sivakumar, “Machine learning techniques based on security management in smart cities using robots,” Work 683 891-902, 2021.

Shikunov, M., & Panov, A. I., “Hierarchical reinforcement learning approach for the road intersection task.,” Biol. Inspired Cogn. Archit. 2019 Proc. Tenth Annu. Meet. BICA Soc., no. 10, 2020.

O. A. Borges D. F. ,. Leite, J. P. R. ,. Moreira, E. M. ,. &. Carpinteiro, “Traffic light control using hierarchical reinforcement learning and options framework,” IEEE Access 9 99155-99165, 2021.

B. Fei L. ,. &. Han, “Multi-object multi-camera tracking based on deep learning for intelligent transportation: A review,” Sens. 238 3852, 2023.

S. Gürsoy, “Machine Learning for Autonomous Vehicle Fleet Management and Optimization,” J. AI-Assist. Sci. Discov. 32 86-106, 2023.

H. M. Louati A. ,. Louati, H. ,. Kariri, E. ,. Neifar, W. ,. Hassan, M. K. ,. Khairi, M. H. ,. Farahat, M. A. ,. &. El-Hoseny, “Sustainable Smart Cities through Multi-Agent Reinforcement Learning-Based Cooperative Autonomous Vehicles,” Sustain. 165 1779, 2024.

A. Syed A. S. ,. Sierra-Sosa, D. ,. Kumar, A. ,. &. Elmaghraby, “Making cities smarter—optimization problems for the iot enabled smart city development: A mapping of applications, objectives, constraints,” Sens. 2212 4380, 2022.

I. B. Al-Mashhadani, “Waste material classification using performance evaluation of deep learning models,” J. Intell. Syst. 321 20230064, 2023.

Kumar, C. S., Mohan, C. R., Kumar, B. S., Singh, N., Lavanya, P., & Banerjee, D., “Novel Deep Learning Approaches to Environmental Management with Sustainability.,” 2024 Int. Conf. Commun. Comput. Sci. Eng. IC3SE, 2024.

F. H. Busch J. V. ,. Latzko, V. ,. Reisslein, M. ,. &. Fitzek, “Optimized traffic light management through reinforcement learning: Traffic state agnostic agent vs,” Holist. Agent Curr. V2I Traffic State Knowl., 2020.

Q. Huang R. ,. He, H. ,. &. Su, “An intelligent full-knowledge transferable collaborative eco-driving framework based on improved soft actor-critic algorithm,” Appl. Energy 375 124078, 2024.

L. Mao F. ,. Li, Z. ,. Lin, Y. ,. &. Li, “Mastering arterial traffic signal control with multi-agent attention-based soft actor-critic model,” IEEE Trans. Intell. Transp. Syst. 243 3129-3144, 2022.

B. Nama M. ,. Nath, A. ,. Bechra, N. ,. Bhatia, J. ,. Tanwar, S. ,. Chaturvedi, M. ,. &. Sadoun, “Machine learning‐based traffic scheduling techniques for intelligent transportation system: Opportunities and challenges,” Int. J. Commun. Syst. 349 E4814, 2021.

C. Konstantakopoulos I. C. ,. Barkan, A. R. ,. He, S. ,. Veeravalli, T. ,. Liu, H. ,. &. Spanos, “A deep learning and gamification approach to improving human-building interaction and energy efficiency in smart infrastructure,” Appl. Energy 237 810-821, 2019.

H. Lin Y. H. ,. Lin, M. D. ,. Tsai, K. T. ,. Deng, M. J. ,. &. Ishii, “Multi-objective optimization design of green building envelopes and air conditioning systems for energy conservation and CO2 emission reduction,” Sustain. Cities Soc. 64 102555, 2021.

M. A. Ghofrani A. ,. Nazemi, S. D. ,. &. Jafari, “HVAC load synchronization in smart building communities,” Sustain. Cities Soc. 51 101741, 2019.

S. Mason K. ,. &. Grijalva, “A review of reinforcement learning for autonomous building energy management,” Comput. Electr. Eng. 78 300-312, 2019.

W. L. Mocanu E. ,. Nguyen, P. H. ,. Gibescu, M. ,. &. Kling, “Deep learning for estimating building energy consumption,” Sustain. Energy Grids Netw. 6 91-99, 2016.

Z.-J. M. Deng T. ,. Zhang, K. ,. &. Shen, “A systematic review of a digital twin city: A new pattern of urban governance toward smart cities,” J. Manag. Sci. Eng. 62 125-134, 2021.

S. Tong Z. ,. Ye, F. ,. Yan, M. ,. Liu, H. ,. &. Basodi, “A survey on algorithms for intelligent computing and smart city applications,” Big Data Min. Anal. 43 155-172, 2021.

K. Wolniak R. ,. &. Stecuła, “Artificial Intelligence in Smart Cities—Applications, Barriers, and Future Directions: A Review,” Smart Cities 73 1346-1389, 2024.

P. Sriprateep K. ,. Pitakaso, R. ,. Khonjun, S. ,. Srichok, T. ,. Luesak, P. ,. Gonwirat, S. ,. Kaewta, C. ,. Kosacka-Olejnik, M. ,. &. Enkvetchakul, “Multi-Objective Optimization of Resilient, Sustainable, and Safe Urban Bus Routes for Tourism Promotion Using a Hybrid Reinforcement Learning Algorithm,” Math. 1214 2283, 2024.

M. Xu J. ,. Li, K. ,. &. Abusara, “Preference based multi-objective reinforcement learning for multi-microgrid system optimization problem in smart grid,” Memetic Comput. 142 225-235, 2022.

A. M. Kazmi S. A. ,. Khan, Z. ,. Khan, A. ,. Mazzara, M. ,. &. Khattak, “Leveraging Deep Reinforcement Learning and Healthcare Devices for Active Travelling in Smart Cities,” IEEE Trans. Consum. Electron., 2024.




DOI: https://doi.org/10.31449/inf.v49i20.9776

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.