Adaptive Parallel Processing Algorithm with Dynamic Scheduling for Large-Scale Data Processing in Cloud Environments: Implementation and Performance Evaluation

Abstract

As large-scale data processing tasks continue to grow in volume and complexity, improving the efficiency of computational resource utilization and task execution performance has emerged as a central challenge in cloud computing environments. In response, this study proposes an adaptive parallel processing algorithm that incorporates a dynamic scheduling strategy, designed to optimize task allocation and execution workflows within distributed systems. To assess the algorithm's performance, experiments were conducted across three platforms—Amazon Web Services (AWS), Google Cloud, and a local computing cluster—using three representative large-scale public datasets. These tasks included a structured classification task using the Kaggle Titanic dataset, an image processing task using the Google Open Images dataset (which contains over 90 million images), and a text processing task based on the Common Crawl dataset, which comprises content from billions of web pages. On the Google Cloud platform, the integration of dynamic scheduling reduced execution time to 13.5 hours. It also demonstrated strong adaptability and overall system stability, especially when managing complex task distributions and largescale data. When paired with the adaptive parallel processing algorithm, the dynamic scheduling strategy achieved a 5.2× speedup compared to serial execution. This reduced the total processing time from 12 hours to 2.3 hours, while maintaining high resource utilization and stable task scheduling. These findings underscore the algorithm's substantial potential in enhancing the performance of large-scale data processing and offer practical implications for algorithmic optimization and resource management in cloud-based environments.