Structural Limits to Proto-Life Emergence: Evidence from Maximum-Entropy Reinforcement Learning
Abstract
This study investigates whether failures of proto-life emergence are primarily due to limited optimization orto insufficient model expressivity. We formulate an abstract protocell-inspired environment as a continuouscontrolMarkov decision process and train a Soft Actor–Critic (SAC) agent under a maximum-entropyobjective to act as a strong exploration-preserving optimizer. The environment includes protein-like concentrationdynamics, membrane integrity, abstract functional role coverage, and division-like events, whileexpressivity is varied through three regimes controlling coupling strength, feedback structure, and internalmemory. Performance is evaluated using survival rate, success rate, mean functional role coverage, andmean number of division events. Across regimes, survival reached 100%, but success increased from 67.5%in the low-expressivity regime to 97.5% in the mid regime and 100% in the high regime. Mean division countrose from 7.58 and 9.05 in the low and mid regimes to 34.58 in the high regime, indicating a qualitativetransition toward growth-dominated behavior once stronger feedback and memory were available. Theseresults show that strong entropy-regularized optimization alone does not guarantee life-like organizationwhen essential state variables and interaction mechanisms are absent. Rather, the emergence of sustainedproto-biological behavior is bounded by the representational structure of the model itself. The findingsposition maximum-entropy reinforcement learning as a diagnostic tool for identifying missing structuralingredients in computational origin-of-life models.References
[1] I. Gözen, E. S. Köksal, I. Põldsalu, L. Xue,
K. Spustova, E. Pedrueza-Villalmanzo, R. Ryskulov,
F. Meng, and A. Jesorka, “Protocells: Milestones
and recent advances,” Small, vol. 18, no. 18,
Art. 2106624, 2022. https://doi.org/10.1002/
smll.202106624
[2] W. Hordijk and M. Steel, “Autocatalytic sets and the
origin of life,” Entropy, vol. 20, no. 4, Art. 250, 2018.
https://doi.org/10.3390/e20040250
[3] S. Mann, “Systems of creation: The emergence of
life from nonliving matter,” Accounts of Chemical Research,
vol. 45, no. 12, pp. 2131–2141, 2012. https:
//doi.org/10.1021/ar200281t
[4] E. Camprubí, J. W. de Leeuw, C. H. House, F. Raulin,
M. J. Russell, A. Spang, M. R. Tirumalai, and F. Westall,
“The emergence of life,” Space Science Reviews,
vol. 215, no. 8, Art. 56, 2019. https://doi.org/
10.1007/s11214-019-0624-8
[5] R. S. Sutton and A. G. Barto, Reinforcement Learning:
An Introduction, 2nd ed. Cambridge, MA, USA:
MIT Press, 2018.
[6] J. Terven, “Deep reinforcement learning:
A chronological overview and methods,”
AI, vol. 6, no. 3, Art. 46, 2025. https:
//doi.org/10.3390/ai6030046
[7] S. Han and Y. Sung, “A max–min entropy framework
for reinforcement learning,” in Advances in Neural
Information Processing Systems, vol. 34, pp. 25732–
25745, 2021.
[8] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker,
S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel,
and S. Levine, “Soft actor–critic algorithms and applications,”
arXiv preprint arXiv:1812.05905, 2018.
https://doi.org/10.48550/arXiv.1812.
05905
[9] R. Serra and M. Villani, Modelling Protocells. Cham,
Switzerland: Springer, 2017. https://doi.org/
10.1007/978-94-024-1160-7
[10] W. Hordijk and M. Steel, “Autocatalytic networks at
the basis of life’s origin and organization,” Life, vol. 8,
no. 4, Art. 62, 2018. https://doi.org/10.3390/
life8040062
[11] T. W. Malone and S. A. Smith, “Modeling the performance
of organizational structures,” Operations Research,
vol. 36, no. 3, pp. 421–436, 1988. https:
//doi.org/10.1287/opre.36.3.421
[12] S. McLeod, “Absolute biological needs,” Bioethics,
vol. 28, no. 6, pp. 293–301, 2014. https://doi.
org/10.1111/bioe.12020
[13] H. G. Garcia, J. Kondev, N. Orme, J. A. Theriot,
and R. Phillips, “Thermodynamics of biological
processes,” in Methods in Enzymology,
vol. 492, D. E. Discher and T. J. Mitchison,
Eds. Amsterdam, The Netherlands: Elsevier,
2011, pp. 27–59. https://doi.org/10.1016/
B978-0-12-381268-1.00014-8
[14] W. Hordijk, M. Steel, and P. Dittrich, “Autocatalytic
sets and chemical organizations: Modeling selfsustaining
reaction networks at the origin of life,” New
Journal of Physics, vol. 20, no. 1, Art. 015011, 2018.
https://doi.org/10.1088/1367-2630/aa9fcd
[15] E. Mossel and M. Steel, “Random biochemical networks:
The probability of self-sustaining autocatalysis,”
Journal of Theoretical Biology, vol. 233, no. 3, pp. 327–336, 2005. https://doi.org/10.1016/
j.jtbi.2004.10.011
[16] S. Mann, “The origins of life: Old problems, new
chemistries,” Angewandte Chemie International Edition,
vol. 52, no. 1, pp. 155–162, 2013. https://
doi.org/10.1002/anie.201204968
[17] N. Lehman and S. A. Kauffman, “Constraint closure
drove major transitions in the origins of life,” Entropy,
vol. 23, no. 1, Art. 105, 2021. https://doi.org/
10.3390/e23010105
[18] V. Vanchurin, Y. I. Wolf, E. V. Koonin, and M. I.
Katsnelson, “Thermodynamics of evolution and the
origin of life,” Proceedings of the National Academy
of Sciences, vol. 119, no. 6, Art. e2120042119, 2022.
https://doi.org/10.1073/pnas.2120042119
[19] R. Hasibi, T. Michoel, and D. A. Oyarzún, “Integration
of graph neural networks and genomescale
metabolic models for predicting gene essentiality,”
npj Systems Biology and Applications,
vol. 10, Art. 24, 2024. https://doi.org/10.
1038/s41540-024-00348-2
[20] E. D. Vaishnav, C. G. de Boer, J. Molinet, M. Yassour,
L. Fan, X. Adiconis, D. A. Thompson, J. Z. Levin,
F. A. Cubillos, and A. Regev, “The evolution, evolvability
and engineering of gene regulatory DNA,” Nature,
vol. 603, no. 7901, pp. 455–463, 2022. https:
//doi.org/10.1038/s41586-022-04506-6
[21] H. Lu, F. Li, L. Yuan, I. Domenzain, R. Yu,
H. Wang, G. Li, C. Yu, B. Ji, E. J. Kerkhoven,
and J. Nielsen, “Yeast metabolic innovations emerged
via expanded metabolic network and gene positive
selection,” Molecular Systems Biology, vol. 17,
Art. e10427, 2021. https://doi.org/10.15252/
msb.202110427
[22] H. Bai, R. Cheng, and Y. Jin, “Evolutionary reinforcement
learning: A survey,” Intelligent Computing,
vol. 2, Art. 0025, 2023. https://doi.org/10.
34133/icomputing.0025
[23] E. O. Neftci and B. B. Averbeck, “Reinforcement
learning in artificial and biological
systems,” Nature Machine Intelligence,
vol. 1, no. 3, pp. 133–143, 2019. https:
//doi.org/10.1038/s42256-019-0025-4
[24] A. S. Alshehri and F. You, “Deep learning to catalyze
inverse molecular design,” Chemical Engineering
Journal, vol. 444, Art. 136669, 2022. https:
//doi.org/10.1016/j.cej.2022.136669
DOI:
https://doi.org/10.31449/inf.v50i14.13944Keywords:
Artificial life, Origin of life, Reinforcement learning, Maximum entropy, Soft Actor-CriticDownloads
Published
Issue
Section
License
Authors retain copyright in their work. By submitting to and publishing with Informatica, authors grant the publisher (Slovene Society Informatika) the non-exclusive right to publish, reproduce, and distribute the article and to identify itself as the original publisher.
All articles are published under the Creative Commons Attribution license CC BY 3.0. Under this license, others may share and adapt the work for any purpose, provided appropriate credit is given and changes (if any) are indicated.
Authors may deposit and share the submitted version, accepted manuscript, and published version, provided the original publication in Informatica is properly cited.
