A Lightweight BERT-Variant Optimized for WebGPU-Based Real-TimeInference in Web Browsers
Abstract
Large Language Models (LLMs) achieve state-of-the-art performance across natural language processingtasks but remain computationally intensive, limiting their deployment in browser-based environments.This study investigates the feasibility of real-time transformer inference entirely within web browsers usingWebGPU acceleration. We propose a lightweight BERT-inspired architecture optimized for GPUparallelmatrix operations through TensorFlow.js with the WebGPU backend. The model is evaluated ona 4,000-sample IMDB sentiment classification dataset and achieves 65–70% classification accuracy withper-sample inference latency of 8–9 ms in Google Chrome using an NVIDIA T400 GPU (4 GB VRAM).Batch inference throughput reaches approximately 6,600 inferences per minute, while GPU utilization remainsstable between 26–77%. Compared to CPU-based TensorFlow.js execution, WebGPU significantlyreduces inference latency and enables fully client-side training and inference without server dependencies.Although accuracy is lower than full-scale BERT benchmarks, the results demonstrate that simplified transformerarchitectures can operate efficiently in browser environments under resource constraints. This workestablishes a practical framework for deploying real-time edge NLP applications using open web standardsand GPU acceleration.References
[1] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D.,
& Sutskever, I. (2019). Language models are unsupervised
multitask learners. OpenAI Blog, 1(8), 9.
[2] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J., Dhariwal, P., & Amodei, D. (2020). Language
models are few-shot learners. Advances in Neural
Information Processing Systems, 33, 1877–1901.
[3] Strubell, E., Ganesh, A., & McCallum, A. (2019).
Energy and policy considerations for deep learning
in NLP. arXiv preprint arXiv:1906.02243.
https://doi.org/10.48550/arXiv.1906.02243
[4] Patterson, D., Gonzalez, J., Le, Q., Liang, C.,
Munguia, L.-M., Rothchild, D., So, D., Texier, M.,
& Dean, J. (2021). Carbon emissions and large neural
network training. arXiv preprint arXiv:2104.10350.
https://doi.org/10.48550/arXiv.2104.10350
[5] Knoll, A., & Scheuermann, B. (2020). WebGPU: A
comparison with WebGL. Proceedings of The Web
Conference 2020, ACM, pp. 1256–1267.
[6] Chen, Z., Ma, Y., Shen, H., & Liu, M. (2025).
WeInfer: Unleashing the power of WebGPU
on LLM inference in web browsers. In Proceedings
of The Web Conference 2025, ACM.
https://doi.org/10.1145/3696410.3714919
[7] Ruan, C. F., Qin, Y., Zhou, X., Lai, R., Jin,
H., Dong, Y., & Chen, T. (2024). WebLLM:
A high-performance in-browser LLM inference
engine. arXiv preprint arXiv:2412.15803.
https://doi.org/10.48550/arXiv.2412.15803
[8] Devlin, J., Chang, M. W., Lee, K., & Toutanova,
K. (2019). BERT: Pre-training of deep bidirectional
transformers for language understanding. In Proceedings
of NAACL-HLT 2019, ACL, pp. 4171–4186.
https://doi.org/10.18653/v1/N19-1423
[9] Radford, A., Narasimhan, K., Salimans, T., &
Sutskever, I. (2018). Improving language understanding
by generative pre-training. OpenAI Blog.
[10] Nickolls, J., Buck, I., Garland, M., & Skadron,
K. (2008). Scalable parallel programming
with CUDA. ACM Queue, 6(2), 40–53.
https://doi.org/10.1145/1365490.1365500
[11] Kirk, D. B., & Hwu, W. W. (2016). Programming
Massively Parallel Processors: A Hands-on Approach
(3rd ed.). Morgan Kaufmann.
[12] Owens, J. D., Houston, M., Luebke, D., Green, S.,
Stone, J. E., & Phillips, J. C. (2008). GPU computing.
Proceedings of the IEEE, 96(5), 879–899.
https://doi.org/10.1109/JPROC.2008.917757
[13] Micikevicius, P., Narang, S., Alben, J., Diamos, G.,
Elsen, E., Garcia, D., & Houston, M. (2018). Mixed
precision training. arXiv preprint arXiv:1710.03740.
https://doi.org/10.48550/arXiv.1710.03740
[14] Ben-Nun, T., & Hoefler, T. (2019). Demystifying parallel
and distributed deep learning: An in-depth concurrency
analysis. ACM Computing Surveys, 52(4),
1–43. https://doi.org/10.1145/3320060
[15] Zhang, Y., Sun, S., Yang, H., & Li, M. (2022).
Optimizing large language models for realtime
applications: A review of techniques and
challenges. IEEE Access, 10, 78945–78960.
https://doi.org/10.1109/ACCESS.2022.3193506
[16] Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling
the knowledge in a neural network. In Proceedings of
NIPS 2015, MIT Press.
[17] Han, S., Pool, J., Tran, J., & Dally, W. (2015). Learning
both weights and connections for efficient neural
networks. In Proceedings of NIPS 2015, MIT Press.
[18] Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang,
M., Howard, A., Adam, H., & Kalenichenko, D.
(2018). Quantization and training of neural networks
for efficient integer-arithmetic-only inference. In Proceedings
of CVPR 2018, IEEE, pp. 2704–2713.
https://doi.org/10.1109/CVPR.2018.00286
[19] NVIDIA Corporation (2020). NVIDIA A100 Tensor
Core GPU Architecture. NVIDIA Technical White
Paper.
[20] Jouppi, N. P., Young, C., Patil, N., et al. (2017). Indatacenter
performance analysis of a tensor processing
unit. In Proceedings of ISCA 2017, ACM, pp. 1–
12. https://doi.org/10.1145/3079856.3080246
[21] Wang, T., Zhao, R., Yu, Y., Liang, Y., & Wei, S.
(2021). Hardware accelerator for transformer-based
NLP on FPGAs. In Proceedings of FCCM 2021,
IEEE.
[22] Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., &
Han, S. (2023). SmoothQuant: Accurate and efficient
post-training quantization for large language models.
In Proceedings of ICML 2023, PMLR.
[23] Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D.,
Chen, D., & Dao, T. (2024). Medusa: Simple
LLM inference acceleration framework with multiple
decoding heads. arXiv preprint arXiv:2401.10774.
https://doi.org/10.48550/arXiv.2401.10774
[24] Li, Y., Han, X., Zhao, Z., Yang, X., & Li, Y. (2024).
SnapKV: LLM knows what you are looking for before
generation. arXiv preprint arXiv:2404.14469.
https://doi.org/10.48550/arXiv.2404.14469
[25] Li, X., Chen, L., Wang, J., & Huang, T. (2023).
Chrysalis: A specialized dataset for training LLMbased
debugging assistants in hardware development.
In Proceedings of MLSys 2023.
[26] Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao,
L., & Li, Z. (2023). OmniQuant: Omnidirectionally
calibrated quantization for large language
models. arXiv preprint arXiv:2308.13137.
https://doi.org/10.48550/arXiv.2308.13137
[27] Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh,
D. (2022). GPTQ: Accurate post-training
quantization for generative pre-trained transformers.
arXiv preprint arXiv:2210.17323.
https://doi.org/10.48550/arXiv.2210.17323
[28] Huang, Y., Wan, L. J., Ye, H., Jha, M., Wang,
J., Li, Y., Zhang, X., & Chen, D. (2024).
New solutions on LLM acceleration, optimization,
and application. arXiv preprint arXiv:2406.10903.
https://doi.org/10.48550/arXiv.2406.10903
[29] Chavan, A., Magazine, R., Kushwaha, S., Debbah,
M., & Gupta, D. (2024). Faster and lighter
LLMs: A survey on current challenges and
way forward. arXiv preprint arXiv:2402.01799.
https://doi.org/10.48550/arXiv.2402.01799
[30] Shen, H., Chang, H., Dong, B., Luo, Y., &
Meng, H. (2023). Efficient LLM inference
on CPUs. arXiv preprint arXiv:2311.00502.
https://doi.org/10.48550/arXiv.2311.00502
[31] Ma, Y., Xiang, D., Zheng, S., Tian, D., &
Liu, X. (2019). Moving deep learning into web
browser: How far can we go? In Proceedings of
The Web Conference 2019, ACM, pp. 1234–1244.
https://doi.org/10.1145/3308558.3313639
[32] Bai, T., Liang, H., Wan, B., Yang, L., Li, B.,
Wang, Y., & Zhang, W. (2024). A survey of
multimodal large language model from a datacentric
perspective. arXiv preprint arXiv:2405.16640.
https://doi.org/10.48550/arXiv.2405.16640
[33] Dong, B., Liu, T., Li, B., Zhou, X., Wang,
S., & Xu, Z.-D. (2023). WebInf: Accelerating
WebGPU-based in-browser DNN inference
via adaptive model partitioning. In Proceedings
of ICPADS 2023, IEEE, pp. 2499–2506.
https://doi.org/10.1109/ICPADS60453.2023.00345
[34] Wang, Q., Jiang, S., Chen, Z., Cao, X., Li, Y., Li,
A., Ma, Y., Cao, T., & Liu, X. (2024). Anatomizing
deep learning inference in web browsers. ACM Transactions
on Software Engineering and Methodology,
33(5). https://doi.org/10.1145/3649596
[35] Yuan, Z., Shang, Y., Zhou, Y., Dong, Z., Zhou,
Z., Xue, C., Wu, B., Li, Z., Gu, Q., Lee, Y.
J., Yan, Y., Chen, B., Sun, G., & Keutzer, K.
(2024). LLM inference unveiled: Survey and roofline
model insights. arXiv preprint arXiv:2402.16363.
https://doi.org/10.48550/arXiv.2402.16363
[36] Zhang, H., Ning, A., Prabhakar, R. B., & Wentzlaff,
D. (2024). LLMCompass: Enabling efficient hardware
design for large language model inference. In
Proceedings of ISCA 2024, IEEE, pp. 1080–1096.
https://doi.org/10.1109/ISCA59077.2024.00080
[37] Liu, W., Zhou, P., Zhao, Z., Wang, Z.,
Deng, H., & Ju, Q. (2020). FastBERT: A
self-distilling BERT with adaptive inference
time. arXiv preprint arXiv:2004.02178.
https://doi.org/10.48550/arXiv.2004.02178
[38] Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng,
A. Y., & Potts, C. (2011). Learning word vectors for
sentiment analysis. In Proceedings of ACL-HLT 2011,
ACL, pp. 142–150. https://aclanthology.org/P11-
1015
[39] Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar,
S., Usman, M., Akhtar, N., Barnes, N., &
Mian, A. (2025). A comprehensive overview of large
language models. arXiv preprint arXiv:2307.06435.
https://doi.org/10.48550/arXiv.2307.06435
[40] Xi, Z., Chen, W., Guo, X., He, W., Ding, Y.,
Hong, B., & Gui, T. (2025). The rise and potential
of large language model based agents:
A survey. arXiv preprint arXiv:2309.07864.
https://doi.org/10.48550/arXiv.2309.07864
[41] Zhang, J., Xie, R., Hou, Y., Zhao, W. X.,
Lin, L., & Wen, J. R. (2025). Recommendation
as instruction following: A large language
model empowered recommendation
approach. arXiv preprint arXiv:2305.07001.
https://doi.org/10.48550/arXiv.2305.07001
[42] Sanh, V., Debut, L., Chaumond, J., &
Wolf, T. (2019). DistilBERT, a distilled version
of BERT: Smaller, faster, cheaper and
lighter. arXiv preprint arXiv:1910.01108.
https://doi.org/10.48550/arXiv.1910.01108
[43] Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y.,
& Zhou, D. (2020). MobileBERT: A compact
task-agnostic BERT for resource-limited
devices. arXiv preprint arXiv:2004.02984.
https://doi.org/10.48550/arXiv.2004.02984
DOI:
https://doi.org/10.31449/inf.v50i14.10934Keywords:
Large language model, LLM acceleration, WebGPU, BERT architecture, Transformer model, deep learning model, inferenceDownloads
Published
Issue
Section
License
Authors retain copyright in their work. By submitting to and publishing with Informatica, authors grant the publisher (Slovene Society Informatika) the non-exclusive right to publish, reproduce, and distribute the article and to identify itself as the original publisher.
All articles are published under the Creative Commons Attribution license CC BY 3.0. Under this license, others may share and adapt the work for any purpose, provided appropriate credit is given and changes (if any) are indicated.
Authors may deposit and share the submitted version, accepted manuscript, and published version, provided the original publication in Informatica is properly cited.
