\documentclass[a4paper,11pt,twoside]{article}
\usepackage{informat}
\usepackage{epsfig}
\usepackage[flushleft]{threeparttable}
\usepackage{amsfonts}
\usepackage{cite}
\usepackage[cmex10]{amsmath}
\usepackage[caption=false,font=footnotesize]{subfig}
\usepackage{epstopdf}
\usepackage[intoc]{nomencl}
\usepackage{authblk}
\usepackage{subcaption}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{lscape}
\usepackage{ragged2e}

\begin{document}
  \title{Efficiency Boosted SVM By  Sparsification From LP \& Stochastic Functional $\xi$-Analysis}

\author[1]{*Rezaul Karim}
\author[1]{Zakir Hossain}
\author[1]{Amit Kumar Kundu}
\author[1]{Ali Ahmed Ave}
\affil[1]{Department of Electrical and Electronic Engineering, Uttara University, Dhaka, Bangladesh}
\affil[*]{Corresponding author; E-mail: rezkar@uttarauniversity.edu.bd}

  \titleodd{EBSVM By Sparsification From LP \& Stochastic\ldots}
  \authoreven{R. Karim et al.}
  \keywords{classifier, stochastic, LP, QP, sparse, SVM, EBSVM(Efficiency Boosted SVM), sccuracy, MAC, GFR, compact, data, mapping.}
  \received{, }

  \abstract{Abstract-- While SVM being a solid mathematical and state-of-the-art sparse but powerful stochastic machine with top accuracy, Vapnik's LP based SVM is sparser than the QP generated one though offers similar accuracy. Yet, further sparsity is essential to efficiently work with very complicated and large datasets. In our work, we impose additional sparsication on this sparser LPSVM maintaining optimal complexity of the machine to further reduce classification cost while preserving similar accuracy by analyzing the stochastic auxiliary variable, $\xi$  after a compact deterministic mapping of data. Experiments with benchmark and synthetic data reveals that our model demands classification cost much smaller than the cutting-edge prominent classifiers while posing similar accuracy despite of being simpler to be implemented. On average, it costs 12.4\% of  the standard QPSVM, 50.3\% of the VLPSVM, 68.1\% of the EESVM\cite{kundu2023escalating}, 98.1\% of the greedy SpSVM-2\cite{keerthi2006building}. It insists the least MAC (Machine Accuracy Cost) value as well as GFR(Generalization Failure Rate) considering these machines. This machine is also expected to work with high efficiency in case of very large and complex data satisfying constraints in real-time.}
  \abstractSi{}

  \maketitle

\section{Introduction \& Selected Related Work}
In stochastic learning, the principal objective in a probabilistic data-classification problem is in general to model a discriminator function that effectively
represents the generalized relationship of the input objects or patterns with their matching class-labels  applying the given dataset having some
example patterns with their associated class-labels.
 While in such indeterministic  frameworks,  performance from kernel-based classifiers is influenced by the size, randomness of distribution and topological variations of the dataset, the complexity of constructing such classifiers and determining its decision boundary  by involving kernel operating patterns is also related to the probabilistic nature of data, classifiers’ geometry and available computational resources. Thus different machines show different performances on data detection demanding different computational cost. In this regard while Support Vector Machines (SVMs) lead the field of data classification by offering top level accuracy through finding global optima,  a key stochastic property of SVMs is their sparsity, which enables the construction of the optimal classifier using only a probabilistically selected subset of training patterns, known as support vectors (SVs). This sparsity reduces computational demands but as the number of SVs increases, computational cost from nonlinear SVM classifiers also gets higher. However, although Vapnik’s linear programming (LP) based SVM is even sparser and demands much less computational cost, it also demands the expensive kernel execution task.  Moreover,  as science and technology have advanced, the size as well as the complexity of data have grown rapidly, and as a result, the number of support vectors usually rises with the growth of the the size of training dataset
 \cite{steinwart2004sparseness}.
This makes the further but compact sparsity of the support vector set a critical and continuous consideration for large-scale datasets, especially in scenarios constrained by computational resources and time. As a result, developing methods to address this challenge has become a central focus of research, aiming optimal performance by balancing stochastic generalization with computational efficiency.
\\
 Tom Downs et al.\cite{downs2001exact} offers a method to detect linearly dependent support vectors (SVs), retaining only one and discarding the rest but the number of this linearly dependent SVs tends to decrease as data dimensionality and complexity increase. Joachims-Yu \cite{JoachimsY09}, Keerthi et al. \cite{keerthi2006building},and  Cotter et al. \cite{CotterSS13}  develop iterative algorithms for reduced SVMs that deliver impressive results, demanding only a very small amount of kernel execution to classifying a single pattern whereas \cite{CotterSS13} reported a memory run out from \cite{keerthi2006building},\cite{JoachimsY09} applying on bulky dataset  while their approach introduces  a notable deviation from the original methodology  with massive parameter choosing.  Romdhani et al. \cite{romdhani2004efficient} and Wu et al. \cite{wu2006direct}  propose to approximate the decision function of a full SVM by identifying a novel denser set of patterns to replace the full set of support vectors, whereas Raetsch et al. \cite{ratsch2005over}  explore wavelet-based estimations for these vectors to efficiently compute dot products between the kernel generating pattern vectors. These approaches, however, rely heavily on some complex optimization methods that are indeed prone to the starting point, learning rate, etc.
Heisele et al. \cite{heisele2003hierarchical} suggests a structured SVM-oriented classification method that uses a series of linear SVMs in the early stages and a nonlinear SVM in the last stage, while optimizing thresholds for enhancing both classification time and accuracy 
whereas Karim et al. \cite{karim2007greedy}  
presents that a carefully organized cascade of two nonlinear SVMs considerably lowers classification costs compared to using a single nonlinear SVM. Arreola et al. \cite{arreola2006fast} models a linear SVM-based decision tree.  Sahbi-Geman \cite{sahbi2006hierarchy}  designs a tree-structured hierarchical SVM using threshold selection as well as reduced set techniques, applied across partitioned pattern spaces. Moreover, there are also some other noteworthy works focusing on developing cost-efficient SVM variants. Where, Maji et al. \cite{MajiBM13}  presents that histogram and additive kernels based SVMs outperform linear SVMs in speed and accuracy. Ladicky and Torr \cite{LadickyT11} propose a locally linear SVM with bounded curvature and a smooth decision boundary, balancing the number of anchor points against classifier complexity to prevent overfitting and runtime issues. Xu et al. \cite{XuGTW15} suggests an additional training algorithm to further compress the trained SVMs using a small set of parameters. Li and Zhang \cite{Li013}
designs a fast object detector using cascaded stages with logistic regression acting as a weak base learner focusing on efficient training method.
Raykar et al. \cite{RaykarKY10} develops a cascaded system where the classifiers discard patterns based on probabilities calculated in previous stages achieving a balance between feature learning costs and accuracy. 
EESVM by Kundu-Karim \cite{kundu2023escalating}, HESVM by Karim- kundu \cite{karim2024sequential}
 achieve a significant advancement in SVM efficiency, which is particularly suitable for applications involving large-scale and complex datasets  though their implementation complexity is also notable. Fu et al. \cite{FuRZ10} combines linear SVMs to classify complex data showing that the prediction phase efficiency of linear SVMs rivals that of nonlinear SVMs. Karim and Kundu \cite{Karim-Kundu-2018-1}  propose a second-order SVM by performing quadratic and linear programming in sequence, achieving compactness and high classification accuracy. 
Later, they introduce 
a different method\cite{karim2019computational} for support vector reduction by LP and computational analysis, demonstrating on benchmark datasets that their model achieves comparable accuracy to standard nonlinear SVMs with significantly improved prediction efficiency.
Franc et al.\cite{franc2003greedy} proposed Reduced SVMs (RSVMs) with lower computational costs compared to standard SVMs to classify patterns by leveraging a small subset from the support vectors of the  conventional SVM  while Gu et al.\cite{GuH13}  introduced Clustered SVMs, employing weighted combinations of LSVMs that are trained on the subsets of the training data that are clustered to achieve localized separation. Karim et al. \cite{karim2023lp} introduces a novel similarity function to enhance the efficiency of Linear Programming based Support Vector Machine. Zhou \cite{zhou2022sparse} proposes an  approach to enhance the efficiency of SVMs through sparsity constraints that offers significant computational speed while using a practically challenging assumption in the optimization method. Dries et al. \cite{geebelen2012reducing}
offers innovative approaches for support vectors reduction in SVM classifiers through two different methods that gives some computational benefits. However, their modification of training data limits their applicability to non-separable datasets. Wang et al. \cite{wang2022support} offers a significant advancement in linear SVM algorithm by tackling fundamental issues with traditional hinge loss but initiates new challenges, specially in terms of optimization and scalability. Liu et al.\cite{liu2022nonlinear}
offers a significant advancement in kernel SVM methodology by incorporating the 0-1 soft margin loss into a nonlinear kernel framework that gives a promising alternative for complex classification tasks while the inherent challenges associated with their non-convex optimization and algorithmic complexity limits the model's practicality and robustness. Wu et al. \cite{wu2024nonlinear} proposes an innovative approach to SVM classification by excluding the involvement of kernel functions and directly incorporating the 0-1 loss function that gives improved performance. However, on top of the challenges from their non-convex optimization and algorithmic complexity, their assumption that a quadratic hyper-surface is suitable for separating classes, which may not hold true for all datasets, potentially bounds its applicability. Akhtar et al. \cite{akhtar2024roboss} gains generalization performance by coupling their RoBoSS loss function in SVM while issues about implementation complexity, parameter tuning, scalability, and data quality should be taken into account seriously during its application. Lin et al. \cite{lin2024kernel} gets support vector reduction 
from modification of conventional SVMs by integrating the $\ell_0$-norm into the hinge loss function but the challenges regarding their non-convex optimization, computational complexity should be considered when applying this approach. Wang et al. \cite{wang2023sparse},\cite{wang2023fastTran}, \cite{wang2024fast}, \cite{wang2023fast} 
propose  effective mechanisms for handling large-scale classification problems by balancing robustness, sparsity, and efficiency  but their non-convex models bring challenges during implementation.
\\In spite of the efficiency of these classifiers, further sparsification with convenient implementation is very essential for datasets that are complex and very large.
\\
In this work, we focus to build up an exceptionally sparse classifier within a stochastic framework.
By analyzing the SVM learning process through stochastic principles with pattern space spanning and leveraging computational and probabilistic insights, we propose a simple but very productive learning algorithm under uncertainty. 
Being generated from the outcome of the analysis of a stochastic variable, this algorithm employs methods to model a highly efficient classifier from a very compact and dense set of stochastic decision points 
that significantly reduces classification costs while preserving statistical robustness and strong generalization capabilities.
\\The remaining part of the paper is arranged in the following way: Section \ref{basics} covers the relevant foundations while Section \ref{method} details the proposed algorithm with an in-depth analysis. The experimental results are presented in Section \ref{results} whereas Section \ref{conclusion} concludes with a discussion and suggestions for future work.


\section{Related Basics}\label{basics}
\subsection{Stochastic Learning Fundamentals}
Stochastic learning lies in the Machine Learning system to train the machine where model parameters are determined under uncertainty using the training data (figure\ref{GeneralProbabilisticMacine_figure}). With  constant model complexity, for small training sets, the model may get insufficient information to capture underlying patterns, just memorizes the training dataset well rather than learning
generalizable patterns, leading to high variance or overfitting and underperform while  with larger sample size, the generalization error typically decreases because the model is trained on more representative data, improving its ability to generalize and reducing overfitting. Further, when the model has low dimensionality, it may not have the capacity to fit the training data adequately, resulting in high bias (underfitting) where the Generalization error gets highly influenced by the inability to capture data complexity whereas when the model has high dimensionality, it can perfectly fit the training data, including noise, leading to high variance (overfitting) and the Generalization error increases as the model fails to perform well on test data.
Hansen \cite{hansen1993stochastic}, Karim \cite{karim2004theoretical} showed that to avoid higher Generalization Error in Stochastic Linear Learning, for a problem with sample size $N$, model dimension $d$, one must have $N>>d$ and this error goes unbounded for $N\le d+1$.
However, the ideal balance is achieved when the model dimension matches the effective complexity of the data, given a sufficient sample size and the relationship among generalization error, sample size, and model dimension in stochastic learning is governed by the interplay among bias, variance, and the complexity of the model. 

\begin{figure}[t]
\includegraphics[scale=.4]{GeneralProbabilisticMacine}%uslogo(.eps,.pdf)
\caption{%
Training and Generalization Phase of a Statistical Learning Machine  Model. \cite{kecman2005support}}\label{GeneralProbabilisticMacine_figure} %% no full stop at the end of caption
\end{figure}


\section*{Generalization Error While Modeling A Stochastic Machine}

\subsection*{Problem Set Up}
The key objective in machine learning is to approximate an unknown function $f(x)$ 
that assigns an output $y$ for an input $x$ 
while the observed outputs $y$ are noisy due to inherent randomness or measurement errors:

$$ y = f(x) + \epsilon $$%delta $$

where  $f(x) $ is the true underlying function, $\epsilon $ is the random noise following a zero-mean normal distribution with  variance 
$\sigma^2_\epsilon \): \( \epsilon \sim \mathcal{N}(0, \sigma^2_\epsilon) $.

We approximate \( f(x) \) using a model \( \hat{f}(x) \), which is learned from data, aiming at measuring how well  $\hat{f}(x)$ performs by analyzing its expected squared error.

\subsection*{Expected Error Forming}
The expected squared-error at a point $x$ can be calculated as:

$$
\mathbb{E}[(y - \hat{f}(x))^2] $$

where the expectation is taken over the randomness in both $ \hat{f}(x)$  and \( y \) (due to noise \( \epsilon \)).

Substituting  $ y = f(x) + \epsilon $ in the above equation we get

$$
\mathbb{E}[(y - \hat{f}(x))^2] = \mathbb{E}[(f(x) + \epsilon - \hat{f}(x))^2].
$$

Expanding the squared term we get, 
$$
(f(x) + \epsilon - \hat{f}(x))^2 = (f(x) - \hat{f}(x))^2 + 2(f(x) - \hat{f}(x))\epsilon + \epsilon^2.
$$

Now taking the expectation and using $ \mathbb{E}[\epsilon] = 0 $,  the middle term becomes $ \mathbb{E}[2(f(x) - \hat{f}(x))\epsilon] = 0 $ while the last term gives
$ \mathbb{E}[\epsilon^2] = \sigma^2_\epsilon $, which is the noise variance and the whole expected squared error simplifies to

$$
\mathbb{E}[(y - \hat{f}(x))^2] = \mathbb{E}[(f(x) - \hat{f}(x))^2] + \sigma^2_\epsilon.
$$

\subsection*{Error Term Decomposition}
The term $ \mathbb{E}[(f(x) - \hat{f}(x))^2] $ representing the error in approximating $ f(x) $ with $\hat{f}(x)$, can be further decomposed by adding and subtracting $ \mathbb{E}[\hat{f}(x)] $ (the expected prediction of the model over all possible datasets) as
$$
\mathbb{E}[(f(x) - \hat{f}(x))^2] = \mathbb{E}[(f(x) - \mathbb{E}[\hat{f}(x)] + \mathbb{E}[\hat{f}(x)] - \hat{f}(x))^2].
$$
Now, expanding the squared term, we get
\begin{multline*}
$$
(f(x) - \mathbb{E}[\hat{f}(x)] + \mathbb{E}[\hat{f}(x)] - \hat{f}(x))^2 = (f(x) - \mathbb{E}[\hat{f}(x)])^2 + \\ 2(f(x) -
\mathbb{E}[\hat{f}(x)])(\mathbb{E}[\hat{f}(x)] - \hat{f}(x)) + (\mathbb{E}[\hat{f}(x)] - \hat{f}(x))^2.
$$
\end{multline*}
Now applying the expectation, we get the cross-term $\mathbb{E}[2(f(x) - \mathbb{E}[\hat{f}(x)])(\mathbb{E}[\hat{f}(x)] - \hat{f}(x))] = 0$ as $ \mathbb{E}[\hat{f}(x)] - \hat{f}(x) = 0$.
whereas the first term $ (f(x) - \mathbb{E}[\hat{f}(x)])^2$  is the \textbf{Bias squared}, the second term $ \mathbb{E}[(\mathbb{E}[\hat{f}(x)] - \hat{f}(x))^2] $ is the \textbf{variance}.

Thus, we can write 
$$
\mathbb{E}[(f(x) - \hat{f}(x))^2] = (\text{Bias})^2 + (\text{Variance}),
$$
where, $ \text{Bias}^2 = (f(x) - \mathbb{E}[\hat{f}(x)])^2 $,
$ \text{Variance} = \mathbb{E}[(\hat{f}(x) - \mathbb{E}[\hat{f}(x)])^2] $.

Eventually, the full expression for the expected squared error becomes
$$
\mathbb{E}[(y - \hat{f}(x))^2] = (\text{Bias})^2 + (\text{Variance}) + \sigma^2_\epsilon.
$$

\subsection*{ Generalization Introducing Input Distribution}
Considering all inputs $ x $, we  have to take the expectation over the input distribution $ p(x) $ as
$$
\text{Generalization Error} = \mathbb{E}_{x, \epsilon}[(y - \hat{f}(x))^2].
$$

Expanding this we can write the the final expression as 

\begin{eqnarray}
\mathbb{E}_{x, \epsilon}[(y - \hat{f}(x))^2] \nonumber = \underbrace{\mathbb{E}_x[(f(x) - \mathbb{E}[\hat{f}(x)])^2]}_{\text{Bias}^2} \\ + \underbrace{\mathbb{E}_x[\mathbb{E}_\epsilon[(\hat{f}(x) - \mathbb{E}[\hat{f}(x)])^2]]}_{\text{Variance}} + \underbrace{\sigma^2_\epsilon}_{\text{Irreducible Error}}.\label{BiasVar_StochastMachin}
\end{eqnarray}



\subsection*{Intuition Behind Each Term \& Key Observations} 

\begin{itemize}
    \item High bias takes place  while the model is undertrained or too simple to capture data complexity and as model complexity increases, \textbf{Bias} decreases because the model can better approximate \( f(x) \). Underfitting from high bias can lead to a situation, where both training and test errors are high. 
    
    \item  \textbf{Variance} increases because the model turns to be more responsive to the training data. Variance measures how sensitive the model is to changes in the training data. High variance can result in overfitting, where the model performs well on the training data but poorly on unseen test data.
    
    

    
    \item  \textbf{Irreducible error} (\( \sigma^2 \)) remains constant, independent of model complexity.
    \item A model having high bias typically has low variance and
vice versa. The optimal model 
is a properly tuned model that achieves good generalization with reasonable training performance minimizing the total error by balancing bias and variance. This is the \textbf{bias-variance tradeoff},  a fundamental concept in machine learning that explains the relationship between a model's ability to generalize to unseen data and its performance on training data. It is very important to consider during the construction of a stochastic machine.

\end{itemize}

Next we discuss about Support Vector Machine (SVM), which is very prominent managing this bias-variance trade off successfully. 


\subsection{Support Vector Machine (SVM)}
Due to their robustness, strong generalization capability, and  convexity of the mathematical model to find unique global optimum, SVMs
 probably have gained the highest popularity among  machine learning approaches for supervised learning though having very simple principle. The problem that drove its initial development occurs in various forms as the bias variance trade-off \cite{geman1992neural}, 
capacity control \cite{guyon1991structural}
, overfitting \cite{montgomery2021introduction} while the core idea is the same \cite{burges1998tutorial}.
Thus, it is mathematically quite strong basing on the concept of statistical
learning theory as well as optimization
\subsubsection{SVM With Statistical  Learning Theory}

Suppose that we have two classes of objects. Now we have received a novel object for which we 
are to allocate a class from one of these two classes.
Such a problem can be considered as a \textit{(binary, due to two classes) pattern recognition} problem that can be formed in the following way: having a set of observational  data  
%\begin{eqnarray}
   $ (x_1, y_1), \dots, (x_m, y_m) \in \mathcal{S} \times \{\pm1\}$, 
%\end{eqnarray}
  we need to design a \textit{decision (discriminator) function} $ f : \mathcal{S} \to \{\pm1\} $, where \( \mathcal{S} \) is a set of elements from which the  \( x_i \) patterns have been drawn, generally referred to as the \textit{domain} whereas \( y_i \) are designated as the \textit{targets} or \textit{class-labels}.
A well defined decision function is supposed to be able to \textit{generalize} to the unseen novel data instances that poses a possibly low value while measuring the \textit{risk}  

\begin{eqnarray}
 R[f] = \frac{1}{2}\int  | f(x) - y | \, dP(x, y)  \label{eqn_GenRisk} 
\end{eqnarray} 
where \( dP(x, y) \) can be written as \( p(x, y)dxdy \) when a density \( p(x, y) \) exists.

Expressing linguistically, 
on average across an unspecified distribution \( P \) that is considered to generate both of the training and test data, we aim to achieve a low error. Here,  we calculate the error  using the \textit{binary loss function}  
$
q(x, y, f(x)) := \frac{1}{2} | f(x) - y |
$, which gives 1 when \( (x, y) \) is not correctly classified and 0 when it is correctly classified.
\\
It is to note that up to now, 
the patterns can take almost any form
, and no assumptions have been made on \( \mathcal{S} \) except it is a set equipped by a probability measure \( P \).  Additionally, without having an estimate of  \( P(x, y) \), although (\ref{eqn_GenRisk}) represents the true mean error in a nice way,
it does not tell about the way either to find
a function posing a small risk or even to asses the risk of a specified function, and thus becomes not so useful. Hence,for risk minimization, we need to move towards an induction principle .
Now, we incorporate an additional form of structure related to the information we actually have, that is the training data. 
And while the quantity \( R[f] \) is usually referred to as the expected risk, or simply the risk or here, we  term it as the actual risk, this newly introduced quantity will be called the empirical risk (the training error) , noted by \( R_{\text{emp}}[f] \), which  is defined as the calculated mean error rate over the full training set (considering a fixed and finite number of observations); that is,  
\begin{equation}
R_{\text{emp}}[f] = \frac{1}{m} \sum_{i=1}^{m} |f(x_i) - y_i|
\end{equation}
It is to be noted that, here, no probability distribution shows up and \( R_{\text{emp}}[f] \) is a fixed number for a specific  \( f \) and for a specific training data set \( \{x_i, y_i\} \).

However, when the number of training examples $m$ is limited, minimizing this empirical risk does not assure small actual risk as small error over the training set  does not automatically
imply good generalization ability,  that is, a small amount of error on any independent and novel test data. This phenomenon is commonly known as overfitting \cite{bishop1995neural}. 

To get the maximum benefit from a limited amount of data, various statistical methods are being developed over many years. Vapnik-Chervonenkis,  through their VC theory \cite{Vapnik1995} in Statistical Learning Theory, emphasis on  the necessity of limiting the set of functions from which $f$ is selected to one having a \textit{capacity} that well fits the available training data. It establishes \textit{bounds} on the test error, which depends on the empirical risk as well as the capacity of the function class while minimizing these bounds 
guides to the \textit{structural risk minimization} principle \cite{vapnik1979estimation}. Considering the discussed problem, SVMs can be regarded as an application of this principle where it is tried to minimize the "Structural Risk", which is
a combination of the  \textit{empirical risk}, 
$ R_{\text{emp}}[f] $
and a "capacity term" found for the set of hyperplanes from a dot product space \( \mathcal{H} \)  as stated next. For some \( \eta \) such that \( 0 \leq \eta \leq 1 \) and $m > h$, with probability \( 1 - \eta \), the following bound holds :
\begin{equation}
R[f] \leq R_{\text{emp}}[f] + \sqrt{\frac{h \left( \log(2m/h) + 1 \right) - \log(\eta/4)}{m}} \label{Risk_Bound}
\end{equation} 

where \( h \), the Vapnik-Chervonenkis (VC) dimension (or "model complexity"), which is an integer that cannot be negative and is used to asses the above mentioned  capacity. Right-hand side of (\ref{Risk_Bound}) is called the  the "risk bound" and the second term of this part is called the "model complexity penalty" or "VC confidence".

Several key points can be noted about this bound. First, it is free of the distribution \( P(x, y) \) and here only assumption is made that  both the training and test data are taken independently following this distribution. 
Second, the left hand side of the bound is usually not possible to compute whereas the right hand side of it can be computed if we know $h$. 
Third, learning process is said to be consistent if both of the expected risk and the empirical risk converge to the minimum value of the risk, which can be found for $m \to \infty$ and at last, for given some diverse learning machines (equivalently, family of functions),
selecting the one, which minimizes the right hand part of (\ref{Risk_Bound}) will be taking the machine that gives  the least upper bound imposed on the actual risk in its left hand part . 
\\Moreover, going into a bit more explicit about the error or risk of the machine depending on its model complexity  we see that  the \textit{VC (Vapnik-Chervonenkis)-dimension}, \( h \) of a set of functions describing their \textit{capacity} controls the main characteristic of a machine.  SVM, by  applying structural risk minimization (SRM) \cite{vapnik1964note} principle, which  is an inductive method that chooses a model to learn from a training dataset with finite examples, acting as  an indicator of this capacity control by proposing a trade-off between the empirical error and  this VC dimensions, $h$, equivalently, the hypothesis of space complexity. Having a form of convex optimization, SRM applies a set of  sequential models with an ascending order of complexity.  figure \ref{Error_Model_figure}  briefly displays how the total model error changes depending on the  model-complexity-index of a machine that can be analyzed from the relation (\ref{Risk_Bound}). For low-complex models (similar to high bias but low variance in relation (\ref{BiasVar_StochastMachin}) during a stochastic machine  learning), the error gets higher as simpler models are unable to learn all of the data-complexity leadingto an underfitting situation. On contrary, for higher model indices (similar to high variance but low bias in relation (\ref{BiasVar_StochastMachin}) during the learning of a stochastic machine), the structure tries to adapt  its learning model with higher concentration to the very specific  training data leading to an overfitting condition, which reduces the amount of training error with increasing the model VC but  at the cost of a worsening performance on the test data. Thus the optimal model index, $h^*$ lies between these higher and lower values of this model index where the model gets down to its lowest value.

\begin{figure}[t]
\centering
\includegraphics[scale=.4]{Fig_SVM_ErroTrend_ModelComplexity}%uslogo(.eps,.pdf)
\caption{ Error Trends On Model Complexity. \cite{Awad2015}}
  \label{Error_Model_figure} %% no full stop at the end of caption
\end{figure}



\subsubsection{Kernels With SVM}
In the very simple pattern recognition problems, SVM utilizes a linear separating hyperplane to form a classifier with the largest margin. To achieve this, the learning problem for SVM will be formulated as a constrained nonlinear programming problem where the objective function are quadratic and the constraints are linear, that means, it will be needed  to solve a QP(quadratic programming) problem. On the other hand, when a dataset cannot be separated linearly in the original input space, the soft-margin SVM struggles to manage a reliable separating hyperplane that both minimizes misclassified data points and generalizes effectively. But following Cover’s theorem\cite{ cover1965geometrical}, data points get more probable to be in the form for being linearly separable after they are nonlinearly mapped into a higher-dimensional space. In this regard, SVM uses Kernel method, which was first introduced in \cite{Aizerman1964}\cite{Du2024Exploring} and  implicitly  performs a nonlinear transformation of the input training data from its existing space to a much higher-dimensional feature space (or kernel space) by replacing the inner product by an optimum positive definite function (or kernel function, strongly related to Mercer's Theorem \cite{ mercer1909functions}), where data will be linearly separable (figure \ref{HigerDimensionalMapping_Kernel} on page \pageref{HigerDimensionalMapping_Kernel}) 
rather than solving a high-order separating hypersurface in the input space and this is accomplished before solving the learning problem through a convex optimization. This approach is preferred because a nonlinear separating boundary in the input space escalates computational demands during the optimization phase. Nonlinear kernel functions lessen the curse of dimensionality\cite{Bellman1957} productively. However, selecting and setting the right kernel function  are vital for SVM optimality.
\begin{figure}[h]
\includegraphics[scale=.32]{higerDimensionalMapping}%uslogo(.eps,.pdf)
\caption{Non-linear transformation of data through kernel methods: mapping input space to a much
higher dimensional feature space, realizing linear separation of data-classes
in the feature space, which is not linear in input space.} \label{HigerDimensionalMapping_Kernel}

\end{figure}
\section*{Mercer's Theorem for Kernels}
Mercer's Theorem is a fundamental concept from Functional Analysis to provide the theoretical basis for using kernels in SVM, which can be described in the following way. Let \( K(x, y) \) be a continuous function defined on a compact set \( X \times X \). Mercer's Theorem states that if \( K(x, y) \) is \textit{positive semi-definite} (i.e., for all \( d \)-dimensional vectors \( \mathbf{v} \), the matrix \( [K(x_i, x_j)] \) is positive semi-definite), then there exists a \textit{feature space} and a \textit{mapping} \( \phi(x) \) such that:

\[
K(x, y) = \langle \phi(x), \phi(y) \rangle
\]
Explaining in another way, the function \( K(x, y) \) can be expressed as the inner product of two vectors in some high-dimensional (even possibly, infinite-dimensional) feature space (equivalently, RKHS) where some key concepts are described below:
\subsection*{Key Concepts}

\begin{itemize}
    \item \textbf{Positive Semi-Definiteness}: A function \( K(x, y) \) is positive semi-definite if, for any finite set of points \( \{ x_1, x_2, \dots, x_d \} \), the matrix formed by evaluating \( K(x_i, x_j) \) for all pairs \( (x_i, x_j) \) is positive semi-definite. This means that all eigenvalues of the matrix are non-negative.
    
    \item \textbf{Reproducing Kernel Hilbert Space (RKHS) \cite{Aronszajn1950, Girosi1997, Heckman1997, Wahba1990}}: The feature space \( \mathcal{H} \) associated with the kernel function \( K(x, y) \) is called the RKHS. The kernel \( K \) defines an inner product in this space, and the function \( \phi(x) \) maps the original input space to this RKHS.
    
    \item \textbf{Significance for Machine Learning}: Mercer's Theorem justifies the use of kernel functions like the \textit{Gaussian (RBF) kernel}, \textit{polynomial kernel}, and others in machine learning. The theorem ensures that these kernels correspond to valid inner products in high-dimensional feature spaces, enabling the use of linear methods in these spaces.
\end{itemize}

Most commonly used kernels are:
\textbf{Polynomial kernel:} \( K(x, y) = (x^T y + c)^d \)
    \&  \textbf{Gaussian (RBF) kernel:} \( K(x, y) = \exp\left(-\frac{\|x - y\|^2}{2\sigma^2}\right) \)

 \subsection*{For further kernels: }
It is possible to construct more complex kernel functions by combining simpler kernels.
\begin{itemize}
    \item \textit{Non-negative Linear Combination 
     of Mercer kernels  also produces a Mercer kernel
    :} If \( K_1(x, y), K_2(x, y), \dots, K_n(x, y) \) are Mercer kernels and \( \alpha_1, \alpha_2, \dots, \alpha_n \geq 0 \), then \( K(x, y) = \sum_{i=1}^n \alpha_i K_i(x, y) \) is also a Mercer kernel.
    \item \textit{Product of Mercer kernels result in a Mercer kernel   
    :} If \( K_1(x, y) \) and \( K_2(x, y) \) are Mercer kernels, then \( K(x, y) = K_1(x, y) \cdot K_2(x, y) \) is also a Mercer kernel.
\end{itemize}


\subsubsection{SVM With Mathematical Programming}
Being mathematically solid and hence powerful, SVM has become one of the most popular supervised machine learning method for  classification where the problem is framed as estimating a decision boundary under uncertainty. Due to its high power and minimal classification error despite sparsity, it has gained high popularity. Initially, it was introduced through Quadratic Programming (QP), known as QPSVM. However, as the $L_1$-norm is more likely to produce sparser solutions than $L_2$-norm \cite{boyd2004convex}, Vapnik modelled an alternative SVM model using Linear Programming (LP) to design the objective function, called VLPSVM. This part addresses  SVM and its two core forms (QPSVM and VLPSVM), with further particulars provided in \cite{vapnik1998}.


\subsubsection*{Quadratic Programming (QP) Based SVM}
This is the standard SVM, a leading algorithm in the field of machine learning. QPSVM uses structural risk minimization concept from statistical learning theory implementing margin maximization between two classes that aims to find the data depending separating hyperplane, which is as far as possible from the nearest data point. Let us consider a binary classification problem using a set of given training patterns $\{\big (x_i, y_i\big)\}_{i= 1}^N$; $x_i \in \mathbb{R}^d$ and $y_i \in \{ -1,1\}$. To divide these patterns into two classes effectively, SVM determines a classifier based on a decision function having the form $ f(x) = w \cdot \phi(x) + b$, leading to $class(x) = sgn(f(x))$, where $K(x_i, x_j) = \phi(x_i)\cdot \phi(x_j )$ is a kernel operation. The primal domain formulation of this QPSVM is as follows:
\vspace{-0.40cm}
\begin{gather}
\min_{w,b,\zeta} f_P(w) ~=~ \frac{1}{2} \|w\|^2 ~+~C \sum_{i=1}^{N} \zeta_i
\label{Vapnik_QP_CSVC_primal_1}
\\ s.t. ~~ y_i \big(w \cdot\phi(x_i) + b \big) ~~ \ge ~ 1-\zeta_i
\label{Vapnik_QP_CSVC_primal_2}
\\\zeta_i ~ \ge ~0 ; ~i = 1,2,...,N
\label{Vapnik_QP_CSVC_primal_3}
\end{gather}

where the auxiliary variables $\zeta_i > 0$ work mainly for patterns that remain outside their respective class-margins (or we call it here, "margin-outward-deviated patterns") with $C > 0$ is a data-influenced parameter to regularize the classifier following bias-variance tradeoff concept in stochastic learning. The corresponding dual problem becomes:

 \vspace{-0.20cm}
\begin{gather}
\max_\alpha f_D = \sum_{i=1}^{N} \alpha_i -\frac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_i \alpha_j y_i y_j \phi(x_i) \cdot \phi(x_j)   \label{Vapnik_QP_CSVC_dual_1}
\\s.t ~~ \sum_{i=1}^{N}\alpha_i y_i = 0 \label{Vapnik_QP_CSVC_dual_1_constrain_1}
~~~~~~~~\\0 \le \alpha_i \le C ; ~~i = 1,2,...,N \label{Vapnik_QP_CSVC_dual_1_constrain_2}
\end{gather}


which is also a QP problem  and the patterns for which $\alpha_i > 0$ are the support vectors (SVs). They can also be called as the stochastic decision points as they act as the bases of the final decision function  through their coefficients $\alpha$, which are used to determine the bias term, $b$ and the weight variable $w$ as  $w = \sum \alpha_i y_i \phi(x_i)$. An illustrative depiction of this
 margin based SVM is provided in figure \ref{QP_SVM_figure} on page \pageref{QP_SVM_figure}. 

\begin{figure}[t!]
\includegraphics[scale=.5]{QP_figure.eps}%uslogo(.eps,.pdf)
\caption{A visual representation of the standard SVM is shown, where patterns for the positive and negative classes are depicted by squares and triangles, respectively. The green straight line represents the optimal separating hyperplane, and the brown lines mark the margins for the classes. Patterns that lie outside their respective classes are referred to as outward-deviated patterns, where the depth of this deviation  is represented by a non-negative variable, $\zeta$. The goal of this QP-based SVM is to maximize the margin while minimizing the training error as much as possible. The green line is also known as the decision boundary from the training output.}\label{QP_SVM_figure} %% no full stop at the end of caption
\end{figure}
 

\subsubsection*{Linear Programming (LP) Based SVM}

Vapnik proposed an LP model to find a separating hyperplane, which we name here VLPSVM and effectively incorporates an indirect $L_1$ norm while modeling the cost function, promoting even more sparsity while maintaining accuracy comparable to QPSVM that uses the $L_2$ norm. As a result, VLPSVM shares similarities with QPSVM but demands less computational effort for classifying patterns. If $w$ represents the weight vector of the hyperplane for VLPSVM, its decision function is given by:
$f(x) = w \cdot \phi(x) + b$ leading to $class(x) = sgn(f(x))$. The primal domain formulation of this VLPSVM model is as follows:


\vspace{-0.50cm}
\begin{gather}
\min_{\beta,\zeta, b}  ~~\sum_{i=1}^{N} \beta_i  ~+~C \sum_{i=1}^{N} \xi_i
\label{Vapnik_LP_CSVC_primal_1}
\\ s.t. ~~ y_i \Big(\sum_{j=1}^{N} \beta_j y_j  \phi(x_j)\cdot\phi(x_i) + b \Big) ~~ \ge ~ 1-\xi_i
\label{Vapnik_LP_CSVC_primal_2}
\\\beta_j ~ \ge ~0 ; ~j = 1,2,...,N
\label{Vapnik_LP_CSVC_primal_3}
\\\xi_i ~ \ge ~0 ; ~i = 1,2,...,N
\label{Vapnik_LP_CSVC_primal_4}
\end{gather}

In this case, the auxiliary variables $\xi_i > 0$  are assigned  to handle with the unity-outward-deviated patterns (i.e., the training patterns that give $ ClassLabel(x_i) \cdot f(x_i) <  $1) and these $\xi$ are also optimization variables for the problem (\ref{Vapnik_LP_CSVC_primal_1})-(\ref{Vapnik_LP_CSVC_primal_4}) that are found with the other variable $\beta$s, $b$ depending on the data-dependent stochastic parameters, kernel parameter and penalty parameter. 
To avoid confusion and for sharp clarification, here, training examples ($x_j$) of this VLPSVM, for which, coefficients $\beta_j> 0$ are labeled as Expansion Vectors (EVs), which
are quite similar to the support vectors (SVs) in QPSVM 
while being different from both topological and physical points. These optimum value of $\beta$s are used to determine $w = \sum \beta_j y_j \phi(x_j)$. The number of EVs found from the VLPSVM algorithm are usually very much smaller compared to the total number of training patterns proving further sparsity of VLPSVM model. Our experiments on benchmark data as well as other works \cite{nefedov2009experimental} show that VLPSVM has nearly the same performance as QPSVM despite of being more efficient due to less kernel execution in classification stage
showing VLPSVM to be more productive for many cases, specially, when classification cost, speed, resources, etc are under constraints. Further, this LP-SVM simplifies the optimization problem by using linear programming, making it computationally efficient for certain stochastic processes.
and for this model, the kernel, $K(.,.)$ does not need to satisfy Mercer's conditions \cite{vapnik1998}

A graphical representation of  this (kernel based) SVM is given in figure \ref{ClassificationWithKernelSVM_figure} on page \pageref{ClassificationWithKernelSVM_figure}. 

\begin{figure}[t]
\includegraphics[scale=.32]{ClassificationWithKernelSVM}%uslogo(.eps,.pdf)
\caption{Classification with a kernel based SVM. The learning method is completely expressed by the used kernel function $k$ along with the penalty parameter $C$. Having training data, the machine produces the  
 coefficients for the decision function.} \cite{Suttorp2009MultiObjectiveOptimization}
  \label{ClassificationWithKernelSVM_figure} 
\end{figure}

\section{Proposed Method With Stochastic Functional $\xi$- Analysis}\label{method}
\subsection{Research Gap, Motivation \& Our Direction}
 Despite of being the most mathematically sound machine learning algorithm by using the famous statistical learning theory, maximum margin concept, and mathematical programming, while working on probabilistic data, implementing total risk minimization principle, SVM does not directly control the model  complexity but it automatically determines it by selecting the type and number of stochastic decision points (support vectors (SVs)) depending on the data-dependent parameters. These SVs use kernel operations
that highly boosts the expressive capability of the decision function while preserving the underlying linearity that ensures the learning to remain tractable. However, on top of the mounted cost by excessive kernel computations from a large number of SVs, this increased flexibility raises the risk of overfitting since the selection of separating hyperplane turns to be increasingly ill-posed because of the number of degrees of freedom \cite{christianini2000svm}. This becomes more serious in case of large and complex data with noise.
Thus managing the machine from getting excessive number of SVs is necessary regarding both cost and accuracy.
\\Moreover, as the standard SVM algorithm concentrates on error performance indirectly through cost-function and constraints using data-dependent parameters, these parameters play a very crucial role in the algorithm, specially, finding the best trade-off between model-complexity and generalization. However, as there is no exact and analytic expression for these stochastic parameters, we have to find their values  using cross-validation method that does not give the exact best and unique values of theses parameters. On top of this, 
it is also followed that to improve a very little amount of classification accuracy on complex dataset, SVM generates a relatively larger number of support vectors (represented in figure \ref{fig_NSV_TrPR}). 
Additionally, because of the continuous raise in the heavy size and complexity of data in parallel with modern technological development, further speed up of the classification process is being demanded continuously, which stipulates one to work on reducing the number of SVs as much as possible.

Thus, while the existing methods for SVM cost reduction are either 1)complex method that is very expensive and not so convenient to implement or 2) bear uncertainty to get as being non-convex or 3) cannot gain sparsity and actuary simultaneously upto that level, a mathematical and computational analysis based suitable operation on the first-step solution from SVM algorithm may also be productive and useful. One of the simplest but heavily productive approach in this direction could be to reduce the support vectors by discarding the least significant SVs, misleading SVs, or such kernel computing vectors by sacrificing an insignificant amount of classification accuracy
(even in case it becomes demanded, the worst scenario). But while discarding SVs without considering their contribution can create negative impact on decision-making, in practice, training data may contain noise or outliers that stay far from their main cluster and as the SVM decision boundary relies on SVs, the presence of outliers in the SVs set can make the decision function loaded with more misleading info and shape the  boundary less smooth, or may even cause overfitting, especially with complex datasets. Hence, a method to reduce the number of support vectors (as well as the machine-misleading noise due to the noise in the  data) after analyzing the outlier's right-leading or misleading on the decision function is rewarding with respect to both cost and accuracy.
In this direction, we start with the basis of Statistical Learning Theory where we try to further reduce the Right Hand Side of (\ref{Risk_Bound}) by reducing both of its 1st term and 2nd term simultaneously that eventually lowers the upper bound  on the actual risk, leading towards a machine having better generalization performance than before
while also reducing the classification cost.

 This is achieved by using some mathematical and computational analysis with the stochastic parameter based values of the  discriminator function, auxiliary variable, and co-efficient of the stochastic decision points found from the first step SVM solution to select the insignificant SVs, misleading and noisy SVs and 
 discard from them as many as possible while preserving the machine's efficiency and performance. 

Experimental results demonstrate that our proposed method significantly reduces classification costs while posing classification accuracy analogous to the standard SVM, particularly with large and complex training data that contains many outliers.

\begin{figure}[t!]
\includegraphics[scale=.24]{BananaTrPrRateOnVariousSigma.png}%uslogo(.eps,.pdf)
\caption{The number of SVs fluctuating with training accuracy. The plots are normalized by dividing them by their respective maximum values, bringing within 0 and 1.
} 
  \label{fig_NSV_TrPR} 
\end{figure}


\subsection{Deterministic Mapping of  Data}
As the numerical values of raw data varies widely, scaling it may help to avoid numerical instability by dropping data redundancy and irregularity that is useful for the objective function to work properly and to improve the convergence of the optimization algorithm. This also gets further motivated from the reporting about data preprocessing for higher accuracy by Alaa et al.\cite{Tharwat2016} and  about data scaling 
to improve the condition
number of the matrix in the optimisation problem \cite{Gunn1998}. However, while doing this, we must try for such a scaling that may not hamper the crucial properties of data. For example, we can try for a careful scaling of data that may allow a bit change in its topological spanning without noticeably losing its main stochastic property. But we also know that the mean of a scaled dataset is scaled by the scaling factor in each dimension and the elements of the co-variance matrix of the scaled dataset is scaled by the square of the scaling factor (proof is given in the later part of this section). Hence, we decide first to pick the maximum absolute of all the feature values and then divide each entry by this, which maps each entry of the column to lie in [-1,1]. This kind of deterministic transformation would help to evade data leakage in the stage of model testing but may turn out to be sensitive to the outliers that are very common in the real-world problem. Hence, by scaling we bring feature values $\le$ 1 and then throw the machine vectors for which corresponding stochastic auxiliary variables $\xi$ give values $\ge$ 1 by following some numerical analysis based on the stochastic functional analysis described next. A pictorial view of before and after scaling of a benchmark and a  synthetic data set is given in figure \ref{beforeAfter_DataScaling_banana_twoSpiral} on page \pageref{beforeAfter_DataScaling_banana_twoSpiral}

\begin{figure*}[t!]
  \centering  
  \subfloat[Banana - Before Scaling]{\includegraphics[width=8cm,height=5cm]{Banana96_+ve_-ve_class_Without_Normalize_z.eps}} \hfil
  \subfloat[Banana - After Scaling]{\includegraphics[width=8cm,height=5cm]{Banana96_+ve_-ve_class_With_Normalize_z.eps}} \hfill
  \subfloat[Two Spiral - Before Scaling]{\includegraphics[width=8cm,height=5cm]{Two_Spiral_noise_label=3.0_+ve_-ve_class_Without_Normalize_z.eps}} \hfil
  \subfloat[Two Spiral - After Scaling]{\includegraphics[width=8cm,height=5cm]{Two_Spiral_noise_label=3.0_+ve_-ve_class_With_Normalize_z.eps}} 
  
  \caption{Visualization of the effect on 2D data by our way of scaling. It can be clearly observed that after data scaling, in case of Benchmark Banana dataset\cite{data} the 
  mean is scaled by $\frac{1}{3.25}$ and Co-variance is scaled by $(\frac{1}{3.25})^2$ whereas for the Synthetic Two-spiral dataset\cite{kools2025datasets} the mean is scaled by $\frac{1}{13.98}$ and Co-variance is scaled by $(\frac{1}{13.98})^2$. Note that the topological spanning of the datasets are reduced by contracting its spatial scattering but the stochastic property of the datasets are not harmed noticeably.}
\label{beforeAfter_DataScaling_banana_twoSpiral}
\end{figure*}


\subsubsection*{Effect on Mean and Variance by Scaling a Multidimensional Dataset:}
Let \( \mathbf{X} = \left\{ \mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_n \right\} \) be a set of \( n \) data points, where each data point \( \mathbf{x}_i = (x_{i1}, x_{i2}, \dots, x_{id}) \) is a vector in \( \mathbb{R}^d \) (a \( d \)-dimensional space). If We scale this dataset by multiplying with a constant factor $\lambda> 0 $ , then each of the data points will be multiplied by $\lambda$ and the new dataset becomes:
$\mathbf{x}'_i = \lambda \cdot \mathbf{x}_i = (\lambda x_{i1}, \lambda x_{i2}, \dots, \lambda x_{id})$. So, the scaled dataset $\mathbf{X}'$ contains the scaled vectors $\mathbf{x}'_i \in \mathbb{R}^d $ 

Now, we know that, in case of the original (unscaled) data, the mean vector of $\mathbf{X}$ be $\boldsymbol{\mu}_\mathbf{X} = \mathbb{E}[\mathbf{X}]$ and the covariance matrix of $\mathbf{X}$ be:
    \[
    \mathbf{\Sigma}_\mathbf{X} = \mathbb{E}[(\mathbf{X} - \boldsymbol{\mu}_\mathbf{X})(\mathbf{X} - \boldsymbol{\mu}_\mathbf{X})^\top].
    \]
Then in case of the scaled data, the mean vector of $\mathbf{X'}$ be $\boldsymbol{\mu}_\mathbf{X'} = \mathbb{E}[\mathbf{X'}] = \mathbb{E}[\lambda\mathbf{X}] = \lambda \mathbb{E}[\mathbf{X}] =  \lambda\boldsymbol{\mu}_\mathbf{X}$ and the covariance matrix of $\mathbf{X'}$ be:
  %  \[
  $  \mathbf{\Sigma}_\mathbf{X'} = \mathbb{E}[(\mathbf{X'} - \boldsymbol{\mu}_\mathbf{X'})(\mathbf{X'} - \boldsymbol{\mu}_\mathbf{X'})^\top]
=\mathbb{E}[(\lambda\mathbf{ X} - \lambda\boldsymbol{\mu}_\mathbf{X})(\lambda\mathbf{X} - \lambda\boldsymbol{\mu}_\mathbf{X})^\top]
= \lambda^2 \mathbb{E}[(\mathbf{ X} - \boldsymbol{\mu}_\mathbf{X})(\mathbf{X} -\boldsymbol{\mu}_\mathbf{X})^\top]
=\lambda^2 \mathbf{\Sigma}_\mathbf{X} 
$.
%\]
That is, the covariance matrix of the scaled random vector is scaled by $\lambda^2$. But the \textit{variance} of a single dimension of a dataset is simply the diagonal element of the covariance matrix. Then in case of the original dataset $\mathbf{X}$, 
$ \text{Var}(x_j) = \Sigma_{X,jj} $ and for the scaled dataset $\mathbf{X'}$, 
%the variance in each dimension is the diagonal element of  $\mathbf{\Sigma}_{X'} :$
$
\text{Var}(x'_j) = \Sigma_{X',jj} = \lambda^2 \cdot \Sigma_{X,jj}
$
Thus, the variance in each dimension is scaled by a factor of $\lambda^2 $. 
We scale the dataset by dividing it by the maximum value of the elements of $\mathbf{X}$, that is in our case, if $\beta = max(max(abs(\mathbf{X}))) \ge 1$ then $\lambda = 1/\beta$ and the new dataset $\mathbf{x}'_i = \lambda \cdot \mathbf{x}_i =  \frac{1}{\beta} \cdot \mathbf{x}_i =  \frac{1}{max(max(abs(\mathbf{X})))} \cdot \mathbf{x}_i \le \mathbf{x}_i$ with $\mu_{X'} = \lambda \cdot \mu_X = \frac{1}{\beta} \cdot \mu_X  = \frac{1}{max(max(abs(\mathbf{X})))} \cdot \mu_X \le \mu_X $ and $\mathbf{\Sigma}_{X'} = \lambda^2 \cdot \mathbf{\Sigma}_X = \frac{1}{\beta^2} \cdot \mathbf{\Sigma}_X = \frac{1}{[max(max(abs(\mathbf{X})))]^2} \cdot \mathbf{\Sigma}_X \le \mathbf{\Sigma}_X $. 


\subsection{Analyzing Stochastic Function }%{Basic Idea}
In case of learning from probabilistic data, we need to model a data dependent stochastic parameter based discriminator function with the capacity to master the training data well to build optimum generalization ability that can classify novel patterns with least error. For this, it is needed to model the problem set using error-related constraints. Thus in case of both LPSVM and QPSVM, while the decision function are expressed as  $f(x)=w.\phi(x)+ b$, their  error-associated constraints from (\ref{Vapnik_QP_CSVC_primal_2}) \& (\ref{Vapnik_LP_CSVC_primal_2}) can be represented by the relation, $y_i f(x_i) \ge 1-\xi_i \implies \xi_i  \ge  1- y_i f_(x_i)$. 
\\Now, considering any wrongly classified training example $x_i$, if $y_i f(x_i) = - \delta_i$ with $\delta_i > 0$,  then $ \xi_i \ge 1 + \delta_i$. Thus, for $M$ wrongly classified training examples, $\sum_{i=1}^{M} \xi_i \ge M + \sum_{i=1}^{M} \delta_i$. Hence, suppressing $\sum_i \xi_i$ through regularizer will curb $M + \sum_{i=1}^{M} \delta_i$, not only just $M$, which is the total training error. It is possible that suppressing $M + \sum_{i=1}^{M} \delta_i$ could take place during the following three conditions: i) Minimizing $M$ faster than $\sum_{i=1}^{M} \delta_i$, which eventually decrease the training error rate, ii) Minimizing $M$ comparatively slower than $\sum_{i=1}^{M} \delta_i$, which may cause unexpected phenomena on training performance, iii) Minimizing $M$ and $\sum_{i=1}^{M} \delta_i$ with the same rate and preference, which may not lower the training error rate expectedly. Thus, throwing $P$ ($P \le M \le N$ ) outlying patterns apparently decreases $P$ training errors while also lowers $P$ kernel executions in classification stage when these example patterns are also elements of the final classifier. Nevertheless, how this outlier throwing impacts the classification of other patterns is an important issue to address. We examine it through a careful analysis of the contribution as well as confusion caused by them in the accurate classification of other patterns, which we refer to as $\xi$-analysis, and it is explained in the following sections.\\\\
Further, the discriminators from both VLPSVM and QPSVM can be rewritten using the following common expression by introducing a shared term, "Kernel Computing Vector (KCV)", for Support Vector(SV), Expansion Vector(EV), or such other kernel operating vectors:
 $f(p) = \sum_{l \in \text{KCV-set}} \beta_l y_l K(x_l,p) + b$, for \( p \) representing any arbitrary pattern with \( K(x_l, p) = \phi(x_l) \cdot \phi(p) \) is a kernel function (for example, in case of RBF: $\phi(x_l) \cdot \phi(p) = e^{-\frac{(x_l - p)^2}{2\sigma^2}}$), which is the
 leading contributor to computational expenses
  with all \( \beta_l > 0 \) involved in the aggregation of time during the classification stage. Following this, we aim to identify a subset from this kernel operating \( x_l \) patterns that have \( \beta_l > 0 \) and discard those patterns that do not contribute meaningfully to decision-making because of being outliers. This is the main focus of our \textit{stochastic \( \xi \)-analysis}, which is discussed in the next section.

\subsection{Numerical Analysis  of  Stochastic Auxiliary Variable, $\xi$}
A model should ideally take the data's inherent signal and exclude any noise or erratic patterns, which is not a simple task as data is often far from perfect like containing outliers, irrelevant variables, etc, which are related to the stochastic property of data. This is efficiently handled by introducing the auxiliary variable, $\xi$ in the mathematical model.
In case of LP SVM, its value comes from the solution of an optimization  problem using probabilistic data dependent stochastic parameters. This variable quantifies the degree of violation of the margin constraint for the patterns, which
can be interpreted as a measure of uncertainty or noise in the data and following its values,
patterns’ class-position, patterns’ role as well as actual significance in the final discriminator can be analyzed.
\subsubsection{Patterns' class-position following $\xi$}
As we have described before that an "outlier" pattern is the one that lies beyond the limits of its class boundary. Thus, a pattern is considered  "inlier" if it stays within its own class boundary. Hence, for any arbitrary training pattern $x_i$ having the class-label $y_i$ with the final decision function-value $f(x_i)$ is considered as an inlier when $y_i = sgn\big(f(x_i)\big)  \implies  1= y_i sgn\big(f(x_i)\big)  \implies  y_i f(x_i) > 0 $ and is considered as an outlier if $y_i = -sgn\big(f(x_i)\big)  \implies  -1= y_i sgn\big(f(x_i)\big)  \implies  y_i f(x_i) < 0$. Expressing the SVM-error-constraints in (\ref{Vapnik_QP_CSVC_primal_2}) or in (\ref{Vapnik_LP_CSVC_primal_2}) in a common single form utilizing the stochastic auxiliary variable $\xi$ as $  y_i f(x_i) ~ \ge ~ 1-\xi_i \implies \xi_i  ~ \ge ~ 1- y_i f(x_i)$, we can write for outlier $\xi_i  > 1$  while in case of inlier $0 \le \xi < 1 $
with the next two scenarios: i) inlier with $ y_i f(x_i) < 1  \implies   1 - y_i f(x_i) > 0  \implies  ~  \xi_i > 0 $ ii) inlier with $ y_i f(x_i) > 1  \implies  1 - y_i f(x_i) < 0  \implies  \xi_i = 0 $ \text{as $\xi_i$ cannot be negative}.

From the above discussed points, it is evident that patterns with \( \xi > 1 \) belong to  different classes rather than their original classes. Now, we examine their significance by evaluating the roles they play in the discriminating function in case of both LPSVM and QPSVM:


\subsubsection{Roles of the $\xi > 1$ patterns in the discriminating function}

\paragraph{Role while working with QPSVM:}

QPSVM has a KKT condition as $ \alpha_i \Big(y_i \big(w \cdot \phi(x_i) + b \big)~ -1 +\zeta_i\Big) = 0$. Using $ \alpha_i \ne  0$ \& $ \alpha_i \Big(y_i \big(w \cdot \phi(x_i) + b \big)~ -1 +\xi_i\Big) = 0$ leads to  $ y_i \big(w \cdot \phi(x_i) + b \big)~ -1 +\xi_i = 0 \implies y_i \big(w \cdot \phi(x_i) + b \big)= 1 -\xi_i $. From this, it can be found that 
QPSVM gives three types of KCV patterns (that is, patterns having $ \alpha_i \ne  0$) considering the values of $\xi$: i) $ \xi  = 0 $  ii) $ 0 < \xi  \le 1  $  iii)  $ \xi >1 $. But after a little calculation with another KKT condition, it can be seen that if $ \xi  > 0 $ for any pattern, its corresponding co-efficient value, $\alpha$ becomes equal to the penalty~parameter, $C$ that is the upper limit of $\alpha$ values. Thus, discarding patterns where $ \xi >1 $ not only ignores a KKT condition but also removes crucial elements of the decision function that have the largest coefficient values, potentially impacting the decision-making process significantly. Additionally, removing patterns having $\alpha = C $ could significantly disrupt the condition $\sum_{i=1}^{N}\alpha_i y_i = 0$, which ensures the balance between the support vector patterns and their associated coefficients, similar to a mechanical system, thereby impacting the optimal margin and the final decision boundary. Therefore, discarding any pattern from this machine is not in our option. 
 
\paragraph{Role while working with VLPSVM:}
    In this case, unlike QPSVM, values of this stochastic auxiliary variables $\xi$ are found by solving an optimization problem directly using its primal from where no direct involvement among KCV-coefficients, KKT conditions, and $\xi$ are seen. Moreover, there is no other constraint-based restrictions involving the KCV coefficients.
  This provides an opportunity to work further on this machine. Experience with VLPSVM shows that patterns with \( \xi > 1 \) have relatively small corresponding coefficients \( \beta \) in the final decision function. This aligns with the two main parts of its objective function: $\min_{\beta, \xi, b} \sum_{i=1}^{N} \beta_i + C \sum_{i=1}^{N} \xi_i$. Nevertheless, these coefficient values do not fully vanish, meaning the corresponding patterns continue to be part of the decision function, with each of them requiring kernel evaluation for classification, 
  even though they may not have a considerable constructive impact on the final decision. This brings the arguments about whether they should be included in the final discriminator, considering the trade-off between classification accuracy and computational cost. To address this issue, we examine their contribution as well as the confusion they create in accurate classification, as explained below:
  
  
\subsubsection{Contribution and Confusion from the patterns having $\xi > 1$ while working with VLPSVM}

We have seen earlier that these patterns consistently stay beyond the boundaries of their classes, produce comparatively larger kernel values while in operation with patterns from the opposite classes, leading to confusion. On contrary, they generate lower kernel values while in operation with patterns from their same classes, which produces weaker contribution in the final discriminating function. However, there may be confusion from the inlier KCVs by very weakly attracting patterns from the other classes, we ignore this here as they are negligible in this circumstance and efficiently handled by the complete machine.

Now, the final discriminating function that decides about any arbitrary training pattern $x_a$ that has class-label $y_a$ is 
$f(x_a) = \sum_{l \in KCV-set} \beta_l y_l  K(x_l, x_a) + b$. Considering $ m, n : m \cup n = \{l\} , m \cap n = \{\} , y_m = - y_n $, then $x_m, x_n$ are KCVs. Now as $\beta_m, \beta_n > 0$ \& $K(x_m,x_a), K(x_n,x_a) > 0$, if $\frac{\beta_m y_m K(x_m,x_a)} {y_a} < 0 \implies \frac{y_m} {y_a} < 0\implies x_a ~\text{and}~ x_m$ do not belong to the same class and $x_m ~\text{pulls}~ x_a$ outwards from $x_a$'s own class (i,e., towards, the opposite class of $x_a$) and if $\frac{\beta_n y_n K(x_n,x_a)} {y_a} > 0 \implies \frac{y_n} {y_a} > 0\implies x_a ~\text{and}~ x_n$ are from the same class with $x_n ~\text{pulls}~ x_a$ towards the $x_a$'s home class. However, from these KCV patterns, those with \( \xi > 1 \) remain within the boundaries of their opposite classes, produce comparatively larger kernel values while considering samples from the opposite classes, which misleads the discriminating function. Simultaneously, they generate lower kernel values while considering patterns from their same classes, poorly guiding the discriminating function. Thus, to enhance both classification accuracy  and speed, we decide to discard these $\xi>1$ KCV patterns after evaluating their level of confusion (which pushes training patterns away from their respective classes) as well as their contribution (which pulls training patterns towards their own classes) \cite{karim2019computational}, taking into account all training patterns through the following mathematical relations: \\
Contribution of a $\xi > 1$ KCV, $x_O$ with label $y_O$ on all training patterns, $Contr(x_O) = \frac{1}{Card\{i\}} \sum_{\{i: y_i.y_O = 1\}} \beta_{x_O} K(x_O,x_i)$, whereas Confusion of that KCV, $x_O$ on all training patterns, $Conf(x_O) = \frac{1}{Card\{i\}}\sum_{\{i: y_i.y_O = -1\}}\beta_{x_O} K(x_O,x_i)$ \\
 
Extensive experiments conducted on the benchmark datasets using the mathematical formulations above reveal that, in almost all cases, $\xi>1$  KCV patterns pose confusion more than contribution value. This agrees with our decision to remove $\xi>1$  KCV patterns from the final discriminating function. figure \ref{fig_cont_conf1} on page \pageref{fig_cont_conf1} provides an example of a contribution-confusion sketch. 

\begin{figure*}[t!]
  \centering
  \subfloat[Banana]{\includegraphics[width=8cm,height=5cm]{banana56_z.eps}} \hfil
  \subfloat[Breast Cancer]{\includegraphics[width=8cm,height=5cm]{breast27_z.eps}} 
  
  \caption{Confusion \& Contribution values for the Patterns having $\xi \geq 1$ from the training set.}
\label{fig_cont_conf1}
  % \vspace{-0.00cm}
\end{figure*}


\subsubsection{Contribution, Confusion as a function of Data Mapping}
We have discussed about our data mapping and its motivation earlier, here we discuss and check our hypothesis that compact mapping reduces the cardinality of the representer set, that is, information space with respect to the representer set gets denser.
 It is likely that the KCVs identified after data mapping may form a subset of the KCVs derived from the original non-scaled data or may contain part of it and the KCVs obtained after data mapping should carry more information. To check this, 
  we have computed contribution to confusion ratio (\( CCR \)) for the intersecting basis vector sets from before and after mapping. For each of common vectors \(KCV_{b}\) = \( B \cap B' \), \(CCR\) and \(CCR'\) for pre and post mapped basis pattern \(x_b\), \(x_b'\) with label \(y_b\), \(y_b'\) can be described as follows:

\[
CCR = \frac{\frac{1}{\text{card}\{i\}} \sum_{\{i: y_i.y_b = 1\}} \beta_{x_b} K(x_b, x_i)}{\frac{1}{\text{card}\{i\}} \sum_{\{i: y_i.y_b = -1\}} \beta_{x_b} K(x_b, x_i)}
\]

\[
CCR' = \frac{\frac{1}{\text{card}\{i\}} \sum_{\{i: y_i.y_b' = 1\}} \beta_{x_b'} K(x_b', x_i)}{\frac{1}{\text{card}\{i\}} \sum_{\{i: y_i.y_b' = -1\}} \beta_{x_b'} K(x_b', x_i)}
\]


Where, $(x_{b},y_{b})\in$\(B\), and $(x_{b}', y_{b}')\in$\(B'\) \\
 
Our findings show that after data mapping, \(KCV_{b}\) set have a higher contribution-to-confusion ratio, such that \( CCR' > CCR \), as illustrated in figure \ref{conf-contr-ratio} on page \pageref{conf-contr-ratio}. This observation supports the idea that the contribution of the reduced basis set becomes significantly higher than the confusion, thereby highlighting the effectiveness of our data mapping.

\begin{figure}[t]
  \centering
  {\includegraphics[width=9cm,height=5cm]{banana-2_contribution_before_z.eps}}
  \caption{Confusion and Contribution ratio for Common KCVs from Before and After Mapping of Data.}
\label{conf-contr-ratio}
% \vspace{-2.00cm}
\end{figure}

\subsubsection{Updating Bias}
Weight vector of our sparser machine $w_{EBSVM}$ is found by taking KCV set, $S_R$ as a subset of the KCV set, $S$ from  the
 full scale LP SVM solution so that $S_R \subseteq S$ and if $R$ and $M$
be the elements of
$S_R$ and $S$ respectively, then $R\le M$. Now, if $w$ be the weight vector of this full scale LP SVM machine, it is very most likely that $w_{EBSVM} = \sum_{j=1}^{R} \beta_j y_j \phi(x_j)  \ne w =\sum_{p=1}^{M} \beta_l y_l  \phi(x_l)$ and thus for any pattern $x$, $w_{EBSVM} \cdot \phi(x) + b  \ne w \cdot \phi(x)+b$. Hence, it is important to provide an optimally updated bias for our proposed EBSVM. 
\\Now, as $R\le M$, it is normal that  model complexity of EBSVM is less than the model complexity of the full scale LP SVM, $\|w_{EBSVM}\|^2 \le \|w\|^2$ and hence margin of EBSVM is larger than the margin of full scale LP SVM. In another way, as $R\le M$, 2nd term or the VC-confidence term of the risk bound (\ref{Risk_Bound}) of EBSVM becomes smaller than that of the full scale LP SVM. But this reduction in the VC- confidence term due to lesser model-complexity generally raises the 1st term, $R_{emp}[f]$ of the risk bound (\ref{Risk_Bound}). Hence, we need to focus to reduce this part. We can do it by imitating the  discriminator function (value) of a complex machine and here we select the full scale LP SVM one as it is the origin from where EBSVM comes after modification. We do it in the following way:
\\Let the variable bias value of EBSVM is $b_R$, whose optimum value would be the final value of our EBSVM. Now if $\delta$ be the total squared difference between the function values from the full scale LP SVM and variable bias based EBSVM using all training patterns, then we can write,
\begin{equation}
\begin{split}
  \delta =  \sum_{i=1}^{N} \Bigg [\Big(\sum_{p=1}^{M} \beta_l y_l  \phi(x_l)\cdot\phi(x_i) + b \Big) \\ -\Big(w_{EBSVM}\cdot\phi(x_i)+b_R \Big)\Bigg ]^2 
\end{split}
\end{equation}

At the minimum of $\delta$, we should get 
\begin{gather}
    \frac{\partial\delta}{\partial b_R}=0 \\ \implies \sum_{i=1}^{N} \Bigg [\Big(\sum_{p=1}^{M} \beta_p y_p  \phi(x_p)\cdot\phi(x_i) + b \Big) \nonumber \\ -\Big(w_{EBSVM}\cdot\phi(x_i)+b_R \Big)\Bigg ] = 0 \nonumber 
    \\ \implies b_{R}^{*} = \frac{1}{N}\sum_{i=1}^{N} \Bigg [\sum_{p=1}^{M} \beta_p y_p  \phi(x_p)\cdot\phi(x_i) \\ \nonumber+ b -w_{EBSVM}\cdot\phi(x_i)\Bigg ]
\end{gather}
This $b_{R}^{*}$ is the value of the bias term of the proposed, EBSVM. Algorithm 1 on page \pageref{algo} outlines the steps of the proposed EBSVM whereas figure \ref{fig_decisionboundary1}, \ref{fig_decisionboundary2}, \ref{fig_decisionboundary3}, \ref{fig_decisionboundary4} show the decision boundaries using both the synthetic and benchmark datasets, from QPSVM, VLPSVM as well as the proposed EBSVM, along with their corresponding KCV counts and training error rate.
\\

%ALGORITHM
\begin{table}[!t]
\label{algo}
\rule{235pt}{3pt}
\begin{tabular}{l}  % Use only necessary columns
    \multicolumn{1}{c}{\textbf{Algorithm 1: Proposed EBSVM}} \\
    \multicolumn{1}{c}{\textbf{(Efficiency Boosted SVM)}} \\
\end{tabular}
\rule{235pt}{1pt}
\\1:~\textbf{Input:}~ A set of training data $\{\big (x_i, y_i\big)\}_{i= 1}^N$
\\2:~\textbf{Output:}~ A discriminator  $f_{EBSVM}(\cdot)$
\\3:~Scale the training set efficiently
\\4:~Select the best values of the data dependent parameters:\\ $(Penalty-parameter,Kernel-parameter)\equiv(C,\sigma)$
\\5:~Run LP based SVM (VLPSVM) using training data and solve the following optimization problem as below:
\begin{gather}
\min_{\beta,\xi, b}  ~~\sum_{l=1}^{N} \beta_l  ~+~C \sum_{l=1}^{N} \xi_l
\notag
\\ s.t. ~~ y_i \Big(\sum_{l=1}^{N} \beta_l y_l  \phi(x_l)\cdot\phi(x_i) + b \Big) ~~ \ge ~ 1-\xi_i
\notag
\\\beta_l ~ \ge ~0 ;~l = 1,2,...,N; \xi_i ~ \ge ~0 ; ~i = 1,2,...,N \notag
\end{gather}
\\6:~Obtain the KCVs along with their labels $\{\big (x_p, y_p\big)\}_{p= 1}^M$ and the bias term, $b$ derived from the VLPSVM applying $\beta_l > 0$
\\7:~Find $\{\beta_p | \xi_p > 1 \}$ and discard the corresponding patterns belong to the KCVs set and form a new smaller set of KCVs (RKCVs), $\{ x_p | \xi_p \le 1 ~with~ \beta_p > 0\}$
\\8:~Retrieve the RKCVs with their labels $\{\big (x_j, y_j\big)\}_{j = 1}^R$
\\9:~$w_{EBSVM} \gets \sum_{j=1}^{R} \beta_j y_j \phi(x_j) $ 
\\10.~$b_{EBSVM} \! \gets argmin_{b_R} \sum_{i=1}^{N} \Bigg [\Big(\sum_{p=1}^{M} \beta_p y_p  \phi(x_p)\cdot\phi(x_i) + b \Big) -\Big(w_{EBSVM}\cdot\phi(x_i)+b_R \Big)\Bigg ]^2 $
\\11:~Return $f_{EBSVM}(\cdot)= w_{EBSVM} \cdot \phi(\cdot) + b_{EBSVM}$
\rule{235pt}{1pt}
\end{table}


%%%% decision Boundary

\begin{figure*}[t!]
\centering

    \hspace*{-20pt}
    \subfloat[QPSVM]
    {\includegraphics[width=6.8cm,height=3.5cm]{banana96_qpsvm_rbf_z.eps}} 
    \hspace*{-19pt}
  \subfloat[VLPSVM]{\includegraphics[width=6.8cm,height=3.5cm]{banana96_lpsvm_rbf_z.eps}}  
  \hspace*{-19pt}
  \subfloat[Proposed EBSVM]{\includegraphics[width=6.8cm,height=3.5cm]{banana96_rnlpsvm_rbf_z.eps}}  
  
  \caption{Pattern Classification Boundaries on Banana data along with their respective KCVs counts and training error rates for different machines.}
  \label{fig_decisionboundary1}
  % \vspace{-0.00cm}
\end{figure*}
%
\begin{figure*}[t!]
  \centering
    \hspace*{-20pt}
    \subfloat[QPSVM]
    {\includegraphics[width=6.8cm,height=3.5cm]{3.qp_tS_3.3.eps}} 
    \hspace*{-19pt}
  \subfloat[VLPSVM]{\includegraphics[width=6.8cm,height=3.5cm]{3.lp_tS_3.3.eps}}  
  \hspace*{-19pt}
  \subfloat[Proposed EBSVM]{\includegraphics[width=6.8cm,height=3.5cm]{3.rnlp_tS_3.3.eps}} 
  
  \caption{Pattern Classification Boundaries on synthetic Two spiral data (with noise level 3.3) along with their respective KCVs counts and training error rates for different machines.}
  \label{fig_decisionboundary2}
  % \vspace{-0.00cm}
\end{figure*}
%
\begin{figure*}[t!]
  \centering
    \hspace*{-20pt}
    \subfloat[QPSVM]
    {\includegraphics[width=6.8cm,height=3.5cm]{2.qp_tS_4.3.eps}} 
    \hspace*{-19pt}
  \subfloat[VLPSVM]{\includegraphics[width=6.8cm,height=3.5cm]{2.lp_tS_4.3.eps}}  
  \hspace*{-19pt}
  \subfloat[Proposed EBSVM]{\includegraphics[width=6.8cm,height=3.5cm]{2.rnlp_tS_4.3.eps}} 
  
  \caption{Pattern Classification Boundaries on synthetic Two spiral data (with noise level 4.3) along with their respective KCVs counts and training error rates for different machines.}
  \label{fig_decisionboundary3}
  % \vspace{-0.00cm}
\end{figure*}
%
\begin{figure*}[t!]
  \centering
    \hspace*{-20pt}
    \subfloat[QPSVM]
    {\includegraphics[width=6.8cm,height=3.5cm]{1.qp_tS_4.6.eps}} 
    \hspace*{-19pt}
  \subfloat[VLPSVM]{\includegraphics[width=6.8cm,height=3.5cm]{1.lp_tS_4.6.eps}}  
  \hspace*{-19pt}
  \subfloat[Proposed EBSVM]{\includegraphics[width=6.8cm,height=3.5cm]{1.rnlp_tS_4.6.eps}} 
  
  \caption{Pattern Classification Boundaries on synthetic Two spiral data (with noise level 4.6) along with their respective KCVs counts and training error rates for different machines.}
  \label{fig_decisionboundary4}
  % \vspace{-0.00cm}
\end{figure*}
%

\section{Experimental Set-up and Results} \label{results}
\subsection{Set-up Method}

In this section, the effectiveness of the suggested EBSVM algorithm is demonstrated by a comparison of its experimental outcomes with those of many state-of-the machines, including 
QPSVM\cite{Vapnik1995}, VLPSVM\cite{vapnik1998}, EESVM\cite{kundu2023escalating} and SpSVM-2 \cite{keerthi2006building}. The outcomes are broken down into two dimensions: the algorithm's classification cost and it's performance. Ten publicly available benchmark datasets—Heart, Twonorm, Titanic, Thyroid, Breast Cancer, Banana, Flare Solar, German, and Waveform are used to evaluate the algorithm's performance, as referenced in \cite{data}. Each of these datasets includes 100 realizations of both training and test sets. Scaling is executed on each dataset by dividing with the maximum among the absolute values of the features, as discussed in the Data Scaling section.
All studies are conducted using a Gaussian kernel on these scaled datasets. A varied collection of $C \in \{2^{0},2^{2},...,2^{20}\}$ and $\sigma \in \{2^{-2},2^{0},2^{2},...,2^{8}\}$ is employed in fivefold cross-validation, wherein each $C$ value is normalized by the scaling factor of the respective dataset, thereby calibrating the computation of $C$ for each dataset pattern. Fivefold cross-validation is employed to determine the optimal parameters, extracting the best values of $C$ and $\sigma$ for each dataset. The initial training set among 100 is utilized in cross-validation. The mean number of KCV values and the test error rates for QPSVM, VLPSVM, SpSVM-2, EESVM, and the proposed approach are presented in Table \ref{tab:svm_comparison} page \pageref{tab:svm_comparison}. The proposed EBSVM achieves similar accuracy with all of these SOTA machines while  reducing the KCV count dramatically compared to the standard SVM (QPSVM, which is sparse itself), significantly  compared to the sparser VLPSVM while considerably compared to the further sparser EESVM and SpSVM-2. Details of KCV counts rate with corresponding error rates are described in caption of table \ref{tab:svm_comparison}.


\begin{table*}[t]
\huge
    \centering
    \caption{Number of KCVs as well as Test Error Rates for Various state-of-the art Machines and Our Proposed Method (EBSVM)}
    \label{tab:svm_comparison}
    \resizebox{\textwidth}{!}{%
    \begin{tabular}{lcccccc}
        \toprule
        Name of Dataset, & QPSVM\cite{Vapnik1995} & VLPSVM\cite{vapnik1998} & EESVM\cite{kundu2023escalating} & SpSVM-2\cite{keerthi2006building} & EBSVM \\
        (\#Training Examples, \#Testing & (KCVs, TestErR) & (KCVs, TestErR) & (KCVs, TestErR) & (KCVs, TestErR) & (KCVs, TestErR) \\
        Examples, \#Dimension) & & & & & \\
        \midrule
        Banana (400, 4900, 2) & (102.26, 10.61) & (15.08, 10.75) & (13.62, 11.53) & (17.30, 10.87) & (16.64, 10.65) \\
        \hline
        Breast Cancer (200, 77, 9) & (200, 28.53) & (18.89, 26.05) & *NR & (12.10, 29.22) & (\textbf{8.01}, 27.56) \\
        \hline
        Diabetes (468, 300, 8) & (263.22, 23.28) & (12.58, 23.40) & *NR & (13.80, 23.47) & (\textbf{8.81}, 23.66) \\
        \hline
        Flare Solar (666, 400, 9) & (609.98, 32.56) & (251.66, 32.41) & *NR & (8.40, 33.90) & (\textbf{3.87}, 32.32) \\
        \hline
        German (700, 300, 20) & (438.75, 23.70) & (26.47, 24) & (27.61, 24.16) & (14.00, 24.90) & (19.35, 24.28) \\
        \hline
        Heart (170, 100, 13) & (68.23, 16.60) & (21.94, 17.44) & (7.45, 15.33) & (4.30, 15.50) & (7.42, \textbf{15.23}) \\
        \hline
        Ringnorm (400, 7000, 20) & (77.00, 2.23) & (15.99, 1.73) & (15.83, 2.54) & (12.9, 1.97) & (15.84, 1.86) \\
        \hline
        Splice (150, 2051, 3) & (943.65, 12.34) & (292.15, 12.39) & *NR & *NR & (\textbf{283.25}, 12.39) \\
        \hline
        Titanic (150, 2051, 3) & (148.50, 22.68) & (83.91, 22.91) & (39.19, 23.18) & (3.30, 22.68) & (5.70, 23.12) \\
        \hline
        Waveform (400, 4600, 21) & (228.33, 13.15) & (20.81, 11.18) & (10.67, 12.72) & (14.40, 10.66) & (13.01, 10.81) \\
        \midrule
        \textbf{Overall Average} & (307.99, 18.55) & (75.95, 18.24) & (- , -) & (- , -) & (\textbf{38.19}, 18.37) \\
        \textbf{Average (1) w.r.t SpSVM-2} & (237.36, 19.24) & (51.93, 18.89) & (- , -) & (11.17, 19.24) & (\textbf{10.96}, 19.03) \\
        \textbf{Average (2) w.r.t EESVM} & (177.18, 14.83) & (30.7, 14.67) & (19.06, 14.91) & (- , -) & (\textbf{12.99}, 14.55) \\
        \bottomrule
    \end{tabular}%
    }
    \tiny \rightline{*NR: Not Reported}
 \justifying{Table \ref{tab:svm_comparison} provides a detailed comparison of Kernel Computing Vectors (KCVs) and test error rates (TestErR) across various SVM models including QPSVM, VLPSVM, EESVM, SpSVM-2, and the proposed EBSVM. The results highlight that EBSVM achieves significant computational gain while maintaining strong classification accuracy. Notably, in the breast cancer, flare solar, heart, and splice datasets, EBSVM demonstrates remarkable efficiency, significantly reducing KCVs compared to other models while maintaining competitive error rates. Looking at the overall average, excluding datasets where values were not reported, EBSVM reduces KCVs by 87.6\% compared to QPSVM and 49.7\% compared to VLPSVM, with just a minimal 0.7\% and 0.71\% increase in test error rates, respectively. When compared to Average (1) i.e., w.r.t SpSVM-2, EBSVM requires 1.2\% fewer KCVs and achieves a 0.21\% lower error rate. Against EESVM of Average (2), EBSVM reduces KCVs by 31.5\% while also achieving a 2.4\% lower test error rate. These findings confirm that EBSVM is a highly efficient approach, striking the right balance between computational cost and classification accuracy, making it a strongly desired classifier in the filed of modern machine learning.}
\end{table*}



\subsection{Result Analysis} %Subsection is not in place
\subsubsection{Error-Accuracy In Stochastic Classification }
For any arbitrary pattern $x$ having class label $y$, our discriminator  models the decision function in the form:
$f(x) = w^T \phi(x) + b$ with the predicted class label as
$Classified~Class~Label(x) = {y}'= sgn(f(x)) $. Thus a  novel pattern is classified correctly if its decision function value has the same polarity as its actual class label, i.e., \( f(x) > 0 \) if it is from positive class  and \( f(x) < 0 \) if it is from negative class and consequently, during this detection (to accept positive patterns and reject negative patterns) by classifications of  patterns, two types of errors are to be considered first. They are:\\
\textbf{False positive error:} This is the error when a Negative pattern is considered as a Positive pattern by mistake. We denote this rate or probability as $P_0(x=1)$. Mathematically, we can define it as,

\begin{equation}
 P_0(x=1)=~
\begin{array}{c}
\textit{Number of Falsely} \\ \textit{accepted Negative Pattern} \\
\hline
\textit{Total number of Input} \\ \textit{ Negative Pattern}
\end{array}
\end{equation}

\begin{equation}
\begin{array}{ll}
\Rightarrow P({y}'=1|y=-1)~\\\\=~\begin{array}{c}
\textit{Number of Falsely} \\ \textit{accepted Negative Pattern} \\
\hline
\textit{Total number of Input} \\ \textit{ Negative Pattern}
\end{array}
\end{array}
\end{equation}


\begin{equation}
\begin{array}{ll}
    \Rightarrow P({y}'=1|y=-1) 
      \\\\=\frac{\frac{Number~ of~Falsely~ accepted~Negative~Pattern}{Total~ number~ of~ Input~Pattern}}{\frac{Total~number~ of~ Input~Negative~Pattern}{Total~ number~ of~ Input~Pattern}}
\end{array}
\end{equation}

\begin{equation}
\Rightarrow P({y}'=1|y=-1)~=~\frac{P({y}'=1,y=-1)}{P(y=-1)}
\end{equation}
Where $P({y}'=1|y=-1)$  
 is the conditional probability that the predicted class label ${y}'$ is positive but the actual class label $y$  is negative, $P({y}'=1,y=-1)$ is the joint probability distribution of this predicted and actual class labels, $P(y=-1)$ is the marginal probability distribution of negative class labels.   
This can also be thought as $1-Negative~Rejection~Rate$.  \emph{Negative Rejection rate} is often termed as \emph{power}.
\\ \textbf{False negative error:} This is the error when a Positive pattern is considered as a Negative pattern by mistake. Sometimes,it is also called as "missed detection". We denote this rate or probability as $P_1(x=0)$. Mathematically, we can define it as
\begin{equation}
 \begin{array}{ll}
 \begin{array}{ll}
      P_1(x=0) \\ =\frac{Number~ of~ Rejected~ Positive~Pattern}{Total~ number~ of~ Input~Positive~Pattern}
 \end{array}\\
\\\Rightarrow P({y}'=-1|y=1)\\\\=~\frac{Number~ of~ Rejected~ Positive~Pattern}{Total~ number~ of~ Input~Positive~Pattern} \\
\\\Rightarrow P({y}'=-1|y=1)\\\\=~\frac{\frac{Number~ of~ Rejected~ Positive~Pattern}{Total~ number~ of~ Input~Pattern}}{\frac{Total~ number~ of~ Input~Positive~Pattern}{Total~ number~ of~ Input~Pattern}} \\
\\\Rightarrow P({y}'=-1|y=1)~=~\frac{P({y}'=-1,y=1)}{P(y=1)}
\end{array}
\end{equation}
Where $P({y}'=-1|y=1)$  
 is the conditional probability that the predicted class label ${y}'$ is negative but the actual class label $y$  is positive, $P({y}'=-1,y=1)$ is the joint probability distribution of this predicted and actual class labels, $P(y=1)$ is the marginal probability distribution of positive class labels.
\\Using the above two terms, we can also find the total error probability during each detection as below

\begin{eqnarray} 
P(Error) = P(Classified~Class~Label(x)\\ \nonumber \neq Actual~Class~label(x)) )
\end{eqnarray}

\begin{equation}
\begin{array}{ll}
 P({y}'\neq y) = P({y}' = 1, y= -1)\\+P({y}' = -1, y= 1) \\\\= P({y}' \nonumber
 = 1| y= -1)P(y= -1) \\+ P({y}' = -1| y= 1) P(y=1) \\\\=\nonumber
 ~\frac{Number~ of~ accepted~ Negative~Pattern}{Total~ number~ of~ Input~Negative~Pattern}\\\\~\frac{Total~Number~of~ Negative~Pattern}{Total~ number~ of~ Input~Pattern} \\\\\nonumber~+~\frac{Number~ of~ Rejected~Positive~Pattern}{Total~ number~ of~Input~Positive~Pattern}\\\\~\frac{Total~Number~of~ Positive~Pattern}{Total~ number~ of~ Input~Pattern}\\\\ \nonumber
 =~\begin{array}{c}
\textit{Number of accepted Negative Pattern} \\ \textit{+Number of Rejected Positive Pattern} \\
\hline
\textit{Total number of Input Pattern}
\end{array}
 % =~\frac{Number~ of~ accepted~ Negative~Pattern~+~Number~ of~ Rejected~Positive~Pattern}{Total~ number~ of~Input~Pattern}\\\\ \nonumber
 % \Rightarrow P(Error)=~\frac{Total~Number~of~wrong ~prediction}{Total~ number~ of~Input~Pattern} \nonumber
 \end{array}
\end{equation}

and accordingly, the total accuracy
probability during each detection is $$P(Accuracy) = 1-P(Error)$$
However, in our evaluation, we measure the error rates and accuracy in percentages ($\%$) as
\[
Error~Rate = \frac{1}{N} \sum_{i=1}^ N\mathbf{1}[y_i \neq {y_i}'] \times 100
\] where $N$ is the total number of classified patterns and $\mathbf{1}$ is the indicator function.
\subsubsection{Machine Accuracy Cost (MAC)} 
A powerful yet cost-effective computing model is always really preferred. Considering this the objective is to design a discriminator that minimizes kernel evaluations during testing while preserving high accuracy. To strike a balance between an expensive machine with superior accuracy and a highly cost-effective system with similar efficiency, the concept of MAC\cite{Karim-Kundu-2018-1} is introduced for kernel-based models, defined as follows: 

\begin{equation}
MAC = \frac{Number~~of~~KCVs}{Test~~Accuracy}    
 \end{equation}

A lower MAC value is always desirable, as it signifies that the machine attains optimal test accuracy with minimal kernel computations (KCVs). The values of MAC for different systems are provided in Table \ref{tab:MAC_Cmp} in page \pageref{tab:MAC_Cmp}.


\begin{table*}[h]
\centering
\caption{MAC (Machine Accuracy Cos) Table}
 % \resizebox{\textwidth}{!}{%
\begin{tabular}{lcccccccc}
\toprule
Dataset & QPSVM & VLPSVM & EESVM & SpSVM-2 & EBSVM \\ \midrule
Banana & 1.144 & 0.169 & 0.154 & 0.194 & 0.186 \\
Breast Cancer & 2.798 & 0.255 & - & 0.171 & \textbf{0.111} \\
Diabetes & 3.431 & 0.164 & - & 0.180 & \textbf{0.116} \\
Flare Solar & 9.025 & 3.732 & - & 0.127 & \textbf{0.057} \\
German & 5.750 & 0.348 & 0.364 & 0.186 & 0.256 \\
Heart & 0.818 & 0.266 & 0.088 & 0.051 & 0.088 \\
Ringnorm & 0.788 & 0.163 & 0.162 & 0.132 & 0.162 \\
Splice & 10.765 & 3.335 & - & - & \textbf{3.233} \\
% thyroid & 0.459 & 0.095 & 0.095 &       & 0.112 & \textbf{0.091} \\
Titanic & 1.921 & 1.088 & 0.510 & 0.043 & 0.074 \\
Waveform & 2.629 & 0.234 & 0.122 & 0.161 & 0.148 \\ \bottomrule
\end{tabular}
\\ \noindent
 \tiny \justifying{\\Table \ref{tab:MAC_Cmp} presents the MAC values for the different machines, including the proposed algorithm. According to the results, the proposed algorithm completely outperforms all other classifiers in terms of MAC values for the Flare Solar, Diabetes, Breast Cancer, and Splice datasets while giving much smaller MAC values compared to the QPSVM, VLPSVM for all datasets. However, in case of other six datasets, it gives very similar MAC values compared to the other two further sparser machines, EESVM and SpSVM-2.This shows that compared to the reported classification algorithms, the suggested model achieves a more cost-efficient accuracy.}
\label{tab:MAC_Cmp}%
\end{table*}


\subsubsection{GFR(Generalization failure rate)}
We know that the optimal stochastic machine is designed by properly tuned from the bias-variance tradeoff and gives least generalization error ensuring that the model is not overfitting to the training data and can perform well on novel, unseen data. We have also tried to activate this on our model basing on some suitable mathematical formulations. However, as it is further sparsified from a sparser machine, its bridging capability between the training and new unseen data becomes more into interest. In this regard, we have used the term GFR \cite{Karim-Kundu-2018-1} to asses the  generalization deficiency of our machine as below, which is defined considering  two points simultaneously: i) how badly the machine overfits, and ii) how poor it performs on test patterns. :
\begin{equation}
\begin{array}{ll}
    Overfitting~Tendency,~OT \\\\= \frac{Test~Error~Rate-Train~Error~Rate}{Train~Error~Rate} 
\end{array}
\end{equation}\label{eqnOT}

\begin{equation}
   GFR = \frac{OT}{Test~~Accuracy}\label{eqnGFR} 
\end{equation}

\subsubsection*{Why less GFR or good Generalization Performance from our proposed EBSVM:}
 As we have seen that $w = \sum \beta_j y_j \phi(x_j)$,  for LP SVM, throwing KCVs with $\beta_j> 0$  gives less $||w||$, hence higher margin and reduced VC confidence, 2nd part of the RHS of the risk bound (\ref{Risk_Bound}) becomes smaller. This usually increases the 1st part of the RHS of the risk bound (\ref{Risk_Bound}) ,  $R_{emp}[f]$ due to the reduced model complexity.  But in our algorithm,  we  generate the  bias value of our final discriminator by imitating the function value of the complex and non-reduced discriminator, which leads us to realize the final discriminator generated from an optimally further larger margin  and sparser class based hypothesis where the hyperplane is shifted towards the complex model reducing the value of $R_{emp}[f]$ as well.
 Thus the overall value of the right hand side of (\ref{Risk_Bound}) gets lower and we manage to get the machine having reduced upper bound of the actual risk that gives less generalization error.


\begin{table*}[h]
 \centering
\caption{Generalization Failure Rate (GFR) for Different Machines}\label{tab:GFR_Cmp}
 % \resizebox{\textwidth}{!}{%
\begin{tabular}{lcccc}
\toprule
Dataset & QPSVM & VLPSVM & EESVM & EBSVM \\
\midrule
Banana & $5.97\times 10^{-3}$ & $6.14\times 10^{-3}$ &  $1.8\times 10^{-3}$  & $2.18\times 10^{-3}$ \\
Breast Cancer & $1.82\times 10^{-3}$ & $2.89\times 10^{-3}$& - & $\mathbf{1.23\times 10^{-3}}$ \\
Diabetes & $1.36\times 10^{-3}$ & $1.39\times 10^{-3}$ & - & $\mathbf{1.18\times 10^{-3}}$ \\
Flare Solar & $-7.67\times 10^{-5}$ & $-9.5\times 10^{-5}$ & - & $-9.06\times 10^{-5}$ \\
German & $1.44\times 10^{-3}$ & $7.18\times 10^{-3}$ & $9\times 10^{-4}$ & $\mathbf{8.7\times 10^{-4}}$ \\
Heart & $1.27\times 10^{-3}$ & $1.94\times 10^{-3}$ & $1\times 10^{-3}$ & $\mathbf{8.98\times 10^{-4}}$ \\
Ringnorm & $1.22\times 10^{-1}$ & $1.03\times 10^{-2}$ & $1.62\times 10^{-2}$ & $2.21\times 10^{-2}$ \\
Splice & $4.11\times 10^{-1}$ & $9.42$ & - & $9.42$ \\
Titanic & $1.01\times 10^{-3}$ & $1.49\times 10^{-3}$ & $1\times 10^{-3}$ & $1.59\times 10^{-3}$ \\
Waveform & $1.1\times 10^{-2}$ & $4.25\times 10^{-3}$ & $1.8\times 10^{-3}$ & $2.38\times 10^{-3}$ \\
\bottomrule
\end{tabular}
\\ \noindent
\tiny \justifying{\\Table \ref{tab:GFR_Cmp} presents the GFR values of various algorithms, including the proposed EBSVM, alongside QPSVM, VLPSVM, and EESVM ( SpSVM-2 is not reported here due to the lack of information). The results indicate that for the Breast Cancer, Diabetes, German, and Heart datasets, the proposed machine achieves lower GFR values compared to all other machines. However, for the remaining datasets, the proposed machine generally performs better in most cases.}
\end{table*}%
 
\section{Conclusion and Future Work}\label{conclusion}
In this paper, we have discussed one of the most mathematically solid statistical machine learning approaches, Support Vector Machine (SVM) that gives decision under uncertainty depending on the given probabilistic data and their dependent stochastic parameters. We have gone through briefly using its two main mathematical bases, statistical theory and mathematical programming. Basing on these concepts, we have gone a bit further to produce a very efficient detector having both high classification accuracy and speed that is realized from further sparsification 
of Vapnik's LP SVM which is indeed sparser itself. This proposed machine requires kernel computation as small as  0.6\% of the sparse QPSVM,  1.5\% of the sparser VLPSVM, 14.5\% of the sparser EESVM, and 46.1\% of the sparser SpSVM-2 by Keerthi et al\cite{keerthi2006building}, whereas in average  12.4\% of the sparse QPSVM,  50.3\% of the sparser VLPSVM, 68.1\% of the sparser EESVM, and 98.1\% of the  sparser SpSVM-2  though producing very similar classification accuracy despite of being very straight forward by demanding minimum effort for training. It also poses the least MAC (Machine Accuracy Cost, a term to assess the kernel load with respect to accuracy) value considering these machines while giving least GFR(Generalization failure rate) values compared to these SOTA machines as well, which proves high generalization quality too.
\\It is seen from the experiment that after our compact  deterministic mapping of probabilistic data, contribution increases relatively higher compared to the confusion from the same KCVs. This supports our idea that by our mapping, topological spanning of the data shirks without harming its stochastic nature countably and a basis set with comparatively less number of patterns becomes sufficient to represent the dataset. Hence relatively fewer number of KCVs can efficiently  generate the discriminator machine having optimum generalization capability. It is also seen that in case of the outlier KCVs, contribution gets smaller than confusion and hence these misleading (playing undesired role in the final discriminator) or least significant outlier KCVs become potential to be thrown away from the basis-set of the final discriminator for further sparsification of the sparser machine, which simultaneously helps to maintain high classification accuracy by reducing the probability of over-fitting and boost up the classification speed.
Our method recovers the small-scale gap that exists from SVM model which are: i) It focuses on minimizing the error-depth instead of emphasizing the classification-accuracy directly.
ii) There is no complete certainty to find
the best values of the regularization parameters from their range of continuous and real values through cross-validation process where a small deviation from the best values may lead to the solution that selects more patterns to be Support Vectors which drags the model towards over-fitting while increasing computational cost as well. 
iii) Having a highly deviated and noise-contaminated heavy-outlier may change the SVM solution dramatically from the training set without having it.
\\While some state-of-the-art machines pose the risk of decreasing classification accuracy by sparsification from throwing some significant bases vectors, our model does not do so as its sparsification is done by following a smart stochastic mathematical analysis. 
\\By throwing outlier KCVs, our model reduces machine-misleading noise, 
gives a hypothesis from a class with less VC dimension, larger margin with reduced model complexity and VC confidence term in (\ref{Risk_Bound}). Further, updating the bias term of our machine optimally, we have also managed to reduce the empirical risk from our model simultaneously. These jointly,lowers the upper bound on the actual risk from our model that eventually produces a machine with excellent generalization performance that is experimentally demonstrated from the GFR (Generalization Failure Rate) and Test Error Rate of our machine on benchmark data.
In case of a highly complex  and large dataset, a well generalized SVM is likely to get more outliers that are also included into the SVs-set making the machine more complex and computationally expensive, which is efficiently tackled by our stochastic $\xi$ analysis to make this machine optimally sparse and complex while in case of the dataset is less complex without having many outliers, our method will produce sparser solution with less number of KCVs naturally. This certainly proves the significance of our model.
\\Finally, while a classifier with high accuracy and speed like this proposed EBSVM is really indispensable
to satisfy constraints in real-time, next, we will try to make the KCV set $\nu$-sparse, where $\nu$ is a user defined variable.


\bibliographystyle{IEEEtran}
\bibliography{REF_EBSVM_DOI}
\end{document}
