An Adaptive Image Inpainting Method Based on the Weighted Mean

Imaging inpainting is the process of digitally filling-in missing pixel values in images and requires carefully crafted image analysis tools. In this work, we propose an adaptive image inpainting method based on the weighted mean. The weighted mean is assessed to be better than the median because, for the case of the weighted mean, we can exclude the values of the corrupted pixels from evaluating values to fill those corrupted pixels. In the experiments, we implement the algorithm on an open dataset with various corrupted masks and we also compare the inpainting result by the proposed method to other similar inpainting methods – the harmonic inpainting method and the inpainting by directional median filters – to prove its own effectiveness to restore small, medium as well as fairly large corrupted regions. This comparison will be handled based on two of the most popular image quality assessment error metrics, such as the peak signal to noise ratio, and structural similarity. Further, since the proposed inpainting method is non-iterative, it is suitable for implementations to process big imagery that traditionally require higher computational costs, such as the large, high-resolution images or video sequences.


Introduction
Image inpainting [1,2,3,4,5,6] is one of the interesting problems of image processing [7,8] that has attracted a lot of attention. Image inpainting or image interpolation is a process of filling the damaged and/or missing parts. Image inpainting has a wide range of applications in practice such as watermark removal [9], image disocclusion [10,11], restoring old images corrupted by dust, scratches, etc., image zooming, image super-resolution [8], etc.
To solve the image inpainting problem, there are some intensive approaches [12,13,14] such as partial-differential-equation-based (PDE-based) methods [1,8,15], calculus-of-variation-based methods [8], graph-based methods, stochastic methods, etc. and machine-learning-based methods. In this paper, we only focus on non-learningbased methods, because the comparison of methods based on machine-learning to non-learning-based methods is not fair. In non-learning-based methods, the PDE-based and variation-based methods are highly effective to treat this problem. However, these methods are iterative and the accuracy, performance, execution time depends much on the tolerance.
In this paper, we study an inpainting method based on the weighted mean [16]. Our method is non-iterative, hence, the computational complexity associated with traditional iterative inpainting methods. This advantage is especially appealing as it paves the way for us to apply inpainting for larger images or high-resolution images, such as Full HD 1080, 2K, 4K, and higher resolution and/or video sequences data.
We propose the method to fill the corrupted parts by the weighted mean because it excludes the value of the corrupted pixels from the evaluating process. The pixels of the corrupted regions are always different from the neighboring pixels' values. So, their values are not useful for evaluating the values to fill the corrupted regions. This idea is better than using the median. The proposed inpainting method is effective on various corrupted masks: from a small, medium up to large area.
In our experiments, we compare our inpainting results from the proposed method to the harmonic inpainting method [1,15] (PDE-based inpainting) and the inpainting method by directional median filters [17]. Our results prove that the adaptive inpainting based on the weighted mean is a novel method and performs favorably to other state-of-the-art inpainting methods.
The rest of the paper is organized as follows. Section 2 introduces the general image inpainting problem that is formulated in the form of the optimization problem, and the proposed adaptive inpainting method based on the weighted mean. Section 3 presents the experiments and the comparison of the inpainting result to another similar method. Finally, section 4 concludes.
As can be observed from the minimization model, to solve the above inpainting model, we need some complex optimization methods. The algorithms based on this model are iterative manners that the accuracy depends much on the tolerance.

The proposed inpainting method
In this paper, we propose the inpainting method based on the adaptive weighted mean filter. The adaptive weighted mean filter [16] is proposed by Zhang and Li to solve the denoising problem. Like other median-based and meanbased filters, this method evaluates the corrupted pixels values on the clipping windows by the mean value. However, this method considers the weight of pixels values to evaluate the mean. So, it is called to be the weighted mean.
For the inpainting problem, the corrupted regions are the connected regions with a small, medium or large area. This is quite different from the denoising problem: the corrupted pixels are always separated pixels or connected pixels on a small area (for the high noise levels).
Like the adaptive weighted mean filter for denoising, we also evaluate the corrupted pixels values based on the pixels values of the considered clipping windows.
Let , be indices of pixels of all image domain Ω and of the corrupted regions , respectively. We call ( ) a clipping window centered at the pixel ( , ) with a size of 2 + 1:   Our proposed inpainting method starts from the clipping windows of every corrupted pixel ( , ) ∈ . If the corrupted pixels lay on the position ( , ) with − ≤ 0 or + ≥ or − ≤ 0 or + ≥ , we need to extend the pixels matrix to top, bottom, left or right more rows or columns of pixels. The values to fill to the padding regions are taken by the symmetric method. In MATLAB, we can use the built-in function padarray. Figure 2 shows an example of the padding pixels matrix of the image 3×2 (the gray pixels area) with the padding is 2.
Next step, we need to evaluate the maximum, the minimum and the weighted mean values in every clipping window centered at the corrupted pixels: Finally, with every pixel of the corrupted regions ( , ) ∈ , we replace the corrupted pixel value by the weighted mean value .
The proposed image inpainting method has a smaller number of calculations than the denoising method by the adaptive weighted mean filter because we only consider the clipping windows of the corrupted pixels. The algorithm of the proposed image inpainting method by the adaptive weighted mean is presented in Algorithm 1.

Algorithm 1. The inpainting method base on the adaptive weighted mean.
Input: The corrupted image . Output: The restored image .
As can be seen in Algorithm 1, the values of the corrupted pixels are replaced by the weighted mean value. The weighted mean value will not contain the values of corrupted pixels in the clipping windows. We choose the weighted mean value because the values of the corrupted pixels always have no similarity with the values of other neighboring pixels in the same clipping window.

Experiments
We implement the proposed inpainting method and the harmonic inpainting method to recover the corrupted image by MATLAB 2018a. The configuration of the computing system is Windows 10 Pro with Intel Core i5, 1.6GHz, 4GB 2295MHz DDR3 RAM memory. For the harmonic inpainting method, we set the tolerance = 10 −5 . This value will balance the accuracy and execution time.

Image quality assessment metrics
To compare the inpainting result of the proposed method to other similar inpainting methods, it is necessary to assess image quality after inpainting based on the error metrics. The popular error metrics are PSNR and SSIM that were used in many works [20,21,22,23,24,25,26,27]: = 10 log 10 (   2   ) , is the mean squared error with 0 is the original (latent) image, denotes the maximum value, for e.g. for 8bit images = 255; ( ) and 0 ( ) are pixels values of and 0 at every pixel ( , ). The difference of 0.5dB is visible. The higher PSNR (measured in decibels -dB), the better image quality. Structural similarity (SSIM) is proven to be a better error metric for comparing the image quality and it is in the range [0, 1] with a value closer to one indicating better structure preservation. This metric based on the characteristic of the human vision. The SSIM is computed between two windows and of common size × ,

Image datasets and test cases
We test the performance of the proposed inpainting method on a dataset of natural images with artificial corrupted masks. The dataset is well-known as BSDS of UC All images are stored in JPEG format, grayscale and have size 481×321 pixels. Figure 3 shows the selected image of UC Berkeley for the tests. The corrupted masks are presented in Figure 4. In this case, the black areas indicate the corrupted regions. Note that, we use 10 original images and 2 masks. Hence, our experiment will be handled on 20 images. This size is suitable for non-learning-based methods without any trained data.   In the case of the mask 1, the corrupted ratio is 5%. The size of the corrupted parts (white square) is 14×14 pixels. The inpainting results are presented in Figure 5. The values of the PSNR metric are presented in table 1. The value of the SSIM metric is shown in table 2. We can see that the inpainting result by the harmonic method is good by the PSNR metric, but it fails for SSIM. The SSIM of the harmonic inpainting method is smaller than one of the corrupted images. However, in Figure 5, the quality of the inpainting result by the harmonic is really better than the corrupted image. The harmonic method changes whole the image (including the uncorrupted regions). The restored image gets smoother. This is the reason to make SSIM metric of the harmonic inpainting method to be lower. The inpainting result by the directional median filters is better, but there are still some defects: flowers on the dress of woman (ID 157055), on leaves and branches of the tree (ID 295087).   For our proposed method, the inpainting result is very good. Our method only recovers the value of the corrupted pixels. Because the corrupted area is not large, our proposed method gives perfect inpainting result. By both PSNR and SSIM metrics, our proposed method is better than the harmonic method and directional median filters. In the case of the mask 2, the corrupted ratio is 10%. The lengths of some corrupted regions are very long. The corrupted areas, in this case, are bigger than one of the cases of the mask 1. The inpainting results are presented in Figure 6. The PSNR and SSIM metrics are presented in Table 3 and Table 4, respectively. The harmonic method still made the whole image to be smoother. The inpainting result by the directional median filters looks better. The inpainting result of our proposed method is best. Our proposed method preserves the structure of the image because it did not change the pixels' values outside the corrupted regions. Both PSNR and SSIM metrics of the proposed method, in this case, are also better than ones of the harmonic method and the directional median filters.

IDs
From the above experiments, we can see that the proposed method can work well on various masks with small and medium corrupted regions. For the large corrupted regions, our proposed method also works well. However, if the corrupted area is too large, our method will work ineffectively, because the proposed inpainting method uses the clipping windows to evaluate. If a lot of pixels in the clipping windows are corrupted, the recovery process cannot get high accuracy. Figure 7 shows the inpainting results of the proposed method on the large corrupted regions (the corrupted ratio is 42%, the corrupted sizes are 30 pixels by width and 30 pixels by height). As can be seen, our proposed method creates a "paintbrush" effect. By human eyes, the inpainting result is good, and the values of PSNR and SSIM metrics of the inpainting result for this case by the proposed inpainting method are good enough too PSNR=21.9475, SSIM=0.7029.

Original (ID 157055)
Original (ID 295087) Corrupted (ID 157055) Corrupted (ID 295087) By harmonic method By harmonic method By directional median By directional median By proposed method By proposed method Figure 6: The inpainting results on the corrupted images with the mask 2.
We have to notice that, in real applications of image inpainting, the most important task is to detect the mask. The mask is not only the corrupted regions on images but also is objects that need to be removed. This task depends on every problem. It can be made manually or automatically. For example, in order to segment skin lesions of dermoscopic images [28], we need to detect hairs and remove them to improve image quality. The hairs can be detected by many methods, including machine learning methods, curvatures-based techniques, etc.
As we discussed above, our proposed method is a noniterative manner, and it only considers the clipping windows of the corrupted pixels, so it can work very fast. In all the tests, the execution time is under 1 second. The harmonic inpainting method spent up to 40 seconds with above tolerance 10 −5 . This is promising as the method is suitable to process large, high-resolution images or video sequences that are our future applications.

Original (ID 157055)
Original (ID 295087) Corrupted (ID 157055) Corrupted (ID 295087) By harmonic method By harmonic method By directional median By directional median By proposed method By proposed method Figure 5: The inpainting results on the corrupted image with the mask 1.

Conclusion
In this paper, we proposed an adaptive inpainting method based on the weighted mean. The proposed method can restore the images with small and medium corrupted regions effectively. It only recovers the corrupted pixels and do not touch other pixels. Hence, the image structure is unchanged. For the large corrupted areas, our proposed method works well, but it creates an artificial effect -the paintbrush effect.
The proposed inpainting method is non-iterative manner, so it can work very fast. Otherwise, the proposed method only considers the clipping windows of every corrupted pixels, so it requires less memory. These advantages are useful to process large, or high-resolution images or video sequences.
In future works, we would like to improve the proposed method to remove the paintbrush effect during restoring the images with large corrupted areas.