Backlit Keyboard Inspection Using Machine Vision

Der-Baau Perng, Hsiao-Wei Liu, and Po-An Chen

1. Introduction

3C (computer/communication/consumer) products have widely been used in the information booming era. The keyboard still is an inevitable accessory for the computer hobbyist. The backlit keyboard has been presented to the users to bring them more convenience and efficiency. The backlit keyboard uses different color LED as illuminating light source to have the keys be seen easily in poor light or dark environment. Users can be divided into four groups that are gamers, explorers, optimizers, and balancers. The modern backlit keyboard could be more attractive to these groups of users, especially the gamers[1].

As shown in Fig. 1, the laser marking keyboard with backlit can be categorized into three main regions of function keys, normal keys, and number keys. These backlit keyboards were done through manufacturing and assembly processes.

Fig. 1. Backlit keyboard with keys processed by the laser marking.

However, some types of defect might happen on the backlit keyboard, such as incorrect illuminating area,incorrect luminance, incorrect intensity, and non-uniformity.Fig. 2 illustrates a fragment of backlit keyboard, including function keys and normal keys.

The “Esc” key misses the character “c”, as indicated by the circle of Fig. 2, which forms an incorrect illuminating area. Such incorrect illuminating area defect might be caused by a laser marking error. The second type of incorrect luminance and intensity defect might happen to any key. The intensity of each key has been pre-defined by manufacturers and may appear in red, green, blue, or white.Once the intensity of a key in the inspection region (IR) is beyond the pre-defined specification, it is identified as the incorrect intensity defect. The final defect of nonuniformity is focused on some zone or area of the backlit keyboard. In the region of function keys of Fig. 1, the intensity of “F5” is lower than other function keys which might be the less uniformity of function keys region.

Fig. 2. Illustration of the defect in the fragment of backlit keyboard image.

Traditionally, the above mentioned types of defect were checked by human inspector. In mass production, human inspector usually cannot keep consistent inspection results.Besides, not all human inspectors were keen on the intensity variation of the keys. In this paper, an automated system for backlit keyboard inspection is proposed and implemented.The rest of this paper is organized as follows. The related survey of image processing is discussed in Section 2. The research methodology of backlit keyboard inspection is presented in Section 3. The experiments and discussions are demonstrated in Section 4. Finally, the conclusions and further researches are given in Section 5.

2. Related Surveys

Red, green, and blue components can be represented by the brightness values that are obtained through three separate filters, red, green, and blue filters, given by

where FR, FG, and FBare the color filters on radiance E(λ)of the incoming light, and λ is the wavelength. The RGB color space can be represented in the 3D cube, as shown in Fig. 3[2]. Each color or point in the cube is represented by corresponding red, green, and blue values.

Fig. 3. RGB color space represented in the three dimensional cube[2].

The connected component labeling[3]is executed based on the pixel connectivity of four neighbors, as shown in Fig. 4.

Fig. 4. Component labeling: (a) binary image and (b) connected components labeling.

After executing the connected components labeling, we can select the interested component and calculate its contour. Based on the contour’s coordinates [xi, yi] where i belongs to contour, the extremely points can be found. One useful bounding box is a rectangle with horizontal and vertical sides. These sides touch the topmost, bottommost,leftmost, and rightmost points of the connected components,as shown in Fig. 5. Let xminand xmax, respectively, be the leftmost and rightmost point x coordinate; and let yminand ymax, respectively, be the upmost and bottommost point y coordinate, Then the minimal bounding box can be generated by these points of (xmin, ymax) and (xmax, ymax)[4].

Fig. 5. Illustration of the minimal bounding box algorithm.

The operation of binary morphology is useful in processing the regions derived from the binary image. The basic idea of morphology is to probe an image with a structuring element and to quantify the manner in which the structuring element fits within the image.

Let S be a structuring element, the convolution kernel represented by a binary image, and B be a binary image.The shape and size of the structuring element are needed to be decided to grow or shrink the region in image B. The operations of morphology including dilation, erosion,opening, and closing are as follows[3]-[5]:

3. Backlit Keyboard Inspection System

3.1 Structure of the Proposed Backlit Keyboard Inspection System

Fig. 6 demonstrates the hardware prototype of the proposed vision inspection system for the backlit keyboard.This prototype consists of two components: a host computer and an image capturing system that is with a webcam in a black chamber.

The software system of the proposed backlit keyboard inspection system is shown in Fig. 7. The inspection system is composed of an engineer module and an operator module.Based on the engineer module, a supervised engineer needs to pre-train the parameters of inspection and set proper inspection specification. As to the operator module, the to-be-inspected backlit keyboard is loaded and unloaded for inspection without setting any parameter when the in-line inspection is executed.

Fig. 6. Hardware prototype of backlit keyboard vision inspection system.

The proposed software system was programmed in Visual Studio 2008 C++ and linked with the Open CV 1.0[6].The execution system was implemented via the Universal Testing System (UTS) of the company C-FER. Critical processes of the proposed inspection method are described in the followings.

Fig. 7. Flowchart of the backlit keyboard inspection system.

3.2 Training Process of the Backlit Keyboard Inspection System

A. Specify Inspection Region

The image of a golden backlit keyboard was first captured by a webcam, as shown in Fig. 8 (a). Then a supervised engineer selected the IRs manually. As shown in Fig. 8 (b), four IRs corresponding to Fig. 8 (a) were generated by a supervised engineer. Based on the specified region, the uniformity of that region could be calculated accordingly.

Fig. 8. Golden backlit keyboard: (a) the image and (b) five IRs of(a) specified manually.

B. Image Segmentation

The binary image was obtained by applying the Otsu threshold method[7]on the golden image. Then, the opening operation, as given in (7), was applied on the binary image for noise removal. Next, the operation of dilation, by using(4), was applied iteratively on the opening image to obtain the approximate region of all the keys. We labeled and calculated the number of connected components after each dilation operation. Since the total number of keys of the to-be-inspected backlit keyboard was known in advance,we could determine the number of iterations in accordance with the number of obtained connected components.

Obtaining the labeled components, we used the minimal bounding box algorithm to get the contour of each labeled component. Fig. 9 shows the obtained result of total one-hundred and twenty regions of the labeled components in Fig. 8 (a).

Fig. 9. Bonding rectangle and connected component labeling.

Then each key, corresponding to a region, can be segmented. Some segmented keys of “N”, “C”, “T”, and“U” are demonstrated in Fig. 10.

Fig. 10. Segmented keys of “N”, “C”, “T”, and “U” from the bounding box.

C. Feature Extraction

Based on the segmented keys, the features of each key could be extracted. Some features are (a) the area of each key, (b) the mean intensity of red, green, and blue colors of each key, and (c) the uniformity of some inspection region(IR) that is to be calculated first.

Let K(i,j)be the ith segmented key on the jth IR from the golden image. The binary key could be obtained by using the threshold method:

where (a, b) represents the x and y coordinates. The sample binary regions, corresponding to Fig. 10, are obtained by setting T=42, as shown in Fig. 11. The area of the ith segmented key,αi, can be calculated by

Fig. 11. Four binary key images obtained by thresholding the four sample keys of Fig. 10.

Based on the area value αiof the ith key, we can calculate the average intensity of red, green, and blue of each key by using (10) to (12). Letr(i,j),g(i,j), and b(i,j),respectively, be the mean intensity of red, green, and blue of the ith segmented key on the jth IR. Then the luminance[2]r(i,j),g(i,j),b(i,j), and y(i,j)can be calculated by using the linear combination, given by

The final feature to be calculated is the uniformity on the jth IR. The minimum luminanceand maximum luminanceof the jth IR on the region are calculated by

Then we obtain the uniformity βjof the jth IR by

算下来，这一年里，他到全球各主要产区考察参观，单是中国酒庄就跑了七八十家；他在全国各地举办葡萄酒课程，至于具体多少场，他说自己都数不清了。平均下来，每个月他只有一周时间可以待在家里，对于一个新手爸爸来说，这是一种折磨。

where βj∈[0,1]. The more the value of βjis close to 1,the more uniform the jth IR is. On the contrary, the more the value ofβjis close to 0, the more non-uniform the jth IR is.

Once the values ofαi,y(i,j),r(i,j),g(i,j),b(i,j),andβjof the ith key on the jth IR are obtained from the golden backlit image, a supervisor has to set related specification of these features and to save the coordinates of IR and the bounding box of every key accordingly.

3.3 Testing Process of the Backlit Keyboard Inspection System

In the testing phase, the training parameters should first be loaded by the UTS. Then the operator can execute the in-line inspection after a to-be-inspected backlit keyboard was loaded. Then the system captured the image of the to-be-inspected backlit keyboard automatically via a webcam to get each of the keys according to the trained coordinates of the bounding boxes.

Next, the features of α′i,y′(i,j),r′(i,j),g′(i,j),b′(i,j),andβ′jof each key on the to-be-inspected image were calculated. Then, the following steps were executed:

A. Area Inspection

B. Illumination or Mean Intensity Inspection

C. Uniformity Inspection

4. Experiments and Discussions

In Section 4.1, the preliminary experiments are described to evaluate the impact of the iterative dilation process. In Section 4.2, 228 backlit keyboard images are used to demonstrate the effectiveness and robustness of the proposed inspection algorithm. The backlit keyboard of“US” type with 123 keys is used to evaluate the proposed method.

4.1 Impact of the Iteration Number of Dilation

The relationship of the number of iteration and the labeled components is shown in Fig. 12. When the iteration number n is nine, the number of labeled components is mostly close to 123 keys. As a result, n is set to nine as for a golden backlit keyboard image.

Fig. 12. Relationship of the number of iteration and labeled components.

4.2 In-Line Experimental Results

Five IRs selected for inspection is shown in Fig. 8 (b).They are M keys region, LED region, left keys region,center keys region, and right keys region, which are indexed as ROI(insp)-i, and i ∈{1, 5}, as in Fig. 8 (b).

We used 222 blue backlit keyboards to evaluate the proposed method. The inspection result is given in Table 1.

Table 1. Result of backlit keyboard inspection

The yield rate is up to 98.2%. In average, it takes under 50 seconds to complete all the inspection items for each backlit keyboard.

Four NG cases can be found in the M keys region and in the LED region. Fig. 13 (a), Fig. 13 (b), and Fig 13 (c)show the NG area and NG intensity of the 4th-, 9th-, and 3rd-key, respectively. Such three NG cases are recognized as LED failure. In addition, due to the area of the 9th-key is beyond the specification, it is classified as failure even though its intensity is correct.

Fig. 13. Four NG cases: (a), (b), and (c) the NG area and NG intensity of the 4th-, 9th-, and 3rd-key, respectively, and (d) the NG area of the 9th-key.

As to the result of uniformity inspection, each IR of 222 backlit keyboards was detected. The line chart of the uniformity to the five IRs is shown in Fig. 14, in which the 2nd-IR of LED region has the highest uniformity. The uniformity of the remained region is roughly within 50% to 70%.

Fig. 14. Line chart of the uniformity to the five IRs.

Table 2 lists the average and deviation of the uniformity of the five IRs. The 5th-IR has the largest deviation value,representing that the keys in the right region illuminate less uniformly than the other four IRs.

Table 2: Uniformity for five IRs

5. Conclusions and Further Researches

A machine vision system for backlit keyboard inspection has been proposed and implemented. The features of the backlit keyboard to be inspected in this paper include the illuminating area, illumination uniformity of specified IR, and specified luminance and intensity of individual key. The inspecting time for each backlit keyboard is under 50 seconds. The yield rate is up to 98.2%.The experimental results reveal that the proposed auto-inspection system for backlit keyboard is robust and useful.

Further works may be focused on (a) inspecting the exposure backlit keyboard and (b) implementing the proposed inspection system to various features of backlit products.

Acknowledgment

This research was supported by the Logitech Far East Ltd.,Hsinchu. We would like to thank particularly the Asia FE Engineering Group of Logitech Far East Ltd. for providing valuable samples and experiences.

[1] D.-B. Perng, P.-A. Chen, and H.-W. Liu, “Backlit mouse defect inspection using machine vision,” in Proc. of 2011 Int.Conf. on Multimedia Technology, Hangzhou, 2011, pp.500-503.

[2] H.-D. Cheng, X.-H. Jiang, Y. Sun, and J.-L. Wang, “Color image segmentation: Advances and prospects,” Pattern Recognition, vol. 34, pp. 2259-2281, Dec. 2001.

[3] L. G. Shapiro and G. C. Stockman, Computer Vision, Upper Saddle River, USA: Prentice Hall, 1992, ch. 3, pp. 69-75.

[4] M. Moganti, F. Ercal, C. H. Dagli, and S. Tsunnekawa,“Automatic PCB inspection algorithms: A survey,” Computer Vision and Image Understanding, vol. 63, no. 2, pp. 287–313,1996.

[5] D.-B. Perng, S.-H. Chen, and Y.-S. Chang, “A novel internal thread defect auto-inspection system,” Int. Journal of Advanced Manufacturing Technology, vol. 47, no. 5–8, pp.731-743, 2010.

[6] G. Bradski and A. Keahler, Learning OpenCV: Computer Vision with the OpenCV Library, 1st ed. O’Reilly Media,USA: Sebastopol, 2008.

[7] N. Otsu, “A threshold selection method from gray-level histogram,” IEEE Trans. on Systems, Man and Cybernetics,vol. 9, no. 1, pp. 62-66, 1979.