Automated Mobile ph Reader on a Camera Phone

Transcription

1 Automated Moble ph Reader on a Camera Phone B.Y. Loh, N.K. Vuong, S. Chan and C.. Lau AbstractA robust classfcaton algorthm that apples color scence and mage processng technques s developed to automatcally dentfy the ph level on a test strp. hs algorthm s mplemented on a camera phone that captures color mages of ph test strps for healthcare or medcal purpose. he pre-nstalled and platform ndependent program n the camera cell phone then processes the mages captured and s able to nform vsually-challenged users of the ph level of the strp. Expermental results show that ths new approach s more robust and effcent n handlng reflecton, skewed placements, as well as dfferent types of color reference. Index ermsph test, color mage processng, moble applcaton. I. INRODUCION ph tests are commonly used n chemcal laboratores to measure the acdc or alkal levels of solutons. ph levels typcally range from 1 to 14 wth 7 beng neutral; 1 s hghly acdc and 14 s hghly alkalne. In medcal practce, ph tests are also used as a health ndcator for human bengs due to the fact that the body fluds of a healthy person should nether be too alkalne nor acdc [1]. For nstance, the ph of a healthy body s blood, salva and spnal flud s at 7.4. Other readngs from 7.0 to 7.5 ndcate that the person s non-defcent and healthy whle a readng from 4.5 to 6.5 ndcates calcum defcency of agng and lfestyle defects [2]. ph tests are also used for urnalyss. A hghly acdc urne ph can be due to dabetes, darrhea, dehydraton and other alments. Conversely, hghly alkalne urne may be sgns of chronc renal falure, urnary tract obstructon, respratory dsease and other dseases [3]. hus, montorng the ph level of body fluds such as salva or urne s a smple and effectve way to check for early ndcaton of degeneratve dseases whch are caused by unhealthy acdc or alkalne levels n the body. An ndvdual can then adjust hs or her det to mantan the body s ph at healthy levels, or seek medcal treatment f necessary. he equpment used to measure ph levels range from the smple ltmus papers to electronc ph meters. Ltmus papers are affordable, easy to use and could be easly purchased from varous retal outlets. herefore, t s wdely used by medcal professonals and consumers to determne the human s body ph level smply by dppng a ph test strp n the flud. he ph level s ndcated by the color change on the test strp. By vsually comparng the test strp s color aganst a color reference, the ph level of the flud under test s known. For example, orange s level 3, green s level 7 Manuscrpt receved July 08, 2011; revsed July 15, B.Y. Loh, N.K. Vuong, S. Chan and C.. Lau are wth the School of Computer Engneerng, Nanyang echnologcal Unversty, Sngapore. (phone: ; fax: ; e-mal: {vuon0004, asschan, asctlau@ntu.edu.sg). and blue s level 10. However, some of colors n the reference color chart are very smlar to each other, whch may potentally cause problem n dentfyng the rght ph level usng just the human eyes. For ndvduals who fnd t dffcult or mpossble to compare the colors vsually, especally the elderly and color blnd, we propose a convenent soluton usng a camera phone - a devce that many people own and use n ther daly lfe nowadays. We have earler developed and presented a novel assstve technology system for measurng and classfyng ph levels from a dgtal camera phone mage [4]. Varous steps have snce been taken to enhance the robustness of the system [5,6]. Fg. 1 shows the overall structure of the system. An mage of the ph test strp and the color reference chart s frst captured by a moble camera phone. he mage processng unt (IPU) runnng on the J2ME enabled moble phone that supports the MMAPI extenson classfes the ph level by matchng the color of the ph test strp wth one of the colors on the reference chart. he result of the automatcally dentfed ph level s dsplayed on the phone. At the same tme, the ph result can be stored n the phone or transmtted to the doctors to seek further medcal attenton f necessary. est strp Image captured Moble ph Applcaton Image Process Unt Result Fgure 1. Structure of the proposed moble ph applcaton he moble ph applcaton ams to provde assstance to ndvduals who montor ther body fluds for medcal and healthcare reasons. hs tool s partcularly helpful for colorblnd patents and elderly people wth poor eye sght. It s also benefcal to those wth normal eyesght but have dffcultes n dstngushng colors that are smlar. hs automated applcaton may also be helpful to medcal professonals such as nurses or doctors who have to conduct and nterpret numerous tests daly. Addtonally, the proposed system has several advantages. Frstly, t does not ncur much cost snce t requres no specal hardware other than the camera phone tself. Moreover, t s easy to use wth no techncal sklls requred. hrdly, ths software soluton can be readly ntegrated nto a comprehensve moble healthcare system to provde holstc servce whch can be

2 customzed for ndvdual patents. Most mportantly, ths software tool enables patents to montor ther health regularly n the comfort of ther home thus savng tme and money that would otherwse be ncurred for medcal consultatons and laboratory vsts. II. PREVIOUS WORKS he ph test kt comes wth both rectangular and crcular color reference charts (Fg. 2 and 3). Earler works [4-6] appled color scence and mage processng algorthms to dentfy the correct ph level usng rectangular charts (Fg. 2). he orgnal approach we employed to process mages of the ph test strps [4] was based on edge detecton flters. Frst, we appled Sobel edge detecton to locate those pxels on the edges of the test strp and the color reference strps. Subsequently, smple scan-lne and thresholdng technques were used to determne the boundares of the test strp and the color targets. Eventually, we classfed the strp s ph level by measurng color dstances between the strp color and color targets. he color patch that had mnmum color dstance from the one of the test strp revealed the ph level of the test strp. Expermental results produced by the edge detecton based (EDB) approach were consstent wth the ground truth estmaton [4]. However, the EDB algorthm could not process mages whose test strps were skewed more than 7 angular degrees to the horzontal axs. Fgure 2. Sample est Image wth Rectangular Color Reference Chart We then developed an mproved approach known as CQB [5] based on Wu s color quantzaton [7]. It comprsed two steps. he frst step was to quantze the entre color set n the orgnal mage to two clusters n order to remove the background as well as some nose. Next, usng the pror knowledge of the number of color patches n the test mage, we agan quantzed the colors n the resdual mage to 11 clusters (see Fg. 2). he cluster wth the largest number of pxels was thus dentfed as that of the ph test strp. he computaton speed of ths approach was about 20 tmes faster than EDB. It was also able to handle test strps that were placed n skewed orentatons. However, ths method dd not work well f parts of the mage suffered from reflecton. Also, CQB dd not explctly dentfy the numercal ph level but reled on the user to vew the result from the resdual mage dsplayed. o overcome the above lmtatons, we propose a new approach (Sequence Based approach or SB) that can robustly handle mages n whch the test strps are placed n a skewed manner or mages that are partally affected by reflecton. Frst, we use the approach smlar to EDB to obtan the boundares of the test strp and the color reference strps. Next, we use K-means clusterng to separate the edge ponts nto three clusters. Usng pror knowledge of the relatve lengths of the color reference and test strps, the edge ponts are automatcally dentfed as three separate sets: one belongng to the test strp, one belongng to the longer color reference strp (ph1-6) and the last one belongng to the shorter color reference strp (ph7-11). hs step contrbutes a major mprovement over the prevous algorthm because ts functon s robust even when the strps are placed n a skewed manner. Next, the color reference strps are further parttoned nto 11 segments (ph1-11) usng the edge ponts obtaned earler. Wth pror knowledge of the colors n the color reference strps, a color lookup table s then used to recognze the color reference segments. hs s an mportant enhancement as t helps to recognze the color reference strps even when they are placed n arbtrary orders or orentatons. Moreover, by matchng aganst the color lookup table, even f one or two segments of the color reference strps are affected by reflecton or other llumnaton effects, the reference strps can stll be correctly dentfed and the color dstortons of the affected segments can be rectfed. hs greatly mproves the robustness of our algorthm. In the fnal step, the average color of the test strp s matched to one of the 11 color segments n the reference strp to automatcally dentfy the test strp s ph level. One of the advantages of the enhanced SB approach s t also can be used to process test mages wth crcular reference charts (Fg. 3). In fact, we have appled smlar processng technques of SB approach and successfully dentfed the ph level of mages that come wth crcularshaped reference chart. he method used comprses edge detecton technques accompaned by corner detecton methods to locate the postons of the color reference chart and the test strp. hereafter, post color processng and flterng are done to the regon of nterest to evaluate the ph value of the test strp. he processng technques for two categores of mages (rectangular and crcular reference charts) together wth ther expermental results wll be dscussed n the next sectons. Fgure 3. Crcular Color Reference Chart

3 III. IMAGE PROCESSING UNI he mage captured by the phone camera wll go through the sequence of processes as shown n Fg. 4. Edge Detecton he output of SED produces an mage that outlnes all the edges. A further processng returns a set of XY coordnates whch dentfes all the edge pxels n the mage. he results of mages wth rectangular and crcular reference charts after applyng SED are shown n Fg. 5. 5a. Rectangular reference chart K-means Clusterng Post K-means Clusterng Pont-n-Polygon est Converson to CIELab 5b. Crcular reference chart Determne Color Chart ph value Fnal Classfcaton of est Strp ph value Fgure 4. Sequence of operatons n the IPU A. Edge Detecton he IPU begns wth Sobel Edge Detecton (SED) [8] on the mage captured. SED s generally used for grayscale mages and modfcaton to the algorthm s requred for t to be used on color mages. he modfed process depcted n Algorthm 1 s obtaned from J2ME-EDB approach [4]. Input: Color mage, h and weght w where 0 w 1 for each color band Output: Bnary mage contanng edges for the nput mage G G y x1 1. G x G x , G y Gy G x3 Gy Z1 z11 z12 z13 2. for each 3x3 mage sub-area Z 2 z21 z22 z23 of the Z 3 z31 z32 z33 nput color mage: 3. for each color band : 4. compute: 5. and 6. compute: f G x f G y f x1( Z1 ) Gx2( Z2 ) Gx3( Z3 ) y1( Z1 ) Gy2( Z2 ) Gy3( Z3 ) f w x 2 f w y 7. f f h then mark the pxel z 22 as an edge pxel 8. else z 22 s a non-edge pxel 9. return bnary mage contanng only edge or non-edge pxels for the nput mage Algorthm 1. Pseudo-code for SED of color mages 2 B. K-Means Clusterng Fgure 5. Results after applyng SED K-means Clusterng s a cluster analyss method that separates a set of ponts nto k clusters or regons where each regon has a centrod. An nteger k representng the numbers of centrods s frst chosen. Next, the dstance of a pont to a centrod s calculated. Each pont has k dstances to k centrods. A regon conssts of all the ponts that are nearest to the cluster s centrod. At the end of ths frst teraton, all the ponts n a regon are used to calculate the new poston of the centrod. he process s repeated by computng the dstance of each pont to the new k centrods. he teraton stops when there s no change to the ponts n each regon as llustrated n Fg 6. Start Choose number of cluster k Determne Centrods Dstance of each pont to centrods Put ponts nto dfferent clusters Yes Change n cluster s members No Fgure 6. K-means clusterng algorthm End

4 In our applcaton, we set k = 3 regons for mages wth rectangular reference chart; namely est strp, Color reference ph 1-6, Color reference ph Usng k-means clusterng, we are able to determne the boundng areas of each strp n 3 teratons as shown n Fg 7. Centrods - ndcated by the red dots est strp Color ref ph 1-6 Color ref ph ) Images wth Crcular Reference Color Chart Smlarly, the 4 corners of the rectangular test strp are determned by computng the dstance of each edge pont to the 4 corners of the mage. After separatng the test strp and the crcular color chart nto two regons of nterest, we dvde the annulus regon between the two concentrc crcles (see Fg. 5b) nto 11 regons to segregate each ph level nto ts ndvdual regon shown n Fg. 9. Each ph regon s bounded by ts 4 corners of the approxmate trapezums, and ther coordnates are stored for use n future steps. he 4 corner coordnates of the test strp are also stored n ths case. Fgure 7. Complete clusterng n 3 teratons We realze that applyng K-means clusterng on mages wth crcular reference chart s not necessary as such mages contan only the test strp and the color chart. After usng SED, we can determne the four corners of the test strp by usng corner detecton technque, whch s presented n the next processng step. hose pxels whch belong to the color chart can be deduced after separatng the rectangular test strp. C. Post K- Means Clusterng 1) Images wth Rectangular Reference Color Chart After separaton of all ponts nto 3 regons, the 4 corners of each rectangular strp are determned by computng the dstance of each edge pont to the 4 corners of the mage. he edge pont nearest to the respectve mage corner s labeled as a corner and the bounded area of the strp s then marked out as shown n Fg. 8. he strp wth the longest length s the test strp, followed by the ph 1-6 reference strp and the shortest length s the ph 7-11 reference strp. Next, the ph 1-6 reference strp s dvded nto 6 regons and the ph 7-11 reference strp s dvded nto 5 regons to segregate each ph level nto ts ndvdual regon. Each ph regon s bounded by ts 4 corners, and ther coordnates are stored for use n future steps. he 4 corner coordnates of the test strp are also stored. Fgure 8. Bounded regons for mages wth rectangular reference chart Fgure 9. Bounded regons for mage wth crcular reference chart D. Pont-n-Polygon est and Converson to CIELab Edge Detecton Images from most dgtal cameras use RGB encodng and a 3x3 lnear color transformaton s performed to map all the pxels n RGB color space P to CIELab reference values M usng equaton (1). M * A L 11 M * a A21 M * b A31 A12 A22 A32 A13 Pred A23 Pgreen A33 Pblue hs step s performed because the dfference between any two colors n Lab format can be approxmated by treatng each color as a pont n a three dmensonal space (wth three components: L, a, b) and takng the Eucldean dstance between them [9]. Each ph regon n both categores of mages, rectangular and crcular reference charts, conssts of pxels whch are bounded by the 4 corners whose coordnates have earler been determned. A pont-n-polygon check s performed to determne f the pxel s wthn the ntended regon. he applcaton converts all pxels wthn each regon nto Lab color space. Pxels that fall outsde the regons are dscarded. he entre procedure s performed for all the 12 regons and n addton, a two-pass Lab converson procedure avalable n our prevous verson of the applcaton [4] s used to further elmnate nose and the text n the test strp and color reference strps. (1)

5 E. Determnaton of Color Chart ph Value 1) Images wth Rectangular Reference Color Chart It mght be noted that the poston of the 3 strps n ths type of mages can be placed n any order. However, t s requred that the applcaton has knowledge of the regon type and the orentaton of the color reference chart. hs allows correct dentfcaton of the ph value of the respectve regon after the test strp has been matched to one of the ph regons. Each ph regon s matched to a pre-defned average Lab color as shown n able 1. he average Lab values for the 11 regons are computed based on mages obtaned from 3 dfferent moble phones, each wth 7 samples taken. able 1. Average Lab values ph Regon L a b he procedure begns by computng the root mean square (RMS) Eucldean dstances between ph 1 regon n the frst color reference chart (ph1-6 strp) and the pre-defned average color values of ph1-6. hs procedure s repeated for the other fve regons n the frst color reference chart. Snce ph1 and ph6 have dverse color values, we are able to determne whch extreme end of the frst color chart belongs to ph1 or ph6 based on the RMS result. Wth ths nformaton, the system wll also know whch regon belongs to ph2, ph3, ph4 and ph5. he same procedure s repeated for the second color reference chart (ph7-11) as llustrated n Algorthm 2. In addton to determnng the ph value of each regon n the color reference chart, the 11 average Lab values n the color table are used to resolve wrong ph value classfcaton when a part of the captured mage has suffered reflecton. If the ndvdual ph regon Lab value dffers by more than 20% from the correspondng Lab value n the average color table, the ndvdual ph regon Lab value s replaced wth that n the average color table. In cases when the mages are affected by reflecton or llumnaton problem, ths process provdes a more accurate Lab values for the fnal ph matchng and classfcaton. Input: ph1-6 color ref chart coordnates (R1 where =1 to 6), ph7-11 color ref chart coordnates (R2 where =7 to 11), Orgnal Lab Matrx Vector (L where =1 to 11), Color Lookup able Matrx Vector (C j where j=1 to 11) Output: Sorted Lab Matrx Vector 1. for each Lab Matrx Vector, L where =1 to 6 2. compute: rmscol1[][j] = RMS of L to C j where j=1 to 6 3. set mn1 = rmscol1[1][1] and mnindex1=1 4. f mn of rmscol[][1] < mn1 then mn1 = rmscol[][1], mnindex1=; =1 to 6 5. set mn2 = rmscol1[1][1] and mnindex2=1 6. f mn of rmscol[][1] < mn2 then mn2 = rmscol[][1], mnindex2=; =1 to 6 7. f mnindex2=6 or (mnindex1 >= 4 AND mnindex2 <= 3) then swop the Lab Matrx for L and color reference chart coordnates R1 where =1 to end for 9. repeat 1 to 6 where =7 to f mnindex2=11 or (mnindex1 >= 9 AND mnindex2 <= 9) then swop the Lab Matrx for L and color reference chart coordnates R2 where =7 to end for 12. return Sorted Lab Matrx Vector L Algorthm 2. Pseudo-code for computaton of ph regon value for mages wth rectangular color chart 2) Images wth Crcular Reference Color Chart As the crcular reference chart can be placed n any drecton, t s mportant to determne whch regon belongs to the correspondng ph level after segregatng the reference chart nto 11 regons of nterest. Smlarly, we also use the pre-defned lookup table of the average Lab colors n order to determne the ph level of each regon. In the prevous step, each regon s average Lab values are stored n sequental order n an array. For each unknown regon n the ph chart, the average Lab value s used to match aganst the most smlar ph n the lookup table, based on the closest Eucldean dstance calculated. For each match, the dfference between the poston n the ph array and the poston of the matched ph n the lookup table s then calculated. he dfference that has the hghest frequency wll determne the number of rotaton shft the ph array has to be re-arranged. he procedure s llustrated n Algorthm 3. Input: ph1-11 color ref chart coordnates (R where =1 to 11), Orgnal Lab Matrx Vector (L where =1 to 11), Color Lookup able Matrx Vector (C j where j=1 to 11) Output: Sorted Lab Matrx Vector 1. for each poston n the pharray, where = 1 to compute the DstanceLAB[][j]to all the ph n the LookUpable j, where j=1 to for each poston n DstanceLAB[], fnd the mnmum DstanceLAB[][j] 4. set CorrespondngpH[]=j 5. for each CorrespondngpH[], dfference x = (-j) where x=1 to f dfference x <0, dfference x = dfference x +11 (clockwse shft) 7. for each unque value dfference x, fnd the frequency that t occurs 8. set dfference x that as the hghest fequency to be an nterger K 9. for Count=1 to 11 and newarray z where z=1 to shft=count+k 11. f shft<11, shft=shft newarray[z]=imagearray[shft] Algorthm 3. Pseudo-code for computaton of ph regon value for mages wth crcular color chart F. Fnal ph Classfcaton Wth the enhanced Lab values of the 11 ph color chart regons and the knowledge of each regon ph value and ts coordnates, the fnal step s to compare the test strp wth the 11 ph regons. he RMS Eucldean dstances between the test strp Lab values and the 11 ph regons Lab values are compared. he regon wth the smallest RMS ndcates that t s the best match to the test strp and s consdered as the fnal ph classfcaton. Fg. 10 and 11 show the results

6 wth Lab values n yellow. he classfcatons ndcate that the test strps have ph level of 3 and 7 respectvely. vsually by the respondents n able 2. he ph classfcaton results acheved by our applcaton are consstent wth those ndcated by the respondents. able 2. Expermental results of mages wth rectangular reference chart Soluton Index Respondent Respondent Respondent Respondent Respondent Applcaton Classfcaton Images wth reflecton and slanted strp placements as shown n Fg. 12 have been tested and the results ndcate that the ph values of the test strps are correctly classfed by our applcaton. Fgure 10. Fnal classfcaton of ph=3 Fgure 12. Images wth reflecton (left) and slanted strp placements B. Images wth crcular reference color chart Seven test solutons were tested each usng 3 dfferent ph strps [10]. able 3 shows the lst of solutons used n the experment. Fgure 11. Fnal classfcaton of ph = 7 IV. RESULS AND DISCUSSION A seres of experments were conducted to evaluate the effectveness of the applcaton. A. Images wth rectangular reference color chart Eght test solutons were ndvdually tested wth 5 dfferent ph test strps [10] and 5 respondents were tasked to vsually match the test strp to the ph value on the color reference chart. Each respondent vewed a dfferent test strp tested on each of the 8 solutons and ther responses are collated n able 2. he same ph strps vewed by the respondents were captured by the phone camera and classfed usng the applcaton. Each soluton has 5 sample mages for evaluaton. For evaluaton of our ph classfcaton applcaton, we compare the results aganst those obtaned able 3. Lst of solutons used n the experment Index Soluton 1 OPI-FREE Contact Lens Soluton 2 Domestc Bleach 3 Mult-Purpose Cleaner 4 Vnegar 5 Nal Polsh Remover 6 Persl Washng Detergent 7 Freshly Squeezed Lme he ph classfcaton was performed on 2 smart phones. 3 respondents were nvted to vsually dentfy the ph of the soluton on the test strp. he results are shown n able 4.

7 able 4. Experment results for mages wth crcular reference chart Soluton Index Respondent A Respondent B Respondent C Applcaton Classfcaton by Phone 3GS [8] I.E. Sobel, Camera models and machne percepton, Ph.D. dssertaton, Stanford Unversty, Stanford, Calf, USA, 1970 [9] M. kalcc, and J.F. asc, Color spaces perceptual, hstorcal and applcatonal background, EUROCON 2003 [10] Johnson est Paper, Applcaton Classfcaton by Noka As we can see from the values n able 4, the results yelded from the predcted ph by the applcaton are close to the results vsble to the human eyes, wth a devaton of 1 ph value. Certan colors, especally those n the ph range 6-7, can be rather dffcult to classfy even by the human eyes. Also, envronmental condtons such as humdty may contrbute to the change n colors of the test strp and make t drop or ncrease by one ph level after a short perod of tme. In general, we conclude that the applcaton predcts an accurate and relable ph. V. CONCLUSION hs paper has presented a robust soluton to the problem of moble ph classfcaton. We have overcome the major lmtatons n two earler approaches. hs new algorthm s able to handle test strps or color reference strps that are placed n a skewed manner or arbtrary order. It can also automatcally dentfy the correct ph level even when a part of the mage s affected by reflecton. Addtonally, the applcaton s also ntegrated and customzed to operate on dfferent types of ph test color reference charts and produce results accurate and consstent wth the ground truth estmaton. hs s one major step closer to accomplshng an effcent, robust, low cost, accurate, and ntellgent moble ph reader that s of great use to the elderly or colorblnd people. REFERENCES [1] J.A. Smervlle, W.C. Maxted, and J.J. Pahra, Urnalyss: a Comprehensve Revew, Amercan Famly Physcan, vol. 71(6), pp , 2005 [2] Salva ph est. (N.D.). Retreved from Alkalze For Health: [3] Rnceus. Urne ph. Retreved March 2011[Onlne] [4] N.K. Vuong, S. Chan, C.. Lau, Classfcaton of ph Levels Usng a Camera Phone, he 13 th IEEE Internatonal Symposum on Consumer Electroncs, 2009 [5] N.K. Vuong, S. Chan, C.. Lau, ph Levels Classfcaton by Color Quantzaton on a Camera Phone, Internatonal Conference on Communcatons and Moble Computng, 2010 [6] B.Y. Loh, N.K. Vuong, S. Chan, C.. Lau, Robust Classfcaton of ph Levels on a Camera Phone, Lecture Notes n Engneerng and Computer Scence: Proceedngs of he Internatonal MultConference of Engneers and Computer Scentsts 2011, IMECS 2011, March, 2011, Hong Kong, pp [7] X. Wu, Color Quantzaton by Dynamc Programmng and Prncpal Analyss, ACM ransactons on Graphcs, vol. 11(4), pp , 1992