I.J. Information Tehnolog and Computer Siene, 2011, 4, 29-35 Published Online August 2011 in MECS http://.mes-press.org/) Researh on Virtual Vehile Driven ith Vision Youquan Liu and Yu Cui Shool of Information Engineering, Chang an Universit, Xi an, China Email: ouquan@hd.edu.n, uiu08@126.om Enhua Wu Institute of Softare, Chinese Aadem of Sienes; Fault of Siene and Tehnolog, Universit of Maau, China Email: ehu@uma.mo Abstrat This paper presents a traffi-oriented modeling and simulation frameork. With this tool, users an reate a traffi sene quikl and easil, and then drive the vehile in the reated sene ith phsis simulated. Espeiall, the vehile ould be driven ith vision tehnolog, hih is equipped ith a ebam and a paper printed ith ontrolling markers. And the paper is used as the virtual heel to ontrol the motion of the vehile. At suh lo ost, man people an enjo the driving fun; it also provides an eas and interesting a for driving video games ith promising business values. Inde Terms-amera; traffi modeling; vision interation; vehile driving; simulator I. INTRODUCTION As one of the most important virtual realit environments, driving simulator provides users a useful tool for training and digital entertainment. With suh kind of sstems, people ould attain driving eperiene safel at lo ost or get fun easil. And it is also apable to evaluate the drivers performane. The National Advaned Driving Simulator at the Universit of Ioa and the Toota one are among the largest ground vehile driving simulator in the orld[1]. Hoever beause these industrial simulators are equipped ith omple eletro iruit sstem, mehanial and fluid drive sstems, the are epensive and onl available to professionals. For ordinar people, the a to get driving fun is to pla video games, like 3D Driving-Shool[2], Need for Speed[3], Euro Truk Simulator[4] et. And often keboard and mouse are the interation devies; further jostiks like simulation steering heel are used to ahieve more realisti driving eperiene. In traffi area, amera is often used as video apture devie, for eample, Ono et al. [5] used amera to apture the real sene and ombine the real image ith the virtual geometr data to give a more realisti rendering for driving simulation. Beside apturing video, amera sometimes is also used as an interation a in digital entertainment area, suh as Eeto[6] b Son, Kinet[7] b Mirosoft, gesture interation. It is seldom used to ontrol the motion of a omple virtual objet, suh as virtual vehile. The losest ork to ours is that a game from GoArm ebsite[8], it provides a driving game hih is ontrolled b ebam. Hoever it just provides limited steering onl. In this paper e present a frameork to model the traffi sene and drive the vehile. With suh a tool a sene ould be set up easil ith phsis dnami properties. And a mied realit based virtual driving method ith amera ontrolling is illustrated, hih gives an innovative interation a to ontrol the vehile in the omputer-generated environment. Instead of other epensive interation devies, e are using the heap ebam as the interation devie, hih is also almost available in ever omputer sstem. Beside the vision ontrolled dnami simulation of driving, a traffioriented visual modeler is also presented. With this tool, a it traffi sene an be onstruted quikl. With a printed paper visible in front of ebam, e reonstrut the spatial oordinates of the marker loated from the aptured 2D image, and then alulate the angle hange from the frameork hih is used as the steering; and some other markers used to ontrol the braking, baking, honking operations. Suh kind of a provides users an interesting interation mode for vehile driving, hih is heap and available to man users. And there is not muh limitation to the position of the ebam, and the onl restrition is that the markers on the paper should be visible to the ebam. In the folloing setion the hole sstem ill be desribed firstl, and then traffi-oriented visual modeler and the vision interation ill be disussed in Setion 3 and Setion 4. The last to parts ill give our eperiment and disussion. II. SYSTEM DESIGN As a driving simulation frameork, our sstem onsists of to parts, the first one is the visual it traffi modeling tool, hih is used to interativel design the it traffi sene quikl. The seond part is the ebam interation tool to ontrol the driving. Copright 2011 MECS I.J. Information Tehnolog and Computer Siene, 2011, 4, 29-35
30 Researh on Virtual Vehile Driven ith Vision Figure Frameork of traffi oriented modeler sub-sstem Obviousl the modeling tool an be designed separatel to avoid developing ompleit. Fig.1 presents the sheme of our traffi oriented modeler sstem. With this modeler, e an reate a sene ith traffi information quikl. Different from other general 3D modeling tools like 3DS Ma, this tool is designed speiall for the purpose of it traffi. This part ill be disussed in detail in Setion 3. In Fig.2, the sheme of the ebam ontrolling subsstem is given. The interation devie is the ebam ith a ommon paper printed ith markers, as shon in Fig.10 the virtual vehile is ontrolled ith a paper. B rotating the paper under the amera, e an steer the diretion of the vehile. And b alternating the visibilities of some markers ith hands, e an do some pre-defined operations, suh as baking, honking, braking, and shifting. This ill be disussed in detail in Setion 4. Figure 3. Interfae of the Traffi Oriented Modeler With phsis from PhsX[9] integrated, the ne added omponents ill be plaed rationall ithout rossing or overlapping, reduing users burden to adjust their positions. And their phsis properties an be also set up onvenientl hih ill be used for dnami driving, suh as mass, frition et. So the reated sene is dnami ith vehiles moving. Figure 2. The ebam ontrolling sub-sstem sheme In both sub-sstems, to give the realisti motion information, e integrate the phsis effets into the sene so that ollisions ould be deteted and responded realistiall. III. TRAFFIC-ORIENTED VISUAL MODELER As desribed above, the traffi-oriented visual modeler provides users an eas a to reate a it traffi sene quikl. Fig.3 gives a sene designed ith our modeler. In Fig.3, its funtion modules are illustrated. With this tool, e an load the eternal models designed b others and put them together ith dnami properties. Figure 4. Funtion modules of the modeler To failitate the modeling of traffi sene, e also provide a speial pre-designed librar hih inludes man traffi signs, different rossas, plant sights, users onl need to lik and drag in most situations as shon in Fig.. Fig.4 gives one eample. Copright 2011 MECS I.J. Information Tehnolog and Computer Siene, 2011, 4, 29-35
Researh on Virtual Vehile Driven ith Vision 31 C 1 P 3 C 2 D C D C 1 D C 2 P 2 A A P 1 B 2 B B 1 A B 1 B 2 P 4 Figure 7. The initial points ith mouse input at A, B, C, D Figure 5. Pre-designed librar ith man omponents To avoid the problem mentioned above, the ontour tehnique is used to onstrut the geometr mesh of the road. To help users to manipulate the objets in onvenient a, several draggers are provided, suh as Plane Dragger hih onstrains the motion on the plane, Trakball Dragger hih ontrols the rotation of the objet, Translating Dragger hih translates the objet in 1D, 2D or 3D spae, Bo Dragger as the most omple dragger hih is used to sale or translate the objet. Fig. 6 illustrates that the stop sign model is adjusted to the proper size and orientation. Figure 8. The geometr generation for the road a d Figure 6. Eample of manipulating ith to draggers from a-b--d We an not reate a it model ithout road. So road geometr onstrution is neessar. Hoever manual road onstrution is tedious obviousl. In this paper e present an eas a to minimize the labor b just mouse liking at ke points. With simple liks, ke points of the road are defined as shon in Fig.7, like A, B, C, D. Obviousl, if e just etend the road idth along the normal diretion as in Fig.7, some parts ill get ollided like P 1, P 2, some parts ill break like P 3, P 4. b As shon in Fig.8, point A n-1,a n,a 1 are defined ith mouse liks. To onstrut the road surfae, the boundar points a n-1,1, a n-1,2 and a 1,1, a 1,2 are alulated firstl, and then the boundar points a n,1, a n,2 of A n is alulated. Beause line segment an 11, a is normal to n 1,2 line segment A n-1 A n,similarl, an 1,1a is normal to + 1,2 A n A 1. With the road idth, the oordinates of a, n 1,1 an, a 1,2 n +, 1,1 a n + an be alulated ith the folloing 1,2 equations. An. an 1, = + *. An 1. ) + An. 1. ) An. an 1, = *. An 1. ) + An. 1. ) A. A. a A a = A + + n n 1 n 1,2. = n *. An 1. ) + An. 1. ) An. n 1,2. n *. An 1. ) An. 1. ) A. A. a A a = A + + 1 n 1, = * A 1. An. ) + A 1. An. ) A An. 1, * A 1. An. ) A 1. An. ) A. A. a A A. A. a = A + 1 n 1,2. = + * A 1. An. ) + A 1. An. ) 1 n 1,2. * A 1. An. ) A 1. An. ) Copright 2011 MECS I.J. Information Tehnolog and Computer Siene, 2011, 4, 29-35
32 Researh on Virtual Vehile Driven ith Vision a, a n,2 these to orner points, speial To n,1 proessing is needed. The ommon tangents of irle A n-1 and A n, A n and A 1 are found to interset to get a n,1, n,2 a as shon in the folloing equations. π α an, = sin β α + + os 2 π α an, = os + β + α os 2 α π an,2. = sin β α + os 2 α π an,2. = os β + α os 2 here A An. An. α = artan artan A An. An. An. β = artan An. After these steps, the ke points of the road geometr are read. To onstrut the geometr mesh of the road, Delauna triangulation is used to onstrut the final mesh so that it an be integrated into phsis orld. Here just 2D equations are given to alulate the verties of the mesh, hoever it is straightforard to etend them to 3D spae. And teture mapping is used to enhane the realism. With different tetures and idths e an onstrut different lanes road as shon in Fig.9. Figure 9. Road onstruted ith simple liks. Figure 10. The user interfae of virtual driving 4.1 Frameork of oordinate sstem Sine ebam is used to apture the motion of the paper ith markers, there are three frameorks used, orld, amera, image spae. The relationship among these three frameorks is shon in Fig.1 With omputer vision knoledge, ebam is modeled ith pinhole assumption. So a point in 3D orld spae an be transformed into image spae ith the folloing equation: X u f 0 r11 r12 r13 t1 Y s v 0 f r21 r22 r23 t = 2 Z 1 0 0 1 r31 r32 r33 t 3 1 1) here X, Y, Z ) is the 3D orld spae point, uv, ) is its aording image point in piel metri. And f, f are the foal distane in piel metri,, ) is the base pointoften loated in the enter of the image). Equation 1 ould be ritten further as follos, X u = R Y + t and s v = A 2) z Z 1 z here A is the internal parameter matri, or amera matri hih used to desribe the internal parameters of the amera and independent of the sene. One it is alibrated, it ould be used repeatedl hen its foal distane fied. Transformation matri [ R t ] is the eternal parameters matri hih onsists of translation and rotation. It is used to desribe the rigid motion of the amera in the sene. IV. VECHILE DRIVEN WITH VISION The vision based interation is the ke part of our innovation. All interation information omes from the ebam. Therefore, an effetive reognition algorithm should be available firstl. In our sstem ARToolKit[10] is used as the vision apturing API. It loates the position of entral marker used to restore the rotation information serving as steering angle. With ARToolKit API the visibilities of surrounding markers are translated into ommands used to ontrol the vehile. The details ill be given belo. Figure 1 The relationship among the three oordinate frameork Copright 2011 MECS I.J. Information Tehnolog and Computer Siene, 2011, 4, 29-35
Researh on Virtual Vehile Driven ith Vision 33 With the help of ARToolKit API, e an apture the orld position of the markers. But to ontrol the steer of vehile, the orld position has to be transformed into angle information. Fig.6 presents the virtual heel used in our sstem, ertainl it is a ommon paper printed ith different marks, hih are labeled ith red irles. something else, its state ill hange from visible to invisible, that s, some aording operation ill be eeuted. The ontrol of speed is realized b the left number sub-markers. The number of visible number submarkers indiates the gears. Less visible sub-markers, less speed ill be. Bak, brake, honk these three operations are the simplest ases in our sstem, hih are assoiated ith marker R, B, S respetivel. If maker R is invisible in the amera, then the vehile ill stop and then bak up. If maker B is invisible, the vehile ill be braked to take a stop. S means a honk. Figure 12. Markers distribution The aptured information inludes the loation of the entral marker, the visibilit of other markers. The steering is ontrolled b the entral marker labeled ith Wheel, its loation relationship matri beteen orld frameork and image spae ould be ritten as, z p z p z z zz p z Figure 13. Rotation angle alulation With the folloing analsis e an separate the steering angle from this matri: 1)The first three olumns are the,, z ais vetors respetivel. 2) Sine the paper moves on the desktop, e an alulate the rotation of ais from dot produt of the ais of front frame and bak frame as shon in Fig.7. To get the eat rotation angle, e use the ais vetor of the first frame as the original referene 0. B alulating r f r, r r 0 b, the delta angle beteen the front frame 0 and bak frame is Δ a = aros r f r 0) aros r b r 0). V. SCENE DISPLAY AND PHYSICS SIMULATION Our displa is based on OpenSeneGraph[11], hih is used both in the it modeling and virtual driving. Fig.14 gives a simple demo sene reated b our modeling tool ith more than 5000 verties. Sine the sene data are independent of the driving simulator, our sstem an also inorporate other data like the Boston it model as shon in Fig.15, hih omes from [11] ith 0.6M verties. Beause the 3D sene an onl provide the driver the loal vie around the ab, to provide the global position information, 2D map is given hih is onsistent ith the 3D sene. Taking the Boston sene as an eample, the sstem integrates Google Map as the 2D map as shon in Fig.15. As a result, the driver not onl an ath the sene folloing the vehile, but also an loate the position in the hole it ith GIS information. As another onsideration, to provide the bak vie ameras, to more vies are used to displa the sene hih means the sene ill be rendered in to more passes. For realisti motion, PhsX is integrated into the frameork to provide phsial reasonable ollision detetion and response, suh as the effets beteen vehile and environment, vehile and vehile, the vehile s shaking, et al. VI. EXPERIMENTS AND DISCUSSION The hole sstem is developed ith Visual Studio 2005, C++. The sene is displaed ith OpenSeneGraph, phsis effets ith PhsX, UI ith Qt for the modeler, and 2D map ith Google map API b HTML + JavaSript. The mahine is equipped ith Intel Quad CPU Q8300@ 2.50GHz and NVIDIA GeFore 9800 GT graphis ard. As to the Boston sene, the number of triangles is more than 0.6M ith three vies enabled. The frames per seond are above 60. 3)Hoever the angle diretl alulated from dot produt is unertain. e have to transfer the angle to [-π, π] sine the vehile an turn left or right arbitraril. Here e alulate r r 0 at the same time, hih is used to determine the diretion. After e get the angle, then steering of vehile ould be done. Eept the steering of vehile, other behaviors are ontrolled b the visibilit state of aording markers to the amera. If some marker is oluded b hand or Copright 2011 MECS I.J. Information Tehnolog and Computer Siene, 2011, 4, 29-35
34 Researh on Virtual Vehile Driven ith Vision Figure 14. Simple sene set up b our modeling tool Though it is not realisti enough ompared to professional simulators or simulation steering heels, it is different, interesting and better than the keboard-mouse a. Suh kind of driving pattern gives a novel a for driving, hih ould be used in other tpes of video games, not onl in driving games. It should have potential business values. Hoever, the limitation of this method is also obvious. It an not provide realisti operation feeling ith fore feedbak. And it also is eas to be influened b the lighting environment. These drabaks onstrain its appliations in the digital entertainment, aa from professional driving simulators. In the future ork, e ould like to integrate GIS into our modeler to enable adding muh useful information to drivers. Other visual features ill also be added to the modeler to provide a more realisti virtual environment. And the traffi simulation ill be integrated into the sstem to simulate the traffi flo ith man vehiles. VIII. ACKNOWLEDGE The ork is supported b the National Grand Fundamental Researh 973 Program of China No.2009CB320802), National Natural Siene Foundation of China No. 60973066, No.50978030), Program for Changjiang Sholars and Innovative Researh Team in UniversitNo. IRT0951). Figure 15. Whole sstem interfae ith Boston sene During eperiments, the environment light ondition plas an important role on the reognition sine the sstem is vision based. But ho to improve the reognition aura is not the researh topi of this paper, readers an refer to other referenes, e ill not disuss here. But generall the method in our paper is feasible. What s more, markers are blak-hite bloks ith distint ontrast hih are eas to reognize. Hoever, if the paper is out of the vie of the amera, the sstem ill fail to detet the vision ommands. If so, the last suessful state ill be used to avoid failure. When the paper omes bak, the state ill refresh automatiall. The hole interfae of the sstem is shon in Fig.15 ith red ords illustrating the main purpose of eah indo. And different tpe of markers is enhaned ith different olor shon in the amera previe indo. The operation demo is available at: http://imlab.hd.edu.n/projets/projets.htm. VII. CONCLUSION AND FUTURE WORK In this paper a traffi-oriented modeling and simulation frameork is presented. With suh a frameork e an design a virtual traffi sene quikl and then drive the vehile in suh a sene. And a vision based driving a is demonstrated. Onl a paper and ebam are needed. With the popularit of ebam, e onl need to print a paper to ontrol the virtual vehile. REFERENCES [1] http://en.ikipedia.org/iki/driving_simulator,2010. [2] http://.3dfahrshule.de/uk_inde.htm,2010. [3] http://.needforspeed.om/,2010. [4] http://.sssoft.om/,2010. [5] Shintaro Ono, Koihi Ogaara, Masataka Kagesaaz, Hiroshi Kaasaki, Masaaki Onuki,Junihi Abeki, Toru Yano, Masami Nerio, Ken Hondak and Katsushi Ikeuhi, A Photo- Realisti Driving Simulation Sstem for Mied- Realit Traffi Eperiment Spae, Proeedings of IEEE Intelligent Vehiles Smposium, 2005.June 2005, pp.747-752. [6] http://en.ikipedia.org/iki/eeto,2010. [7] http://en.ikipedia.org/iki/kinet,2010. [8] http://.goarm.om/raeforstrength,2010. [9] http://.nvidia.om/objet/phs_ne.html, 2010. [10] http://.hitl.ashington.edu/artoolkit/, 2010. [11] http://.opensenegraph.org/,2010. [12] http://.triangraphis.om,2010. Youquan Liu reeived his Ph.D from the Institute of Softare, Chinese Aadem of Sienes in 2005. He no is an assoiate professor in Chang an Universit, his researh interests inlude omputer graphis, virtual realit, urrentl he has more than 30 papers published. Copright 2011 MECS I.J. Information Tehnolog and Computer Siene, 2011, 4, 29-35
Researh on Virtual Vehile Driven ith Vision 35 Yu Cui reeived her Bahelor degree from Chang an Universit in 2008. No she is a master student in Chang an Universit. Her researh interests inlude omputer graphis, virtual realit. Enhua Wu reeived his Ph.D from Universit of Manhester in 1976. He is a Professor at Universit of Maau & the State Ke Lab of Computer Siene, Institute of Softare, Chinese Aadem of Sienes. His researh interests inlude omputer graphis, virtual realit and Sientifi Visualization. He is an IEEE member. Copright 2011 MECS I.J. Information Tehnolog and Computer Siene, 2011, 4, 29-35