Data warehouse on Manpower Employment for Decision Support System

Transcription

1 Data warhous on Manpowr Employmnt for Dcision Support Systm Amro F. ALASTA, and Muftah A. Enaba Abstract Sinc th us of computrs in businss world, data collction has bcom on of th most important issus du to th availabl knowldg in th data; such data has bn stord in databas. Databas systm was dvlopd which ld to th volvmnt of hirarchical and rlational databas followd by Standard Qury Languag (SQL). As data siz incrass, th nd for mor control and information rtrival incras. Ths incrass lad to th dvlopmnt of data mining systms and data warhouss. This papr focuss on th us of data warhous as a supporting tool in dcision making. W to study th ffctivnss of data warhous tchniqus in th sns of tim and flxibility in our cas study (Manpowr Employmnt). Th study will conclud with a comparison of traditional rlational databas and th us of data warhous. Th fundamntal rol of data warhous is to provid data for supporting dcision-making procss. Data in data warhous nvironmnt is multidimnsional data stor. W can simply say that data warhous is a procss not a product, for assmbling and managing data from various sourcs for th purpos of gaining a singl dtaild viw of part or all an stablishmnt. Th data warhous concpt has changd th natur of dcision support systm, by adding nw bnfits for improving and xpanding th scop, accuracy, and accssibility of data. Th warhous is th link btwn th application and raw data, which is scattrd in sparat databas but now is unifid. Th objctivs of this work ar to study th impact of using data warhous on Manpowr Employmnt Dcision Support Systm, in th sns as far as th data quality concrn. W will focus on th bnfits gaind from using data warhous, and why it is mor powrful than th us of traditional databass in dcision making. Th cas study will b th Libyan national manpowr mploymnt agncy. Th data warhous will collct databas scattrd from diffrnt sourcs in Libya in ordr to compar th prformanc and tim. words Data warhous, Tim-variant, On Lin Transaction Procssing (OLTP). S I. INTRODUCTION INCE th lat 1960s, th ara of Databas Managmnt Systm (DBMS) has mrgd in rspons to th nd of many organizations to manag and bnfit from ths hug amounts of data, which hav bn collctd and gnratd by ths organizations. Th concpt of tabular orintd rlational databas was Amro F. ALASTA, Faculty of Scinc - Misurata Univrsity, Zlitn, LIBYA. Amr_hard@yahoo.com Muftah A. Enaba, Faculty of Education -Univrsity, Misurata, LIBYA. niba_muf@yahoo.com. introducd in th arly 70s by Dr. Td Cod. Th rlational databas modl has rcivd much attntion and dvlopmnts du to its simpl mathmatical basis (th st thory). Commrcially viabl, rlational databas managmnt systms wr availabl in th markt by arly 80s. Although, in th arly 1980s, most of th commrcial databas systms wr basd on rlational modls, svral altrnativs in databas modls wr also proposd. On of thos altrnativs for rlational databas was th smantic data modl. Anothr altrnativ is th Objct Orintation (OO) modl, th purpos bhind both th dvlopmnt of smantic data modls and th dvlopmnt OO modls is to modl th ral world as closly as possibl. In OO data modling, ach ral world ntity of problm domain is rprsntd by a st of objcts with rlations and oprations. Each objct consists of part of objcts or sub-objcts that rlats objcts to ach othr (rlation rprsntation). In th arly 1990, rlational databas managmnt systms wr mor popular than hirarchal and ntwork databas managmnt systms. Som of th incrasd advantags of th rlational databas managmnt systms wr its functionality and flxibility and th us of cach up in prformanc. In currnt databas managmnt systms, objct orintd tchniqus bcom mor popular bcaus of its ncapsulation of th data and th functions bing prformd on ths data. Rcntly, advancs in tchnology hav bn rvaling nw applications of databas systms, such as picturs, vido clip, and sound mssag; can now b stord by multimdia databas. In addition, maps, wathr data, and satllit imags can b stord and analyzd by Gographic Information Systm (GIS). As data siz incrass, th nds for mor control and information rtrival also hav incrasd. Ths incrass hav ld to th dvlopmnt of Data Warhouss (DW), Data Mining (DM) systms and Knowldg Discovry in Databas (KDD) systms. "Data warhous (DW) and On Lin Analytical Procssing (OLAP) systm ar usd in many companis to xtract and analyz usful information from vry larg databas for dcision making. Ral-tim and activ databas tchnologis ar usd in controlling industrial and manufacturing procsss. Furthrmor, databas sarch tchniqus ar bing applid to th World Wid Wb (WWW) to improv th sarch for information that is ndd by usrs browsing through th intrnt." [1]. Nw gnration of intgratd information systms hav bn appald to IJCCIE.E

2 computr usrs sinc th yar of Thr ar many dfinitions of data mining, on of th most common dfinitions dscrib data mining as, "data mining is th procss of discovring intrsting knowldg from larg amounts of data stord ithr in databass, data warhouss, or othr information rpositoris." [5]. Anothr dfinition of data mining is givn in [3], stats it as; th sarch for rlationships and global pattrns that xist in larg databass that ar hiddn among th vast amount of data, such as th rlationship btwn patint data and thir mdical diagnosis. All abov dfinitions and othr ons put strss on th discovry of rlationships, pattrns and trnds in vast amount of data in figur. Fig. 2 Illustrat th Data Warhous Architctur Th middl layr of th architctur is th global data warhous. In this layr a historical rcord of data is stord aftr bing rsultd from som oprations such as: transformation, intgration, and aggrgation of dtaild data found in th data sourcs. Th data warhous is populatd with clan and homognous data. Fig.1 Data mining as a procss of knowldg discovry II. DATA WAREHOUSE ARCHITECTURE Many rsarchrs and practitionrs shar th undrstanding that a data warhous (DW) architctur can b formally undrstood as layrs of matrializd viws on top of ach othr. DW architctur xhibits various layrs of data in which data from on layr ar drivd from data of th lowr layr. [17]. Hr, w will illustrat th layrs that constitut a data warhous and as dpictd in figur-2.. Th lowst layr of th data warhous architctur is calld th data sourcs layr, which usually consists of th oprational databass. This layr may consist of structurd, unstructurd or smistructurd data stord in fils or othr storag systm. Th data in this layr is xtractd to crat th data warhous. III. ON LINE TRANSACTION PROCESSING (OLTP) VS. ON LINE ANALYTICAL PROCESSING (OLAP). Th purpos of On Lin Transaction Procssing (OLTP) systms is to allow high concurrncy btwn uss which mak it possibl for many usrs to accss th sam data at th sam tim. As th nam implis, ths systms allow transactions to b procssd against th data. In othr words, ths systms control th changs of th data du to som oprations such as: insrtion, updat and dltion during businss procsss. Figur-3. Dpicts a basic OLTP systm: Th figur shows that numrous clint applications can accss th databas to gt th ndd pics of information. Th brokn lins btwn th clint applications and th DBMS symbolizd that ths connctions can physically b implmntd in many diffrnt ways. IJCCIE.E

3 V. OLTP SYSTEM STRUCTURE Each local Scrtary of Manpowr in ach city in Libya has its own databas, ach of which contains various databas tabls. As an xampl, tabl-1 dpicts Misurata Citydatabas tabls. TABLE I MISURATA CITY DATABASE Tabl Nam Tabl Dscription Fig. 3 Dpicts a basic OLTP systm. IV. OLD SYSTEM PROBLEMS Th job skrs apply for th suitabl job according to thir spcialization (major), ducation lvl and th jobs offrd in thir city of rsidnc. Th applicants hav to apply for job in any of th (GPCM) cntrs in ach city. Th Gnral Popl's Committ of Manpowr is doing its bst in manually form to collct th rports producd by th computrizd systms in ach city. Howvr ach city in Libya has its own computr systm to hlp in un mploying popl, but thr ar two main problms with ths systms which ar: 1. Th rporting modul taks a lot of tim to xtract som statistical rports about th applicants. 2. Th systm in ach city works in an individual mannr with its own databas that diffrs from th othr databass in othr citis, which maks it vry difficult to xtract, and collct all th statistical rports ndd for all citis togthr. According to th currntly usd information systm in th Gnral Popl's Committ of Manpowr (GPCM) and basd on our study of a sampl of thr cntrs in Tripoli, Misurata and sirit popularitis, w ralizd that ach cntr has its own sparatd databas that diffrs in structur, coding, data typs and fild lngths, from th ons usd in othr cntrs. For xampl in Tripoli th fils usd ar flat fils data format, in Misurata a rlational databas is usd, whil in sirit nonrlational databas is usd. Thrfor w hav ralizd that th currnt systm of th Gnral Popl's Committ of Manpowr has th following drawbacks: 1. Th administration of data is complx. 2. Thr is data inconsistncy. 3. Solving inconsistncis is xpnsiv and too slow. 4. Th ndd informational data for th dcision support taks long tim to acquir. 5. Th dcision support nds an informational data not a raw data. 6. Th traditional dcision support nvironmnt has faild to provid complt, accurat, intgratd, and timly information to th scrtary of manpowr Ths dficincis motivatd us to invstigat th ida of using data warhous in th dcision support sctor of th Gnral Popl's Committ of Manpowr. Applicant Spcialty Job-Group Moahl Education-lvl Mothamr Srvic This tabl contains prsonal data of th applicant such as; ID-numbr, nam addrss, sx. This tabl contains th information rlatd to th spcialty (major) of th applicant. jobgroup spcifid in ach city sctor to which, applicants wr dirctd to. ducation qualification of th applicant. ducation lvl of applicant, lik primary prparatory, scondary, congrsss in ach city. Contains th attributs that ar rlatd to th military srvic rcord for mals or national srvic for fmals. VI. METHODOLOGY In ordr to build th rquird data warhous, w startd by collcting information from scattrd databass in th abov mntiond thr cits targtd in our study and stor th data undr a unifid schma. As w will illustrat bllow, w constructd data warhous via a procss of data claning, transformation, intgration, rduction, loading, and priodic data rfrshing. Th dsign of th rlation databas rquirs th us of Entity Rlationship (ER) modl, which is appropriat for On-Lin Transactional Procssing (OLTP). Data warhous dsign rquirs subjct schma that is appropriat for On Lin Analytical Procssing (OLAP). "Th most popular data modl for data warhous is a multidimnsional modl. Such modl can xist in th form of star schma, a snowflak schma, or a fact constllation schma." [5]. According to [11], most data warhouss us a star schma to rprsnt th multidimnsional data modl, so w adopt this typ of schma in th dsign of th Gnral Popl's Committ of Manpowr data warhous. VII. DATA PREPROCESSING Typically, th Manpowr databass, as an xampl of ral- IJCCIE.E

4 world databass that ar highly suscptibl to nois, missing, and inconsistnt data du to thir normally hug siz. So it is bttr to improv th quality of th data by prprocssing and in turn this will improv th mining task. During th implmntation of th GPCM data warhous, w wr facd with two main problms. Th first problm is th rdundancy of th applicant's rcords, whr applicants hav applid in mor than on city. W hav to ovrcom this problm by th us of clansing procdur in our data warhous tools that dlts any rdundant rcords. Th scond problm is that of ntry data discrpancy. Bcaus thr ar about 25 cntrs for data ntry and ach oprator has its own styl for data ntry. W hav solvd th abov mntiond problms by som prprocssing procdurs in our data warhous tools to prform clansing, intgration, transformation and data rduction. numbr of data rduction tchniqus could b applid such as; dimnsion rduction, whr th irrlvant or rdundant attributs ar rmovd. Anothr data rduction tchniqu which is known as data cub aggrgation might b also usd whr aggrgation oprations ar applid to th data and th rsult is stord in a multidimnsional data cub. For xampl, th manpowr's databas contains data for dirctd applicants for diffrnt jobs pr quartr for th yars from 2000 to Howvr, somtims it s of mor intrst if th total dirctd applicant by yar rathr than by quartr, so th aggrgatd totals ar rsultd in a smallr volum without losing any information. This typ aggrgation is illustratd in figur-5. VIII. DATA CLEANING Thr ar many possibl rasons for noisy data, at most human rror occurring at data ntry, th data claning routins work to clan th data by filling in missing valus using a global constant, a most probabl valu, or using an attribut man for numric valus. W usd a global constant approach, which is rplacing all th missing valu by th sam constant. Evn though this approach is simpl and asy to implmnt, it is not rcommndd bcaus it might affct th mining procss bcaus it could b mistaknly takn to b as an intrsting concpt. IX. DATA TRANSFORMATION Data transformations involv multipl tchniqus lik, smoothing, aggrgation, gnralization, normalization and attribut construction. In our implmntation, gnralization is usd to rplac low-lvl data by highr-lvl concpts by th us of concpt hirarchis. For xampl catgorical addrss in which districts ar gnralizd to high-lvl concpt congrss, as dpictd in figur-4. Fig. 5 Aggrgation is illustratd Concpt hirarchy tchniqu is also usd in data rduction. For xampl in our cas study th valus of th attributs city, congrss, district form a concpt hirarchy with multipl lvls as dpictd in figur-6. Fig. 4 A concpt hirarchy for attribut addrss in Misurata city X. DATA REDUCTION Th manpowr's data is hug and complx, so it taks a long tim to b procssd, that maks it impractical for data mining. So, it is important to rduc th data siz and rmov th irrlvant attributs for th mining procss, in a way that prsrvs th intgrity of th original data. To rduc th data, IJCCIE.E

5 XI. BUILDING GPCM DATA WAREHOUSE In th procss of building th Gnral Popl's Committ of Manpowr data warhous, it is sufficint to us star schma modl, which containts a singl fact tabl and a numbr of dimnsional tabls. Tabl II and tabl III, dpicts th structur of th fact tabl and th dimnsion tabls rspctivly. Attribut nam City_id TABLE II STRUCTURE OF THE FACT TABLE Dscription Each city has an uniqu numbr _id Each sctor has an EduLvl_id Cong_id Srvic_id Tim_id Total numbr of applicant Numbr of skrs Numbr of dirctd Education lvl Military srvic uniqu numbr Each ducation lvl has an uniqu numbr Each congrss has an uniqu numbr Each military srvic has an uniqu numbr Each tim unit has an uniqu numbr Numbr of all applicants Numbr of all skrs Numbr of all dirctd Numbr of applicants for spcific ducation lvl Numbr of applicants for a spcific military srvic Rmark Masur Masur Masur Masur Masur Th fact tabl contains a numbr of masurs that constitut th output. Ths masurs ar: Total numbr of applicants rprsnts th numbr of applicants that can b rportd accuratly in a vry short tim. Numbr of skrs rprsnts th numbr of applicants that sk to b mployd for ach city and/or for a spcific priod of tim. Numbr of dirctd applicants rprsnts th numbr of applicants that hav bn dirctd to a spcific job. Th numbr of dirctd applicants can b tabld in many ways such as; th numbr of dirctd applicants to a spcific sctor or in a givn city or in a givn priod of tim. Education lvl count rprsnts th numbr of applicants in ach ducation lvl. Military srvic count rprsnts th numbr of applicants in ach military srvic catgory in a givn city or for all citis in a givn priod of tim. City Dimnsion Nam Dscn_ Education_lvl Congrss Srvic Tim TABLE III STRUCTURE OF THE DIMENSION TABLE Dimnsion Dscription city sctor to which th applicant is dirctd. ducation lvl congrss in ach city military srvic for mal or social srvic for fmal tim dimnsion, ach yar consists of 4 quartrs. XII. GPCM MULTIDIMENSIONAL DATA MODEL Th GPCM multidimnsional data modl can b visualizd as data cub with svral dimnsions, as dpictd in figur- 7. Th city dimnsion consists of th citis in Libya in our cas study. Th tim dimnsion divids th yar into four quartrs (Q1, Q2, Q3 and Q4). Th ducation lvl dimnsion is dividd into six ducational lvls. Citis Tim (2000) Sirit Misurata Tripoli Q1 Q2 Q3 Q Education Lvl Th numbr of candidat in Misurata in Q2 in yar 2000, with ducation lvl 3 Fig. 7 GPCM Multidimnsional data modl Nxt figur-8 dpicts th total numbr of job skrs, according to thir prfrrd sctors in all citis from th yar of 2000 to th yar of Figur-9, rprsnts a comparison btwn skrs for jobs and th dirctd applicants to diffrnt sctors from th yar 2000 to 2006 in all citis. Th prvious graphical rprsntations and rports ar th rsults of our data warhous systm which will hlp th dcision makrs in taking th right dcisions in th right tim IJCCIE.E

6 with no dlay according to th nw information availabl. Gnral Gnral Educatin Educati n Trainin g Training Privat compa ny Hlt Privat company Job Skr dirctd Application Fig. 8 Graphical rprsntation for th numbr of applicants and dirctd applicants in all citis from th yar 2000 to 2006 XIII. CONCLUSION "As our world is now in its information ra, a hug amount of data is accumulatd vryday. A ral univrsal challng is to find actionabl knowldg from a larg amount of data. Data mining is an mrging rsarch dirction to mt this challng. Many kinds of knowldg (pattrns) can b mind from various data", [15]. Th objctivs of this work ar to study th impact of using data warhous on a hug amount of data in Manpowr Employmnt Dcision Support Systm in which th data quality ar concrnd. In our rsarch, w compard th data warhous with th OLTP systm to know th issus and impacts of using th data warhous systm on th information systm. In fact, w found that w cannot compltly rplac th information systm with th data warhous but th data warhous is just for hlping th information systm in th procss of dcision-making. W masurd th rporting prformanc btwn th two systms and found that th data warhous is mor powrful than th ordinary old systm. Th old systm has its faturs, which th data warhous cannot do it (as an xampl add a rcord, updat a rcord or dlt a rcord), this is bcaus it is not its functions. W cannot us th old systm instad of th data warhous spcially in dcision support rports or multidimnsional rports, which nd a lot of tim from th Information Systm staff, and at th nd, ths rports ar static rports not dynamic ons. W hav accomplishd th objctivs of th study by focusing on th bnfits gaind from using data warhous, and why it is mor powrful than th us of traditional databass in dcision-making. Th cas study tackld th impact of using th data warhous on mploying popl in Popl's Committ of Manpowr.W hav analyzd and dsignd it by V.B studio 6.0, and SQL srvics programming languags, in ordr to compar th prformanc and tim. W tstd our systm with ral, synthtic and scattrd data that was collctd from diffrnt sourcs in Libya with diffrnt typs (accss databas, DBF fils and flat fil) and sizs (15.56 MB, 6.85 MB and 28.1 MB). Th problms that fac th old systm usrs, which w mntiond in sction 4.1 motivatd us to us data warhous systm in th dcision support organization. Th advantags of this systm wr shown in sction 4.4, whr th rsultd rports and figurs mphasiz thos advantags and th ability of such systm to hlp dcision making which is not asy to achiv using th old systm. Th rports and figurs producd by th prsntd systm show how clar th ovrall situation of th mploymnt procss. So dcision makrs can tak th right dcisions in th right tim with no dlay according to th nw information availabl. REFERENCES [1] Al-saiad Grgawi, Data Intgration in databas systms, faculty of nginring, Tanta Univrsity, Egypt, 2004, pp. 1. [2] Matthias Jark, Yannis Vassiliou, Data Warhous Quality. 2nd Confrnc on Information Quality. Massachustts Institut of Tchnology, Cambridg, 1997, p. 4. [3] Aml Bakry Abd El Alm. MSC, Data Mining For Improving Data Capabilitis, Dpartmnt of Computr & information scinc, Institut of Statistical Studis and Rsarch(ISSR), Cairo Univrsity, Egypt, 2001, p. 9. [4] Ramz Elmasri, Shamkant Navath, Fundamntals Of Databas Systms, Th Bnjamin/Cummings, Inc, California, 1989, pp [5] Jawi Han and Michlin Kambr, Data Mining: Concpts and Tchniqus, Simon Frasr Univrsity, Canada, Morgan Kaufmann, 2000, pp [6] Robrt Viirra, profssional SQL Srvr 2000 Programming, A John Wily & Sons, Inc, Indiana, 2000, pp [7] Mical J. A. Brry, Gordon S. Lin off, Data mining tchniqus, for markting, sals, and customr rlationship managmnt, scond dition, publishd by Wily Inc, Indiana, U.S.A, 2004, pp [8] Karttih J., whit papr, Data Mirror Bnfits of Transformational Data Intgration, Data mining corporation, Toronto, Canada, 2002, pp [9] Lori oviatt, Margo Crandall, Dsigning and implmnting of data warhous using Microsoft SQL srvic 7.0 dlivry guid, Microsoft Corporation, 1999, pp [10] Halim Habib Hanna, MSC, Data Modling in databas and convrsion btwn modls, Institut of Statistical Studis and Rsarch (ISSR), Cairo Univrsity, Egypt, 1994, pp [11] Yin Jnny Tam, Data cub Its Implmntation and Application in OLTP, Simon Frasr Univrsity, Canada, Dpartmnt Computr Scinc, 1998, pp [12] Ahmd mashri El Diab, Studying Prformanc of Data Mining of ral Databas Using Th Classification Tchniqu, Faculty of Enginring, Cairo Univrsity Giza, Egypt, Jun 2003, pp, [13] M. Brry and G. Linoff, Introduction to Data Mining and Knowldg Discovry, Third Edition by Two Crows Corporation, U.S.A, 1999, PP [14] Danil T. Laros, Discovring knowldg in data, Cntral Conncticut Stat Univrsity, A John Wily & Sons, Inc., Canada, 2004, pp [15] Jian Pi, Pattrn-Growth Mthods For Frqunt Pattrn Mining, Simon Frasr Univrsity, Canada, Jun 13, 2002, pp [16] Olivia Parr Rud, Data Mining Cookbook, a John Wily & Sons, inc, Nw York, 2001, pp [17] Panos Vassiliadis, Mokran Bouzghoub, Christoph Quix, Towards Quality-Orintd Data Warhous Usag and Evolution, DWQ: Foundations of Data Warhous Quality, (CAISE 99), Hidlbrg, Grmany, 1997, pp 1-3. [18] Karl Abrr, Klmns Hmm, A mthodology for building a data warhous in a scintific nvironmnt, first IFCIS intrnational confrnc on coopration information systms, Brussls Blgium, pp, IJCCIE.E