Internatonal Journal of Informatcs Socety, VOL.5, NO.2 (203) 7-80 7 Desgn and Development of a Securty Evaluaton Platform Based on Internatonal Standards Yuj Takahash and Yoshm Teshgawara Graduate School of Engneerng, Soka Unversty, Japan {e08d5203, teshga}@soka.ac.jp Abstract - To obtan securty attestaton, organzatons evaluate securty products by usng systems based on nternatonal standards. However, they currently must use ndvdual systems correspondng to dfferent versons of these standards. Therefore, we have been studyng a platform that enables evaluaton for dfferent standard contents and evaluaton targets by focusng on changes of the standards used as evaluaton crtera. We developed and mplemented the platform takng nto consderaton the herarchcal structure and reference relatons of the standards. The platform provdes functons such as a reference-related arrangement of the whole standard, the dsplay of a reference tree, and score calculaton. In addton, n order to produce the pertnent nformaton for data converson, we calculated the smlarty between two standards. Expermental evaluaton shows that coverng all tems and avodance of human error can be acheved by supplementng techncal knowledge and by utlzng vsual effects. The valdty of the platform s also confrmed. Keywords: Securty management, Informaton securty, Internatonal standard, ISO/IEC 27000 BACKGROUND AND PURPOSE OF RESEARCH In recent years, the scope of securty management s expandng from self-defense for protectng the assets of an organzaton to preventng becomng the target of attackers who cause damage to the organzaton. As a result, t has become mportant to have the status of the mplementaton of safety and securty measures assessed by an external agency []. There s a specfc standard for such assessment, called ISO/IEC 2700 and the number of organzatons that are beng accredted by ths standard s contnuously ncreasng. By June 202 more than 7,000 companes had been accredted worldwde, more than 4,000 of them n Japan [2]. For most of the securty certfcatons, standards such as ISO/IEC 2700, ISO/IEC 27002, and JIS Q 500 are taken as references and organzatons are accredted by satsfyng all the tems that are descrbed n those standards. In addton, securty assessment systems are used to valdate the achevement of crtera n the certfcaton process [3]. However, the tems of the standard are frequently changed as tme passes. Compared wth other standards, securty related standards are changed more frequently because they are not tested precsely; user comments are taken nto consderaton and changes are made accordngly. In addton, because the certfcaton process dffers dependng on the sze of the organzaton and other factors, the crtera for assessment also dffer. If the organzaton and the objectve of the assessment change, changes such as revson of the standard wll create a stuaton where a new system has to be created for redong each certfcaton usng ndvdual tools or personnel. Hence consderable tme, personnel and money are requred and ths leads to problems that have huge personnel and monetary mpacts on company actvtes. The need for a general assessment tool, allowng changes n the organzaton and the purpose of assessment, n place of ndvdual securty assessment tools, has been ncreasng. We have been studyng a securty assessment platform that enables the realzaton of partcular securty assessments by replacng only raw data (herenafter referred to as fundamental data) that address the fundamental standard wthout dependng on the target standard [ 4 ]. In ths platform, by focusng on the herarchcal structure of sentences, whch s a characterstc structure of standards documents, the tems of the standard as well as statements ndcatng the detaled condtons and references to other tems (herenafter reference relatons) were organzed n a herarchcal structure. Securty assessment needs to be done wthout dependng on the type of reference standard, and a method of estmatng the assessment level wthout dependng on the type of standard s requred. In the platform the assessment level s estmated usng the herarchcal structure and reference relatons, so we have been studyng a platform that ams to be sutable for achevng securty requrements. In partcular, we have developed an approprate platform system and regstered the data for the ISO/IEC 27000 seres [5]. In ths study, securty assessment s conducted by changng the mpact of assessment wth respect to each component of the reference tree as descrbed below. On the bass of expermental results for securty assessment methods that consder the dstance n the reference trees as well as the relaton of each tem wth assessment tems, we found that changng mpact s effectve. For consderng the relaton of each tem wth assessment tems, we proposed varous methods of estmatng mpact and expermented usng these methods [5] [6] [7]. Thus, for those users who do not have a deep knowledge of attestaton, we expermented wth changng the sample provdng functon and the data mgraton functon, usng relevant nformaton based on past cases, n order to support countermeasure selecton and mplementaton, and we confrmed the valdty of the proposed platform [7] [8]. We found that the effectveness of the data mgraton functon can be ncreased by nterlockng wth the sample functon ISSN883-4566 203 - Informatcs Socety and the authors. All rghts reserved.
72 Y. Takahash et al. / Desgn and Development of a Securty Evaluaton Platform Based on Internatonal Standards [8]. In ths paper, we evaluate the results of the experments done for each functon separately and n combnaton. 2 ANALYSIS AND UTILIZATION OF STANDARDS 2. Relevant standards In ths paper, the experments and verfcatons are performed manly usng the securty standard data that have been summarzed as the ISO/IEC 27000 seres. Ths has adapted the concept of the Plan-Do-Check-Act (PDCA) cycle, whch s wdely used n the standards of securty management and s represented n nformaton securty management systems (ISMSs). Ths securty assessment platform s ntended to be used not n a sngle phase of the PDCA cycle but n every phase where the platform s applcable. If t s appled n the Plan stage, the loopholes n the countermeasures can be checked by enterng the results of the present data analyss. In the Do stage when t s recognzed that enforcng countermeasures does not cover the plannng tem, the loophole can be verfed n ts entrety by means of checkng those tems. In the Check stage, the functonalty of each countermeasure can be checked accordng to the plan made n the countermeasure enforcement stage. The loopholes can be checked by summng those changes n the correspondng condtons that match the condtons n practce. In the Act stage, as n the Plan stage, the loopholes of the correspondng countermeasures that were re-defned can be checked. 2... ISO/IEC 27000-seres The ISO/IEC 27000-seres s an nformaton securty standard famly, establshed by the collaboraton between the Internatonal Organzaton for Standardzaton (ISO) and the Internatonal Electrotechncal Commsson (IEC). Ths seres s broad n scope, coverng prvacy, confdentalty and nformaton technology securty ssues. Therefore, t s applcable to organzatons of all szes and types. To obtan securty attestaton n ths seres, organzatons frst assess ther nformaton securty rsks and then mplement approprate nformaton securty controls accordng to ther needs. Gven the dynamc nature of nformaton securty, the ISMS concept ncorporates contnuous feedback and mprovement actvtes based on the PDCA cycle. As of June 20, 0 standards of ISO/IEC 27000 had been developed and many other standards are now under development [9]. ISO/IEC 27000 s a standard reference n many areas and t shows the mportance of the PDCA cycle to ISMSs. 2..2. ISO/IEC 2700 The objectve of ISO/IEC 2700 s to provde a model for the establshment, mplementaton, operaton, montorng, revew, mantenance and mprovement of an ISMS [0]. In addton, the contents shown n each tem of ths standard n the operatonal manual created durng the process of ISMS attestaton, corresponds to the securty requrements. It should cover all the tems ncludng those that specfy what s outsde the scope of the attestaton. Durng the nspecton for ISMS attestaton, the securty countermeasures correspondng to each tem of ths manual wll be subject to nspecton. 2.2 Standard confguraton Generally the body n the relevant standard has often been descrbed n a herarchcal structure of three phases: 'Chapter', 'Secton' and 'Item'. In a 'Chapter', the assessment targets are roughly classfed. In a 'Secton' the assessment targets are descrbed n detal and n an 'Item' the contents are further descrbed n more detal. However, there are many ndvdual tems whch are not only descrbed as separate tems but also as condtons or supplementary matters that refer to other tems. For nstance, Secton 7. of ISO/IEC 2700 contans a reference to Item 4.3.3 and ths relatonshp s expressed n the reference tree used n ths study, as shown n Fg.. 7. General 4.3.3 Control of records 7. Management revew of the ISMS 7.2 Revew nput 7.3 Revew output Fgure : Reference-related example of ISO/IEC 2700 2.3 The problems of coverng tems related to countermeasures and ther soluton In securty attestaton, the crtera should be comprehensvely covered. Dependng on the framework of each chapter of the confguraton, the requred polcy decsons, such as mplementaton of countermeasures and acceptance of the rsk, wll be made. At that tme, snce there s a need for the comprehensve cover of the standard for each chapter, t s necessary to capture precsely the herarchcal structure of each chapter and reference relatons for each tem. However, n all standards, not just ISO/IEC 2700, there are many references and there s a wde varety of content (tems) whch should be covered. Hence, understandng all of them precsely and choosng comprehensve measures becomes dffcult. Therefore, t s desrable to manage collectvely all the tems covered n each chapter. To solve these ssues, we propose a platform that can collectvely manage all the tems to be covered by usng the herarchcal structure and the reference relatons. Snce the herarchcal structure and reference relatons that are descrbed n the
Internatonal Journal of Informatcs Socety, VOL.5, NO.2 (203) 7-80 73 platform descrbe nformaton from the standard wth smlar characterstcs, the platform can cope f there s a change n the standard or even f the standard s a dfferent one. 3 OVERVIEW OF THE PLATFORM 3. Structure of the platform Ths platform conssts of three parts namely, the data nput unt, the data management unt and the score calculaton unt. The confguraton of the platform s shown n Fg.2. In the data nput unt, the fundamental data of the standard, structural nformaton, reference nformaton, countermeasure nformaton and other relevant nformaton are entered. Intally the nput of countermeasure nformaton can be based on sample nformaton created by the data management unt. Based on these fundamental data and the structural nformaton, the data management unt organzes the data, develops the reference relatons by usng the reference nformaton and confgures the reference tree. In the score calculaton unt, the calculated assessment values (score data) are managed. Also, based on the countermeasure nformaton or other relevant nformaton that has been nput, the sample data are generated. In the score calculaton unt, based on the reference nformaton stored n the reference tree and the nformaton about the regstered countermeasures, the assessment value s calculated and the calculated data are passed to the data management unt. Data nput Data management Score calculaton Standard raw data Structure nformaton Reference relevant nformaton Measure nformaton Pertnent nformaton Reference tree Sample data Score data Fgure 2: Structure of proposed platform 3.2 Behavor of the platform Score Frst, the standard s fundamental data from the data nput unt are stored. Then, the structural nformaton based on the herarchcal structure descrbed n secton 2.3 s stored along wth prevously regstered data. Subsequently, the herarchybased nformaton and the drect reference nformaton (herenafter referred to as drect references) that are descrbed n the standard document are regstered. The regstered multple crtera (standards) are related to each other and, f relevant nformaton s provded showng the requrement of measures, n terms of whch tems of each crteron are related to other tems, that nformaton s also regstered. After data regstraton s completed, the regstered data are delvered to the data management unt and then mgrated to the next operaton. In ths platform, the herarchy s defned usng levels. Chapters are defned as level '' and the followng stages as level '2' and so on. Level 'm' s assumed to refer drectly to the tems of level 'm+'. In ths study, ths type of herarchcal structure s also defned as a part of the reference relatons. The basc tree s confgured wth an tem that has a drect reference as the root (herenafter referred to as the parent reference) and the descrbed tems that should be referenced (herenafter referred to as a reference) are the leaves of the tree. If the leaf of a basc tree becomes the root of another basc tree, a new tree combnng the part of the leaf of the former tree wth the root of the latter tree s confgured. Durng confguraton, a leaf may have the same tem as a reference as the root of the tree. If ths repeated reference relaton has multple references at multple locatons wth the same feld as a reference, t wll cause a reference loop to occur when the tree s confgured. When these references occur, the part that overlaps s desgnated as the leaf and the confguraton of the tree s contnued. Thus, bndng of the tree s contnued untl t becomes mpossble to bnd further and the largest tree becomes the reference tree. In a reference tree, the relaton between the tems s expressed as a dstance. The dstance of those that are referenced drectly s and for each teraton of the followng references the dstance between the tems ncreases gradually. Subsequently, a standard for securty attestaton usng that reference tree s created n the score calculaton unt. The crteron s ntended to provde an assessment value for the entre reference tree. In fact, n the data nput unt, nformaton about the countermeasure mplementaton, based on the nformaton of the reference tree, countermeasures n past projects ncluded n the sample data and the complance status of each tem n the standard, s suggested. Based on the countermeasure nformaton and the reference tree nformaton that has been nput, an assessment value s calculated. In addton, f the sample data are set n the data management unt durng that tme, countermeasures and the supportng data wthn the complance status nformaton of each tem wll be stored as sample data. After that, when the complance status of the correspondng countermeasure s nput by another user, the sample data can be nput referrng to the sample data that have already been provded. If relevant nformaton on other crtera s referred to when the assessment s done under new crtera, by means of the data mgraton functon n the data management unt, sample data are generated based on the complance status data of the underlyng crtera and the data can be nput whle browsng.
74 Y. Takahash et al. / Desgn and Development of a Securty Evaluaton Platform Based on Internatonal Standards 3.3 Features of the platform In ths platform, when there s a change n the standard, the nformaton n the data nput unt s updated. After updatng the nformaton, the reference tree wll be automatcally reconfgured n the data management unt. In the score calculaton unt, reassessment and the recalculaton of the score can be done n accordance wth the changed contents of the standard. In addton, the relatonshps between the tems can be vsualzed by confgurng the reference tree. Choosng the countermeasures whle checkng the reference tree can help to set the effectve countermeasures. In the sample data dsplay functon, managers who may not have suffcent expertse can share nformaton. In the data mgraton functon, durng the reassessment process, the sample data that can be used as reference can be generated wthout any extra effort. 3.4 System confguraton of the platform Ths platform has been developed n Vsual Basc, and varous experments have been performed so far. Frst, the entre platform s confgured as a sngle program. The program s composed of ndependent subprograms: a subprogram that composes nformaton for confgurng the reference tree, after regsterng the crtera, herarchcal structure nformaton and reference relaton nformaton; a subprogram that dsplays the reference tree; a subprogram that organzes the status of the countermeasures; and a subprogram that performs assessment value calculaton. These subprograms ensure smooth runnng of the system by runnng n the background. For nstance, when the data are frst regstered or when any change s made to the data, changes are made to the reference relatons of all the crtera n the background and so, even durng the process of makng changes, the hstory of the data can be vewed. In addton, the body of the platform can always be run by operatng the tme consumng subprograms, such as dsplayng the reference tree and changng the status of the countermeasures, as ndependent programs durng the process. In addton, as the assessment value calculaton s a separate subprogram, t can be easly changed to a new method. Ths s useful when ntroducng or testng multple methods of assessment value calculaton. Smlarly, n the dsplay functon of the reference tree, nstead of replacng the entre program to meet the user's demands, a dsplay program that matches the user's preferences can be easly ntroduced. 4 METHOD OF CALCULATING IMPACT OF EACH COMPONENT OF THE REFERENCE TREE In ths study, whch focuses on the number of tems n the reference tree and the dstances between them, the value of the assessment can be compared usng the securty assessment method that changes the mpact of each component. In addton when tems from other chapters are referred to n the reference tree, t s possble to determne the mpact on the calculaton results of those tems due to the change n the calculaton method. There are four methods tested so far. Method focuses only on the component number. The numbers of exstng measures, measures n progress and measures yet to be mplemented, n the reference tree whch s the root of the estmated tem s called 'n'. The th component s gven the value x, where x s equal to f the estmaton tem s applcable and s equal to 0 otherwse. The evaluaton value Score s gven by Score Method 2 s an estmaton method dependng on the maxmum dstance. For the th component of the reference tree whch s the root of the evaluaton tem, the dstance s d, and the maxmum dstance s d max. As for Method the component number s 'n' and the th component has the value x, whch s equal to f the estmaton tem s applcable and equal to 0 f t s not applcable. The degree of mpact of the th tem s taken as d max d,+. The evaluaton value Score 2 s gven by Score n = 2 = n = In ths method, though there s change n the mpact based on the maxmum dstance n the reference tree, dependng on the dstance, the mpact of the degree of assessment s determned n monotoncally decreasng form. Characterstcally, the mpact of each tem on the assessed tem falls slowly. Method 3 uses the recprocal of the dstance. The assessment value Score 3 s gven as follows: Score In ths method, the mpact of each component s not affected by the maxmum dstance of the reference tree; mpact s determned purely by dstance. In ths method the dstance between the tems has a great effect for small dstances and, as the dstance gets larger, the mpact slowly falls. In Method 4, when the evaluaton tem represented n the herarchcal structure and the chapter of the component are the same, the degree of mpact s reduced; when the represented chapter n the reference structure s dfferent, sudden reducton n the degree of mpact occurs n accordance wth the dstance. In addton, when a smlar n = = { x ( d ( d max n max n = 3 = n = x d d x d d + )} + ) () (2) (3)
Internatonal Journal of Informatcs Socety, VOL.5, NO.2 (203) 7-80 75 concept s referred to n the reference structure, the calculated degree of mpact s relatvely hgh. 5 Fg.3). Then, by morphologcal analyss, ths text nformaton s resolved nto morphemes and extracted ((2) n Fg.3). These are the ndex terms (tems representng the contents of the document) []. One morphologcal analyss program s ChaSen [2] developed by the Nara Insttute of Scence and Technology. Then, the words that become dssonant are removed as unnecessary words. ((3) n Fg.3). In addton, the extracted words are weghted ((4) n Fg.3). For the weghtng method, ndex word frequency Term Frequency (TF) and Inverse Document Frequency (IDF), or a combnaton of these, TFIDF, are often used []. Fnally, the smlarty between texts, whch are converted to vectors or matrces by weghtng, s calculated ((5) n Fg.3). CALCULATION OF SIMILARITY 5. Smlarty calculaton In studes of the classfcaton of documents many methods of calculatng the smlarty have been proposed. In ths paper, we adopt the most commonly used technque as our smlarty calculaton method. The general procedure for calculatng the smlarty s shown n Fg.3. Frst of all, when calculatng smlarty, the text nformaton n each document s to be determned (() n ()Determnaton of problem nformaton [Queston group] Queston (2) Extracton of word Morphologcal analyss 問題2 Q. t 連体詞 t2 名詞-サ変接続 t2t3 Noun-Excepton 名詞-一般 t3 Noun-General 問題2 Q. t t t2 t2t3 t3t4 tt t2t2 t3 t3t4 t Attrbute Queston 2 (3) Deleton of unnecessary word Hgh frequency word Low frequency word Keyword Problem sentence Choce (correct and wrong answer) Sngle noun / Compound noun (4) Dgnty attachment Queston t2 t6t7 t9 4.7 4.73 2. 2.63.00 (5) Smlarty calculaton Index word frequency IDF Entropy TF/IDF Queston Q.6 Q.2 Q. Cosne Inner product Dce coeffcent Jaccard coeffcent 0.982 0.783 0.450 Fgure 3: General procedure for calculatng smlarty Table : Evaluaton values for all methods Category standard value evaluaton value evaluaton value 2 evaluaton value 3 evaluaton value 4 4. Informaton securty management system 2.32%.92% 0.45% 3.98% 5. Management responsblty 5 3.24% 0.79% 3.36% 8.06% 6. Internal ISMS audts 3.24% 0.34% 0.4% 4.9% 7. Management revew of the ISMS 8. ISMS mprovement 3.24% 8.78% 8.59%.84% Table 2: Dfferences for all proposed types Category standard value dfference dfference 2 dfference 3 dfference 4 4. Informaton securty management system 2-8.68% -8.08% -9.55% -6.02% 5. Management responsblty 5-36.76% -39.2% -36.64% -3.94% 6. Internal ISMS audts 3.24% 0.34% 0.4% 4.9% 7. Management Revew of ISMS 8. ISMS mprovement 3.24% 8.78% 8.59%.84%
76 Y. Takahash et al. / Desgn and Development of a Securty Evaluaton Platform Based on Internatonal Standards 5.2 Applcaton example When experments are carred out usng a dfferent standard, the data on the countermeasure s status n the already assessed standard are assumed. The followng applcaton example can be consdered. If the standard s updated, t s possble to locate the tems of the revsed chapter or the tems moved to a newly created chapter. Suppose global crtera of an nternatonal standard are taken as the base. Whle creatng the local crtera of the nternal standard, the platform verfes the extent to whch the underlyng contents of the standard can be reflected, as well as whether any loophole has occurred. If an nternal standard s provded and the am s to obtan securty attestaton, the platform can be used to check how close the current nternal crtera are close to the target crtera for attestaton. 6 EXPERIMENTS BASED ON EACH FUNCTION 6. Experment : Evaluaton value calculaton We compared the evaluaton value usng the securty evaluaton method, whch adds a weght factor to each tem payng attenton to the number of tems and dstance of a reference tree usng Methods 4. In addton, we found that there was an mpact on the results for tems, when tems n other chapters were beng referred to wthn a reference tree. 6... Outlne of experment Frst, we asked an evaluator who has expert securty knowledge to evaluate the securty of an organzaton, and we summarzed the results n a table for every category. Next, we used Methods, 2, and 3, evaluated for the same securty countermeasures, and compared these evaluaton values wth the evaluator's assessment. We also nvestgated whether an mprovement n a value could be obtaned by usng Method 4, based on the knowledge acqured from the experment. 6..2. Expermental results ) Calculaton of evaluaton values usng Methods, 2, 3, and 4 We nput the securty countermeasures nto the platform, and calculated the evaluaton value by each method. These values are called evaluaton values, 2, 3, and 4. The results are shown n Table. 2) Comparson of evaluaton values We compared standard values wth evaluaton values, 2, 3, and 4, and we nvestgated whch method gves a value closest to the standard value n each management feld. The dfferences from the standard value for each evaluaton value and each category are shown n Table 2 as dfferences, 2, 3, and 4. Snce a lower absolute value of dfference ndcates a result that s closer to a standard value Method 2, out of methods 3, s the most effectve n Category 4. Ths means that countermeasures for the tems of the category are n place. On the other hand, Method 3 s the most effectve n Categores 5, 6 and 8, whch means that reference tems nstead of the tem of the category are beng addressed. It was never the case that Method, 2 or 3 was the most effectve n all the categores. Therefore, we used Method 4 as an mpact calculaton method n the form where the features of each method were harnessed. Ths produced an mprovement n all categores. 6.2 Experment 2: Sample presentaton 6.2.. Outlne of experment We nvestgated the correspondence of the countermeasures to tems of standards by showng that sample data could be generated by admnstrators who do not have n-depth knowledge of securty attestaton. Ths experment was executed n the form of role play. The sample data were generated by an author who had experence n general securty operatons and knowledge of securty standards. Countermeasure data were generated by a graduate student n our laboratory who has general knowledge about securty but does not have n-depth knowledge of securty standards. 6.2.2. Expermental results ) Analyss of the countermeasures by the admnstrator Frst, we asked an admnstrator to manually dstngush tems of standards correspondng to the countermeasures. Snce the admnstrator's knowledge of securty standards was not suffcent, he chose tems focusng on hs noton of a countermeasure. Therefore, the selected results have many effectve tems for every countermeasure. Then, the same task was undertaken whle vewng reference-related nformaton n a reference tree. Items wth low relevance compared wth the man tems n each management measure were rejected. The same task was undertaken once agan whle vewng the sample data. The sample data were dsplayed n two forms, n whch the data generated when extractng a countermeasure and the data generated by the admnstrator were dstngushable. A further reducton n the number of tems judged correspondng to the countermeasures was obtaned. 2) Intervew of the admnstrator We ntervewed the admnstrator concernng hs changng selecton crtera and the results. He was able to determne the relatonshp among tems by usng the platform and selected tems wth confdence after presentaton of the sample data. In addton, he sad that he left the data that were not n samples wth confdence n hs judgment n practcal jobs.
Internatonal Journal of Informatcs Socety, VOL.5, NO.2 (203) 7-80 77 Table 3: Reproduced rates and assurance of tems wth a relaton Number of pertnent tems Number of extracton tems OK FN FP NG Reproduced rates Assurance Top category 0 8 8 2 0 0 8 0 Mddle category 3 28 25 5 2 80.65% 89.29% Bottom category 6 97 95 9 0 2 8.9 97.94% 6.3 Experment 3: Data converson 6.3.. Outlne of experment Frst, we compared countermeasures from two vewponts: "the ISO/IEC 2700 Annex A" and "an ISMS attestaton standard Ver.2.0 attachment". Next, we checked the results by carryng out data converson from each dataset. We asked a graduate student who s an admnstrator of our laboratory to partcpate n an experment usng the countermeasures adopted n our laboratory. 6.3.2. Expermental results We used about 20 countermeasures. The number of dfferent tems wth a correspondence was a lttle more than 20. We could obtan all patterns, ncludng opposte selecton and one sde selecton. By analyzng the contents of tems that showed a dfference, we could classfy the dfferences nto the followng sx patterns.. The contents of the tem were specfed n detal.. The contents of the tem became ambguous.. If the contents of an tem at a hgher level to an tem dffer; those tems to whch t ponts also dffer. v. The contents are expressed dfferently; the meanng does not change. v. The same contents are vewed from another aspect. v. An tem does not belong to the same category n both standards. 6.4 Experment 4: Pertnent nformaton extracton by smlarty 6.4.. Outlne of experment We calculated the smlarty between two standards, the ISO/IEC 2700 Annex A (herenafter Standard A) of the nternatonal standard and an ISMS attestaton standard Ver.2.0 attachment "detaled management measure" (herenafter Standard B) whch s part of a Japanese standard. The pertnent nformaton n the two standards s already specfed. We defned tems that have the maxmum calculated smlarty between the two standards as "tems wth a relaton" and we checked how many specfed relatons were reproduced. We classfed tems that were not reproduced nto three categores: False Negatve (FN), whch means they were not extracted although there s a relaton; False Postve (FP), whch means they were extracted although there was no relaton; and NG, whch ndcates that the wrong tem was extracted. 6.4.2. Expermental results A comparson of the pertnent nformaton n Standards A and B the tems extracted as tems wth a relaton s shown n Table 3. The reproduced rates exceed 8 n the top, mddle and bottom categores. Each assurance has a value exceedng 89%. We nvestgated the 3 errors (26 FN,2 FP and 3 NG) to determne the cause. We found that most of the combnatons that cause errors have low smlarty. In the top category, whch contaned few techncal terms, f a more sutable judgment could be made, we could transfer one of FNs to the correct combnaton. Also, f smlar words could be correctly dstngushed between tems, we could also transfer the other FN to the correct combnaton. Each of three combnatons detected as FP and NG n the mddle category had smlartes less than 0.5. Moreover, the NG tem was extracted usng only an tem name, so the smlarty was, and full match was carred out. However smlarty was decreased by combnng the name wth a porton of the detaled descrpton. For the FNs there were also cases whch showed concdences or hgh smlarty of tem names. Other causes of FNs were a low maxmum smlarty vewng from both standards A and B, or a maxmum smlarty vewng from one sde whereas the smlarty s the second or thrd value from the other sde and could not be detected because of ts small margn. In the bottom category, FPs dd not appear. The two combnatons n the NG category showed the maxmum smlarty seen from one sde, and had second or thrd smlarty values seen from the other sde. We could classfy most of the 9 FNs nto the same two cases as for the mddle category The followng knowledge was acqured from these analytcal results.. An tem whch has a maxmum smlarty less than 0.5 does not have a related tem n many cases.. When a descrpton s dvded nto an tem name and detaled descrpton, the smlarty of the tem name becomes more mportant.. Related tems can be detected n many cases f they nclude an tem wth hgher smlarty, even when the
78 Y. Takahash et al. / Desgn and Development of a Securty Evaluaton Platform Based on Internatonal Standards maxmum smlarty from both sdes ndcates that there s no related tem. 6.5 Dscusson 6.5.. Experment Through these experments based on the thnkng of an evaluator, we found an nfluence on achevement level n the management category from tems at a large dstance n the reference tree of the platform. These tems make reference to tems outsde of the management category. In addton, we found that usng reference trees s an effectve way to avod human errors, such as overlookng the nfluence of tems referrng to other categores. Moreover, the evaluaton value has been mproved n Category 5. "Management responsblty" by changng a method to reflect comments from an ntervew. However, the dfference between the partcpant s evaluaton value and the standard result s stll large. Ths may be because possbltes are added as evaluaton crtera, or because contents other than actual evaluaton crtera may be reflected n a result. 6.5.2. Experment 2 There was a tendency for a partcpant wth nsuffcent professonal knowledge to select more tems for countermeasures. Through an ntervew we found out that relatonshps between standards were dffcult to dscern for the admnstrator who had nsuffcent knowledge. We also found out that t s effectve to express relatonshps vsually usng reference trees and that the presentaton of sample data was useful. 6.5.3. Experment 3 From tems,, and n Secton 6.3.2, snce changes may come out n countermeasures by expressonal range, we recognze that t s not approprate to smply change data. From tems v, v, and v, we see that errors can be avoded by showng the sample data. 6.5.4. Experment 4 We confrmed that hgh reproducblty can be obtaned by extractng tems that have relatons based on text smlarty. We found n partcular that the assurance of the tems extracted was very hgh. Some of the causes of errors were due to mproper range dvson of words at the tme of the analyss of wordng. In addton, dfferent words wth the same meanng cannot be automatcally judged because techncal terms are used and the smlarty s low. In spte of usng a smple smlarty calculaton, a hgh reproducton rate and hgh assurance were obtaned. So t appears that t s effectve to use the technque of extractng related tems to determne the smlarty between standards by smlarty calculaton methods currently used n the feld of natural language processng. Moreover, t s expected that stll hgher reproducton rates and assurance can be obtaned by creatng pertnent nformaton usng a more sophstcated technque. Once the data for sample presentaton are generated, t seems to be mportant to reduce FP and NG tems even f FN ncreases. Ths s because we assume users who do not have much specalzed knowledge. For example, the followng technques may mprove the results. When the standard document s dvded nto tem names and detaled descrptons, mportance should be placed on the tem name nstead of employng the weghtng used n our experments. Snce a standard has a herarchcal structure, the smlarty and detecton of relatons of tems n hgher categores should also be taken nto consderaton. 6.5.5. All experments From experments, 2, and 3 we found out that platform s effectve n preventng human error. The errors that can be prevented are dfferent n each experment, but what s consdered as the prmary cause of errors depends on the complcated composton of the standards used as the base document whch s one of the targets of ths research. In ths approach, vsual correspondence was provded by usng reference trees, and contrbuted to problem solvng. In partcular, vsual support was provded by reference trees n experments and 2, and ths contrbuted to the preventon of errors. In experments 2 and 3, vsual support was provded by the presentaton of the sample data, whch also contrbuted to the preventon of human error. Moreover, n experment 4, by usng the technque of text smlarty calculaton, pertnent nformaton was extracted from the standards, even where relatons between the standards are not ndcated. We confrmed that pertnent nformaton can be generated from varous standards, such as a global standard and a local standard. These experments are hghly flexble and ther applcaton s not lmted to securty-specfc standards. However, snce experment 2 s desgned for choosng the relaton between securty countermeasures and a standard, the securty vewpont s strongly reflected here. 7 FUTURE WORK The sample presentaton functon has basc ssues, such as determnng a sample collectng rule and relablty. Currently, we are consderng solutons based on practcal use rather than techncal consderatons. Regardng the sample collecton rule we have proposed a rule n whch the data generated from the sample data are provded as a new sample. Regardng relablty, we have proposed the followng method, whch uses a central server, n order to mprove the relablty of the sample data. If entres correspondng to the same tem about the same measures are stored more than a fxed number of tmes, the server wll automatcally judge that the sample data are relable, and adopt them as the sample data. Otherwse, the data are checked by a human and adopted f ther valdty can be confrmed. We have expermented wth usng the phases of gap analyss and present data analyss. However, there are many phases n whch securty evaluaton can be carred out. Some
Internatonal Journal of Informatcs Socety, VOL.5, NO.2 (203) 7-80 79 of examples are the phase n whch the detaled rsk analyss s conducted, and the phase n whch attestaton acquston has already fnshed and the PDCA cycle correspondng to the phase that carres out securty evaluaton has already been employed. Therefore, we wll also conduct a securty evaluaton experment of an organzaton wth other phases, and examne the valdty of the platform. We conducted an experment usng a calculaton of smlarty, and we used a standard that has pertnent nformaton. Nevertheless, errors occurred. We wll try to avod these errors by usng semantc smlarty and rasng text analyss accuracy. For example, we could use the structural nformaton (e.g., about herarchcal structure) and reference nformaton of a standard. We are plannng experments n whch we wll calculate the smlarty by assumng that the tem name s more mportant, f the standard tem conssts of a name and detaled descrpton. 8 CONCLUSION In ths paper, we verfed, based on the expermental results, not the valdty of an ndvdual functon but the valdty of the whole platform. We found that the platform s effectve for such problems as oversght and nsuffcent knowledge by usng vsual support that presents reference trees or samples. In ths platform, we confrmed that potental nfluence s expressed usng reference-related nformaton n cases where nfluence may be overlooked even f the evaluator has expert knowledge. In addton, we recognzed that the provson of vsual nformaton by reference trees and sample presentaton was effectve for oversght and avodance of msjudgment, when knowledge was nsuffcent. Furthermore, we found that each functon can be utlzed more effectvely by nterlockng two or more functons, such as sample presentaton and data converson. In ths study, we expressed the status of countermeasures by the two choces "done" and "not yet" for smplcty. In addton, we expect that evaluaton of potentalty can be mproved ascertanng the optmal rate of "not yet" f potentalty and the state of beng under way are expressed by usng a thrd choce of "dong". We conducted experments usng two standards where pertnent nformaton was clearly specfed. We recreated the pertnent nformaton wth a hgh reproducton rate and hgh assurance. By determnng such pertnent nformaton through a smlarty calculaton technque, we were able to lessen the rollback of the reapprasal carred out when the standard changes. It s expected that better results can be obtaned by usng a more sophstcated technque. We wll contnue to examne the adaptablty of our platform to varous phases and we wll try to mprove ts valdty. REFERENCES [] JIPDEC, The nternatonal trend of ISMS, and the actual condton of a measure <2004 edton>, (2005). [2] Informaton Management Systems Promoton Center (IMSPC), The number transton of attestaton acquston organzatons, The attestaton acquston organzaton of a certfcate authorty excepton and a prefecture level, http://www.sms.jpdec.jp/lst/nd/su.html. [3] IPA, Securty desgn evaluaton supportve tool V03, http://www.pa.go.jp/securty/fy3/evalu/cc_system/cc tool_v03/secevtoolv03.htm. [4] Y. Takahash, and Y. Teshgawara, A Study on a Securty Evaluaton Platform Based on Internatonal Standards, IPSJ Computer Securty Symposum 2008 The 2nd separate volume of collected papers, pp. 85 89 (2008). [5] Y. Takahash, and Y. Teshgawara, A Study on an Effectveness of Securty Evaluaton Platform Based on Internatonal Standards, IPSJ SIG Techncal Report, Vol. 2009-CSEC-46, No.3, pp.-8 (2009). [6] Y. Takahash, and Y. Teshgawara, A Study of Securty Evaluaton Method Based on Reference Relatonshps among Internatonal Standards, IPSJ SIG Techncal Report, Vol. 200-DPS-42, No. 53, pp.-8 (200). [7] Y. Takahash, and Y. Teshgawara, A Study on Measures Presentaton Functon for Non-Professonal Persons of Securty Evaluaton Method Based on Reference Relatonshps among Internatonal Standards, Multmeda, Dstrbuted, Cooperatve, and Moble Symposum.(DICOMO20), pp. 27 34 (20). [8] Y. Takahash, and Y. Teshgawara, A Study on Data Converson Functon of Securty Evaluaton Method Based on Reference Relatonshps among Internatonal Standards, IPSJ Computer Securty Symposum 20(CSS20), pp. 666 67 (20). [9] Informaton Management Systems Promoton Center (IMSPC), Internatonal trend "ISO/IEC 27000 famly, http://www.sms.jpdec.or.jp/27000famly_20220.p df. [0] ISO/IEC 2700, Informaton technology - Securty technques - Informaton securty management system Requrements, (2005). [] T. Tokunaga, Informaton retreval and language processng, Unversty of Tokyo Press (999). [2] Yuj Matsumoto, Akra Ktauch, Tatsuo Yamashta, Yoshtaka Hrano, Hrosh Matsuda, and Masayuk Asahara, Japanese Morphologcal Analyss System ChaSen 2.0 Users Manual, NAIST Techncal Report, NAIST-IS-TR9902, Nara Insttute of Scence and Technology (999). (Receved October 20, 202) (Revsed January 7, 203)
80 Y. Takahash et al. / Desgn and Development of a Securty Evaluaton Platform Based on Internatonal Standards Yuj Takahash receved the B.E and M.E from Faculty of Engneerng, Soka Unversty n 200 and 2003. He s currently dong hs Ph.D project at Soka Unversty. Hs research nterests are securty management and nternatonal standard of securty. He s a member of Informaton Processng Socety of Japan (IPSJ). Dr. Yoshm Teshgawara s a Professor of Department of Informaton Systems Scence, Faculty of Engneerng at Soka Unversty snce 995, He began hs professonal career n 970 at NEC Corporaton, engaged n the desgn and development of computer networks. From 974 to 976, Dr. Teshgawara was a Vstng Research Afflate wth ALOHA System at the Unversty of Hawa where he dd research on packet rado and satellte networks. He served Dean of Faculty of Engneerng and Dean of Graduate School of Engneerng at Soka Unversty. Hs current nterests are network securty, e-learnng, and ubqutous sensor networks. Dr. Teshgawara receved hs PhD from Tokyo Insttute of Technology, Japan, n 970. He s a fellow of Informaton Processng Socety of Japan as well as Japan Operaton Research Socety. He s a member of IEEE and ACM.