Anton Kosyanenko. Начальник инспекции. Aleksey Solodov

Transcription

1 Accounts Счетная Chamber палата Российской of the Russian Федерации Federation ОПЫТ АУДИТА DEVELOPMENT ГОСУДАРСТВЕННЫХ OF STANDARD ИНФОРМАЦИОННЫХ FOR STATE INFORMATION СИСТЕМ SYSTEMS И AND ПРОЕКТОВ PROJECTS AUDIT Anton Kosyanenko Начальник инспекции Нехорошкин Nikolay Николай Nekhoroshkin Иванович Aleksey Solodov 1

2 Project outline Project name: Development of Standards for State Information Systems and Projects Audit Time frame: Project Leader: SAI of Russia Project Member SAIs: South Africa, USA, Poland, Slovakia, Japan, India Validity of project activities: The degree of the solutions reinforcement with required evidence within the life cycle of the project activities. Bench-marketing: The way to conduct the action research of the economic entities in terms of comparison of performance of the estimated object with the comparable performance (often better) of other objects.

3 Architecture of IT-project audit Financial audit Assessment on the fact Performance audit Strategic audit Forecast assessment Project audit activities Project audit of socio-economic development (SED) is a framework of external state control on grounds of financial audit, performance audit and strategic audit of projects and programs of socioeconomic development (project events). It includes: estimation of indicators values for quality of project management in controlled points, based on analysis of obtained results corresponding to the predetermined requirements within the constraints of funds amount, time and other resources; - Audit organization according to project management principles. 3

4 Stages of IT-project assessment 3. Validity of project activities assessment 2. Goals, tasks and key indicators assessment 1. Assessment of the IT-project environment 4. Legality of resources use assessment 11. Assessment of consequences of the IT-project realization 5. Contents and implementation procedure assessment 10. Assessment of potentials of socio-economic development capabilities 6. Assessment of project activities connectivity 7. Assessment of effectiveness, productivity, economy and unthrift 8. Assessment of resources allocation expediency 9. Assessment of project activity feasibility and risks 4

5 IT-project evaluation phases 5

6 Possible classifications of IT-project audit processes Comprehensive evaluation Evaluation spheres Project audit Goal setting IT-project Management Organization Evaluation stages IT-project Management Technology Evaluation phases IT-project Realization

7 Evaluation steps Architecture of IT-project audit Comprehensive assessment Aggregate assessments Individual assessments Range of assessments Poor Satisfactory Good Evaluation spheres Audit activities support Evaluation stages 7

8 Audit processes matrix Stages Phases Evaluation of goal setting Organization evaluation Technologies evaluation Realization evaluation 1. Assessment of the IT-project environment 2. Goals, tasks and key indicators assessment 3. Validity of project activities assessment 4. Legality of resources use assessment 5. Contents and implementation procedure assessment 6. Assessment of project activities connectivity 7. Assessment of effectiveness, productivity, economy and unthrift 8. Assessment of resources allocation expediency 9. Assessment of the IT-project feasibility and risks 10. Assessment of socio-economic development capabilities 11. Assessment of consequences of the IT-project realization 8

9 Financial audit Stages Phases Evaluation of goal setting Organization evaluation Technologies evaluation Realization evaluation 1.Assessment of the IT-project environment 2. Goals, tasks and key indicators assessment 3. Validity of project activities assessment 4. Legality of resources use assessment 5. Contents and implementation procedure assessment 6. Assessment of project activities connectivity 7. Assessment of effectiveness, productivity, economy and unthrift 8. Assessment of resources allocation expediency 9. Assessment of the IT-project feasibility and risks 10. Assessment of socio-economic development capabilities 11. Assessment of consequences of the IT-project realization 9

10 Performance audit Stages Phases Evaluation of goal setting Organization evaluation Technologies evaluation Realization evaluation 1.Assessment of the IT-project environment 2. Goals, tasks and key indicators assessment 3. Validity of project activities assessment 4. Legality of resources use assessment 5. Contents and implementation procedure assessment 6. Assessment of project activities connectivity 7. Assessment of effectiveness, productivity, economy and unthrift 8. Assessment of resources allocation expediency 9. Assessment of the IT-project feasibility and risks 10. Assessment of socio-economic development capabilities 11. Assessment of consequences of the IT-project realization 10

11 Strategic audit Stages Phases Evaluation of goal setting Organization evaluation Technologies evaluation Realization evaluation 1.Assessment of the IT-project environment 2. Goals, tasks and key indicators assessment 3. Validity of project activities assessment 4. Legality of resources use assessment 5. Contents and implementation procedure assessment 6. Assessment of project activities connectivity 7. Assessment of effectiveness, productivity, economy and unthrift 8. Assessment of resources allocation expediency 9. Assessment of the IT-project feasibility and risks 10. Assessment of socio-economic development capabilities 11. Assessment of consequences of the IT-project realization 11

12 Audit activities support - Methodological support - Information support - Software support - Mathematical support - Linguistic support - Organizational support - Regulatory support - Technical support 12

13 Governance Схема processes процессов creation управления and operation созданием и IT-system эксплуатацией figure (in IT-системы( IAS ARIS) в ИАС ARIS) Жизненный цикл управления автоматизированными системами Automatized system development Разработка автоматизированной системы Automatized Эксплуатация system автоматизированной operation системы Automatized system modernization Модернизация автоматизированной системы Automatized system utilization Утилизация автоматизированной системы Development of terms Разработка технического of задания reference на автоматизированную систему automatized system Development of terms of Разработка технического reference Automatized задания на модернизацию for automatized system автоматизированной системы system modernization Automatized system engineering Проектирование автоматизированной системы Development of software methodic Разработка программной методики испытаний автоматизированной automatized system системы approbation Estimation of quality Проведение оценки index показателей automatized качества автоматизированной системы system Automatized system Проектирование modernization модернизации автоматизированной системы engineering Development of software Разработка программной methodic методики испытаний modernized Automatized модернизированной system automatized system автоматизированной системы approbation Estimation of quality Automatized index modernized system automatized system Проведение оценки показателей качества модернизированной автоматизированной системы Prototype testing of automatized system Опытная эксплуатация автоматизированной системы Последствия проектного мероприятия: Результаты реализации проектного мероприятия, способные оказать Operating modernized влияние Operating на окружение automatized проектного мероприятия. Принятие в эксплуатацию автоматизированной системы system acceptance Prototype testing of Опытная эксплуатация modernized Automatized модернизированной automatized system автоматизированной системы system Принятие в эксплуатацию модернизированной автоматизированной системы Automatized automatized system acceptance 13

14 Individual assessments by stages and spheres of audit 1. Individual assessments by spheres and stages of audit with color codes 2. Brief information on the audit 3. Aggregate and comprehensive assessments with color codes 4. Color codes legend 14

15 Comprehensive assessment 1. Comprehensive assessment with color codes 2. Aggregation procedure for stages assessments in form of a tree 3. Color codes legend 15

16 1. Assessment of the IT-project environment Goal: the identification of subjects and negative environmental factors that affect feasibility and consequences of project activity. Tasks to be solved : identification of subjects, operating in the project activity environment and potentially affecting feasibility and consequences of project activity; identification of negative environmental factors affecting feasibility of project activity; carrying out of the cluster, semantic and frequency analysis of the facts development of proposals to enhance the effectiveness of project activities. Methods to be used: the cluster, semantic and frequency analysis methods; methods of finding patterns in an unstructured data (eg. Internet) (Text Mining, Data Mining) Project activities environment: The system of external and internal circumstances of the environment of the project under consideration, which determine the conditions of operation (implementation) of the project activities. The circumstances should include economic, political, social, technological, regulatory, cultural and other factors. 16

17 2. Goals, tasks and key indicators assessment Goal: to assess the quality of goal-setting and tasks-setting procedures of the ITproject, the IT-project key indicators target values and their validation Tasks to be solved : assess the quality of IT-project goals and tasks setting; assess the IT-project key indicators target values; assess the validity of planned (target) values of the IT-project key indicators. Methods to be used: methods of ontologic and semantic analysis are used. 17

18 3. Validity of project activities assessment Goal: assess financial, economic and technical validity of the IT-project resource allocation Tasks to be solved : assess the level of development of the financial, economic and technical feasibility of the IT-project; assess the adequacy of the IT-project documentation, including terms of reference for creation of information systems, required formats of estimates, budgets, plans, projections, applied calculations, explanatory notes, etc., revealing the key indicators, other effects from realization of the IT-project; verify the compliance of the IT-project in question with relevant laws and other regulatory acts, in particular the legislation on personal data copyright protection. Methods to be used: financial and systemic analysis, benchmarking. 18

19 4. Legality of resources use assessment The legality of budgetary funds expenditure: Compliance of actions focused on the use of budget funds for the project activity with the budget legislation. 19

20 5. Contents and implementation procedure assessment 1. IT-project activities schedule (Gantt chart) 2. IT-project contracts 3. IT-project transactions 4. Performance indicators Resource capacity: The amount of resources required for the implementation of project activities. Availability of resources: Value that characterizes the ratio of the volume of resources allocated to the implementation of project activities and resource capacity. Operation - an element of the project activities for the execution of which the resources are to be allocated. 20

21 Assessment of operations 1. Audit questions 2. Assessments for individual questions 3-6. Aggregate assessments 21

22 6. Connectivity of project activities assessment Goal: assessment of efficiency of IT-project performance by using characteristics of connectivity and ability of architecture of project management system to reallocate resources between different activities within IT-project to achieve goals in the case of project risk events realization with limited resources and time; assessment of interconnections between activities of IT-project 22

23 7. Assessment of effectiveness, productivity, economy and unthrift This module is designed to automate assessment of efficiency of project activity realization using such indicators as effectiveness, productivity, economy и unthrift. Efficiency Effectiveness Economy Unthrift Data requested during the control activity ACTUAL RESULT ACTUAL TIME PLANNED RESULT PLANNED TIME ACTUAL COST PLANNED COST 23

24 8. Assessment of resources allocation expediency Project activities environment: The system of external and internal circumstances of the environment of the project under consideration, which determine the conditions of operation (implementation) of the project activities. The circumstances should include economic, political, social, technological, regulatory, cultural and other factors. 24

25 9. Assessment of project activities feasibility and risks 1. Resources constrains 2. Probability to successfully complete ITproject by a given date 3. IT-project activities schedule with risk events and bottlenecks 4. Probability of successful completion Project activities feasibility : 1) the ability to achieve the objectives of the project activities with existing resources, time and other constraints; 2) characterization of the implementation of project activities for any given stage. Risks of project activities: The potential for occurrence causing the impact on the architecture and technology of project activity, leading to deviations from its purpose or limits. 25

26 10. Assessment of socio-economic development capabilities Goal: to assess the IT system s ability to ensure by virtue of IT-project effective implementation the attainment of the objectives of its development and use. Tasks to be solved : analyze and assess of the non-contradiction of objectives; analyze the IT-project efficiency and attainment of its goals; determination of IT infrastructure elements having the greatest impact on IT system capability. Methods to be used: simulation, hybrid (discrete-continuous) modeling, vector stratification and global sensitivity assessment methods Project activities environment: The system of external and internal circumstances of the environment of the project under consideration, which determine the conditions of operation (implementation) of the project activities. The circumstances should include economic, political, social, technological, regulatory, cultural and other factors. 26

27 11. Assessment of consequences of the IT-project realization Goal : assessment of positive and negative consequences of project activity. Tasks to be solved : identify the list of subjects, who are affected by project activity realization; detect potential budget costs for execution of unscheduled events, triggered be the negative consequences of project activity; evaluate the consequences of project activity. Methods: methods of cognitive modeling and vector stratification, T.SAATI methods. Consequences of project activities realization : The results of the implementation of project activities that may have an impact on the environment of the project activities. 27

28 Assessment of the quality of IT-infrastructure Goal : assessment of the ability of the proposed or existing IT-infrastructure to provide problem solving domain; conformity assessment of the existing IT-infrastructure to the planned requirements; assessment of the effects of the establishment or operation of IT-infrastructure; assessment of the ability of the IT-infrastructure to respond to changes. Tasks to be solved : identification of indicators to assess the infrastructure; analysis of IT infrastructure : the composition of software and hardware, the list of licenses, the amount and quality of information, the composition and qualifications of users; analysis of the software used and the relevancy of its choice; analysis of the effectiveness of the use of the applied information systems for the coverage of the required functionality; analysis of the current organizational and functional structure of the IT-department in terms of its optimality and compliance to the tasks entrusted to it; analysis of the system of internal control of IT and risk management systems; analysis of the effectiveness of the development of the application code; analysis of the effectiveness of technical support; measurement of key indicators of the software quality for evaluation of the application, satisfaction, performance, availability, reliability, security, modification capabilities 28

29 IT-audit of government international system of fuel and energy complex Note: 39% not classified due to the early stage of the life cycle of a GIS TEK Evaluation of IT processes GIS TEK 29

30 Technology assessment management Lack of formalization of problems There are no system requirements There are no system to subsystems The purpose, goals, objectives, results System requirements: architecture activities, process, roles Requirements for sub-systems: IT architecture, integration Life cycle creating IT systems Acceptance process owners during commissioning Acceptance of an IT Architects by components Customer acceptance and objectives No responsible customers by task No responsible consumers results Not responsible for architectural elements Draft ToR requires significant improvement Requirements for elements: functions, reports, technology Acceptance responsible for implementing the function in the process of Not responsible for detailed function 30 30

31 Main publications 1. Napreenko V.G. Nekhoroshkin N.I Audit of the risk of projects and programs using underdetermined models. The problems of control and modeling in complex systems: Proceedings of the X International Conference (Samara, June 23-25, 2008) / Ed: Acad. E.A. Fedosova, Acad. N.A. Kuznetsova, prof. V.A.Wittich. - Samara: Samara Scientific Center of RAS, p., S Nehoroshkin N.I. Challenges and opportunities of information and analytical support for the audit of projects and programs. / / Bulletin of Aksor,. 1, S Nehoroshkin N.I.Technologies of project audit. / Bulletin of PMSOFT. Journal of Project Management professionals., Number 8. C Nehoroshkin N.I. Standard of the project audit. / XXI Conference of the Association of control and audit bodies of the Russian Federation, Rostov-on-Don, March 25, Nekhoroshkin N., Solodov А. Application of special software for IT-audit/ 7 th Performance Auditing Seminar of INTOSAI Working group on IT Audit (Vilnius, Lithuania, April 2013). 6. Nehoroshkin N.I. Piskunov A.A. Basics of methodical maintenance for projects and programs. / / Bulletin of Aksor,. 2, S Nehoroshkin N.I. INTOSAI auditing Standart for the audit of government information systems and projects. / XII International Conference on Project Management. Moscow, 30-May 31, 2013 / 8. Nehoroshkin N.I. Informational Technologies of the state audit. The collection of reports of the second conference "Information technologies in the service of the military-industrial complex of Russia." Moscow, April 2013 c 10-11/ 9. Nehoroshkin N.I. Solodov A.V. Formation of methodological approaches to the audit of state information systems and projects at the international level: the role of the Accounts Chamber of Russia. / Bulletin of Aksor, 2, S Nehoroshkin N.I. Informational Technologies of audit of projects and programs. / / Bulletin of Aksor,. 3, S