How to Measure Progress & Impact: Network Mapping

Transcription

1 How to Measure Progress & Impact: Network Mapping Professor Robyn Keast Chair Collaborative Research Network: Policy and Planning for Regional Sustainability, Southern Cross University

2 Measuring Collec/ve Impact: Using Social Network Analysis for Diagnos/c and Evalua/ve Func/ons Professor Robyn Keast Chair: Southern Cross University Collabora8ve Research Network: Policy and Planning for Regional Sustainability Keast, February 2014

3 Background Working together & collec8ve effort holy grail Many models that aim for par8cipa8ng bodies to leverage from their shared resources, build common purpose and generate collec8ve impact. require many different actors and organisaitons to work together and therefore func8on on a different opera8ng logic: demanding changed thinking, behaviours and ways of evalua8ng CONNECTING VIA RELATIONSHIPS! Despite promise collec8ve models are hard to establish and even harder to sustain. Research indicates that in some industries % of joint efforts fail. Two main factors have been isolated as contribu8ng to the failure: incorrect decisions on when and how to work together and a lack of tools from which to empirically assess levels of connec8vity as well as (individual) and collec8ve impact.

4 Today. Understand and iden8fy the different ways of working together, their various merits and limita8ons Be aware of the suite of tools available to determine levels and types of connec8ons & structures between agencies and diagnose points of weakness Demonstrate, using social network analysis how to uncover, map, empirically measure and analyse service connec8ons and rela8onships in order to op8mally configure service arrangements to achieve collec8ve impact and Examine their network connec8ons in order to strategically leverage for op8mal outcomes. Keast, February 2014

5 Integra8on unpacked: connec8ons Keast February 2014

6 Keast February 2014

7 Integra)on Con)nuum Integra8on rela8onship con8nuum COOPERATIVE COORDINATIVE COLLABORATIVE Low trust unstable rela)ons Medium trust based on prior rela)ons High trust stable rela)ons Infrequent communica)on flows Structured communica)on flows Thick communica)on flows Known informa)on sharing Project related and directed informa)on sharing Tac)c informa)on sharing Adjus)ng ac)ons Joint projects, joint funding, joint policy Systems change Independent/autonomous goals Semi- independent goals Dense interdependent rela)ons and goals Power remains with organisa)on Power remains with organisa)ons Shared power Resources remain own Shared resources around project Pooled, collec)ve resources Commitment and accountability to own agency Commitment and accountability to own agency and project Commitment and accountability to the network first Rela)onal )me frame requirement short- term Rela)onal )me frame medium- term open based on prior projects Rela)onal )me frame requirement long- term 3-5 years Keast, February 2014

8 Fit- for- purpose designs All rela8ons have merit & applica8on Approaches & rela8onal strength must be requisite to purpose Independent, adjus8ng ac8ons & informa8on sharing: coopera8on Do same, but more efficiently through joint working, aligned resources & ac8on: coordina8on Systems change: collabora8on Also need to consider the ver8cal rela8ons Keast, February 2014

9 Key rela8onal essen8als Trust n Companion trust: goodwill & friendship based on associa8on n Competence trust: confidence in capacity of others to fulfil tasks n Commitment trust: contractual or enforceable agreement n Shared values: - the most essen8al Reciprocity: give & take not an I will if you will (self interested) Take a risk & put something on the table Reputa8on hard to will/fast to go Mutuality: replace independent interests collec8ve mutually beneficial outcomes Keast, February 2014

10 When to work together Some services are best provided by single organisa8on/ department What are the 8pping points? Nature & complexity of presen8ng problem n Crisis- mandate doing it because it is the right thing to do Sense of interdependency (can t do this alone) Willingness to engage in new thinking & behaviours new ways of working Keast, February 2014

11 EVALUATING COLLECTIVE IMPACTS: SOME APPROACHES Keast, February 2014

12 Need to assess the impact of different ways of working Different ways of working different ways of evalua8ng Emphasis first is on the rela8onships (intangibles) This is not to say that conven8onal evalua8on is discounted (tangibles) Looking at: the rela8onships and processes that enable wtg the level of par8cipa8on and engagement of members how well the structure allows par8cipants to contribute to and influence work and outcomes Keast, February 2014

13 Check list Rela)onships and processes Are there good rela8ons between members? What is the trust level? Is 8me spent on members gemng to know each other and their problems/limita8ons? Do members feel a strong or weak bond, or commitment, to each other? Are there processes in place to enable these bonds? Is rela8onship building (internal and external) an accepted part of the work program? Do members communicate openly and frequently? Do members have a sense of commitment to the collabora8on as well as their own organisa8on? What are the power rela8ons? Is power shared or does it appear to rest with specific members of the collabora8on? Are there mechanisms to resolve conflict? Is there a culture of learning? Par)cipa)on Level Do all members par8cipate in the collabora8on, in terms of decision- making and resource provision? Are there barriers to par8cipa8on? Are there processes in place to check engagement level? Are people par8cipa8ng as much as they can/ wish Structure and control Is the way the collabora8on is set up appropriate for the aims? Is the structure too 8ght (strangling), too loose (lacks cohesion) or just right (facilitates ac8on)? Where/how are most decisions made? Democra8cally or centralised? Is there support for the collabora8on by key actors outside the collabora8on, eg: parent organisa8ons powerful stakeholders & respected people in the community? Keast, February 2014

14 Contribu8on assessment This tool helps to uncover and understand the level of contribu8on and commitment that members make to the collabora8on, what resources (financial, skill, exper8se, knowledge and materials) are available to the collabora8on, and where (or with whom) they are located and how they can be used. The tool: iden8fies the contributors to the collabora8on, e.g. the individual members of the collabora8on, their parent organisa8on or stakeholder groups specifies the aims of the collabora8on and the types of contribu8ons/resources required asks members to indicate their actual and poten8al contribu8ons and how they will deliver on this e.g. by par8cipa8on or funding considers how easily the collabora8on facilitator has been able to shin resources around the collabora8on or leverage from resources to generate added value Assessment can then be done to evaluate: whether the collabora8on has generated the appropriate resources (8me, money, par8cipa8on of key people, staff 8me, support of the parent organisa8on) whether the collabora8on has been successful in facilita8ng the sharing of these resources between members Keast, February 2014

15 Par8cipatory evalua8on processes Par8cipatory evalua8on involves collec8ve members more directly through a process of self- reflec8on on ac8ons and behaviours as well as uncovering the cri8cal stages and events of the collabora8on. Members are asked to reflect on issues such as: how far strategies and understandings of the collabora8on /collec8ve context are shared how far the informa8on, ideas, documents and resources and analysis circula8ng within the collabora8on have been distributed and their impact on cri8cal moments how members have been able to work crea8vely and collec8vely how connected members are to others in the network Keast, February 2014

16 Contribu8on assessment This tool helps to uncover and understand the level of contribu8on and commitment that members make to the collabora8on, what resources (financial, skill, exper8se, knowledge and materials) are available to the collabora8on, and where (or with whom) they are located and how they can be used. The tool: iden8fies the contributors to the collabora8on, e.g. the individual members of the collabora8on, their parent organisa8on or stakeholder groups specifies the aims of the collabora8on and the types of contribu8ons/resources required asks members to indicate their actual and poten8al contribu8ons and how they will deliver on this e.g. by par8cipa8on or funding considers how easily the collabora8on facilitator has been able to shin resources around the collabora8on or leverage from resources to generate added value Assessment can then be done to evaluate: whether the collabora8on has generated the appropriate resources (8me, money, par8cipa8on of key people, staff 8me, support of the parent organisa8on) whether the collabora8on has been successful in facilita8ng the sharing of these resources between members Keast, February 2014

17 Par8cipatory evalua8on processes Par8cipatory evalua8on involves collabora8on members more directly through a process of self- reflec8on on ac8ons and behaviours as well as uncovering the cri8cal stages and events of the collabora8on. Members are asked to reflect on issues such as: how far strategies and understandings of the collabora8on context are shared how far the informa8on, ideas, documents and resources and analysis circula8ng within the collabora8on have been distributed and their impact on cri8cal moments how members have been able to work crea8vely and collabora8vely how connected members are to others in the collabora8on network Keast, February 2014

18 The SNA way! Keast, February 2014

19 Social networks : uncovering connec8ons

20 Network analysis Levels Metaphor Web of connec8ons, Working together Maps Uncovering actual connec8ons Composi8on Bridges, core periphery Measures Strength of connec8ons density, centrality, centralisa8on

21 History 1934 Moreno sociometry - drawn socio- gram 1970 s advancement in computa8onal power of computers - larger networks Risen to prominence in a number of fields Organisa8onal behaviour, anthropology, sociology & medicine Paqerns of connec8ons between business outcomes e.g. Performance & adop8on of new ideas social networks operate on many levels, from families up to the level of na8ons, and play a crucial role in determining the way problems are solved, people are ac8ng together, organiza8ons are run, and the degree to which individual units succeed in achieving their goals. SNA is focused on uncovering the paqerning of interac8on.

22 What is SNA the deeper details Social Network analysis encompasses theories, models, and applica8ons that are expressed in terms of rela8onal concepts or processes ( ) The unit of analysis in network analysis is not the individual, but an en8ty consis8ng of a collec8on of individuals and the linkages among them ( ) Social network analysis consists of a finite set or sets of actors and the rela8on or rela8ons defined on them. The presence of rela8onal informa8on is a cri8cal and defining feature of a social network. (Wasserman/Faust 2008:4ff.) SNA is (1) guided by formal theory organized in mathema8cal terms, and (2) is grounded in the systema8c analysis of empirical data.

23 Put simply Intui8on not enough! Network Analysis Way to empirically assess/confirm rela8onships Delivers visual representa8ons (maps) metrics (sta8s8cs for analysis/review) Diagnos8c & evalua8ve tool Configure & reconfigure paqerns of sales rela8onships Where to put effort for maximum gain Within networks, across networks & over8me Turbo- charges other methods ie. case studies Keast, May 2011

24 Where else applied... Track terrorist networks Saddam Hussein; Al Qaeda Disease contagion Innova8on diffusion Communica8on flows Business supply chain paqerns Integra8ve services Business clusters Corrup8on Collabora8ve Medical Research Publica8on Outcomes Who is working with whom, on which topics, who are the key links

25 More than methodology In the process of working in this field, network researchers have developed a set of dis8nc8ve theore8cal perspec8ves as well. Following Borgam (2003) some of the hallmarks of these perspec8ves are: focus on rela8onships between actors rather than aqributes of actors sense of interdependence: a molecular rather atomis8c view structure affects substan8ve outcomes emergent effects Social capital theory, theory of weak 8es; social embeddedness; structural holes

26 Conduc8ng SNA: 3 elements Determine the unit of study People involved in sonware development project, community; football matches; transport network Individual en88es are referred to as actors or nodes Interac8ons rela8onships/connec8ons/8es Personal, $, resource exchanges, communica8on, transport Paqern of interac8on = network Aqributes can link rela8ons to aqributes

27 Levels of analysis Community Effec8veness criteria: social capital, aggregate indicators wellbeing Network Effec8veness criteria: membership growth, range of services, rela8onship strength fit for purpose Organisa5on/Actor Effec8veness criteria: client outcomes, linkages to people and services

28 Gathering data & sonware Numerous ways Diary Ques8onnaire/survey Linkage Survey list Ques8on who do you go to for X informa8on? Text analysis - e.g. Official documents Interviews Formal & informal

29 Integrated services Example Agency Shared Informa)on Joint Planning Shared Resources Joint Projects Quality of Interac)on 1-5 Share House CTNC Shelter Connect Dept Housing Medical Centre

30 Colla8ng data: Matrix Actor Sarah Bob Toby Jim Sarah Bob Toby 1 0 _ 1 Jim

31 Key SNA concepts Centrality extent to which a node is in the centre of the network - central people have greater influence, receive more informa8on/ resources, more sa8sfied with jobs Tie strength Social capital Structural holes entrepreneurship Brokerage Core/periphery

32 Key SNA measures & sta8s8cs Centrality Betweenness node others must go through to reach others Highest closeness score goes through fewest In- degree number of nodes coming to a node Density how closely a network is knit. Measure for cohesion & integra8on 0-1, closer to 1 higher Centralisa8on - Average Path Distance number of steps it takes for one person to reach another quasi measure of effec8veness 2.6 (& cohesion) Sub- group iden8fica8on number of closely linked sub- groups or cliques within a larger network Overlapping members, highly segregated sub- groups = lower cohesion

33 SNA in social/human services? Reliance on networked, joined- up, collabora8ve service models Effec8ve networks built fit for purpose Hard to prove integra8on & connec8ons Intui8on not enough topology of links needs to be uncovered to examine actual proper8es of the network to configure & reconfigure

34 Need to assess the impact of different ways of working Different ways of working different ways of evalua8ng Emphasis first is on the rela8onships (intangibles) This is not to say that conven8onal evalua8on is discounted (tangibles) Looking at: the rela8onships and processes that enable connec8ons the level of par8cipa8on and engagement of members how well the structure allows par8cipants to contribute to and influence work and outcomes Keast, May 2011

35 Single network map

36 Alterna8ve view

37 Looking across networks

38 Informa8on sharing Keast, QUT 19 February 2008

39 Shared resources Keast, QUT 19 February 2008

40 Joint planning & programming Keast, QUT 19 February 2008

41 Unpacking roles

42 Metrics Shared Informa)on Shared Resources Shared Planning Network density Av Centrality 54% 39% 33% Av Path Distance Keast, February 2014

43 Density measures Shared Totals Shared Info Shared Funding Service Contracts Contact Referrals Resources Shared Planning and Programming Hi Lo Combined As measure of Density Shared Info Shared Shared Funding Service Contracts Contact Referrals Resources Shared Planning and Programming Hi Lo Combined

44 Ques8ons to consider in analysis Which agencies are most central & are these essen8al for addressing community need? Which core network members have links to important resources via external links? Are some links strong, while others weak what are implica8ons? Does the network include all relevant actors to service area? Which subgroups have strong working rela8onships? Are there core and peripheral actors & implica8ons? Based on compara8ve data has there been reasonable progress been made in building community capacity through development of stronger/durable network 8es? Is the density level of the network sufficient for cohesive ac8on & outcomes?

45 Some important things to think about

46 Be sensi8ve to people & informa8on SNA maps clearly illustrate people/organisa8on s posi8ons & func8ons In or out/close or peripheral Including names can be highly problema8c Uncovering hidden or non rela8onships can be conten8ous Can lead to conflict, unintended consequences & reversion if not handled well Be clear about this upfront & in ethics proposals Respondents have to trust that ethical principles are in place & that the informa8on is being used for up- front purpose Not to cut funding, change structures etc

47 Securing accurate informa8on Data onen collected survey Explicitly disclose their rela8onships (8es) with others Some possibility of respondents giving expected rather than accurate informa8on Be clear in set up & process Some concerns at ability to accurately recollect Studies show able to accurately recollect with list Provide clear instruc8on re aqributes & 8me (e.g. 6 months)

48 Accuracy & gaps SNA uses a host of metrics & measures Commonly used are centrality measures in- degree, betweeness & closeness Robust use of these measures assumes the underlying data sets are accurate Missing data is common 100% response rate is ideal but onen people don't respond to some ques8ons or the whole survey Or spread to informal networks extending the net makes establishing a boundary difficult who is in and not Survey to a network of 45 lead to network of over 300!

49 A picture says a 1000 words Each person can interpret differently from own perspec8ve Use as discussion point/diagnos8c tool rather than defini8ve reality Jumping to conclusions especially based on map & metrics is a danger Qualita8ve work can add flesh to the network bones expanding understanding, explana8on & insights

50 Gemng good response rates Perseverance & patrol Follow- up, again and again and again... Make rela8onship with office personnel Offer something back data, insights, reports, workshops Don t just use as data sources Use your personal rela8ons but with integrity

51 Sharing exper8se Trend for SNA researcher/consultants to be experts Network members are the experts Important to bring them into the analysis Also almost anyone can do the computa8ons the analysis & interpreta8on requires exper8se Understanding of SNA, network theory & context Best achieved through partnerships Sense making is unrelen8ngly social (Weick)

52 High visibility calls for higher ethical & moral standards SNA maps can be very powerful tools E.g. Write into contracts not used for funding cuts etc.

53 SNA other considera8ons Snapshots in Time: As with any data set network data represents the status quo at the point of data collec8on. Since service systems are constantly evolving the results therefore present as a snapshot at that 8me and that any of the paqerns of rela8onships, connec8ons and structures may have changed. As such interpreta8ons of the results should also take into account current contextual considera8ons. Benchmarks: All networks and systems are unique and are contextually determined, reflec8ng the people, history and responsibili8es of those opera8ng within the system. This context can impact on how systems operate and the types and levels of connec8on. Consequently care should be taken comparing across system/network sites. There are no wrong or correct paqerns or metrics. Coupling SNA with other methodologies, par8cularly case studies, is held to value add and turbo charge network results by providing deeper insights and ra8onales for the paqerns. Longitudinal approaches also strengthen the understandings gleaned from SNA studies. The Maps and metrics made possible through network analysis offer a different and important approach to understanding and assessing systems, their paqerns of connec8ons and opera8ons. It is important to appreciate that while they can help answer some ques8ons about the structure and performance of a network, the real benefit of this approach is its ability to s8mulate genuine and informed dialogue and planning.

54 Keast February 2014

55 Some work! Keast, February 2014

56 Ques8ons for considera8on These approaches all rely on building & leveraging rela8onal/ network capital Can/should the collec8ve capital of community networks be leveraged for government outcomes? What role/responsibility does government have? Keast, February 2014

57 Task 1 Iden8fy a project that you are working on now What is its purpose Use the rela8onship (integra8on) con8nuum as a guide to determine: Where are your rela8ons now/with whom? Where should they be to meet the purpose? Structure top down/boqom up What are the op8mal integra8on mechanisms Keast, February 2014

58 Task 2 Building up rela8ons Who do you need to establish rela8ons with? How strong does the link need to be? Borrowing on rela8ons Where do you already have strong rela8ons? How can you use these beqer/use alterna8ve sales formats Use these as links to other sales opportuni8es Dissolving rela8ons Who do you have strong but not produc8ve rela8ons? How could these be ramped up? Should they be dissolved? Are they just habit? Keast, February 2014

59 Strategic rela8onship building Identify those groups/organisations with which your Project/ organisation should have a relationship with to achieve project outcomes. Circle those with whom you already have a relationship Of those circled consider if the current strength or nature of the relationship is sufficient to secure outcome. For relationships that are considered not strong enough, identify possible strategies to strengthen. Similarly it might be necessary to weaken relationships to secure outcomes. For the firms not circled consider (a) what engagement strategies should/could be employed to mobilise commitment/involvement and (b) how strong the relationship needs to be. Keast, February 2014

60 Task 3 What adjustments are required? Systems and processes Behaviours Expectations Management & leadership How will you assess/monitor/evaluate relationships? Individual vs collective impacts Keast, February 2014

61 Some more work Consider you project: Have these issues been iden8fied, discussed & implemented (1) think through some of these factors (2) form into groups to discuss applica8on barriers & enablers What do you need/plan to do? How will you do this champions? Keast, February 2014

62 Evalua8on task Consider your project/program What evalua8on focus have you taken? Will it capture the rela8onship elements How can you draw from some of these alterna8ve evalua8on tools to design an evalua8on that is balanced? Which tool would you use? How would you link/engage ci8zens in this process do they have a role? What are the ethical considera8ons? Keast, February 2014

63 Bright & dark networks what can we learn about connec8ons? Keast February 2014

64 Social Services: Joint planning & programming Keast February 2014

65 Corrup8on Network- Keast February 2014

66 Outlaw Motorcycle Gangs - Keast February 2014

67 Comparison Bright Aim for ^ dense interac8ons High reliance on interpersonal rela8onships as glue that binds & reciprocity Take 8me to build & deliver results, high transac8on costs (constant aqen8on to rela8onship building) Vulnerable to changes in context, shining of actors, funding (+ & - ); poor design Easily disrupted, takes 8me to rebuild reputa8on etc. Dark Based on loose connec8ons One way flows - Rela8onships less important more the exchange & reputa8on Very resilient (last many years), flexible & act fast, low transac8on costs Ceramic very strong - vulnerable to chinks on the edge or major aqacks to the core Hard to disrupt but easy to rebuild (exchange not trust based) Keast February 2014