Museum Analytics Action Research Project (ARP)

Transcription

1 Version: 1.0 Date: Jan 26th, 2014 Rui Guerra and Fransje Pansters INTK Museum Analytics Action Research Project (ARP) Supported by: INTK Neude AD Utrecht The Netherlands Rui Guerra T rui@intk.com

2 TABLE OF CONTENTS 1. Introduction Project Description Online Analytics Maturity Website Analytics Social Media Analytics Participants Feedback Conclusion INTK Neude AD Utrecht The Netherlands Rui Guerra T rui@intk.com

3 1. INTRODUCTION Museums and cultural organization have a public duty to present their stories and collections to a wide public. It is a fact that there is a growing online audience potentially interesting for museums. Not surprising, cultural organizations have been investing a considerable amount of resources on online infrastructures, such as, institutional websites, e-newsletters and social media. One of the most pressing challenges that organizations face in regard to these technologies is how to define, measure and evaluate their effectiveness. The 'Museum Analytics - Action Research Project' tackles these challenges by installing a truly analytic culture in museums, by empowering museum professionals with expertise and tools to measure the impact of their online endeavors. Moreover, it seeks to help gain insights that lead to a deeper understanding of online audiences. The project follows the footsteps of a similar project developed by Culture24 in the UK started in 2011 and still continuing today. The Museum Analytics - Action Research Project took place over a period of approximately 10 months and it included three workshops, two online evaluation surveys and one evaluation meeting. Several museums professionals representing 16 Dutch museums participated in all activities. The followed approach required the active participation of all involved organizations to share their experiences in developing and understanding the impact of online services with the wider project group. The present report is divided into several chapters following the same order of which different topics were discussed in the several workshops. The chapter 'Project Description' presents a detailed account on how the project was developed. It includes the project goals and methodologies, as well as some background information on how the project was initiated. This chapter has been written with the intention to help others that might be interested in launching a similar project. For readers that are focused on how implementing the outcomes of the project in their own organization this chapter can be skipped. The chapter 'Online Analytics Maturity' introduces a model developed by analytics expert Stéphane Hamel and explains how it can be used by museum professionals to assess their current situation in terms of online analytics. Furthermore, it provides an actionable path towards improving competences and leveraging data analyze and decision-making. This chapter includes participants' findings from using the model at the beginning and at the end of the project. The focus of the chapter 'Website Analytics' is to understand whether the participating museums were achieving the goals proposed for their museum/individual institutional websites. The chapter includes the outcome of an online survey conducted to investigate the motivation of online visitors; instruction on how museums can use Google Analytics to measure the performance of their websites and a benchmark of the participants' institutional websites. The participants social media presence is analyzed in the chapter 'Social Media Analytics'. The chapter includes a social media benchmark of all participants as well as social media publishing best practices based on the findings of participants as well as based on information supplied by Facebook and Twitter. The chapter also includes practical information on how museum professionals can use social media to increase website visits and museum attendance. A chapter is included with the participants' feedback about the project. The final chapter includes the projects conclusions and list of future work. The present report aims at inspiring others to develop similar projects in the cultural sector or any other professional sector to which the learnings of the project might be useful. In order to achieve that goal, the report has been intentionally made as detailed as possible both about the methodology used to implement the project, the tools used and its outcomes. Finally, this report is released into the Public Domain. To the extent possible under law, the authors waive all copyright and related or neighboring rights to this report. Museum Analytics - Introduction Page 3 of 39

4 2. PROJECT DESCRIPTION 2.1. PROJECT AIM The Museum Analytics Action Research Project (ARPNL) aims to install a truly analytic culture in the Dutch museum community, by empowering museum professionals with the expertise and tools in order to understand online audiences and effectively measure the impact of online endeavors. This report is the first step towards answering the pressing challenges that museums face in regard to online infrastructures, which is: how to define, measure and evaluate the effectiveness of online activities? Project Focus The focus of the project has been defined based on the assumption that online infrastructures are developed in order to help organizations to fulfill their mission. The relation between the online endeavors and the organization s mission is usually outlined in a document called online strategy. For many organizations, this document is the point of departure when it comes to reporting on their online strategy, as it often includes metrics to measure the effectiveness of such activities. Once metrics have been defined, it is possible to use free software to setup periodic reports also known as digital dashboards. By continuously analyzing the outcomes of online activities, organizations can adjust their online tactics in accordance with their overall mission. The research focus was to: Map overall organizational goals to online activities by creating an online strategy; Setup and use free software to measure and evaluate online activities and online audiences; Analyze and interpret weekly and monthly reports about online activities and online audiences; Understand how to integrate the use of online tactics into the organization overall strategic mission. The focus of the project was defined in a project-outline before the project started. It was presented and agreed on by the participants in a project launch meeting Project Scope The specific scope of the project was informed by the input and ideas of the project group. The following topics reflected the participants interests and defined the project scope: Online analytics maturity (including an analysis of goals, metrics, tools, etc.) Website analytics Social media analytics These priorities set the agenda for several face-to-face meetings and workshops that in turn formed the structure of the research project. Some of the interests expressed by the participants were: Time to reflect on wider organizational goals and the role that online activities can play in fulfilling these (or not). Understanding online user motivation: Why do people visit museum websites? Learn how to set up Google Analytics in accordance with the sector s current best practices. Deeper insights into the use of Google Analytics for web reporting. Gaining insights about the current use of mobile devices to visit museum websites. Improve internal reporting in order to engage staff members with the organization online activities. Being part of a professional network that actively shares knowledge and informs each other about practices. Museum Analytics - Project Description Page 4 of 39

5 While the project was still in premature state, several participants mentioned topics that, although relevant, unfortunately did not fit the scope of the current research project. Some of the topics that came up were e-newsletters and e-commerce. In respect to e- newsletters, participants were interested in sharing information about specific metrics and tools used by other organizations. The discussion about e-commerce focused on existing business models and frameworks used to sell tickets online. Although the participants had the opportunity to discuss these and various other topics, the research project could not include them in its main focus. It was agreed by the project group that such topics are relevant and interesting to be addressed in a possible follow-up research project METHODOLOGY The project methodology has been based on the interests of participants, in order to achieve the active sharing of knowledge. We agreed that the project would be framed around a collaborative approach with several face-to-face meetings, in which participants would take an active role. Between those meetings, an online collaboration tool was used to continue the active exchange of information. The project consisted of two meetings, three workshops and two evaluation surveys over a period of eight months. The workshops were the core of the research project. In order to get the best of the project, it was vital that participants attended all of the meetings and workshops. Different tools were used to define, measure and evaluate the participants online endeavors. An evaluation survey was used to collect participants feedback on the overall project. All participants contributed to the project s funding with a fee. Below, the project methodology is further described Collaborative Approach The project was developed in a participatory and collaborative style, together with sixteen Dutch museums. Each participating museum put forward a member of their staff to act as the liaison point for the research. These individuals were directly involved in the delivery of the organizations digital offer, either at an implementation or strategic level. Furthermore, they were responsible for any other digital staff, freelancers or technical contractors, which allowed them to access specific data and to apply the project s findings immediately. The collaborative project approach demanded the active involvement of each participating organization to share their experiences in developing and understanding the impact of online endeavors with the project group. In practice, this approach meant an investment to being open and honest about online successes and failures, as well as the sharing of internal analytical processes and cooperation with the INTK team to share analytics data from several platforms. The collaborative and active project approach was achieved by requesting each participant to attend all meetings well prepared. Participants were instructed on the necessary preparation well on time. Most preparation assignments were covered over 80% by participants. These assignments stimulated the project group s motivation and dedication. By way of benchmarking, the results of one individual participant would always benefit the results of others, as they were put into context. Furthermore, questions that had arisen during the preparation could directly be discussed during the following meeting or workshop Meetings In total, the project group gathered for five meetings (project launch, 3 workshops, and evaluation) over a period of nine months, from February 2013 until October Three workshop meetings formed the core of the project. Each workshop dealt with one of the agreed focal points and collaboratively formulated priorities, as we agreed on during the first workshop. During two workshops, an international expert was invited to share his/her knowledge on museums and the evaluation of online activities and audiences. More information about these experts can be found in the appendix. All meetings and workshops have been held in the Netherlands and were hosted by one of the participating museums. More museums were willing to host a meeting or workshop than there were opportunities to do so. The final meeting locations were chosen according to geographical location. As a result, the project group gathered in Amsterdam, Otterlo, Den Haag and Utrecht. Most participating museums were located in the Randstad area (Amsterdam, Rotterdam, Den Haag and Utrecht). By spreading the meetings over several cities, it was aimed to encourage participants to visit colleague institutions outside of their own city and to possibly divide traveling time among the group Guided Museum Tours All meetings and workshops were hosted by one of the participating museums. The hosting institutions offered a guided tour as part of the workshop program. These tours differed in character and style, as some described the extraordinary museum architecture, while others exemplified the unique aspects of the presentation of their collection. The tours gave a special touch to each workshop or meeting, as they were instructive and enjoyable. Not only did we get to know each other s working environment, it was also very Museum Analytics - Project Description Page 5 of 39

6 interesting and inspiring to compare different collection presentation styles and approaches. The last meeting was not hosted at a museum, but at the Dutch Game Garden (where the INTK office is located). The Dutch Game Garden is a non-profit organization that supports starting game developers and other creative companies. In order to take advantage of the technological creative environment at the Dutch Game Garden, the tour was replaced by several short presentations by startup companies Online Collaboration In between each meeting or workshop, almost two months passed. Therefore, an online collaboration tool called Basecamp was set up in order to ensure a continuous flow of sharing knowledge, information and experiences among the participants. Most topics discussed on Basecamp were related to one of the workshops, but also issues that fell outside the scope of this research project were shared. Basecamp was thus used as the central platform for communication. As the project further developed and participants got to know each other better, the online platform was used more intensively. Activity on Basecamp can be summarized as follows: 49 people were registered; 37 files were shared; 46 discussions were started; 176 comments were placed. Thanks to the online platform, communication among participants remained open and transparent to the entire group. Files and other information that was shared in the platform, made it a real knowledge hub for all the participants. All in all the online platform truly encouraged the collaboration among participants, as demonstrated by the open and honest sharing of information and experiences among the group ACTIVITIES OVERVIEW From February to September 2013, the following activities took place: a project launch meeting, three workshops and an evaluation meeting. The workshops formed the core of the research project, as each workshop covered one of the topics defined in the project scope. In summary, the first workshop dealt with Online Analytics Maturity (including the analysis of goals, metrics and tools), the second with Website Analytics and the third with Social Media Analytics Project Launch Meeting The Project Launch Meeting was hosted by the EYE Filmmuseum in Amsterdam on February 6th, The meeting brought together more than twenty museums that had shown interest in participating in the research project. Hein Wils (at that time working as project manager at the Netherlands Institute for Sound and Vision) explained the background of ARPNL, while Fransje Pansters (project manager, INTK), described the ARPNL methodology. This meeting offered an opportunity for interested participants to clarify any possible doubts about the project and it s methodology. After the project launch meeting, all attendants were asked to give a final answer about their commitment to the project. From the twenty-seven museums that attended the project launch meeting, sixteen decided to participate. An overview of all museums invited to participate can be found in the appendix. The most important reasons against participating were related to a lack of time to contribute actively to the project, a tight museum staff capacity and/or financial issues regarding the participation fee. EYE Filmmuseum, Amsterdam. Photo by Maarten Noordijk. Museum Analytics - Project Description Page 6 of 39

7 Workshop 1: Online Analytics Maturity The first workshop was hosted by Science Centre NEMO in Amsterdam, on March 15, The workshop was divided into two parts. In the first part of the workshop, Jane Finnis (Chief Executive of Culture24) presented the key findings from the Let's Get Real project. She talked about how we understand online success and how we could relate it to the overall organizational mission. of museums. Furthermore, the scope and research priorities of the research project were further refined. During the second part of the workshop, facilitated by Rui Guerra (co-founder of INTK), participants discussed their organizational strengths and weaknesses in online analytics, following the online analytics maturity model defined by Group picture at the rooftop of the Science Centre NEMO. Stéphane Hamel. By the end of the workshop, each participant had identified their main points of improvement, that were tackled in subsequent workshops. More information about the online analytics maturity model and it s outcomes can be found in the chapter Online Analytics Maturity Workshop 2: Website Analytics The second workshop was hosted by the Kröller-Müller Museum in Otterlo, on May 15, Sebastian Chan (Director of Digital & Emerging Media, Smithsonian Cooper-Hewitt National Design Museum, New York) was leading the workshop. He expounded on how analytic reports have changed and are changing many of the cultural organizations he has worked with over the past years. Together with Sebastian, Google Analytics (GA) was introduced and discussed. Special attention was paid to how GA can be used to keep track of and report about museum goals. While the workshop was highly informative, it was not easy to deliver information tailored to the needs of each and every one of the participants. Some Group picture at the Kröller-Müller Museum. would have liked more specific and in depth information, while others preferred more basic guidance. This devision was to the result of a great difference in expertise. To concede to the different needs of all participants, a blogpost was used to share in depth information about Google Analytics after the workshop. In some specific cases, participants received individual guidance on how to install and configure their Google Analytics account. More information about the lessons learned from Google Analytics with Sebastian Chan can be found in the chapter Website Analytics. Museum Analytics - Project Description Page 7 of 39

8 Workshop 3: Social Media Analytics The third workshop took place on July 4th, 2013 and was hosted collaboratively by the Gemeentemuseum and Museon, which are located next to each other in The Hague. The workshop dealt with social media activities and focussed on measuring performances and evaluating outcomes. Similar to the first workshop, the workshop was divided into two parts. In the first part, Rui Guerra presented and discussed the current state of social media among the project group. Results were contextualized by comparing them with the overall social media usage in the Netherlands. Rui described best practices to publish content on social networks based on statistics collected from Museum Analytics. The second part of the w o r k s h o p f o c u s s e d o n s h a r i n g Gemeentemuseum, The Hague experiences. Charlotte Bosman, Online Media Consultant at Van Gogh Museum, presented her approach to social media. Both presentations were regarded as highly informative and inspirational by the project group. More information about the social media results can be found in the chapter Social Media Analytics. From the previous Website Analytics workshop we learned that participants were eager to discover more about custom Google Analytics reports. Such reports could be used to compose a d a s h b o a r d t h a t s h o w s a n a l y t i c information, tailored to the needs of a specific department or in line with of the organizations goals. Therefore, the second part of the workshop was also directed at a practical exercise that taught participants how to compose a customized Google Analytics dashboard. Starting point was to formulate a dashboard goal to determine what the dashboard should report about. A dashboard can be composed out of many Participant composing a dashboard by cutting and pasting widgets. different metrics (widgets). Those have to be chosen carefully, preferably according a goal, because only 12 widgets can be inserted in a dashboard. The 17 most common widgets reporting on topics such as visits, traffic sources and most popular content, were selected to use during the assignment. These widgets were printed on paper, as cutting and pasting with paper made it easy to select, shift them around and to keep an overview. In the end, three dashboards were created: Social Media Dashboard: focussing on metrics that report on social media activities in relation to the institutional website. Educational Colleague Dashboard: focussing on metrics that report on online activities related to education. Institutional Goals Dashboard: focussing on metrics that report on institutional goals measurable with the website. For example, a dashboard reporting on e-ticket sales if the organizations goal is to increase e-ticket sales. More information about the results of the assignment can be found in the chapter Website Analytics and Social Media Analytics. Museum Analytics - Project Description Page 8 of 39

9 Evaluation Meeting The evaluation meeting took half a day and was hosted by the Dutch Game Garden (DGG) in Utrecht, on September 5th, The meeting dealt with discussing the progress of the project group in terms of online analytics, based on the online analytics maturity model by Stéphane Hamel. Participants compared the outcome of their online maturity model before and after the project took place. This comparison allowed participants to indicate the impact that the research project had had on their own organization, as well as to compare their progress to each other. The final results are being discussed in the chapter Online Analytics Maturity. Furthermore, the evaluation meeting was directed to evaluating the research project itself, as participants were asked for their feedback on the project in general. Group picture at the Dutch Game Garden, Utrecht. The DGG serves as a creative incubator for technological innovation and the development of all kinds of games.therefore, instead of a museum tour, five creative agencies were invited to give a brief talk about some of their innovative projects interesting that were of interest to the participating museums Evaluation Surveys Two evaluation surveys were part of the research project. First, a survey was used to evaluate the progress of participants in terms of their online analytics. This was referred to as the Online Analytics Maturity Model by Stéphane Hamel. See chapter Online Analytics Maturity for more information about the results of this survey. Second, the research project itself was evaluated by the set up of an online survey. This survey collected feedback about the impact that the project had had on the participating organizations and how the project had changed the museums representatives on a personal level. The results of the survey are presented in the chapter Participants Feedback TOOLS Various tools have been used to collect information on website analytics, social media analytics and online audiences. All data was collected automatically. Google Analytics was used as the main tool for the analysis of web traffic. Museum Analytics was used as the tool to collect, visualize and share social media data among participants. Qualaroo was used as a tool to shed light on user behavior and motivation. Furthermore, a blog was created to share some first project results with an interested audience. To measure the participating organizations online analytics maturity, a model developed by Stéphane Hamel was used. As the chapter Online Analytics Maturity goes deeper into the use and results of this model, it will not be further discussed here Google Analytics Google Analytics (GA) is a service offered free of charge by Google. It collects and visualizes information about the traffic on a website. The research project used GA as a main tool for analyzing web traffic among the participants websites. GA is one of the most common tools used for this purpose in the cultural sector and it has been recommended by similar research projects such as the Let s Get Real action research project organized by Culture24. Museum Analytics - Project Description Page 9 of 39

10 Museum Analytics Museum Analytics (MA) is an online platform dedicated to share and discuss information about museums and their audiences. It includes daily updated information about social media usage by museums, as well as information about online and onsite visits updated on an annual bases. Currently MA generates daily, weekly and monthly reports for museums. The platform includes more than 3200 museums worldwide, of which 194 are Dutch museums. During the Museums and the Web conference in 2012, Museum Analytics has been awarded two prices: People s Choice and Best Museum Professional Website Qualaroo Qualaroo is a tool to run online surveys. During the research project, Qualaroo was used to get a better understanding of user motivation, focussing on the question: why do people visit a museum website? The results can be found in the chapter Website Analytics ARPNL Blog Many museum professionals had shown interest in following the outcomes of the ARPNL. Therefore a blog was created with the purpose of publishing and discussing some of the main results the project with a national, as well as an international audience. In line Museum Analytics with the nature of the internet, frequent blogposts were made. The topics were related to issues discussed during the workshops. All blogposts were first shared on Basecamp, to ask the participants for their feedback. The ARPNL blog can be found online. In conclusion, besides the tools that were used by the research project, many other tools exist that could be used to for online analytics. Participants shared and discussed their favorite tools on Basecamp. Some of these were very useful for a day-to-day operation, yet the tools applied in this research project have been chosen because they were most suitable to the project s goals FUNDING MODEL The funding model of the research project was characterized by a collaborative approach. The participants covered approximately half of the projects costs, whereas the other half had been covered with funding form the Mondriaanfonds and the Rijksdienst voor het Cultureel Erfgoed. All participants paid a 1000 participation fee. We were aware of the different budgets of the participating museums. Therefore, the participation fee might have been relatively cheap for some museums, whereas it might have been considerably expensive for others. Few museums had to cancel their intention to participate, due to the fee. Nevertheless, it was difficult to come up with an approach that was fair to all participants. By means of the ARPNL blog, organizations that could initially not afford to participate, still have access to the project outcomes and methods. Furthermore, they are encouraged to follow the guidelines that this report provides. In conclusion, the project methodology was framed by a collaborative approach in which participants took an active role. All participating organizations had at least one representative whom attended all meetings. In general, every two months a meeting took place, over a total period of eight months. The workshop meetings lasted one full day and were hosted by one of the participating museums. Every workshop was held at another museum. Some workshops were so popular that the participating organizations requested to attended with more than one representative. To make sure that the workshops stayed at a manageable size, a maximum of 30 participants was agreed on for each workshop. Museum Analytics - Project Description Page 10 of 39

11 Participants feedback has played an important role throughout the project, as their interests formed the basis of each workshop. Whenever possible, workshop programs have been adjusted to newly raised questions or interests. At the end of the project, the participants improvement on online analytics was evaluated, as well as the project itself was evaluated. To collect analytic information, several tools have been used. The tools have been chosen based on their common usage in the cultural sector and the degree to which they answered to the project goals. The collaborative approach of the project was reflected in the projects funding, which was partly covered by the participating museums and partly by the Mondriaanfonds and Rijksdienst Cultureel Erfgoed BACKGROUND ARPNL followed the footsteps of a similar project that was carried out by Culture24 in the UK in 2011 and The outcomes of the Culture24 Let s Get Real action research project have been presented in cultural heritage conferences, such as, DISH2011 and Museums and the Web The frankness of the Culture24 report was its openness to speak about failure in the sector to really capture the attention of online audiences. The results of this research have been positively received. Several museum professionals in the Netherlands responded to this research with great interest, especially because at that time, a research of this nature had not been carried out yet in the Netherlands. During DISH2011, the possibility was therefore discussed to develop a similar project involving Dutch museums. ARPNL was initiated by Rui Guerra (co-founder of INTK) in April In order to make a benchmark possible, the Dutch research project focused on website analytics and social media analytics, similar to the Culture24 Let s Get Real action research project. Furthermore, both projects took a similar approach to collaborative learning of participants Participants ARPNL was a joint project from INTK, the Mondriaanfonds, Rijksdienst Cultureel Erfgoed and sixteen participating museums. INTK s role was to lead and coordinate the project in close collaboration with the Netherlands Institute for Sound and Vision, who acted as formal project coordinator. Participants and sponsor logo s Museum Analytics - Project Description Page 11 of 39

12 Related Projects In The Netherlands In the past years, several projects related to the ARPNL subject matter have been initiated in the Netherlands. We were aware of these projects, such as De Digitale Feiten (The Digital Facts, DDF) and Museumkompas. De Digitale Feiten research (The Digital Facts, DDF) was carried out in 2008/09 by Digitaal Erfgoed Nederland (DEN). DDF gives an overview of the state of affairs in the field of digitization of cultural heritage, by creating an overall picture of the size and costs of the digitization process in the Netherlands. The report provides a good insight into the order and progress of this digitization process, as well as a statistical framework for measuring it. Different from DDF, the ARPNL focusses on the online presence of participants and the relation with online audiences. How far organizations are in the digitization process was not covered in the research project. Museumkompas was a two-year project implemented by Crossmedialab (Kenniscentrum Communicatie & Journalistiek, Hogeschool Utrecht), Erfgoed Nederland, DEN and several museums. The aim of the project was to support museum professionals in the development of robust new crossmedia services by providing the professional with the knowledge and skills to develop and implement solutions within the context of their own organization. The starting point of Museumkompas is related to ARPNL, but takes a different approach in practice. By closely following the Culture24 research structure, ARPNL could benchmark results in an international context. We have been in contact with Museumkompas to discuss our joint interest and the possibility to join forces. However, at that time, Museumkompas was already about to develop the Crossmedia Monitor Musea, a platform that collects and presents information about social media of over 500 museums in the Netherlands. This platform is very similar to Museum Analytics (that was already in existence then). Unfortunately, both Museumkompas, and ARPNL turned out to be too far ahead in the project s process, in order to find common ground for collaboration. By using Museum Analytics as a platform to evaluate social media, ARPNL could make an international benchmark, as 3000 museums worldwide are part of the platform. In conclusion, ARPNL differs both from DDF and Museumkompas in its emphasis on collaborative learning, which was first implemented by Culture24. Without the active participation that was required from all participants, the project could not have been realized. ARPNL has made its participants aware that many colleagues from different organizations face the same challenges, and that they can be solved easier by joining forces. ARPNL has started a culture of exchange of knowledge and discussion among museum professionals, that has the potential to continue after the end of the project. Museum Analytics - Project Description Page 12 of 39

13 3. ONLINE ANALYTICS MATURITY Museum professionals have recognized the importance of online endeavors to empower their organization to fulfill it s mission. As the museums online presence becomes a fundamental part of the daily museum activities, it is essential to evaluate and report on online performances. There are many tools and metrics available to measure the impact of online activities. In general, online analytics are used to bring knowledge about how a website is used and to study the behavior of online audiences. For example, online analytics can be used to find out what visitors search for on the website, or which artworks of the online collection are viewed most. However, as participants mentioned, it is not always easy to make sense of analytical numbers that report on online performances. There are so many metrics available that the amount of information can become overwhelming, without a clear focus or plan. Indeed, metrics are most useful if it is well understood what they should stand for. With a clear focus, online analytics can be used for sustainable development of the online presence. Many different models can be used to better understand the value of online analytics. To define the participants analytics situation and approach, this project has used a model developed by analytics expert Stéphane Hamel (2009) This model is called: the Online Analytics Maturity Model (OAMM). The OAMM was applied two times throughout the project, once at the beginning of the research project in March 2013 and once towards the end in September 2013, in order to measure and analyze the participants improvement in online analytics. This chapter further describes the OAMM and the approach that was taken to help participants improve on their online analytics situation. Furthermore, an analysis is made of the OAMM results of March and September Finally, the OAMM outcomes of the project are put in an international context ONLINE ANALYTICS MATURITY MODEL (OAMM) The OAMM is a framework that helps organizations to assess their current analytics situation and provides a structured and actionable path towards improving online analytics. Stéphane Hamel proposes six key process areas to take into account when it comes to being successful online. Each key process area is graded by maturity levels, scaling from 0 (damaged analytics) to 5 (analytically addicted). The OAMM was first introduced in the cultural sector by Tate Gallery, as part of a research project developed by Tijana Tasich and Elena Villaespesa. A paper with the outcomes of this research, as well as the slides presented at the Museums and the Web 2012 conference, are available online. As Tate explains in its paper, the OAMM provided a critical framework to visualize the state of analytics approach. Likewise, for this research project, the model clearly demarcated and reflected on the analytics areas of the participants, especially where they were in need of improvement. To stimulate improvement, the six key process areas of the OAMM were transformed into clear actions. Also, the Online Analytics Maturity Model, clear format. Museum Analytics - Online Analytics Maturity Page 13 of 39

14 project s workshops were tailored to some of the areas. Below, the six key process areas are described, along with the improvement actions that participants were recommended to achieve Management, Governance And Adoption The area Management, Governance and Adoption defines the degree to which online analytics are being spread within an organization. Ideally, members of the management team are accountable for promoting online performance analysis among the organization, in order to ensure that analytics find more solid ground. It would be best if one management member is responsible for understanding correlations between metrics, evaluating online performances against the museum s mission and raising awareness among the organization about the growing importance of online visitors. This member is called the web-analytics champion. Metrics are just numbers and indicators without meaning and significance. (Stéphane Hamel) The issue of spreading analytics throughout the organization was defined as one of the key interests of the ARPNL project. Therefore, the participating museums were recommended to select a web analytics champion within their organization. Most of the individuals that had attended the project workshops appointed themselves as web analytics champion. As a possible first step towards raising awareness about online visitors, the champion was advised to create a simple graph that compared the number of onsite visitors to the number of onsite visits, over the past couple of years. This would be a first step towards raising awareness about the online visitors within the organization, as visualizing the online visitors shows their dimension and makes them less abstract Objectives Online analytics data are easiest to interpret in comparison with objectives. These can be defined in accordance with the museum s mission in an online strategy. By defining objectives, participants were encouraged to look only at the metrics that answered their questions, instead of trying to find the objectives by looking at all the answers available (not an uncommon approach towards online analytics). Few participants had defined an online strategy at the start of the project, as most where in the process of developing one. An effective online strategy does not necessarily have to be very elaborate, but should include goals that take the organizational mission as a starting point. Participants were assigned to define a succinct online strategy by creating a one-page document with at least three goals, referring to their museum s overall mission. Questions raised to help participants define goals were: what is the mission of the museum? How can the internet empower the museum staff and the museum audience to fulfill that mission? Also, several existing museum online strategies and social media strategies were distributed to use as inspiration. Participants most common goals were: Increase online visits and engagement to online content; Increase visits to the museum; Increase newsletter subscriptions; Increase e-ticket sales. We made sure that these goals were in accordance with the SMART criteria: Specific, Measurable, Attainable, Relevant and Timesensitive. Goals defined based on the SMART criteria lead to actionable strategies focused on results. As for these goals, they can all be measured with online analytic tools. Based on the analytic results, clear actions could be defined for improvement. For example: the effectiveness of a campaign aimed at increasing e-ticket sales could easily be measured by comparing e-ticket sale numbers before and after the campaign. Instead of an ad-hoc approach towards increasing e-ticket sales, a possible next step could be defined based on the analytic results. Participants who already developed an online strategy shared it among the group, so feedback could be given and others could use it as a starting point Scope The scope defines the areas of the online strategy where the analysis is focused on. It is not unusual for cultural organizations to have a scattered and undocumented online presence. Therefore it was recommended to start inventorying the current situation for Museum Analytics - Online Analytics Maturity Page 14 of 39

15 those participants who had not already done so. Most participants used a simple spreadsheet to document all the websites their museum had created, along with the social media platforms, blogs, apps and other publishing platforms that were being used online. Some possible fields advised to include in the spreadsheet were: platform name, date of creation, current statistic numbers of the specific platform, name of the person responsible for the platform and the URL. Once the participants online presence was mapped, it was advised to choose a couple of platforms to focus the analysis on. Participants were eager to share their spreadsheets, so other participants who did not have one yet, could use their framework as starting point. Example online presence spreadsheet The Analysis Team And Expertise The area Analytics Team and Expertise describes the degree of analytics knowledge and expertise within an organization. Ideally, each organization has an online analytics expert, who is to advise colleagues on how to translate the organizations mission into web analytics requirements. Analytic numbers can be hard to understand for people, unless they are being put into context by an expert. For many people, the stories behind the numbers may be even more valuable than the numbers itself. For example, the expert could help the curatorial team by creating a report that indicates the most popular webpages of the museum s online collection, as well as suggestions on how to improve the less popular pages. Rather than overwhelming the entire organization with customized weekly or monthly reports, it is more effective if simple action can be derived from insights that the reports may provide. In order to improve on the Analysis Team and Expertise, participants were recommended to ask their colleagues what analytical information they could use in their decision making process Continuous Improvement Process And Analysis Methodology The area Continuous Improvement Process and Analysis Methodology determines the level to which a structured approach to a continuous learning process is developed. Information from analytics brings knowledge and insights that could initiate new questions and possibilities. Therefore, it was recommended to define a strategy that ensures continuous learning. Much analytical data about online exists, that can be useful to test and analyze in the context of each organization. Rather than tackle all possible questions at once, it was advised to focus on one improvement point at a time. Participants were recommended to organize a monthly meeting dedicated to a specific goal and to measure it s progress, in order to improve it. Take for example the goal of increasing e-ticket sales. During such a monthly meeting, the museum staff can discuss how many people actually buy tickets online? Can people find the page on the website easily? Can it be made easier for visitors to buy tickets online? Do online ticket sales benefit the queuing time at the museum, or the workload for colleagues at the cash desk? The answers to these questions subsequently help defining the progress of the specific goal, so new actions for improvement can be initiated Tools, Technology And Data Integration The area Tools, Technology and Data Integration specifies the level to which online analytics are used by an organization. The tools become most valuable when they are put into context and when results are being translated into actions by the web analytics champion. Still, the usage of appropriate tools and technology was an important element of the research project. Google Analytics (GA) is one of the most-applied tools to analyze website statistics. During the research project, most participants had Google Analytics installed and configured, following Seb Chan s GA health checklist. Once GA was properly setup, user segments were installed that made sense for the participant s organizations. Seb Chan recommended a set of basic user segments and advanced user segments. More information about GA can be found in the chapter Website Analytics. Apart from Google Analytics, other tools have been used in the research project in order to measure and evaluate participants online endeavors. These were: Basecamp for online collaboration, the OAMM for defining participants analytical situation and improvement, Museum Analytics for social media analytics and Qualaroo for a qualitative online audience investigation. Of course there are many more tools available and in fact, used by participants. Discussion about the tools participants use on a day-to-day basis was encouraged online. The five tools used most, alongside the ones covered in the research project, were: Facebook Insights, YouTube Analytics, Hootsuite, Twitter Counter and Bit.ly. Museum Analytics - Online Analytics Maturity Page 15 of 39

16 In conclusion, to improve participants online analytics maturity, several actions (as described above), were recommended. The areas Management, Governance and Adoption, Objectives and Tools, and Technology and Data Integration, have been discussed during one of the full day project workshops. Recommendations for other areas can only be realized within the scope of an individual organization. Overall, participants were eager to accomplish the suggested actions and they were willing to help each other as is proven by the information, tips and tools that were shared online OAMM RESULTS Participants OAMM results were collected using Hamel s Online Analytics Maturity Self Assessment Tool. The outcomes of the model were visualized in a graph, that shows at a glance how well balanced an organizations analytics situation is. A balanced OAMM result demonstrates that an organization is distributing it s limited resources as equally as possible to secure a sustainable development of its online presence. Therefore, an ideal model is well balanced across the several key process areas. To be successful, executives must recognize web analytics is more than a reporting system and represents an effective way to identify weak points and improvement opportunities. (Stéphane Hamel) Based on the first OAMM outcomes of March 2013, participants were encouraged to define a strategy for improvement by discussing their organizational strengths and points of improvement during the first workshop of the research project. Based on the final OAMM results of September 2013, an analysis of the analytics improvement was made Before The Research Action Research Project The project started by requesting participants to benchmarking their current online analytics situation, using the OAMM. The result of participants average analytics situation in March 2013 was visualized in the following graph. The project started by requesting participants to benchmark their current online analytics situation, using the OAMM. The result of participants average analytics situation in March 2013 was visualized in the following graph. Participants average benchmark, March 2013 Museum Analytics - Online Analytics Maturity Page 16 of 39

17 The OAMM average of the ARPNL participants showed that some development existed in most areas. However, improvement was to be gained on Management, Governance and Adoption, Objectives and 'Continuous Improvement Process and Analysis Methodology'. This showed that very few participants had analytics spread within their organization, that no clear goals were defined to report on online endeavors and that hardly any participant had a strategy to ensure analytics improvement on the long term. In short, the OAMM result made clear that most participants had no structured approach towards online analytics. Although some of the participants had expertise using analytics, their organization did not necessarily discuss the results of analytic reports on a regular basis. In order to spread analytics among the organization, participants were recommended to setup a monthly meeting for the discussion of analytics. Ideally, several museum staff members to whom the results of online analytics could influence their daily work, would take part in such a meeting. The importance and definition of objectives was dealt with in the first workshop. Most participants have a very broad scope of online platforms on which they are active, often including multiple websites, several social media platforms, video platforms and publishing platforms such as blogs. This broad scope reflects the open and inclusive policies that most museums follow. However, to aim at a broad scope might not necessarily represent an organizational strength. In fact, a too broad scope might be detrimental to the museum online strategy. Instead, it was recommended to try to focus on specific platforms, by which more value can be brought to the online visitors active on these platforms, instead of the online audience in general. The OAMM results were discussed during the first project workshop. Participants decided to further develop their continuous improvement process and to better define online objectives. Most participants agreed that the scope area was too broad although it was not completely clear how the scope could become more focused without affecting the museum vision of openness and integration. Overall, the aim was to move towards a more balanced model where resources were equally distributed among all areas September 2013: After The Project Research During the evaluation phase of the project, participants were requested to answer the same OAMM once again. The rational behind this approach was to be able to compare the online analytics improvement throughout with the research project. The average OAMM of all participants showed the progress from March to September 2013 and was visualized in the graph below. Participants progress from March to September, 2013 Museum Analytics - Online Analytics Maturity Page 17 of 39