1 On-line Questionnaires as Lightweight Tools to Extract Collective Medical Knowledge from Daily Practice. Federico Cabitza and Carla Simone Università degli Studi di Milano-Bicocca Viale Sarca 336 Milano, Italy ABSTRACT The paper starts with a critical view of the technologies currently proposed to manage clinical knowledge. The paper highlights two incommensurable approaches and takes a position in favor of the second one. The first approach is based on (complete) models of an objective clinical knowledge; the second one takes a bottom-up perspective in knowledge creation by appreciating the local and contextualized experience of medical care. From a more general perspective, the paper advocates a deeper discussion on the potentially disruptive role of ICT in determining the next ruling medical cosmology. In particular, our approach focuses on Electronic Voting Systems (EVS), which are proposed as a sufficiently mature technology to contribute in defining the contours and boundaries of collective bodies of knowledge in the domain of community-centered research, i.e., in ambits where the direct observation of every member is unfeasible and the opinion of heads-of-household cannot be representative of the ground floor of the community. The paper illustrates how EVS will be put to work in the domain of trans-hospital organizations to check for the validity of this class of technologies to increase the emergence of collective best practices and consensus-based evidences in similar domains. 1. ICT AND MEDICAL KNOWLEDGE: HORSE AND CARRIAGE? When they were first introduced, at the beginning of the 20th century, medical records were sort of ward notes or registers used by doctors as cognitive aids and tools to manage the complexity of alternating work shifts . Although most of today s practitioners could see their paper-based records playing but the same role (because they do play that role), in the sixties Lawrence L. Weed, a physician, was the first to describe an Electronic Medical Record (EMR) in medical literature; he advocated its diffusion for mainly the same reason why it was and often still is disparaged by the majority of colleagues as incompatible with their office practices: not much to reach the well known advantages of electronic documents over paper-based ones (e.g., distributed access to data, reduced cost of storage space, easiness of data aggregation for statistical goals); advantages that in the healthcare domain seem to be often surpassed by the advantages of paper (i.e., flexibility, portability and its non-disruptive nature ). Rather, Weed, a pioneer in several fields, advocated the advent and success of EMRs for nothing less than changing the clinical practice and making it more ordered, less chaotical; and this, not just for the sake of good order. In his vision, this order would have enabled the ordered and meaningful accumulation of physicians annotations over time towards a better and more effective way to build and share clinical knowledge from situated experiences and hence, to improve educational methods. This means that from the very foundation of health informatics, collaborative practices of record keeping and of knowledge production have been envisioned as strictly intertwined ; since then on, moreover, ICTs and medical knowledge have initiated a profound alliance and reached a long-lasting community of interests and aims. Simplistically put, computer systems have been used for decades to create medical knowledge, by analyzing large amounts of data and looking for complex patterns within data to either suggest doctors unexpected associations, facilitate the validation of existing theories or assist the design of new drugs . Moreover, in the thriving field of decision support systems, researchers have at length focused on how doctors can be supported in exploiting medical knowledge  and to this aim they have tried to understand how doctors think 1  and how that knowledge can be systematically formalized in computable constructs. When communication technologies became popular, knowledge sharing became a basic aspect of knowledge management. Actually, if we consider the creation of medical knowledge as a chiefly human, social and collective activity, we can also see how knowledge production blurs with the processes of knowledge promotion and dissemination and therefore with the use of ICT to share and use knowledge sources (e.g., scientific papers). Specifically, in regard to the so called Evidence-Based Medicine (EBM) that is the quite 1 It is no wonder that health informatics is also defined as the rational study of the way [doctors] think about patients, and the way that treatments are defined, selected and evolved. It is the study of how medical knowledge is created, shaped, shared and applied .
2 recent cultural phenomenon that drives doctors to apply the best available evidence gained from the scientific method to their decision making the role of the Internet is deemed so central (e.g., in enabling the coordination of complex clinical trials, moving large quantities of data in almost no time and making the resulting evidences available in any corner of the world in almost no cost) that some authors have even claimed that without one, the other would not exist [7, 8]. 2. COMPLEMENTARITY OR INCOMMEN- SURABLE DIFFERENCES? Following the above technological trend, health care organizations (HCOs) are progressively adopting the vision of ICT that has become very popular in the organization domain , that is the vision of a technology that can be profitably aimed at two, apparently complementary objectives . On the one hand, to help individuals capture, share and convert their collective knowledge into optimal decisions in real time. This goal drives HCOs towards investing in Artificial Intelligence (AI) initiatives aiming to formalize clinical evidences and local constraints in order to embed diagnostic algorithms, treatment guidelines, and locally optimized care processes in computer-based tools of daily use, like triage systems, order entry systems and electronic patient records (see e.g. ). On the other hand, to improve organizational performance by an effective management of the medical knowledge derived from research findings. This goal inspires Knowledge Management (KM) initiatives in the context of HCOs, where healthcare and ICT practitioners collaborate to define effective tools for supporting communication between all the actors involved in patient care , so that doctors can externalize, combine and socialize clinical facts, learnt lessons and best practices  These approaches are often intended as complementing each other. But in this way, their profound divergence is neglected, or worse yet, entirely missed. In fact, AI initiatives assume that there exists an objective medical knowledge that is based on scientific evidences, which are so much immutable and true that they can be converted in computable algorithms and embedded in the tools doctors use, in order to inform (and strengthen) the appropriateness of clinical practice. Conversely, KM initiatives are not seeking to construct a complete model of the content of human communication as the basis of any technological support. On the contrary, they accept a vision of medical knowledge as of something that is essentially narrative in nature and situated that is produced by human professionals who are constrained by their social/historical conditions and by the exigencies of the modes of production that institutionalize and reinforce the power of these professionals . Differently from AI technologies, which spread and consolidate a knowledge based on the power of big numbers, KM technologies support intra-organizational and intra-community communication by letting practitioners discuss their clinical cases and make a reference for routinary tasks out of their (and others ) accounts. In doing so, these technologies allow for the progressive stratification of a procedural and empirical knowledge that is still based on the single patients, their individual histories and on how doctors recollect and represent them. These technologies also reflect and amplify what Jewson termed the object orientated cosmology in his classic work on the historical development and production of medical knowledge ; that is, they contribute in consolidating the ruling medical culture where hospitals (from being shelters for the needy) have become exclusive training centers for the new profession of medicine and influential sites for scientific research. Conversely, AI technologies back and accelerate the transformation between this medical cosmology towards that so called laboratory-medicine, where knowledge has been created by looking for physio-chemical invariants in large samples of patient populations with statistically reliable methods, which make disease processes of patients and risk evaluators of doctors (or, as a head doctor told us nomogram readers ). 3. MORE ABOUT POLITICAL ROLE OF ICT Nowadays, we are witnessing the predominance of two cosmologies : on the one hand, clinical knowledge is being created somewhere else than in single clinical wards but according to outcomes reported from several ones, in the name of a scientific objectivity (cf. the laboratory-medicine cosmology ); and, in the meantime, decisional power is alleged to be slowly returning to the hands of patients (cf. the person-orientated cosmology 2 ) in the name of a better healthcare/service intended for more satisfied patients/ customers. Since power and knowledge, as Foucault pointed out , are inevitably and inextricably intertwined, we believe that ICT can play an important role in tipping the scales in favor of one healthcare model against the other, and of one of the three cosmologies mentioned above (ruled by the patient, doctor and laboratory, respectively). Thus, we, as computer scientists and designers, have to face with two basic alternatives, that have profoundly different political meanings: either we want to contribute to an ICT that acts as a resource to (quite surreptitiously) consolidate and impose scientific evidences; or we want to contribute to an ICT that gives voice and resonance to local positions in the global medical debate where scientific evidences are challenged and doctors lay claims for contingent criteria for appropriateness. In the first case, ICT would be based on the fact that the evidences have been allowed to emerge in impacted and influential literature also thanks to well-targeted funds and human authors and reviewers that found them worth of publication; In the second case, the emphasis would more on the effectiveness of patient care than on the research and management objectives, although it might be the case that some treatments are chosen because more profitable than others and that the advantages of long-pursued standardization in care( comparability of outcomes, cost control and quality monitoring) could be potentially undermined. To try to deal with this issue in our small way, we directed our attention on the relationship between doctors and official medical knowledge. Quite surprisingly (or maybe not), when 160 Cochrane reviews (i.e., one of the most reliable sources for medical evidences) were systematically surveyed, Ezzo et al. found that almost 20% of the interventions applied by doctors worldwide when put under scientific investigation came to be ineffective ( evidence of no effect ) and 21% were not backed by sufficient scientific evidence ( no evidence ). In other words, the survey shows that doctors are often more collectively 2 This cosmology was mentioned by Jewson as the first one, chronologically speaking.
3 intelligent than scientific research can ratify since there is a whole underground practice that doctors do not (or better yet, can not) publish on influential journals and do not get funded by the big pharma. Moreover, the study highlights the fact that doctors prefer to intervene on the body of patients on the basis of tradition, school-membership, long-practiced techniques, lessons learnt from colleagues and good habits (and of the patient s history and condition, of course!), rather than on the basis of evidences coming from clinical trials performed at a thousand miles away or just by some possibly competing colleagues. This is an interesting point: if Collective Intelligence (CI) refers to a collective pool of social knowledge that is fostered by the increasing number of human interactions made possible by ICT, the traditional and institutional modes of medical knowledge ratification could be to a large extent substantially blind to this relevant phenomenon that instead deserves an adequate technology support. 4. LEVERAGING A BOTTOM-UP APPROACH TO KNOWLEDGE The aim to identify a suitable support to CI in medicine, led us to formulate two close research questions. The first is about the medical knowledge content. The second one takes into account that doctors and surgeons who are keenly interested in a particular sub-speciality, disease or surgery technique (e.g., arthroscopy) often affiliate with collective organizations, associations and societies, both at national and international level: their main aims are improving education, research and the clinical exchange between members from various regions and countries in that particular field. Then the two questions are: first, how could ICT be used to i) facilitate the emergence of medical knowledge from the single wards and surgeries where it is at least put to the test of experience; ii) identify bottom-up consensus on best practices while being sure to give voice to every chorister; and iii) get a picture of the extent evidences from the scientific literature have been internalized and included in the daily practice. Second, how could ICT improve the participation of members of an organization, especially when it is highly fragmented in several countries, and how could it foster the internal debate, not only about common initiatives but also towards the externalization and socialization of clinical knowledge and the sharing of relevant experiences and cases, as it usually only happens in (expensive) co-located and periodical workshops and conferences? In other words, is there such a technology that can promote factual clinical knowledge exchange between members of an organization and therefore help organizations understand how much they are real communities of practice (i.e., organizations whose members share practices and co-define them), rather than mere communities of interest (as every medical society certainly is)? Before trying to find an answer to these questions, it is appropriate to better characterize the healthcare domain. Differently from other domains, here there is a stronger connection between research and practice, irrespectively of any of the previous political positions one may take. In fact, in medicine every kind of investigation is necessarily based on some practices around the patient, since virtual experiments are (still) not possible and who needs care cannot be either simulated or replicated. It is no wonder that the branch of AI that is closer to capturing experience, i.e. Case Based Reasoning, is proposed as a compromise between situatedness and standardization. In fact, CBR researchers recognize that to record and share experiences is useful: once again, however, this collected and shared memory is based on a model of the target domain that is pretentiously intended to be exhaustive while experience is captured by an increasing number of cases that are made comparable according to a predefined set of features . The issue of memory is critical in supporting a community of practice, since the strength and cohesion of a community can be assessed on the basis of the effort that members are willing to pay to share knowledge, i.e., in the continuous accumulation, maintenance and fruition of the related experiences and memories . The challenge is then to combine flexibility and structure so that both maintenance and fruition are sustainable: this is another requirement in the game between CI and ICT, beyond sheer communication. The strict relation between research and practice links every doctor in an international network where all doctors (included those who are not directly involved in clinical research) share the need to keep what they know updated while, at the same time, this same network is the arena of their scientific competition and quest for better treatments. This is in general not true for other disciplines where industrial research is more loosely coupled with the academic one, and the two environments keep separated their own arenas of competition. Thus, in the two professional environments the competition game is played and the network of actors is maintained by applying different rules. If we consider the KM approach described above, there are several technologies that can be used to support knowledge sharing within a group or community: e.g., mailing lists, question-answer forums and traditional forums, document on-line publication, and wikis of course, where glossaries, procedures, relevant cases and guidelines are externalized and socialized. These contents are collected in different kinds of repositories where techniques based on agreed keywords, social tagging and the like can be put under the control of end-users and can support this memory management in a lightweight way, i.e., without a reference to an ontology that is imposed from above. That notwithstanding, all these technologies tend to fall short with respect to two important aspects. First of all, efficiency in participation: that is the ratio between participation rates and the incentives spent (or compromises made) to get those rates. In other words, it is always the same old ones who participate in public discussions or wiki drafting on a particular subject, while the others either because of reserved temperament, because not interested in the topic or willing to share what they know do not easily externalize their opinion/knowledge (even if they possess it). Second, efficiency in knowledge extraction. Even by assuming that people like to share what they know about a particular subject in the middle of their conversations or narratives, it can be difficult to distill usable and generalizable knowledge from those ad-hoc interactions, either because positions are not clearly stated nor problems, or because there is no way to reconcile diverging statements. In other words, these technologies interpret the memory as a repository where to record all what has
4 been said and where the only structure provided to support fruition is often based on more or less predefined sets of keywords or categories supporting information retrieval and aggregation. For these reasons, to address the two intertwined questions mentioned above (that is, the two concerns of the emergence of hidden/tacit knowledge from a community and the fostering of participation of their members, respectively), we started focusing on collaborative technologies that allow for single contributions from the members of a group to be aggregated into a collective expression of the group itself. In other words, we focused on electronic vote systems (EVS) that allow users to express a preference on the basis of some personal knowledge (according to some ballot form ), and that aggregate those expressions into a collective expression of knowledge (according to some tally method). In particular, we are focusing on Borda-count EVSs based on Computer-Assisted Web Interviews (CAWI ) that allow users to rank alternative choices on an ordinal scale (e.g., Likert scale) when facing the same question, problem or clinical case. These systems do not force respondents to provide an explicit preference order between alternatives (e.g., all options could receive low grades) and allow for the aggregation of alternatives on the basis of general consensus, rather than on a majoritarian basis. Although academic interest in paper-based vote systems has a long and rich history (vote systems, and the intrinsic concept of suffrage, are at the basis of western democracies), the use of EVS in the context of Web 2.0 technologies is still in their infancy  and their potential in collective intelligence extraction still to investigate. In regard to the shortcomings mentioned above, voting systems can have a role in increasing relevant feedback from as many people as possible while relying on quite simple and lightweight tools. Being light in this ambit is important to avoid known problems of technology appropriation by laymen users: to this respect, electronic voting systems should be plain and straightforward in both scope and aims as their historical and paper-based counterparts have always been. In regard to the efficiency by which knowledge is collected, EVSs require survey designers to formulate questions to ask and statement to assess as clearly and unambiguously as possible, while respondents are asked to choose between a small set of (hopefully) exhaustive alternatives (in very specific and technical domains, like medical specialties, this could be feasible, at least in theory!). In this way, a set of good answers (or at least, the ones appreciated by the majority) can be easily detected, whereas some focused open-choice question can allow for the emergence of new options and original knowledge nuggets. Also the knowledge externalization is facilitated by the structure of the survey and its aggregated outcomes: the reasoning involved in answering the questions constitutes the background for their subsequent retrieval and interpretation, and for an additional reflection on possibly contrasting or incoherent outcomes. In regard to efficiency in participation, EVSs are point-to-point systems that can reach each single member of a community provided that the channel is effective in reaching the potential respondent; their anonymity (or better yet, the perception of the certainty of anonymity they can convey) can facilitate the emergence of opinions even from the most diffident and shy members, while the relatively low social and cognitive effort requested in answering the closed-option questions can be easily repaid by the curiosity of either getting the social ranking of one s personal opinion or learning about the distribution of rankings within the sample of respondents. Obviously we are aware that no EVS could be totally exempt from the shortcomings of every vote systems: e.g., low participation rate, respondents ability to understand either the question or the answers, as well as her benevolence, wariness and sincerity in responding; moreover, that no voting system can guarantee to transfer individual positions/opinions into a single collective preference in a totally fair way , although this could not be a real problem for knowledge emergence/sharing as it is for political voting. Rather, our point is that EVSs could be a feasible and cost-effective tool to break the ice in more articulated and multi-step initiatives both for the extraction of collective intelligence from a loosely distributed organization of experts (question one) and for the identification of wellcircumscribed discussion cases around which to trigger public and possibly moderated debate when either diverging solutions clearly emerge, or no alternative solution really does (question two). In this point, our main assumptions are that in a specialty-driven association/organization: i) members are experts, i.e., they know how to cope with a clinical case; ii) they are motivated to have their own way to cope with a clinical case emerge (or just be socially confirmed) as appropriate; iii) they are also motivated to know what their peers and colleagues deem as an appropriate conduct/technique, under strict anonymity and iv) they are interested in making potential divergences in treatments of choice and attitudes toward clinical appropriateness specific opportunities for further debate and discussion. 5. GIVING EVERY DOCTOR A VOTE To verify the soundness of our point, we have started a collaboration with a group of doctors and surgeons involved in national and international associations in the field of knee / shoulder surgery. We proposed them to administer a short on-line questionnaire to the usually silent ground floor of their members to get a picture of the collective and otherwise hidden knowledge that characterizes their own practices (and the related ability to exploit this knowledge, i.e., intelligence). In this questionnaire we plan to challenge respondents with a set of well-defined and briefly described clinical cases, each associated with a limited set of alternative options (in terms of treatments, interventions and the like). Some cases will be characterized to look as hot cases, i.e., cases that are relevant for their centrality in current debates on the most appropriate interventions in the field. On the other hand, some other cases, shuffled with the former ones, will be characterized by sets of perfectly adequate alternatives among which, quite surreptitiously, also treatments indicated as evidence-based in recent controlled and randomized studies in the specialist literature will be proposed. While answers provided for the hot cases are mainly aimed at creating an informal occasion of provocative idea exchange, the evidence-based cases are conceived to assess the degree of inclusion of scientific evidences into daily practice. In any case, the survey should enable us to detect possible polarizations upon particular choices and al-
5 ternatives among the respondents and, complementarily, to identify grey areas in practice-related knowledge, i.e., cases where the community members are torn between alternative interventions and their uncertainty is greater 3. The emergence of these grey areas could also provide some useful insight on the reliability of a class of new services generally called Medical Second Opinion (MSO 4 ) that are recently drawing an increasing interest (and funding) from both the public opinion and HCOs due to the improvements in quality and confidentiality guaranteed by modern ICTs. 6. SUMMING THINGS UP The paper proposes a critical view of the technologies currently proposed to manage clinical knowledge. This analysis sheds some light on two incommensurable approaches: the first one is based on (complete) models of an objective clinical knowledge; the second one takes a bottom-up perspective in knowledge creation by appreciating the local and contextualized experiences of doctors in care provision. The paper takes a position in favor of the second approach but, at the same time, it criticizes the adoption of technologies that are not intended to support the construction of a community memory and the natural and straightforward retrieval of this collective knowledge, as instead do the information technologies that mainly focus on the accumulation and management of narratives on different media and ways. The challenge posed by this approach to medical knowledge management is to find a way to offer to the users (i.e., to the doctors in the considered domain) pieces of information that are meaningful and structured enough to convey and evoke the experiential knowledge they need to make sense of a given context [13, 18]. From a more general perspective, the paper advocates a deeper discussion among the ICT researchers involved in the healthcare domain on the potentially disruptive role of ICT in determining the next ruling medical cosmology, and in particular it focuses on electronic voting systems and online questionnaires. On the one hand, these technologies can support object orientated cosmology by giving voice and resonance to medical positions that in other times would have hardly emerged from the single consulting rooms and doctor surgeries, up to contributing in shaping a new unspecified and multiform collective intelligence; on the other hand, these technologies can contribute to the patient orientated cosmology, which is nowadays often articulated in terms of patient empowerment, by providing patients, as customers of healthcare services, the ability to rank doctors and their approaches of choice according to subjectively assessed medical outcomes and interaction qualities (e.g., propensity to listening, politeness, clarity). These two cos- 3 or cases where the respondents decision would depend more heavily on elements that could not be reported in, and inferred from neither, the concise textual descriptions of the cases. This could be an occasion to investigate the importance of unwritten or tacit context in medical decisions making. 4 MSO services can be either telemedicine- or web-based services where either doctors or patients seek further clinical opinions by single experts (or pools of anonymous experts) to support the diagnosis or treatment of a particular case, usually on the basis of previous (textual) reports, records and diagnostic imaging that is digitally available for that case. mologies can balance the role of the third one by opening new spaces for the definition of quality in the healthcare domain. In sum, we believe that on-line questionnaires and computer-assisted interviews have reached a sufficient degree of technological maturity: for instance, modern on-line platforms allow to minimize (or at least reduce) question- and answer-order bias, fatigue bias (e.g., with dynamic routing of questions according to run-time answers or by funnel sequencing), non-response bias (e.g., by making a limited set of answers mandatory), acquiescence bias (e.g., by guaranteeing the perception of anonymity) and other non-sampling errors (e.g., Hawthorne effect). These features make the use of these tools suitable in order to contribute in, at least, defining the contours and boundaries of collective bodies of knowledge in the domain of community-centered research, i.e. in domains where either direct observation of every member of a community is unfeasible, or the opinion of the heads-of-household (whoever they are) cannot be representative of the ground floor of the community. In our future work in the domain of trans-hospital organizations, we will report the main findings on this kind of approach in the context of medical knowledge and we will also interpret these findings as part of an explorative study that provides preliminary proof-of-concept for the validity of this class of technologies in increasing the emergence of collective best practices and consensus-based evidences in similar domains. 7. REFERENCES  Berg, M., Winthereik, B.: Waiting for Godot: episodes from the history of patient records. In: Health Information Management: integrating information and communication technology in healthcare work. Routledge, London, UK (2003)  Harper, R.H.R., O Hara, K.P.A., Sellen, A.J., Duthie, D.J.R.: Toward the paperless hospital? a case study of document use by anaesthetists. British Journal of Anaesthesia 78 (1997)  Weed, L.L.: Knowledge Coupling: New Premises and New Tools for Medical Care and Education. Springer-Verlag New York (1991)  Coiera, E.: A Guide to Health Informatics. II edn. Hodder Arnold (2003)  Weed, L.L.: New connections between medical knowledge and patient care. British Medical Journal 315 (1997)  Groopman, J.: How Doctors Think. Houghton Mifflin (2007)  Shigekoto, K.: Internet: an infrastructure of evidence-based medicine in neurology. Rinsho Shinkeigaku 42(11) (2002)  Kvist, M., Kidd, M.: The role of new information and communication technologies in primary care. In: Primary care in the driver s seat? Open University Press (2006)  Stefanelli, M.: The socio-organizational age of artificial intelligence in medicine. Artificial Intelligence in Medicine 23(1) (2001)  Patel, V., Shortliffe, E., Stefanelli, M., Szolovits, P., Berthold, M., Bellazzi, R., Abu-Hanna, A.: The coming of age of artificial intelligence in medicine artificial intelligence in medicine. Artificial Intelligence in Medicine 46(1) (2009) 5 17
6  Panzarasa, S., Maddè, S., Quaglini, S., Pistarini, C., Stefanelli, M.: Evidence-based careflow management systems: the case of post-stroke rehabilitation. Journal of Biomedical Informatics 35(2) (2002)  Abidi, S.: Knowledge management in healthcare: towards knowledge-driven decision-support services. International Journal of Medical Informatics 63 (2001) 5 18  Nonaka, I., Takeuchi, H.: The Knowledge Creating Company. Oxford University Press, Oxford, UK (1995)  McLean, A.: Contradictions in the social production of clinical knowledge: the case of schizophrenia. Soc Sci Med. 30(9) (1990)  Jewson, N.: The disappearance of the sick-man from medical cosmology, Sociology (10) (1976) Reprinted Int J Epidemiol 2009;38:  Foucault, M.: The History of Sexuality. Volume 1. Harmondsworth, Penguin (1981)  Holt, A., Bichindaritz, I., Schmidt, R., Perner, P.: Medical applications in case-based reasoning. The Knowledge Engineering Review 20 (2005) Cambridge University Press.  F. Cabitza, C. Simone, and M. Sarini. Knowledge artifacts as bridges between theory and practice: the clinical pathway case. In Knowledge Management In Action, IFIP 270/2008:37-50, Springer, 2008  Watkins, J.H., Rodriguez, M.A.: A Survey of Web-based Collective Decision Making Systems. In: Studies in Computational Intelligence: Evolution of the Web in Artificial Intelligence Environments. Springer-Verlag (2008)  Manfreda, K.L., Batagelj, Z., Vehovar, V.: Design of web survey questionnaires: Three basic experiments. Journal of Computer Mediated Communication 7(3) (2002)  Arrow, K.: Social Choice and Individual Values. John Wiley and Sons, Chichester (1963)