I- _ US 2009017 1 902A (19) United States (12) Patent Appication Pubication (10) Pub. No.: US 2009/0171902 A1 MacLaurin et a. (43) Pub. Date: J 11]. 2, 2009 (54) LIFE RECORDER (22) Fied: Dec. 28, 2007 (75) Inventors: Matthew B. MacLaurin, Pubication Cassi?cation zizzsaizewa?imzssxt.. ' 61> G06F 17/10 (2006.01) Bathche, Kirkand, WA (US); G 06 F 1 7/30 (2006 01) Boyd Cannon Muterer, Redmond, ' WA (US); John Mark Mier, (52) US. C...... 707/3; 707/04.; 707/E7.009 Kirkand, WA (US); Bret P. O Rourke, Kirkand, WA (U S) (57) ABSTRACT Correspondence Address: A system that can automaticay capture ife experiences of a TUROCY & WATSON, LLP user across a number of senses or perceptions is provided. 127 Pubic square 57th Foor Key Towe. Once the data is captured, it can be annotated and saved for CLEVELAND, Oi 44114 (Us ) subsequent payback. The innovation aso enabes the data to be synchronized to for payback, for exampe, audio can be (73) Assignee; MICROSOFT CORPORATION, time-synced to a corresponding video With a corresponding Redmond, WA (Us) sme, etc. Sti further, the innovation provides for contros that enabe a user to adjust or seect granuarity for capture as (21) App. No.: 11/966,921 We as payback. [ 00 1Q8_1. I "[118" PERCEPTION1 : PERCEPTIONM : I Q I I : I ' : 110 _i\ EXPERIENCEI \ \ J / \\ /1 EXPERIENCE1 4-110 : \ \ \ / / : \ \ / / \ > ' I I / I \ \ \ I : / \ \ / / \ \ 110 _\ EXPERIENCEX L, / \ \ J EXPERIENCEP /: 110 / x I 1 J : EXPERIENCE MANAGEMENT : SYSTEM : I : EXPERIENCE MONITOR 104 I /' i : COMPONENT :/_ 102 M : v : : EXPERIENCE CAPTURE 106 I /_ : COMPONENT :
Patent Appication Pubication Ju. 2, 2009 Sheet 1 0f 12 US 2009/0171902 A1 f 100 1Q8_-_\ q I z:0_8 q : PERCEPTION1 : PERCEPTIONM : I a I u \ \ J / no _i\ EXPERIENCE1 : \\ I, EXPERIENCE1 4 H0 : \\,/ i I \ \ / / I g I I ;> 1. / \ g / I \ \ / \ \ 1 10 _i\ EXPERIENCEN L, / \ \r EXPERIENCEP /: 1 10 I \ I 1 J I- _ : EXPERIENCE MANAGEMENT : : SYSTEM : : EXPERIENCE MONITOR 1 04 I /_ i : COMPONENT :/_ 102 I EXPERIENCE 1 06 : CAPTURE /' i : COMPONENT : I
Patent Appication Pubication Ju. 2, 2009 Sheet 2 0f 12 US 2009/0171902 A1 MONITOR EXPERIENCE(S) /' 202 I CAPTURE DATA RELATED TO / 204 EXPERIENCE I DETERMINE / 206 PERCEPTION TYPE I ANNOTATE /- 208 EXPERIENCE DATA STORE /- 210 ANNOTATED DATA V I STOP ) FIG. 2
Patent Appication Pubication Ju. 2, 2009 Sheet 3 0f 12 US 2009/0171902 A1 ( START ) GENERATE QUERY /_ 302 V SEARCH CAPTURED DATA / 304 VIA ANNOTATIONS RETREIVE CAPTURED DATA /_ 306 SYNCHRONIZE DATA /- 308 RENDER / 310 SYNCHRONIZED DATA FIG. 3
Patent Appication Pubication Ju. 2, 2009 Sheet 4 0f 12 US 2009/0171902 A1 EXPERIENCE MONITOR COMPONENT : PERCEPTION1 I 402 I SENSING }/ } COMPONENT I /- 104 I PERCEPTIONM I SENSING }/ 402 @ I C9P_P9_NP I : :0 CAPTURE > I CONTEXT SENSING }/-4o4 COMPONENT I I ANNOTATION {/- 406 I COMPONENT I FIG. 4
Patent Appication Pubication Ju. 2, 2009 Sheet 5 0f 12 US 2009/0171902 A1 EXPERIENCE MONITOR COMPONENT I PERCEPTION _ u: I SENSING I COMPONENT I 502 : - - - - --i N PHYSIOLOGICAL I I 402 I SENSOR(S) : If I 506 I \NENVIRONMENTALI I I : SENSOR(S) I : : L J / 104 EXPERIENCE S) I I TO CAPTURE I CONTEXT SENSING : : COMPONENT I - - - : \ PHYSIOLOGICAL I I : SENSOR(S) I (-404 L 508 I - \ENVIRONMENTALI I I : SENSOR(S) I i : I. J :- < OTHER I: DATA SOURCES I ANNOTATION COMPONENT :/ 406: L FIG. 5
Patent Appication Pubication Ju. 2, 2009 Sheet 6 0f 12 US 2009/0171902 A1 \J M woo IVIK MmmD HZWEZOMT/ZM FXmELZOU w.05 M M A M mumdommm FZWMMDU m0 mi/em 02:2 HZmmmDE mamome mmvzié >HHMDOmm WEELEF<Q
Patent Appication Pubication Ju. 2, 2009 Sheet 7 0f 12 US 2009/0171902 A1 EXPERIENCE CAPTURE COMPONENT /- 106 : EXPERIENCE I 702 NDEXING IV ' FROM MONITOR E 91%)??? _I TO STORE > : EXPERIENCE I 704 RECORDER If } COMPONENT \ FIG. 7
Patent Appication Pubication Ju. 2, 2009 Sheet 8 0f 12 US 2009/0171902 A1 EXPERIENCE DATA RENDERING MANAGEMENT COMPONENT I QUERY COMPONENT I / 802 I CONTENT I I DEVICE I. CONFIGURATION I. CONFIGURATION I I COMPONENT I I COMPONENT I FIG. 8
Patent Appication Pubication Ju. 2, 2009 Sheet 9 0f 12 US 2009/0171902 A1 CONTENT CONFIGURATION COMPONENT /- 806 : SYNCRONIZATION COMPONENT : : GRANULARITY. SELECTION : { COMPONENT I I- FIG. 9
Patent Appication Pubication Ju. 2, 2009 Sheet 10 0f 12 US 2009/0171902 A1 SYSTEM EXPERIENCE > MONITOR COMPONENT EXPERIENCE CAPTURE COMPONENT MACHINE LEARNING» AND REASONING /' COMPONENT OOZI FIG. 10
Patent Appication Pubication Ju. 2, 2009 Sheet 11 0f 12 US 2009/0171902 A1 f 1100 / 1102 PROCESSING / 1104 i OPERATING SYSTEM : UNIT i f 1132 1108 f- 1106 i 5 APPLICATIONS i SYSTEM MEMORY/_ 1 1 [2 ROM m NTERNAL-IDD _1116 \\\ I_]1_)12_"J I I 1126 EDD /- 1118 w DISK / g /- 1120 /' 1144 m 1128 OPTICAL MONITOR _ / 1146 DRIVE,_1122 \ DISK / F1138 VIDEO KEYBOARD ' ADAPTER f- 1140 / /_ 1142(WIRED/WIRELESS) MOUSE 515515512 [-1158 r1154 r 1148 INTERFACE \ MODEM WAN 1 \ REMOTE COMPUTER(S) / 1156 /- 1152 4 NETWORK 1 LAN 1 S 1150 ADAPTER (WIRED/WIRELESS) - MEMORY/ STORAGE FG.11
Patent Appication Pubication Ju. 2, 2009 Sheet 12 0f 12 US 2009/0171902 A1 1206 K- 1200 1202 1204 \ CLIENT(S) cog/x?giggggb SERVER(S) f A L 1208 1210 V V CLIENT DATA STORE(S) SERVER DATA STQRE(S) FIG. 12
US 2009/0171902 A1 Ju. 2, 2009 LIFE RECORDER BACKGROUND [0001] It is not uncommon for individuas to take photo graphs to create a asting memory of an event. For exampe, photographs captured during a vacation can be viewed to reminisce about a time away in an exotic ocation, a visit With friends, etc. These images can be shared With friends to visu ay te a story of an event that may have occurred during the vacation, for exampe. [0002] Recent deveopments have been directed to optica systems that can be Worn around a user s neck to capture a sequence of sti images reated to events of an individua. HoWever, conventiona systems have been imited to the cap ture of sti or visua images. In other Words, these emerging systems are merey optica devices that can be manuay triggered or aternativey triggered by changes in factors such as ighting, temperature or movement. For instance, if a user Wears one of these traditiona devices, upon eaving (or enter ing) a buiding, the system can detect a change in ambient ighting and thereby prompt the capture of a series of images. These images can ater be used to visuay recreate a user s experience. [0003] For exampe, a user can review images days, Weeks, or months ater to determine a brand of Wine they shared With a friend at dinner. Or, in another exampe, a oved one can review the images to determine activities competed by the edery. Essentiay, sti images of traditiona systems can be reviewed by the Wearer, a oved one or even a heath care professiona to reca, share, or anayze activities of a Wearer. SUMMARY [0004] The foowing presents a simpi?ed summary of the innovation in order to provide a basic understanding of some aspects of the innovation. This summary is not an extensive overview of the innovation. It is not intended to identify key/critica eements of the innovation or to deineate the scope of the innovation. Its soe purpose is to present some concepts of the innovation in a simpi?ed form as a preude to the more detaied description that is presented ater. [0005] The innovation discosed and caimed herein, in one aspect thereof, comprises a system that can capture ife expe riences of a user across a number of senses or perceptions. Once the data is captured, it can be annotated and saved for subsequent payback. Additionay, the speci?cation enabes the data to be synchronized for payback, for exampe, audio can be time- synced to a corresponding video, etc. Sti further, the speci?cation provides for contros that enabe a user to adjust or seect granuarity for data capture as We as pay back. [0006] The abiity to capture ife-reated events enabes repay to better and more fuy comprehend the event. Es sen tiay, the innovation faciitates a repay simiar to those often empoyed in viewing of a sporting event via teevision (e.g., instant repay ). In other Words, if a user Woud ike to re-ive, share or otherwise view a past event, the innovation makes this possibe. Particuary, the innovation can enabe the event to be repayed by synchronizing data captured across a number of perspectives or senses. [0007] In aspects, the innovation provides adjustments and contros that aow different granuarities to be preserved if desired. For instance, events Within the past few hours can be saved at high granuarity Whie events that occurred further back in time can be viewed at coarser granuarities if desired. For exampe, yesterday s events may be viewed one frame at a time at 30 second intervas Where today s events can be viewed in rea time. These granuarities can be automaticay set, preset or inferred on-the-?y based upon factors such as context, content, etc. [0008] The information can be stored in most any format. Additionay, the captured information can be retrieved based upon a query or other search request. Stored memories can be empoyed for ife modeing and can be used in such aspects as ocation gaming, changing or atering one s perception based on the past experiences, utiizing augmented reaities based on the recorded experiences, and even substituting aternative reaities, if desired. [0009] In yet another aspect thereof, machine earning and reasoning component is provided that empoys a probabiistic and/or statistica-based anaysis to prognose or infer an action that a user desires to be automaticay performed. [0010] To the accompishment of the foregoing and reated ends, certain iustrative aspects of the innovation are described herein in connection With the foowing description and the annexed drawings. These aspects are indicative, how ever, of but a few of the various Ways in Which the principes of the innovation can be empoyed and the subject innovation is intended to incude a such aspects and their equivaents. Other advantages and nove features of the innovation Wi become apparent from the foowing detaied description of the innovation When considered in conjunction With the draw 1ngs. BRIEF DESCRIPTION OF THE DRAWINGS [0011] FIG. 1 iustrates a system that faciitates ife record ing in accordance With aspects of the innovation. [0012] FIG. 2 iustrates an exampe How chart of proce dures that faciitate capturing, annotating and storing ife event reated in accordance With an aspect of the innovation. [0013] FIG. 3 iustrates an exampe How chart of proce dures that faciitate searching, synchronizing and rendering captured data in accordance With aspects of the innovation. [0014] FIG. 4 iustrates an exampe bock diagram of an experience monitor component in accordance With an aspect of the innovation. [0015] FIG. 5 iustrates an exampe bock diagram of an experience monitor component that empoys physioogica and environmenta sensing mechanisms in accordance With aspects of the innovation. [0016] FIG. 6 iustrates sampe contextua data that can be used in accordance With aspects of the speci?cation to trigger, annotate or render event-reated data. [0017] FIG. 7 iustrates an exampe experience capture component the empoys anaysis and recorder components to effect data capture in accordance With aspects. [0018] FIG. 8 iustrates an exampe bock diagram of a rendering component that faciitates rendering captured data to a user. [0019] FIG. 9 iustrates an exampe bock diagram of a content con?guration component that enabes synchroniza tion as We as user interface in accordance With aspects. [0020] FIG. 10 iustrates an architecture incuding an arti?cia inteigence-based component that can automate func tionaity in accordance With an aspect of the nove innovation. [0021] FIG. 11 iustrates a bock diagram of a computer operabe to execute the discosed architecture.
US 2009/0171902 A1 Ju. 2, 2009 [0022] FIG. 12 iustrates a schematic bock diagram of an exempary computing environment in accordance With the subject innovation. DETAILED DESCRIPTION [0023] The innovation is now described With reference to the drawings, Wherein ike reference numeras are used to refer to ike eements throughout. In the foowing descrip tion, for purposes of expanation, numerous speci?c detais are set forth in order to provide a thorough understanding of the subject innovation. It may be evident, however, that the innovation can be practiced Without these speci?c detais. In other instances, We-knoWn structures and devices are shown in bock diagram form in order to faciitate describing the innovation. [0024] As used in this appication, the terms component and system are intended to refer to a computer-reated entity, either hardware, a combination of hardware and soft Ware, software, or software in execution. For exampe, a component can be, but is not imited to being, a process running on a processor, a processor, an object, an executabe, a thread of execution, a program, and/ or a computer. By Way of iustration, both an appication running on a server and the server can be a component. One or more components can reside Within a process and/or thread of execution, and a component can be ocaized on one computer and/or distrib uted between two or more computers. [0025] As used herein, the term to infer or inference refer generay to the process of reasoning about or inferring states of the system, environment, and/or user from a set of observations as captured via events and/ or data. Inference can be empoyed to identify a speci?c context or action, or can generate a probabiity distribution over states, for exampe. The inference can be probabiisticithat is, the computation of a probabiity distribution over states of interest based on a consideration of data and events. Inference can aso refer to techniques empoyed for composing higher-eve events from a set of events and/or data. Such inference resuts in the construction of new events or actions from a set of observed events and/ or stored event data, Whether or not the events are correated in cose tempora proximity, and Whether the events and data come from one or severa event and data sources. [0026] Referring initiay to the drawings, FIG. 1 iustrates a bock diagram of an exampe system 100 that enabes ife events and experiences to be monitored and captured based upon context. As Wi be described in greater detai infra, these experiences can be captured in accordance With most any avaiabe perception type. For exampe, visua data, audibe data, scent data, etc. can be captured as reated to a rea-word experience. These data eements can be annotated and/ or tagged and thereafter queried or otherwise retrieved for payback. [0027] As Wi be further described, prior to payback, the perception data can be synchronized so as to create a rich presentation experience to a user. For instance, visua data can be synchronized to audibe data and thereafter con?gured for rendering on a particuar device. The granuarity of the pre sentation can be manuay seected by a user or automaticay seected based upon factors such as, device type, content, context, or other desired factors. [0028] Oftentimes events occur Where it Woud be desirabe to have the abiity to repay the event Within a short period in order to better and more fuy comprehend the event. In other Words, by repaying the event, oftentimes an individua can reive, enjoy or earn from a past experience. Sti further, events can be shared With other individuas to share experi ences, earn from other s experiences or even see or experi ence an event or ocation from another s perspective. [0029] Unfortunatey, events happen in rea time and peope are not given the opportunity to reive the event after it has occurred. Not ony are the events hard to reive, but they aso occur over mutipe dimensions (or perceptions) and human senses. Thus, not ony does one see What is going on but aso hear and fee the event, for exampe. [0030] FIG. 1 iustrates a system 100 that can capture these ife experiences thereby enabing payback or retention for ater use as desired. Essentiay, the innovation can be viewed as a ife recorder that is imbued With tremendous memory capabiity and is aways running, simiar to a Wristwatch. As Wi be described, the recorder can record over mutipe senses such that peope can repay events so they can better appreciate things that may have occurred, earn, share, remi nisce, etc. [0031] Generay, system 100 can incude an experience management system 102 having an experience monitor com ponent 104 and an experience capture component 106. Together, these sub-components (104, 106) monitor percep tions 1 to M and capture 1 to N and to P experiences respectivey, Where M, N and P are integers. In operation, the experience management component 102 can monitor (e.g., via experience monitor component 104) experiences 110 as perceived by perceptions 108. Thereafter, the experience cap ture component 106 can faciitate capture and/or storage of data associated With the experiences. It is to be understood that the data can be stored ocay, remotey (e.g., server based, remote storage, Internet, coud-based) or distributed in a system that incudes a combination of oca and remote storage. [0032] The experience monitor component 104 can incude or empoy sensory mechanisms to monitor experiences 110 that occur. For instance, sensors can be empoyed to capture contextua factors such as current Weather conditions, oca tions, mood, state of mind, etc. These factors can be used to trigger the capture of the information, enhance description of captured information, and to annotate captured information. [0033] The experience capture component 106 can incude adjustments and contros that aow different granuarities of information to be preserved (or subsequenty presented) if desired. For instance, events or experiences Within the past few hours can be saved at high granuarity Where events that occurred further back in time can be viewed at coarser granu arities, if desired. For exampe, yesterday s events may be viewed one frame at a time at 30 second intervas Where today s events can be viewed in rea time. Simiary, different perceptions 108 (e.g., senses) can be captured at different granuarities as desired. [0034] In one embodiment, the captured information can be used to faciitate and otherwise enhance reputation systems in order to provide a richer user experience. For instance, media agents can be empoyed to search and?nd desired media content reated to a given pro?e. This media content can refer, in part, to the information captured by the experience monitor system 102. [0035] Other aspects of the innovation incude the abiity to access information to better understand opinions on a given subject. For instance, this captured information can describe What someone ese thinks about a particuar topic or What type
US 2009/0171902 A1 Ju. 2, 2009 of experience others had upon visiting a particuar ocation. It Wi be understood that the exampe uses and bene?ts of the captured information are countess. Accordingy, exampes provided herein are incuded merey to add perspective to the innovation and are not intended to imit the innovation in any Way. [0036] In other aspects, ideas can be generated based upon socia contexts such as pushing information reated to a per son in an upcoming meeting. Socia suggestions can be made based upon of the pro?es Where pro?es can be automaticay annotated With experiences such as the recorded ife experi ences over time. From past experiences, interests can be dis covered and possibe new interests can be mined over time. Stored memories can be empoyed for ife modeing to be used in such aspects as ocation gaming, changing or atering one s perception based on the past experiences, utiizing aug mented reaities based on the recorded experiences, and even substituting aternative reaities, if desired. It Wi be under stood the security and access contro ists can be empoyed to address privacy concerns reated to capture and access to experience information. In other Words, a user can grant or deny access to capture and/or share information as desired. [0037] FIG. 2 iustrates an exampe methodoogy of moni toring experience data in accordance With an aspect of the innovation. As described With reference to FIG. 1, the inno vation monitors, tracks and captures data reated to rea-ife experiences of a user. The data and information can be tracked or monitored by Way of sensory mechanisms and thereafter captured in accordance With a predetermined or inferred pref erence or poicy. Simiary, these sensory mechanisms can be empoyed to monitor contextua factors Which can essentiay be used to trigger capture of information. These and other aspects Wi become more apparent upon a review of the?gures and exampes that foow. [0038] Whie, for purposes of simpicity of expanation, the one or more methodoogies shown herein, e.g., in the form of a How chart, are shown and described as a series of acts, it is to be understood and appreciated that the subject innovation is not imited by the order of acts, as some acts may, in accordance With the innovation, occur in a different order and/ or concurrenty With other acts from that shown and described herein. For exampe, those skied in the art Wi understand and appreciate that a methodoogy coud aterna tivey be represented as a series of interreated states or events, such as in a state diagram. Moreover, not a iustrated acts may be required to impement a methodoogy in accor dance With the innovation. [0039] At 202, experiences are monitored, for exampe, day-to-day rea-ife events and experiences of a user are monitored or tracked. Essentiay, the innovation enabes an eectronic journa of user experiences to be captured and stored. This eectronic journa can be search, queried, shared, repayed, etc. Features, functions and bene?ts of the innova tion Wi be understood and appreciated upon a review of the?gures that foow. [0040] At 204, data reated to an experience is captured. For exampe, visua (e. g., video, sti images), audio (e.g., spoken Words, auditory sounds), smes, feeings (e.g., temperature, textures) data can be captured. Additionay, contextua fac tors reated to the user physioogica state, environment, etc. can aso be captured. The perception type(s) empoyed for the capture can be determined at 206. Here, a determination can be made if the data Was captured by a sense of sight, hearing, sme, or feeing. [0041] At 208, the captured data can be annotated With metadata or other tags Which describe the data. It is to be understood that this metadata faciitates subsequent ocation for presentation. For instance, the perception type, content of experience, context reated to experience or other such descriptive metadata can be annotated to the data. Finay, at 210, the data canbe stored in a oca or remote storage faciity. [0042] Referring now to FIG. 3, there is iustrated an exampe methodoogy of retrieving and empoying the eec tronic journa in accordance With the innovation. At 302, a query or other search criteria is generated. It is to be appre ciated that, in aspects, a query can be expicity generated by a user. For exampe, suppose a user is preparing to engage in troubeshooting a certain type of Wireess network router in their home. Here, the user can expicity estabish a query With keywords such as, troubeshoot a brand X Wireess network router. In other exampes, the system can automati cay determine or infer the user s action or desire. Subse quenty, a query can be automaticay generated and/or inferred on behaf of the user. It Wi be understood that the exampe features, functions and bene?ts of the innovation are countess. Accordingy, this speci?cation is to incude a feasibe aspects Within the scope of this discosure and caims appended hereto. For instance, in a more simpistic exampe, a user might be interested in baking a cake, or even changing an infant s diaper. These are just two more exampes of how the features, functions and/orbene?ts of the innovation can be empoyed. [0043] At 304, the annotations attached to captured data can be searched based upon a search query. It Wi be under stood that the innovation contempates security and access contro ists to address privacy concerns associated With monitoring actions of individuas. For brevity, the exampes do not discuss these concerns however, it is to be understood that they are to be considered Within the scope of this speci?cation and caims appended hereto. [0044] Accordingy, data can be retrieved at 306. Once data is retrieved, it can be synchronized at 308 and rendered at 310. In other Words, at 308, audio can be matched to video, etc. Thereafter, the synchronized data can be rendered at 310. [0045] Turning now to FIG. 4, an exampe bock diagram of an experience monitor component 104 in accordance With an aspect of the innovation is shown. Generay, the component 104 incudes 1 to M perception sensing components 402, Where M is an integer. Additionay, the experience monitor component 104 incudes a context sensing component 404 and an annotation component 406. Together, these compo nents (402, 404, 406) enabe experience data to be captured and annotated (or otherwise tagged) for indexing and storage. [0046] The perception sensing component(s) 402 can incude sensory mechanisms that capture data reated to the experiences of a user. For exampe, the perception sensing component 402 can capture visua data, auditory data, etc. reated to an experience. In other Words, individua data streams can be captured that represent individua sensory data. In other aspects, muti-perception data can be coected, annotated and stored as desired. [0047] The context sensing component 404 can enabe cap ture of situationa data that reates or is associated to the actua experience data. For exampe, situationa data can incude subject user ocation, time of day, Weather, physioogica data, activity data, demographics, as We as most any other
US 2009/0171902 A1 Ju. 2, 2009 detectabe descriptive data. As Wi be described beow, the contextua data can incude environmenta as We as physi oogica data as appropriate. [0048] The annotation component 406 can be empoyed to attach, embed or otherwise associate descriptive metadata or tagging data to the captured experience. This annotation data can be used to index and ater search the stored data. The annotation data can essentiay be used to expicity or impic ity retrieve data for repay, presentation or other usefu ren dering. [0049] Referring now to FIG. 5, an aternative bock dia gram of an exampe experience monitor component 104 is shown. Essentiay, FIG. 5 iustrates that both the perception sensing component and the context sensing component can incude physioogica (502, 504) and/or environmenta (506, 508) sensors. Sti further, as shown, the annotation compo nent 406 is capabe to obtain other data Which can be empoyed to further describe the captured experience data. [0050] The foowing exampe is provided merey to add perspective to the innovation and is not intended to imit the scope of this speci?cation in any manner. Rather, the foow ing exampe is incuded to iustrate features, functions and bene?ts of the innovation by describing a rea-word appica tion. It is to be understood that other exampes exist and are to be incuded Within the spirit and/or scope of the innovation and caims appended hereto. [0051] In an exampe scenario, experiences during an indi vidua s hiking trip can be captured. During a hike, the per ception sensing component 402 can capture visua data reated to sights experienced during the hike. Simiary, audi tory data can be captured. Other data can be processed by the annotation component 406 Which can be used to annotate the captured experience. For exampe, the Internet can be accessed to determine the ength of the hike, eevation, ter rain, etc. It Wi be appreciated, that some, if not a, of these factors can aso be captured by the environmenta sensors 508. [0052] In aspects, sensors 502, 504, 506 and 508 can be direct indicating sensors. Direct-indicating sensors, for exampe, a mercury thermometer, are human-readabe. Other sensors, such as a thermocoupe, can produce an output vot age or other eectrica output Which can be interpreted by another device (such as a computer processor or software appication). [0053] It Wi be appreciated that sensors are used in every day appications, such as touch-sensitive eevator buttons, automobie ocking mechanisms, biometric?ngerprint read ers, etc. This information can be captured by the experience monitor component 104. A sensor s sensitivity indicates how much the sensor s output changes When the measured quan tity changes. For instance, if the mercury in a thermometer moves 1 cm When the temperature changes by 1, the sensi tivity is 1 cm/ 1. Sensors that measure very sma changes have very high sensitivities. Technoogica progress aows more and more sensors to be manufactured on a microscopic scae as microsensors that use MEMS (microeectrome chanica systems) technoogy. It is to be understood and appreciated that, athough the exampe experience monitor component of FIG. 5 incudes physioogica sensors 502, 504 and environmenta sensors 505, 508, most any sensory mechanisms can be empoyed in accordance With the inno vation. [0054] FIG. 6 iustrates a samping of the kinds of data that can be gathered by the context sensing component, 404 of FIG. 4. In accordance With the aspect iustrated in FIG. 6, the activity context data can be divided into 3 casses: activity context 602, user context 604, and environment context 606. [0055] By Way of exampe, and not imitation, the activity context data 602 incudes the current activity the user is performing. It is to be understood that this activity informa tion can be expicity determined and/or inferred. Addition ay, the activity context data 602 can incude the current step (if any) Within the activity. In other Words, the current step can be described as the current state of the activity. Moreover, the activity context data 602 can incude a current resource (e. g.,?e, appication, gadget, emai, etc.) that the user is interact ing With in accordance With the activity. [0056] In an aspect, the user context data 604 can incude topics of knowedge that the user knows about With respect to the activity and/or appication. As We, the user context data 604 can incude an estimate of the user s state of mind (e. g., happy, frustrated, confused, angry, etc.). It Wi be understood and appreciated that the user s state of mind can be estimated using different input modaities, for exampe, the user can express intent and feeings, the system can anayze pressure and movement on a mouse, content and/or intensity of verba statements, physioogica signas, etc. to determine state of mind. [0057] With continued reference to FIG. 6, the environment context data 606 can incude the physica conditions of the environment (e.g., Wind, ighting, ambient, sound, tempera ture, etc.), the socia setting (e.g., user is in a business meet ing, or user is having dinner With his famiy), the other peope Who are in the user s immediate vicinity, data about how secure the ocation/system/network are, the date and time, and the ocation of the user. As stated above, athough speci?c data is identi?ed in FIG. 6, it is to be understood that addi tiona types of data can be gathered and empoyed in anno tating captured data in accordance With an aspect of the inno vation. As We, it is to be understood that this additiona data is to be incuded Within the scope of the discosure and caims appended hereto. [0058] Turning now to FIG. 7, a bock diagram of an exampe experience capture component 104 is shown. As iustrated, the experience capture component 106 can incude an experience indexing component 702 and an expe rience recording component 704. In operation, each of these sub-components 702, 704 enabe index and storage of anno tated experience data respectivey. [0059] The experience indexing component 702 can estab ish an index based upon annotations provided by the anno tation component (406 of FIG. 4). It is to be understood that most any indexing technique can be empoyed in aspects of the innovation. In one exampe, the index can be based upon keywords derived from an anaysis of the experience aone or in addition to With the contextua data. [0060] Continuing With the aforementioned hiking exampe, the index can be estabished based upon Words that describe a hike, trek, cimb or the ike. As We, the experience data can be indexed based upon the ocation of the hike as We as other contextua data. Mo st any descriptive data (e.g., keywords, maps, image data) can be used to index experience data. [0061] The experience recorder component 704 can be used to store or otherwise maintain the experience data for subsequent access. As described above, the experience data can be maintained ocay, remotey or distributed as desired. Essentiay, once the data is indexed and stored, it can ater be
US 2009/0171902 A1 Ju. 2, 2009 restored or repayed for most any desired reason, incuding but not imited to, repay, sharing, earning, researching, medica reasons, therapy, or the ike. [0062] Additionay, the experience recorder component 704 can enabe expicit or impicit granuarity adjustment of experience data. For exampe, a user can expicity adjust the granuarity of captured data based upon data type, time of day, ocation, content, context or other desired factor(s). Simiary, the component 704 can automaticay adjust the granuarity based upon most any factor incuding but, not imited to type, content, context or the ike. [0063] Referring now to FIG. 8, an exampe rendering man agement component 802 is shown in accordance With an aspect of the innovation. Essentiay, the rendering manage ment component 802 enabes users to retrieve, access, share or otherwise obtain previousy stored experience data. Athough not iustrated, it is to be appreciated that the ren dering component can incude or empoy machine earning and reasoning (MLR) mechanisms. For exampe, the query generation component 804 can empoy MLR mechanisms to automaticay generate and/or con?gure queries on behaf of a user (e.g., based upon context). [0064] As iustrated, the rendering management compo nent 802 can incude a query component 804, a content con?guration component 806 and a device con?guration compo nent 808. Each of these sub-components enabe previousy stored experience data to be accessed by or presented to a user (or appication). [0065] The query component 804 enabes a user to expic ity generate a query. For exampe, if a user Wants to re-ive an experience, a query can be estabished to ocate the saved data. Simiary, a query can be estabished to ocate saved experience data that reates to other s experiences, for exampe, to troubeshoot a Wireess router. Sti further, que ries can be dynamicay inferred or generated using MLR mechanisms based upon historica, statistica and/or contex tua data. Once a query (or other suitabe search) is estab ished, the query component 804 can ocate, access, retrieve or otherwise obtain reevant experience data. [0066] The content con?guration component 806 can be empoyed to prepare the experience data for deivery or pre sentation. Simiary, the device con?guration component 808 can be empoyed to arrange or con?gure the experience data based upon capabiities associated With the target rendering device. For instance, if the target rendering device is a smart phone or persona digita assistant (PDA), the device con?gu ration component 808 can con?gure the data differenty than it Woud for a desktop computer based upon processing power, memory, dispay size/characteristics, etc. [0067] Turning now to FIG. 9, an exampe bock diagram of a content con?guration component 806 is shown. Generay, the component 806 incudes a synchronization component 902 and a granuarity seection component 904. In operation, these sub-components 902, 904 enabe preparation of the experience data for rendering. [0068] The synchronization component 902 can enabe synchronization of muti-perception data for rendering. For exampe, Where visua and audibe data is captured, the syn chronization data can combine the data such that a muti perception data stream can be rendered. As described above, the annotations can be empoyed to effect synchronization in aspects. [0069] The granuarity seection component 904 enabes presentation, payback or rendering granuarity to be dynami cay, expicity or impicity seected. In one aspect, a user can automaticay seect the granuarity for payback. In other aspects, the granuarity can be seected on behaf of a user, for exampe, based upon target device capabiities. [0070] FIG. 10 iustrates a system 1000 that empoys an MLR component 1002 Which faciitates automating one or more features in accordance With the subject innovation. The subject innovation (e. g., in connection With content seection) can empoy various MLR-based schemes for carrying out various aspects thereof. For exampe, a process for determin ing What content to capture, how to annotate, What granuarity to empoy, etc. can be faciitated via an automatic cassi?er system and process. [0071] A cassi?er is a function that maps an input attribute vector, x:(x, x2, x3, x4, xn), to a con?dence that the input beongs to a cass, that is, f(x):con?dence(cass). Such cas si?cation can empoy a probabiistic and/or statistica-based anaysis (e. g., factoring into the anaysis utiities and costs) to prognose or infer an action that a user desires to be automati cay performed. [0072] A support vector machine (SVM) is an exampe of a cassi?er that can be empoyed. The SVM operates by?nding a hypersurface in the space of possibe inputs, Which the hypersurface attempts to spit the triggering criteria from the non-triggering events. Intuitivey, this makes the cassi?ca tion correct for testing data that is near, but not identica to training data. Other directed and undirected mode cassi? cation approaches incude, e.g., nai've Bayes, Bayesian net Works, decision trees, neura networks, fuzzy ogic modes, and probabiistic cassi?cation modes providing different patterns of independence can be empoyed. Cassi?cation as used herein aso is incusive of statistica regression that is utiized to deveop modes of priority. [0073] As Wi be readiy appreciated from the subject speci?cation, the subject innovation can empoy cassi?ers that are expicity trained (e.g., via a generic training data) as We as impicity trained (e.g., via observing user behavior, receiving extrinsic information). For exampe, SVM s are con?gured via a earning or training phase Within a cassi?er constructor and feature seection modue. Thus, the cassi?er (s) can be used to automaticay earn and perform a number of functions, incuding but not imited to determining accord ing to a predetermined criteria When to capture experience data, What experience data to capture, Which perception(s) to empoy, how to annotate the captured data, What granuarity to empoy for capture, What granuarity to empoy for render ing, how to con?gure data for rendering, etc. [0074] Referring now to FIG. 11, there is iustrated a bock diagram of a computer operabe to execute the discosed architecture. In order to provide additiona context for various aspects of the subject innovation, FIG. 11 and the foowing discussion are intended to provide a brief, genera description of a suitabe computing environment 1100 in Which the vari ous aspects of the innovation can be impemented. Whie the innovation has been described above in the genera context of computer-executabe instructions that may run on one or more computers, those skied in the art Wi recognize that the innovation aso can be impemented in combination With other program modues and/or as a combination of hardware and software. [0075] Generay, program modues incude routines, pro grams, components, data structures, etc., that perform par ticuar tasks or impement particuar abstract data types. Moreover, those skied in the art Wi appreciate that the
US 2009/0171902 A1 Ju. 2, 2009 inventive methods can be practiced With other computer sys tem con?gurations, incuding singe-processor or mutipro cessor computer systems, minicomputers, mainframe com puters, as We as persona computers, hand-hed computing devices, microprocessor-based or programmabe consumer eectronics, and the ike, each of Which can be operativey couped to one or more associated devices. [0076] The iustrated aspects of the innovation may aso be practiced in distributed computing environments Where cer tain tasks are performed by remote processing devices that are inked through a communications network. In a distributed computing environment, program modues can be ocated in both oca and remote memory storage devices. [0077] A computer typicay incudes a variety of com puter-readabe media. Computer-readabe media can be any avaiabe media that can be accessed by the computer and incudes both voatie and nonvoatie media, removabe and non-removabe media. By Way of exampe, and not imita tion, computer-readabe media can comprise computer stor age media and communication media. Computer storage media incudes both voatie and nonvoatie, removabe and non-removabe media impemented in any method or tech noogy for storage of information such as computer-readabe instructions, data structures, program modues or other data. Computer storage media incudes, but is not imited to, RAM, ROM, EEPROM,?ash memory or other memory technoogy, CD-ROM, digita versatie disk (DVD) or other optica disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium Which can be used to store the desired information and Which can be accessed by the computer. [0078] Communication media typicay embodies com puter-readabe instructions, data structures, program modues or other data in a moduated data signa such as a carrier Wave or other transport mechanism, and incudes any information deivery media. The term moduated data signa means a signa that has one or more of its characteristics set or changed in such a manner as to encode information in the signa. By Way of exampe, and not imitation, communication media incudes Wired media such as a Wired network or direct-wired connection, and Wireess media such as acoustic, RF, infrared and other Wireess media. Combinations of the any of the above shoud aso be incuded Within the scope of computer readabe media. [0079] With reference again to FIG. 11, the exempary environment 1100 for impementing various aspects of the innovation incudes a computer 1102, the computer 1102 incuding a processing unit 1104, a system memory 1106 and a system bus 1108. The system bus 1108 coupes system components incuding, but not imited to, the system memory 1106 to the processing unit 1104. The processing unit 1104 can be any of various commerciay avaiabe processors. Dua microprocessors and other muti-processor architec tures may aso be empoyed as the processing unit 1104. [0080] The system bus 1108 can be any of severa types of bus structure that may further interconnect to a memory bus (With or Without a memory controer), a periphera bus, and a oca bus using any of a variety of commerciay avaiabe bus architectures. The system memory 1106 incudes read ony memory (ROM) 1110 and random access memory (RAM) 1112. Abasic input/output system (BIOS) is stored in a non-voatie memory 1110 such as ROM, EPROM, EEPROM, Which BIOS contains the basic routines that hep to transfer information between eements Within the com puter 1102, such as during start-up. The RAM 1112 can aso incude a high-speed RAM such as static RAM for caching data. [0081] The computer 1102 further incudes an interna hard disk drive (HDD) 1114 (e.g., EIDE, SATA), Which interna hard disk drive 1114 may aso be con?gured for externa use in a suitabe chassis (not shown), a magnetic?oppy disk drive (FDD) 1116, (e. g., to read from or Write to a removabe diskette 1118) and an optica disk drive 1120, (e.g., reading a CD-ROM disk 1122 or, to read from or Write to other high capacity optica media such as the DVD). The hard disk drive 1114, magnetic disk drive 1116 and optica disk drive 1120 can be connected to the system bus 1108 by a hard disk drive interface 1124, a magnetic disk drive interface 1126 and an optica drive interface 1 128, respectivey. The interface 1 124 for externa drive impementations incudes at east one or both of Universa Seria Bus (U SB) and IEEE 1394 interface technoogies. Other externa drive connection technoogies are Within contempation of the subject innovation. [0082] The drives and their associated computer-readabe media provide nonvoatie storage of data, data structures, computer-executabe instructions, and so forth. For the com puter 1102, the drives and media accommodate the storage of any data in a suitabe digita format. Athough the description of computer-readabe media above refers to a HDD, a remov abe magnetic diskette, and a removabe optica media such as a CD or DVD, it shoud be appreciated by those skied in the art that other types of media Which are readabe by a com puter, such as Zip drives, magnetic cassettes,?ash memory cards, cartridges, and the ike, may aso be used in the exem pary operating environment, and further, that any such media may contain computer-executabe instructions for perform ing the methods of the innovation. [0083] A number of program modues can be stored in the drives and RAM 1112, incuding an operating system 1130, one or more appication programs 1132, other program mod ues 1134 and program data 1136. A or portions of the operating system, appications, modues, and/or data can aso be cached in the RAM 1112. It is appreciated that the inno vation can be impemented With various commerciay avai abe operating systems or combinations of operating systems. [0084] A user can enter commands and information into the computer 1102 through one or more Wired/Wireess input devices, e.g., a keyboard 1138 and a pointing device, such as a mouse 1140. Other input devices (not shown) may incude a microphone, an IR remote contro, a joystick, a game pad, a styus pen, touch screen, or the ike. These and other input devices are often connected to the processing unit 1104 through an input device interface 1142 that is couped to the system bus 1108, but can be connected by other interfaces, such as a parae port, an IEEE 1394 seria port, a game port, a USB port, an IR interface, etc. [0085] A monitor 1144 or other type of dispay device is aso connected to the system bus 1108 via an interface, such as a video adapter 1146. In addition to the monitor 1144, a computer typicay incudes other periphera output devices (not shown), such as speakers, printers, etc. [0086] The computer 1102 may operate in a networked environment using ogica connections via Wired and/ or Wire ess communications to one or more remote computers, such as a remote computer(s) 1148. The remote computer(s) 1148 can be a Workstation, a server computer, a router, a persona computer, portabe computer, microprocessor-based enter tainment appiance, a peer device or other common network
US 2009/0171902 A1 Ju. 2, 2009 node, and typicay incudes many or a of the eements described reative to the computer 1102, athough, for pur poses of brevity, ony a memory/ storage device 1150 is ius trated. The ogica connections depicted incude wired/wire ess connectivity to a oca area network (LAN) 1152 and/or arger networks, eg a wide area network (WAN) 1154. Such LAN and WAN networking environments are commonpace in of?ces and companies, and faciitate enterprise-wide com puter networks, such as intranets, a of which may connect to a goba communications network, e.g., the Internet. [0087] When used in a LAN networking environment, the computer 1102 is connected to the oca network 1152 through a wired and/or wireess communication network interface or adapter 1156. The adapter 1156 may faciitate wired or wireess communication to the LAN 1152, which may aso incude a wireess access point disposed thereon for communicating with the wireess adapter 1156. [0088] When used in a WAN networking environment, the computer 1102 can incude a modem 1158, or is connected to a communications server on the WAN 1154, or has other means for estabishing communications over the WAN 1154, such as by way of the Internet. The modem 1158, which can be interna or externa and a wired or wireess device, is connected to the system bus 1108 via the seria port interface 1142. In a networked environment, program modues depicted reative to the computer 1102, or portions thereof, can be stored in the remote memory/storage device 1150. It wi be appreciated that the network connections shown are exempary and other means of estabishing a communications ink between the computers can be used. [0089] The computer 1102 is operabe to communicate with any wireess devices or entities operativey disposed in wireess communication, e.g., a printer, scanner, desktop and/ or portabe computer, portabe data assistant, communica tions sateite, any piece of equipment or ocation associated with a wireessy detectabe tag (e.g., a kiosk, news stand, restroom), and teephone. This incudes at east Wi-Fi and BuetoothTM wireess technoogies. Thus, the communication can be a prede?ned structure as with a conventiona network or simpy an ad hoc communication between at east two devices. [0090] Wi-Fi, or Wireess Fideity, aows connection to the Internet from a couch at home, a bed in a hote room, or a conference room at work, without wires. Wi-Fi is a wireess technoogy simiar to that used in a ce phone that enabes such devices, e.g., computers, to send and receive data indoors and out; anywhere within the range of a base station. Wi-Fi networks use radio technoogies caed IEEE 802.11 (a, b, g, etc.) to provide secure, reiabe, fast wireess connectiv ity. A Wi-Fi network can be used to connect computers to each other, to the Internet, and to wired networks (which use IEEE 802.3 or Ethernet). Wi-Fi networks operate in the unicensed 2.4 and 5 GHZ radio bands, at an 11 Mbps (802.1 a) or 54 Mbps (802.1 b) data rate, for exampe, or with products that contain both bands (dua band), so the networks can provide rea-word performance simiar to the basic 0 BaseT wired Ethernet networks used in many of?ces. [0091] Referring now to FIG. 12, there is iustrated a sche matic bock diagram of an exempary computing environ ment 1200 in accordance with the subject innovation. The system 1200 incudes one or more cient(s) 1202. The cient (s) 1202 can be hardware and/or software (e. g., threads, pro cesses, computing devices). The cient(s) 1202 can house cookie(s) and/or associated contextua information by empoying the innovation, for exampe. [0092] The system 1200 aso incudes one or more server(s) 1204. The server(s) 1204 can aso be hardware and/or soft ware (e.g., threads, processes, computing devices). The serv ers 1204 can house threads to perform transformations by empoying the innovation, for exampe. One possibe com munication between a cient 1202 and a server 1204 can be in the form of a data packet adapted to be transmitted between two or more computer processes. The data packet may incude a cookie and/or associated contextua information, for exampe. The system 1200 incudes a communication framework 1206 (e. g., a goba communication network such as the Internet) that can be empoyed to faciitate communi cations between the cient(s) 1202 and the server(s) 1204. [0093] Communications can be faciitated via a wired (in cuding optica?ber) and/or wireess technoogy. The cient (s) 1202 are operativey connected to one or more cient data store(s) 1208 that can be empoyed to store information oca to the cient(s) 1202 (e.g., cookie(s) and/or associated con textua information). Simiary, the server(s) 1204 are opera tivey connected to one or more server data store(s) 1210 that can be empoyed to store information oca to the servers 1204. [0094] What has been described above incudes exampes of the innovation. It is, of course, not possibe to describe every conceivabe combination of components or methodoo gies for purposes of describing the subject innovation, but one of ordinary ski in the art may recognize that many further combinations and permutations of the innovation are pos sibe. Accordingy, the innovation is intended to embrace a such aterations, modi?cations and variations that fa within the spirit and scope of the appended caims. Furthermore, to the extent that the term incudes is used in either the detaied description or the caims, such term is intended to be incusive in a manner simiar to the term comprising as comprising is interpreted when empoyed as a transitiona word in a caim. What is caimed is: 1. A system that faciitates recreation of an experience, comprising: an experience monitor component that ob serves a puraity of experiences in view of context; and an experience capture component that empoys at east two perceptions to obtain data associated with a subset of the puraity of experiences. 2. The system of caim 1, the puraity of perceptions incudes at east two of vision, hearing, touch, sme, or taste. 3. The system of caim 1, further comprising a puraity of perception sensing components that faciitate data capture associated with a corresponding perception. 4. The system of caim 3, each of the perception sensing components incudes at east one of a physioogica or envi ronmenta sensor component. 5. The system of caim 1, further comprising a context sensing component that estabishes the context based upon activity context, user context or device context. 6. The system of caim 1, the context sensing component incudes at east one of a physioogica or environmenta sensor component. 7. The system of caim 1, further comprising an annotation component that attaches a tag to a subset of the data wherein the tag faciitates queried retrieva of the subset of the data.