Data Storage Issues in Sensor Networks. Dr. Deepak Ganesan, edited by Dr. Robert Akl

Transcription

1 Data Storage Issues in Sensor Networks Dr. Deepak Ganesan, edited by Dr. Robert Akl

2 Spectrum of Data Management Techniques Communication for Query Processing Local Storage and Flooding or Geography-based Query Processing Local Storage and Distributed Index Distributed Storage and Distributed Index Centralized Storage and Querying Communication for Storage

3 Local Vs Centralized Storage Local storage Merits: Cheaper than transmitting data to a central location. Demerits: High latency for query processing/data visualization Un-reliable if nodes fail. Centralized Storage Merits: Low-latency query processing Highly reliable Demerits: High power consumption

4 Outline Tradeoff between local and centralized storage for vibration monitoring Wisden: A wireless sensor network for structural monitoring Sensys 2004 Distributed Storage Key building blocks DCS: Name-based storage DIMENSIONS: Multi-resolution storage

5 Motivating Example: Long-lived wireless structural monitoring network High data rates compared to limited resources (storage, bandwidth and energy) ~100Hz, 16bit sample Global events Diverse requirements Data collection of interesting event signatures of vibration events. Analysis of data over different time-scales (long-term and shortterm patterns) State of Art: Wireless data acquisition systems using Not scalable, expensive, powerhungry

6 Mote-based Data acquisition systems for wireless structural monitoring Challenge Data collection from all nodes takes significant time Data processing goals: Tuning knobs for reducing latency of data gathering. Data compression for scaling to hundred motes. 15 minutes of vibration event data (100KB each after Huffman coding) from a 20 node multi-hop wireless network of motes takes 4 hours to collect centrally! Wisden: A wireless sensor network for structural monitoring Sensys 2004

7 Approach: Progressive, On-demand Data Collection Progressive Data Acquisition Each node stores its data in its local storage and transmits low-resolution summaries to the base-station immediately after event. User can analyze lowresolution data to determine nodes from which higher resolution data is required. Lossless data is collected from a subset of nodes on-demand within a window of time (before being phased out of nodes local storage) Low-resolution data for 15 minutes of vibration event data can be collected within minutes of event occurrence

8 Are wavelets a good choice for vibration data? Deepak Ganesan (UMass) Most of the energy is concentrated in the lower frequency subbands. Signal decomposition suggests that it is highly appropriate for wavelet compression. Vibration Dataset: CUREE-Kajima Joint Research Program, UCLA

9 Performance: Compression Ratio 17-fold reduction in data size with an RMS error of 3.1 (PSNR: 30dB) Good compression ratios can be achieved with low compression error.

10 Performance: Computation and Memory utilization Deepak Ganesan (UMass) Component of the Codec Wavelet Decomposition + Flash Storage Uniform Quantizer Run-length Encoder Computation Time 6.44ms 0.32ms 6.30ms Memory Req. 288 bytes 7 bytes 20 bytes Efficient mote implementation can be achieved.

11 Spectrum of Data Management Techniques Local Storage and Flooding or Geography-based Query Processing Local Storage and Distributed Index Local Storage Distributed Storage and Distributed Index Centralized Storage and Querying Communication

12 Storage and Search challenges Storage Search Long-term storage in a storage-constrained network Deepak Ganesan (UMass) Search of named events requires a name-search (similar in spirit to P2P systems) Eg: How many trucks passed through an area in the last day? Search for patterns in data requires summaries of data Eg 1: Find an edge in a geographical region. Eg 2: Find 3 dominant frequency components in a vibration signal.

13 Key Building Blocks Geographic Spatial Hierarchies GPSR: Geographic Routing GHT: Geographic Hierarchy Hierarchical data processing DCS: Name-based hash tables DIMENSIONS: Wavelet-based summaries DIFS/DIMS: Indices based on range of data Other multi-scale summaries? Drill-down Queries DIMENSIONS: Multi-scale search

14 GPSR: Greedy Perimeter Stateless Routing C E F x 2 z S B A J G HOLE D 1 3 H I y Packets are marked with position of destination Each node is aware of its own position Greedy forwarding algorithm Perimeter forwarding algorithm

15 Geographic Hashing Hash to a location and send data to the location x Hash point

16 Geographic Spatial Hierarchy Hierarchy construction Recursively split network into non-overlapping square grids. At each level of the hierarchy, Elect clusterhead Cluster-head processes from 4 quadrants Cluster-head propagates processed data to the next level of the hierarchy. Routing protocol: GPSR variant that stores data at node closest to hashed location.

17 Drill-down queries over Spatial Hierarchy Deepak Ganesan (UMass) What is the maximum precipitation between Sept-Dec 2002? Direct query to quadrant that best matches query Queries drill-down from highest level of hierarchy to lowest level until sufficiently accurate answer about the query is obtained Drill-down queries are a popular construct in information extraction.

18 Data Centric Storage Ratnasamy, Karp, Shenker

19 Geographic Rendezvous Node A PDA Hash(TRUCK) = x A, y A Node B Type = POLICE CAR Hash(CAR) = x B, y B

20 Geographic Rendezvous in GHT GHT provides (key, value) based associative memory GHT supports two operations _ Put(k,v)-stores v (observed data) according to the key k _ Get(k)-retrieve whatever value is associated with key k Hash function _ Hash the key in to the geographic coordinates _ Put() and Get() operations on the same key k hash k to the same location Deepak Ganesan (UMass)

21 DIMENSIONS: Multi-resolution Search and Storage

22 Approach: Provide a gracefully degrading storage Deepak Ganesan (UMass) Store data at multiple resolutions to tradeoff query quality for storage requirement. Exploit spatio-temporal correlation to reduce data. Exploit distributed storage capacity of sensor network. Organize data to enable efficient query processing.

23 Architecture Hierarchy: Construct distributed loadbalanced quad-tree hierarchy of lossy wavelet-compressed summaries corresponding to different resolutions and spatio-temporal scales. Drill-down Queries: Queries drill-down from root of hierarchy to focus search on small portions of the network. Data Aging: Progressively age summaries for long-term storage and graceful degradation of query quality over time. PROGRESSIVELY AGE PROGRESSIVELY LOSSY Level 2 Level 1 Level 0

24 Hierarchy: Construction Initially, nodes fill up their own storage with raw sampled data. Increasing time

25 Hierarchy: Construction Tesselate the network space into grids, and hash in each to determine location of clusterhead ([DCS02] Ratnasamy et al). Send wavelet-compressed local time-series to clusterhead. Increasing time

26 Hierarchy: Processing at each level Deepak Ganesan (UMass) Store incoming summaries locally for future search. Get compressed summaries from children. Decode time Wavelet encoder/decoder (other techniques might apply ) x y Re-encode at lower resolution and forward to parent.

27 Hierarchy: Construction Deepak Ganesan (UMass) Recursively send data to higher levels of the hierarchy. Increasing time

28 Hierarchy: Distributing storage load Deepak Ganesan (UMass) Hash to different locations over time to distribute load among nodes in the network. Increasing time

29 Drill-down querying Deepak Ganesan (UMass) User hashes to location where the root is located. The drill-down query is routed down till it reaches base.

30 Aging: Designing an aging policy for summaries Deepak Ganesan (UMass) Eventually, all available storage gets filled, and we have to decide when and how to drop summaries. Local Storage Allocation Res 3 Res 2 Res 1 low query accuracy high compactness Level 2 Level 1 past Time present low high Query Accuracy Local storage capacity Level 0 high query accuracy low compactness How do we allocate storage at each node to summaries at different resolutions to provide gracefully degrading storage and search capability?

31 Match system performance to user requirements User provides a function, Q user that represents desired query quality degradation over time. a set, T, of typical queries posed on the data. System provides a step function, Q system, with steps at times when summaries are aged. Quality Difference Query Accuracy 95% 50% past Time present Objective: Minimize worst case difference between userdesired query quality (blue curve) and query quality that the system can provide (red step function).

32 How do we determine the step function? Deepak Ganesan (UMass) Height: What is the dip in query accuracy when resolution i becomes unavailable? 95% Query Accuracy Quality Difference 50% past Time present Width: How long is resolution i stored within the network before being aged?

33 Storage allocation: constraintoptimization problem Deepak Ganesan (UMass) Objective: Find {s i }, i=1..log 4 N that: Given constraints: min max - < t Storage constraint: Each node cannot store any greater than its storage limit. Drill-down constraint: It is not useful to store finer resolution data if coarser resolutions of the same data is not present. 0 Q user ( t) Q system log 4 i= 1 N s ( t) i S Age Age i+1 i

34 Prior information about sampled data Deepak Ganesan (UMass) full a priori information Omniscient Strategy Infeasible. Use all data to decide optimal allocation. Training Strategy (can be used when small training dataset from initial deployment). Greedy Strategy (when no data is available, use a simple weighted allocation to summaries). Solve Constraint Optimization 1 : 2 : 4 Coarse Finer Finest No a priori information

35 Distributed trace-driven implementation Deepak Ganesan (UMass) Linux implementation of entire system (routing, compression, storage) for ipaq-class nodes uses Emstar, a Linux-based emulator/simulator for sensor networks. 3D Wavelet codec based on freeware by Geoff Davis available at: Geo-spatial precipitation dataset 15x12 grid (50km edge) of precipitation data from , from Pacific Northwest. (Caveat: Not real sensor network data). System parameters Training set: 6% of total dataset. compression ratio: 6:12:24:48 => ~10 time lifetime gain M. Widmann and C.Bretherton. 50 km resolution daily precipitation for the Pacific Northwest,

36 Queries posed over precipitation data Use queries at different spatio-temporal scales to evaluate the performance of schemes Choosing a Query Set GlobalYearlyEdge: look for spatio-temporal feature (edge between high and low precipitation areas). LocalYearlyMean: fine spatial and coarse temporal granularity GlobalDailyMax: coarse spatial and fine temporal granularity GlobalYearlyMax: coarse spatio-temporal granularity

37 How efficient is search? (fine) (coarse) More than 85% accuracy for less then 5% of the network queried for different queries (20x reduction in energy)!

38 Comparing aging schemes Training performs within 1% to optimal. Results with greedy algorithm are sensitive to weights.

39 Data storage in irregular networks Sensor networks are irregular due to its coupling with the physical world. Topology is non-uniform due to terrain terrain and deployment practicalities (power, GPS) Sampling is irregular due to synchronization irregularity (changing temperature/humidity, clocks) How can our storage system deal with such irregularity? Data interpolation Deployment for micro-climate monitoring at James Reserve (CENS, UCLA)

40 Nearest-neighbor re-sampling Algorithm: Pros Each junction node on tree does a nearestneighbor re-sampling of the data to regularize the spatial dataset. Efficient energy-wise. Cons: Can introduce artifacts and be ineffective in highly irregular settings.

41 Performance on irregular topologies Fraction Error (Measured Real / Real) Daily Max Query Level in hierarchy where query terminates Drill-down queries do not always improve results as in the case of regular topology. Query error is greater.

42 Algorithms Used By GHT Geographic hash Table uses GPSR for Routing(Greedy Perimeter Stateless Routing) PEER-TO-PEER look up system (data object is associated with key and each node in the system is responsible for storing a certain range of keys)

43 System goals Deepak Ganesan (UMass) Need smooth transition for researchers who have depended on data collection systems system should retain ability to collect new event signatures on demand. Need to achieve low energy usage for long lifetime system focus has to shift from data collection to in-network data storage and search. Goal: Build a networked storage and search system with flexibility for use as a data acquisition system.