Targeted Resource Management in Multi-tenant Distributed Systems
|
|
|
- Derek Rice
- 7 years ago
- Views:
Transcription
1 Targeted Resource Management in Multi-tenant Distributed Systems Jonathan Mace Brown University Peter Bodik MSR Redmond Rodrigo Fonseca Brown University Madanlal Musuvathi MSR Redmond
2 Resource Management in Multi-Tenant Systems 2
3 Resource Management in Multi-Tenant Systems Failure in availability zone cascades to shared control plane, causes thread pool starvation for all zones April 2011 Amazon EBS Failure 2
4 Resource Management in Multi-Tenant Systems Failure in availability zone cascades to shared control plane, causes thread pool starvation for all zones April 2011 Amazon EBS Failure Aggressive background task responds to increased hardware capacity with deluge of warnings and logging Aug Azure Storage Outage 2
5 Resource Management in Multi-Tenant Systems Failure in availability zone cascades to shared control plane, causes thread pool starvation for all zones April 2011 Amazon EBS Failure Aggressive background task responds to increased hardware capacity with deluge of warnings and logging Aug Azure Storage Outage Code change increases database usage, shifts bottleneck to unmanaged application-level lock Nov Visual Studio Online outage 2
6 Resource Management in Multi-Tenant Systems Failure in availability zone cascades to shared control plane, causes thread pool starvation for all zones April 2011 Amazon EBS Failure Aggressive background task responds to increased hardware capacity with deluge of warnings and logging Aug Azure Storage Outage Code change increases database usage, shifts bottleneck to unmanaged application-level lock Nov Visual Studio Online outage Shared storage layer bottlenecks circumvent resource management layer 2014 Communication with Cloudera 2
7 Resource Management in Multi-Tenant Systems Failure in availability zone cascades to shared control plane, causes thread pool starvation for all zones April 2011 Amazon EBS Failure Aggressive background task responds to increased hardware capacity with deluge of warnings and logging Aug Azure Storage Outage Code change increases database usage, shifts bottleneck to unmanaged application-level lock Nov Visual Studio Online outage Shared storage layer bottlenecks circumvent resource management layer 2014 Communication with Cloudera Degraded performance, Violated SLOs, system outages 2
8 Resource Management in Multi-Tenant Systems Failure in availability zone cascades to shared control plane, causes thread pool starvation for all zones April 2011 Amazon EBS Failure Aggressive background task responds to increased hardware capacity with deluge of warnings and logging Aug Azure Storage Outage Code change increases database usage, shifts bottleneck to unmanaged application-level lock Nov Visual Studio Online outage Shared storage layer bottlenecks circumvent resource management layer 2014 Communication with Cloudera Degraded performance, Violated SLOs, system outages 3
9 Resource Management in Multi-Tenant Systems Failure in availability zone cascades to shared control plane, causes thread pool starvation for all zones April 2011 Amazon EBS Failure Aggressive background task responds to increased hardware capacity with deluge of warnings and logging Aug Azure Storage Outage Code change increases database usage, shifts bottleneck to unmanaged application-level lock Nov Visual Studio Online outage OS Hypervisor Shared storage layer bottlenecks circumvent resource management layer 2014 Communication with Cloudera Degraded performance, Violated SLOs, system outages 3
10 Resource Management in Multi-Tenant Systems Failure in availability zone cascades to shared control plane, causes thread pool starvation for all zones April 2011 Amazon EBS Failure Aggressive background task responds to increased hardware capacity with deluge of warnings and logging Aug Azure Storage Outage Code change increases database usage, shifts bottleneck to unmanaged application-level lock Nov Visual Studio Online outage OS Hypervisor Shared storage layer bottlenecks circumvent resource management layer 2014 Communication with Cloudera Degraded performance, Violated SLOs, system outages 3
11 Containers / VMs 4
12 Containers / VMs Shared Systems: Storage, Database, Queueing, etc. 4
13 5
14 5
15 Monitors resource usage of each tenant in near real-time Actively schedules tenants and activities 5
16 Monitors resource usage of each tenant in near real-time Actively schedules tenants and activities High-level, centralized policies: Encapsulates resource management logic 5
17 Monitors resource usage of each tenant in near real-time Actively schedules tenants and activities High-level, centralized policies: Encapsulates resource management logic Abstractions not specific to resource type, system Achieve different goals: guarantee average latencies, fair share a resource, etc. 5
18 Hadoop Distributed File System (HDFS) 6
19 Hadoop Distributed File System (HDFS) HDFS NameNode Filesystem metadata 6
20 Hadoop Distributed File System (HDFS) HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode Filesystem metadata Replicated block storage 6
21 Hadoop Distributed File System (HDFS) Rename HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode Filesystem metadata Replicated block storage 6
22 Hadoop Distributed File System (HDFS) Rename Read HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode Filesystem metadata Replicated block storage 6
23 Hadoop Distributed File System (HDFS) Rename Read HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode Filesystem metadata Replicated block storage 6
24 HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode 7
25 500 0 time HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode 7
26 500 0 time HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode 7
27 500 0 HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode 8
28 HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode 8
29 HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode 8
30 HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode 9
31 HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode 9
32 HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode 10
33 HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode 10
34 HDFS NameNode HDFS DataNode HDFS DataNode HDFS DataNode 10
35 HDFS DataNode Local storage 11
36 HDFS DataNode Local storage 11
37 HBase RegionServer HDFS DataNode Local storage 11
38 HBase RegionServer HDFS DataNode Local storage 11
39 MapReduce Shuffler Hadoop YARN Yarn Container Yarn Container Container MapReduce Tasks HBase RegionServer HDFS DataNode Local storage 11
40 MapReduce Shuffler Hadoop YARN Yarn Container Yarn Container Container MapReduce Tasks HBase RegionServer HDFS DataNode Local storage 11
41 MapReduce Hadoop ShufflerYARN NodeManager Hadoop YARN NodeManager Hadoop Yarn Hadoop Yarn Hadoop Yarn Container YARN Container Container YARN Container Yarn Container Yarn Container YARN MapReduce Container Tasks Container MapReduce MapReduce HBase RegionServer HBase RegionServer HBase RegionServer HDFS DataNode HDFS DataNode HDFS DataNode Local storage Local storage Local storage 11
42 Goals 12
43 Goals Co-ordinated control across processes, machines, and services 12
44 Goals Co-ordinated control across processes, machines, and services Handle system and application level resources 12
45 Goals Co-ordinated control across processes, machines, and services Handle system and application level resources Principals: tenants, background tasks 12
46 Goals Co-ordinated control across processes, machines, and services Handle system and application level resources Principals: tenants, background tasks Real-time and reactive 12
47 Goals Co-ordinated control across processes, machines, and services Handle system and application level resources Principals: tenants, background tasks Real-time and reactive Efficient: Only control what is needed 12
48 Architecture 13
49 14
50 Tenant Requests 14
51 Workflows Purpose: identify requests from different users, background activities eg, all requests from a tenant over time eg, data balancing in HDFS 15
52 Workflows Purpose: identify requests from different users, background activities eg, all requests from a tenant over time eg, data balancing in HDFS Unit of resource measurement, attribution, and enforcement 15
53 Workflows Purpose: identify requests from different users, background activities eg, all requests from a tenant over time eg, data balancing in HDFS Unit of resource measurement, attribution, and enforcement Tracks a request across varying levels of granularity Orthogonal to threads, processes, network flows, etc. 15
54 Workflows 16
55 Workflows 16
56 Workflows Resources Purpose: cope with diversity of resources 16
57 Workflows Resources Purpose: cope with diversity of resources What we need: 1. Identify overloaded resources 16
58 Workflows Resources Purpose: cope with diversity of resources What we need: 1. Identify overloaded resources 2. Identify culprit workflows 16
59 Workflows Resources Purpose: cope with diversity of resources What we need: 1. Identify overloaded resources Slowdown 2. Identify culprit workflows Ratio of how slow the resource is now compared to its baseline performance with no contention. 16
60 Workflows Resources Purpose: cope with diversity of resources What we need: 1. Identify overloaded resources Slowdown 2. Identify culprit workflows Load Ratio of how slow the resource is now compared to its baseline performance with no contention. Fraction of current utilization that we can attribute to each workflow 16
61 Workflows Resources Purpose: cope with diversity of resources What we need: 1. Identify overloaded resources Slowdown 2. Identify culprit workflows Load Ratio of how slow the resource is now compared to its baseline performance with no contention. Fraction of current utilization that we can attribute to each workflow 17
62 Workflows Resources Purpose: cope with diversity of resources What we need: 1. Identify overloaded resources Slowdown 2. Identify culprit workflows Load Ratio of how slow the resource is now compared to its baseline performance with no contention. Fraction of current utilization that we can attribute to each workflow 17
63 Workflows Slowdown Load Resources (queue time + execute time) / execute time eg. 100ms queue, 10ms execute => slowdown 11 time spent executing eg. 10ms execute => load 10 Purpose: cope with diversity of resources What we need: 1. Identify overloaded resources Slowdown 2. Identify culprit workflows Load Ratio of how slow the resource is now compared to its baseline performance with no contention. Fraction of current utilization that we can attribute to each workflow 17
64 Workflows Resources 17
65 Workflows Resources Control Points Goal: enforce resource management decisions 18
66 Workflows Resources Control Points Goal: enforce resource management decisions Decoupled from resources Rate-limits workflows, agnostic to underlying implementation e.g., token bucket priority queue 18
67 Workflows Resources Control Points Goal: enforce resource management decisions Decoupled from resources Rate-limits workflows, agnostic to underlying implementation e.g., token bucket priority queue 19
68 Workflows Resources Control Points Goal: enforce resource management decisions Decoupled from resources Rate-limits workflows, agnostic to underlying implementation e.g., token bucket priority queue 19
69 Workflows Resources Control Points Goal: enforce resource management decisions Decoupled from resources Rate-limits workflows, agnostic to underlying implementation e.g., token bucket priority queue 19
70 Workflows Resources Control Points Goal: enforce resource management decisions Decoupled from resources Rate-limits workflows, agnostic to underlying implementation e.g., token bucket priority queue 19
71 Workflows Resources Control Points Goal: enforce resource management decisions Decoupled from resources Rate-limits workflows, agnostic to underlying implementation e.g., token bucket priority queue 19
72 Workflows Resources Control Points Goal: enforce resource management decisions Decoupled from resources Rate-limits workflows, agnostic to underlying implementation e.g., token bucket priority queue 19
73 Workflows Resources Control Points Goal: enforce resource management decisions Decoupled from resources Rate-limits workflows, agnostic to underlying implementation e.g., token bucket priority queue 19
74 Workflows Resources Control Points 20
75 Pervasive Measurement Workflows Resources Control Points 1. Pervasive Measurement Aggregated locally then reported centrally once per second 20
76 Retro Controller API Pervasive Measurement Workflows Resources Control Points 1. Pervasive Measurement Aggregated locally then reported centrally once per second 2. Centralized Controller Global, abstracted view of the system 20
77 Retro Controller API Policy Policy Policy Pervasive Measurement Workflows Resources Control Points 1. Pervasive Measurement Aggregated locally then reported centrally once per second 2. Centralized Controller Global, abstracted view of the system Policies run in continuous control loop 20
78 Retro Controller API Policy Policy Policy Pervasive Measurement 1. Pervasive Measurement Aggregated locally then reported centrally once per second 2. Centralized Controller Global, abstracted view of the system Policies run in continuous control loop Distributed Enforcement Workflows Resources Control Points 3. Distributed Enforcement Co-ordinates enforcement using distributed token bucket 20
79 Retro Controller API Policy Policy Policy Pervasive Measurement Distributed Enforcement Workflows Resources Control Points Control Plane for resource management Global, abstracted view of the system Easier to write Reusable 21
80 Policy Example: LatencySLO 22
81 Policy Example: LatencySLO H High Priority Workflows 200ms average request latency 22
82 Policy Example: LatencySLO H High Priority Workflows L Low Priority Workflows 200ms average request latency (use spare capacity) 22
83 Policy Example: LatencySLO H High Priority Workflows L Low Priority Workflows 200ms average request latency monitor latencies (use spare capacity) 22
84 Policy Example: LatencySLO H High Priority Workflows L Low Priority Workflows 200ms average request latency monitor latencies (use spare capacity) attribute interference 22
85 Policy Example: LatencySLO H High Priority Workflows L Low Priority Workflows 200ms average request latency monitor latencies (use spare capacity) throttle interfering workflows 22
86 Policy Example: LatencySLO H High Priority Workflows L Low Priority Workflows 1 foreach candidate in H 2 miss[candidate] = latency(candidate) / guarantee[candidate] 3 W = candidate in H with max miss[candidate] 4 foreach rsrc in resources() // calculate importance of each resource for hipri 5 importance[rsrc] = latency(w, rsrc) * log(slowdown(rsrc)) 6 foreach lopri in L // calculate low priority workflow interference 7 interference[lopri] = Σ rsrc importance[rsrc] * load(lopri, rsrc) / load(rsrc) 8 foreach lopri in L // normalize interference 9 interference[lopri] /= Σ k interference[k] 10 foreach lopri in L 11 if miss[w] > 1 // throttle 12 scalefactor = 1 α * (miss[w] 1) * interference[lopri] 13 else // release 14 scalefactor = 1 + β 15 foreach cpoint in controlpoints() // apply new rates 16 set_rate(cpoint, lopri, scalefactor * get_rate(cpoint, lopri) 23
87 Policy Example: LatencySLO H High Priority Workflows L Low Priority Workflows 1 foreach candidate in H 2 miss[candidate] = latency(candidate) / guarantee[candidate] 3 W = candidate in H with max miss[candidate] 4 foreach rsrc in resources() // calculate importance of each resource for hipri 5 importance[rsrc] = latency(w, rsrc) * log(slowdown(rsrc)) 6 foreach lopri in L // calculate low priority workflow interference 7 interference[lopri] = Σ rsrc importance[rsrc] * load(lopri, rsrc) / load(rsrc) 8 foreach lopri in L // normalize interference 9 interference[lopri] /= Σ k interference[k] 10 foreach lopri in L 11 if miss[w] > 1 // throttle 12 scalefactor = 1 α * (miss[w] 1) * interference[lopri] 13 else // release 14 scalefactor = 1 + β 15 foreach cpoint in controlpoints() // apply new rates 16 set_rate(cpoint, lopri, scalefactor * get_rate(cpoint, lopri) 23
88 Policy Example: LatencySLO H High Priority Workflows L Low Priority Workflows 1 foreach candidate in H 2 miss[candidate] = latency(candidate) / guarantee[candidate] 3 W = candidate in H with max miss[candidate] Select the high priority workflow W with worst performance 4 foreach rsrc in resources() // calculate importance of each resource for hipri 5 importance[rsrc] = latency(w, rsrc) * log(slowdown(rsrc)) 6 foreach lopri in L // calculate low priority workflow interference 7 interference[lopri] = Σ rsrc importance[rsrc] * load(lopri, rsrc) / load(rsrc) 8 foreach lopri in L // normalize interference 9 interference[lopri] /= Σ k interference[k] 10 foreach lopri in L 11 if miss[w] > 1 // throttle 12 scalefactor = 1 α * (miss[w] 1) * interference[lopri] 13 else // release 14 scalefactor = 1 + β 15 foreach cpoint in controlpoints() // apply new rates 16 set_rate(cpoint, lopri, scalefactor * get_rate(cpoint, lopri) 24
89 Policy Example: LatencySLO H High Priority Workflows L Low Priority Workflows 1 foreach candidate in H 2 miss[candidate] = latency(candidate) / guarantee[candidate] 3 W = candidate in H with max miss[candidate] Select the high priority workflow W with worst performance 4 foreach rsrc in resources() // calculate importance of each resource for hipri 5 importance[rsrc] = latency(w, rsrc) * log(slowdown(rsrc)) Weight low priority workflows by their interference with W 6 foreach lopri in L // calculate low priority workflow interference 7 interference[lopri] = Σ rsrc importance[rsrc] * load(lopri, rsrc) / load(rsrc) 8 foreach lopri in L // normalize interference 9 interference[lopri] /= Σ k interference[k] 10 foreach lopri in L 11 if miss[w] > 1 // throttle 12 scalefactor = 1 α * (miss[w] 1) * interference[lopri] 13 else // release 14 scalefactor = 1 + β 15 foreach cpoint in controlpoints() // apply new rates 16 set_rate(cpoint, lopri, scalefactor * get_rate(cpoint, lopri) 24
90 Policy Example: LatencySLO H High Priority Workflows L Low Priority Workflows 1 foreach candidate in H 2 miss[candidate] = latency(candidate) / guarantee[candidate] 3 W = candidate in H with max miss[candidate] Select the high priority workflow W with worst performance 4 foreach rsrc in resources() // calculate importance of each resource for hipri 5 importance[rsrc] = latency(w, rsrc) * log(slowdown(rsrc)) Weight low priority workflows by their interference with W 6 foreach lopri in L // calculate low priority workflow interference 7 interference[lopri] = Σ rsrc importance[rsrc] * load(lopri, rsrc) / load(rsrc) 8 foreach lopri in L // normalize interference 9 interference[lopri] /= Σ k interference[k] 10 foreach lopri in L 11 if miss[w] > 1 // throttle 12 scalefactor = 1 α * (miss[w] 1) * interference[lopri] 13 else // release 14 scalefactor = 1 + β Throttle low priority workflows proportionally to their weight 15 foreach cpoint in controlpoints() // apply new rates 16 set_rate(cpoint, lopri, scalefactor * get_rate(cpoint, lopri) 24
91 Policy Example: LatencySLO H High Priority Workflows L Low Priority Workflows Select the high priority workflow W with worst performance 1 foreach candidate in H 2 miss[candidate] = latency(candidate) / guarantee[candidate] 3 W = candidate in H with max miss[candidate] 4 foreach rsrc in resources() // calculate importance of each resource for hipri 5 importance[rsrc] = latency(w, rsrc) * log(slowdown(rsrc)) Weight low priority workflows by their interference with W 6 foreach lopri in L // calculate low priority workflow interference 7 interference[lopri] = Σ rsrc importance[rsrc] * load(lopri, rsrc) / load(rsrc) 8 foreach lopri in L // normalize interference 9 interference[lopri] /= Σ k interference[k] 10 foreach lopri in L 11 if miss[w] > 1 // throttle 12 scalefactor = 1 α * (miss[w] 1) * interference[lopri] 13 else // release 14 scalefactor = 1 + β Throttle low priority workflows proportionally to their weight 15 foreach cpoint in controlpoints() // apply new rates 16 set_rate(cpoint, lopri, scalefactor * get_rate(cpoint, lopri) 25
92 Policy Example: LatencySLO H High Priority Workflows L Low Priority Workflows Select the high priority workflow W with worst performance 1 foreach candidate in H 2 miss[candidate] = latency(candidate) / guarantee[candidate] 3 W = candidate in H with max miss[candidate] Weight low priority workflows by their interference with W 4 foreach rsrc in resources() // calculate importance of each resource for hipri 5 importance[rsrc] = latency(w, rsrc) * log(slowdown(rsrc)) 6 foreach lopri in L // calculate low priority workflow interference 7 interference[lopri] = Σ rsrc importance[rsrc] * load(lopri, rsrc) / load(rsrc) 8 foreach lopri in L // normalize interference 9 interference[lopri] /= Σ k interference[k] 10 foreach lopri in L 11 if miss[w] > 1 // throttle 12 scalefactor = 1 α * (miss[w] 1) * interference[lopri] 13 else // release 14 scalefactor = 1 + β Throttle low priority workflows proportionally to their weight 15 foreach cpoint in controlpoints() // apply new rates 16 set_rate(cpoint, lopri, scalefactor * get_rate(cpoint, lopri) 26
93 Policy Example: LatencySLO H High Priority Workflows 1 foreach candidate in H 2 miss[candidate] = latency(candidate) / guarantee[candidate] 3 W = candidate in H with max miss[candidate] 4 foreach rsrc in resources() // calculate importance of each resource for hipri 5 importance[rsrc] = latency(w, rsrc) * log(slowdown(rsrc)) 6 foreach lopri in L // calculate low priority workflow interference 7 interference[lopri] = Σ rsrc importance[rsrc] * load(lopri, rsrc) / load(rsrc) 8 foreach lopri in L // normalize interference 9 interference[lopri] /= Σ k interference[k] L Low Priority Workflows Select the high priority workflow W with worst performance Weight low priority workflows by their interference with W Throttle low priority workflows proportionally to their weight 10 foreach lopri in L 11 if miss[w] > 1 // throttle 12 scalefactor = 1 α * (miss[w] 1) * interference[lopri] 13 else // release 14 scalefactor = 1 + β 15 foreach cpoint in controlpoints() // apply new rates 16 set_rate(cpoint, lopri, scalefactor * get_rate(cpoint, lopri) 27
94 Policy Example: LatencySLO H High Priority Workflows 1 foreach candidate in H 2 miss[candidate] = latency(candidate) / guarantee[candidate] 3 W = candidate in H with max miss[candidate] 4 foreach rsrc in resources() // calculate importance of each resource for hipri 5 importance[rsrc] = latency(w, rsrc) * log(slowdown(rsrc)) 6 foreach lopri in L // calculate low priority workflow interference 7 interference[lopri] = Σ rsrc importance[rsrc] * load(lopri, rsrc) / load(rsrc) 8 foreach lopri in L // normalize interference 9 interference[lopri] /= Σ k interference[k] L Low Priority Workflows Select the high priority workflow W with worst performance Weight low priority workflows by their interference with W Throttle low priority workflows proportionally to their weight 10 foreach lopri in L 11 if miss[w] > 1 // throttle 12 scalefactor = 1 α * (miss[w] 1) * interference[lopri] 13 else // release 14 scalefactor = 1 + β 15 foreach cpoint in controlpoints() // apply new rates 16 set_rate(cpoint, lopri, scalefactor * get_rate(cpoint, lopri) 27
95 Other types of policy 28
96 Other types of policy Policy Bottleneck Fairness Detect most overloaded resource Fair-share resource between tenants using it 28
97 Other types of policy Policy Policy Bottleneck Fairness Detect most overloaded resource Fair-share resource between tenants using it Dominant Resource Fairness Estimate demands and capacities from measurements 28
98 Other types of policy Policy Policy Bottleneck Fairness Detect most overloaded resource Fair-share resource between tenants using it Dominant Resource Fairness Estimate demands and capacities from measurements Concise Any resources can be bottleneck (policy doesn t care) Not system specific 28
99 Evaluation 29
100 Instrumentation 30
101 Instrumentation Retro implementation in Java Instrumentation Library Central controller implementation 30
102 Instrumentation Retro implementation in Java Instrumentation Library Central controller implementation To enable Retro Propagate Workflow ID within application (like X-Trace, Dapper) Instrument resources with wrapper classes 30
103 Instrumentation Retro implementation in Java Instrumentation Library Central controller implementation To enable Retro Propagate Workflow ID within application (like X-Trace, Dapper) Instrument resources with wrapper classes Overheads 31
104 Instrumentation Retro implementation in Java Instrumentation Library Central controller implementation To enable Retro Propagate Workflow ID within application (like X-Trace, Dapper) Instrument resources with wrapper classes Overheads Resource instrumentation automatic using AspectJ 31
105 Instrumentation Retro implementation in Java Instrumentation Library Central controller implementation To enable Retro Propagate Workflow ID within application (like X-Trace, Dapper) Instrument resources with wrapper classes Overheads Resource instrumentation automatic using AspectJ Overall lines per system to modify RPCs 31
106 Instrumentation Retro implementation in Java Instrumentation Library Central controller implementation To enable Retro Propagate Workflow ID within application (like X-Trace, Dapper) Instrument resources with wrapper classes Overheads Resource instrumentation automatic using AspectJ Overall lines per system to modify RPCs Retro overhead: max 1-2% on throughput, latency 31
107 Experiments 32
108 Experiments Systems YARN ZooKeeper MapReduce HBase HDFS 32
109 Experiments Workflows MapReduce Jobs (HiBench) HBase (YCSB) HDFS clients Background Data Replication Background Heartbeats Systems YARN ZooKeeper MapReduce HBase HDFS 32
110 Experiments Workflows MapReduce Jobs (HiBench) HBase (YCSB) HDFS clients Background Data Replication Background Heartbeats Systems YARN ZooKeeper MapReduce HBase HDFS Resources CPU, Disk, Network (All systems) Locks, Queues (HDFS, HBase) 32
111 Experiments Workflows MapReduce Jobs (HiBench) HBase (YCSB) HDFS clients Background Data Replication Background Heartbeats Resources CPU, Disk, Network (All systems) Locks, Queues (HDFS, HBase) Systems YARN ZooKeeper Policies LatencySLO MapReduce HDFS HBase Bottleneck Fairness Dominant Resource Fairness 32
112 Experiments Workflows MapReduce Jobs (HiBench) HBase (YCSB) HDFS clients Background Data Replication Background Heartbeats Resources CPU, Disk, Network (All systems) Locks, Queues (HDFS, HBase) Systems YARN ZooKeeper Policies LatencySLO MapReduce HDFS HBase Bottleneck Fairness Dominant Resource Fairness Policies for a mixture of systems, workflows, and resources Results on clusters up to 200 nodes See paper for full experiment results 32
113 Experiments Workflows MapReduce Jobs (HiBench) HBase (YCSB) HDFS clients Background Data Replication Background Heartbeats Resources CPU, Disk, Network (All systems) Locks, Queues (HDFS, HBase) Systems YARN ZooKeeper Policies LatencySLO MapReduce HDFS HBase Bottleneck Fairness Dominant Resource Fairness Policies for a mixture of systems, workflows, and resources Results on clusters up to 200 nodes See paper for full experiment results This talk LatencySLO policy results 32
114 Experiments Workflows MapReduce Jobs (HiBench) HBase (YCSB) HDFS clients Background Data Replication Background Heartbeats Resources CPU, Disk, Network (All systems) Locks, Queues (HDFS, HBase) Systems YARN ZooKeeper Policies LatencySLO MapReduce HDFS HBase Bottleneck Fairness Dominant Resource Fairness Policies for a mixture of systems, workflows, and resources Results on clusters up to 200 nodes See paper for full experiment results This talk LatencySLO policy results 32
115 HDFS read 8k HBase read 1 row HBase read 1 cached row 33
116 HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k HBase read 1 row HBase read 1 cached row 33
117 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 33
118 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 33
119 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k SLO target Time [minutes] HBase read 1 row HBase read 1 cached row 33
120 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k SLO target Time [minutes] HBase read 1 row HBase read 1 cached row 33
121 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k SLO target Time [minutes] HBase read 1 row HBase read 1 cached row 33
122 Slowdown SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k SLO target Time [minutes] HBase read 1 row HBase read 1 cached row Time [minutes] 34
123 Slowdown SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k SLO target Time [minutes] HBase read 1 row HBase read 1 cached row Disk Time [minutes] 34
124 Slowdown SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k SLO target Time [minutes] HBase read 1 row HBase read 1 cached row Disk 10 5 CPU Time [minutes] 34
125 Slowdown SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k SLO target Time [minutes] HBase read 1 row HBase read 1 cached row Disk 10 CPU Time [minutes] HDFS NN Lock 34
126 Slowdown SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k SLO target Time [minutes] HBase read 1 row HBase read 1 cached row Disk 10 CPU Time [minutes] HDFS NN Lock HDFS NN Queue 34
127 Slowdown SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k SLO target Time [minutes] HBase read 1 row HBase read 1 cached row Disk 10 CPU Time [minutes] HDFS NN Lock HDFS NN Queue HBase Queue 34
128 SLO-Normalized Latency SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k SLO target HBase read 1 row Time [minutes] HBase read 1 cached row 10 + SLO Policy Enabled SLO target Time [minutes] 35
129 SLO-Normalized Latency SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k SLO target HBase read 1 row Time [minutes] HBase read 1 cached row 10 + SLO Policy Enabled SLO target Time [minutes] 35
130 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 36
131 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 1 HBase Table Scan
132 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 1 HBase Table Scan
133 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 1 HBase Table Scan
134 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 1 HBase Table Scan HDFS mkdir
135 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 1 HBase Table Scan HDFS mkdir
136 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 1 HBase Table Scan HDFS mkdir
137 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 1 HBase Table Scan HDFS mkdir HBase Cached 0.5 Table Scan 36
138 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 1 HBase Table Scan HDFS mkdir HBase Cached 0.5 Table Scan 36
139 SLO-Normalized Latency HBase Table Scan HDFS mkdir HBase Cached Table Scan HDFS read 8k Time [minutes] HBase read 1 row HBase read 1 cached row 1 HBase Table Scan HDFS mkdir HBase Cached 0.5 Table Scan 36
140 Conclusion 37
141 Conclusion Resource management for shared distributed systems 37
142 Conclusion Resource management for shared distributed systems Centralized resource management 37
143 Conclusion Resource management for shared distributed systems Centralized resource management Comprehensive: resources, processes, tenants, background tasks 37
144 Conclusion Resource management for shared distributed systems Centralized resource management Comprehensive: resources, processes, tenants, background tasks Abstractions for writing concise, general-purpose policies: Workflows Resources (slowdown, load) Control points 37
145 Conclusion Resource management for shared distributed systems Centralized resource management Comprehensive: resources, processes, tenants, background tasks Abstractions for writing concise, general-purpose policies: Workflows Resources (slowdown, load) Control points
Apache HBase. Crazy dances on the elephant back
Apache HBase Crazy dances on the elephant back Roman Nikitchenko, 16.10.2014 YARN 2 FIRST EVER DATA OS 10.000 nodes computer Recent technology changes are focused on higher scale. Better resource usage
Benchmarking Hadoop & HBase on Violin
Technical White Paper Report Technical Report Benchmarking Hadoop & HBase on Violin Harnessing Big Data Analytics at the Speed of Memory Version 1.0 Abstract The purpose of benchmarking is to show advantages
Big Data Technology Core Hadoop: HDFS-YARN Internals
Big Data Technology Core Hadoop: HDFS-YARN Internals Eshcar Hillel Yahoo! Ronny Lempel Outbrain *Based on slides by Edward Bortnikov & Ronny Lempel Roadmap Previous class Map-Reduce Motivation This class
Overview. Big Data in Apache Hadoop. - HDFS - MapReduce in Hadoop - YARN. https://hadoop.apache.org. Big Data Management and Analytics
Overview Big Data in Apache Hadoop - HDFS - MapReduce in Hadoop - YARN https://hadoop.apache.org 138 Apache Hadoop - Historical Background - 2003: Google publishes its cluster architecture & DFS (GFS)
Hadoop Scalability at Facebook. Dmytro Molkov (dms@fb.com) YaC, Moscow, September 19, 2011
Hadoop Scalability at Facebook Dmytro Molkov (dms@fb.com) YaC, Moscow, September 19, 2011 How Facebook uses Hadoop Hadoop Scalability Hadoop High Availability HDFS Raid How Facebook uses Hadoop Usages
Reference Architecture and Best Practices for Virtualizing Hadoop Workloads Justin Murray VMware
Reference Architecture and Best Practices for Virtualizing Hadoop Workloads Justin Murray ware 2 Agenda The Hadoop Journey Why Virtualize Hadoop? Elasticity and Scalability Performance Tests Storage Reference
PEPPERDATA IN MULTI-TENANT ENVIRONMENTS
..................................... PEPPERDATA IN MULTI-TENANT ENVIRONMENTS technical whitepaper June 2015 SUMMARY OF WHAT S WRITTEN IN THIS DOCUMENT If you are short on time and don t want to read the
CS2510 Computer Operating Systems
CS2510 Computer Operating Systems HADOOP Distributed File System Dr. Taieb Znati Computer Science Department University of Pittsburgh Outline HDF Design Issues HDFS Application Profile Block Abstraction
CS2510 Computer Operating Systems
CS2510 Computer Operating Systems HADOOP Distributed File System Dr. Taieb Znati Computer Science Department University of Pittsburgh Outline HDF Design Issues HDFS Application Profile Block Abstraction
MapReduce Job Processing
April 17, 2012 Background: Hadoop Distributed File System (HDFS) Hadoop requires a Distributed File System (DFS), we utilize the Hadoop Distributed File System (HDFS). Background: Hadoop Distributed File
Virtualizing Apache Hadoop. June, 2012
June, 2012 Table of Contents EXECUTIVE SUMMARY... 3 INTRODUCTION... 3 VIRTUALIZING APACHE HADOOP... 4 INTRODUCTION TO VSPHERE TM... 4 USE CASES AND ADVANTAGES OF VIRTUALIZING HADOOP... 4 MYTHS ABOUT RUNNING
Data-Intensive Programming. Timo Aaltonen Department of Pervasive Computing
Data-Intensive Programming Timo Aaltonen Department of Pervasive Computing Data-Intensive Programming Lecturer: Timo Aaltonen University Lecturer timo.aaltonen@tut.fi Assistants: Henri Terho and Antti
Hadoop Ecosystem B Y R A H I M A.
Hadoop Ecosystem B Y R A H I M A. History of Hadoop Hadoop was created by Doug Cutting, the creator of Apache Lucene, the widely used text search library. Hadoop has its origins in Apache Nutch, an open
Use of Hadoop File System for Nuclear Physics Analyses in STAR
1 Use of Hadoop File System for Nuclear Physics Analyses in STAR EVAN SANGALINE UC DAVIS Motivations 2 Data storage a key component of analysis requirements Transmission and storage across diverse resources
Hadoop implementation of MapReduce computational model. Ján Vaňo
Hadoop implementation of MapReduce computational model Ján Vaňo What is MapReduce? A computational model published in a paper by Google in 2004 Based on distributed computation Complements Google s distributed
Cloudera Manager Health Checks
Cloudera, Inc. 220 Portage Avenue Palo Alto, CA 94306 info@cloudera.com US: 1-888-789-1488 Intl: 1-650-362-0488 www.cloudera.com Cloudera Manager Health Checks Important Notice 2010-2013 Cloudera, Inc.
HDFS Under the Hood. Sanjay Radia. Sradia@yahoo-inc.com Grid Computing, Hadoop Yahoo Inc.
HDFS Under the Hood Sanjay Radia Sradia@yahoo-inc.com Grid Computing, Hadoop Yahoo Inc. 1 Outline Overview of Hadoop, an open source project Design of HDFS On going work 2 Hadoop Hadoop provides a framework
Enabling High performance Big Data platform with RDMA
Enabling High performance Big Data platform with RDMA Tong Liu HPC Advisory Council Oct 7 th, 2014 Shortcomings of Hadoop Administration tooling Performance Reliability SQL support Backup and recovery
A very short Intro to Hadoop
4 Overview A very short Intro to Hadoop photo by: exfordy, flickr 5 How to Crunch a Petabyte? Lots of disks, spinning all the time Redundancy, since disks die Lots of CPU cores, working all the time Retry,
HADOOP MOCK TEST HADOOP MOCK TEST I
http://www.tutorialspoint.com HADOOP MOCK TEST Copyright tutorialspoint.com This section presents you various set of Mock Tests related to Hadoop Framework. You can download these sample mock tests at
Communicating with the Elephant in the Data Center
Communicating with the Elephant in the Data Center Who am I? Instructor Consultant Opensource Advocate http://www.laubersoltions.com sml@laubersolutions.com Twitter: @laubersm Freenode: laubersm Outline
Comparing Scalable NOSQL Databases
Comparing Scalable NOSQL Databases Functionalities and Measurements Dory Thibault UCL Contact : thibault.dory@student.uclouvain.be Sponsor : Euranova Website : nosqlbenchmarking.com February 15, 2011 Clarications
Apache Hadoop. Alexandru Costan
1 Apache Hadoop Alexandru Costan Big Data Landscape No one-size-fits-all solution: SQL, NoSQL, MapReduce, No standard, except Hadoop 2 Outline What is Hadoop? Who uses it? Architecture HDFS MapReduce Open
Introduction to Hadoop. New York Oracle User Group Vikas Sawhney
Introduction to Hadoop New York Oracle User Group Vikas Sawhney GENERAL AGENDA Driving Factors behind BIG-DATA NOSQL Database 2014 Database Landscape Hadoop Architecture Map/Reduce Hadoop Eco-system Hadoop
Can High-Performance Interconnects Benefit Memcached and Hadoop?
Can High-Performance Interconnects Benefit Memcached and Hadoop? D. K. Panda and Sayantan Sur Network-Based Computing Laboratory Department of Computer Science and Engineering The Ohio State University,
Architectural patterns for building real time applications with Apache HBase. Andrew Purtell Committer and PMC, Apache HBase
Architectural patterns for building real time applications with Apache HBase Andrew Purtell Committer and PMC, Apache HBase Who am I? Distributed systems engineer Principal Architect in the Big Data Platform
CSE 590: Special Topics Course ( Supercomputing ) Lecture 10 ( MapReduce& Hadoop)
CSE 590: Special Topics Course ( Supercomputing ) Lecture 10 ( MapReduce& Hadoop) Rezaul A. Chowdhury Department of Computer Science SUNY Stony Brook Spring 2016 MapReduce MapReduce is a programming model
Applied Storage Performance For Big Analytics. PRESENTATION TITLE GOES HERE Hubbert Smith LSI
Applied Storage Performance For Big Analytics PRESENTATION TITLE GOES HERE Hubbert Smith LSI It s NOT THIS SIMPLE!!! 2 Theoretical vs Real World Theoretical & Lab Storage Workloads I/O I/O I/O I/O I/O
Hadoop. History and Introduction. Explained By Vaibhav Agarwal
Hadoop History and Introduction Explained By Vaibhav Agarwal Agenda Architecture HDFS Data Flow Map Reduce Data Flow Hadoop Versions History Hadoop version 2 Hadoop Architecture HADOOP (HDFS) Data Flow
Large scale processing using Hadoop. Ján Vaňo
Large scale processing using Hadoop Ján Vaňo What is Hadoop? Software platform that lets one easily write and run applications that process vast amounts of data Includes: MapReduce offline computing engine
EXPERIMENTATION. HARRISON CARRANZA School of Computer Science and Mathematics
BIG DATA WITH HADOOP EXPERIMENTATION HARRISON CARRANZA Marist College APARICIO CARRANZA NYC College of Technology CUNY ECC Conference 2016 Poughkeepsie, NY, June 12-14, 2016 Marist College AGENDA Contents
Deploying Hadoop with Manager
Deploying Hadoop with Manager SUSE Big Data Made Easier Peter Linnell / Sales Engineer plinnell@suse.com Alejandro Bonilla / Sales Engineer abonilla@suse.com 2 Hadoop Core Components 3 Typical Hadoop Distribution
Prepared By : Manoj Kumar Joshi & Vikas Sawhney
Prepared By : Manoj Kumar Joshi & Vikas Sawhney General Agenda Introduction to Hadoop Architecture Acknowledgement Thanks to all the authors who left their selfexplanatory images on the internet. Thanks
Cloudera Enterprise Reference Architecture for Google Cloud Platform Deployments
Cloudera Enterprise Reference Architecture for Google Cloud Platform Deployments Important Notice 2010-2016 Cloudera, Inc. All rights reserved. Cloudera, the Cloudera logo, Cloudera Impala, Impala, and
... ... PEPPERDATA OVERVIEW AND DIFFERENTIATORS ... ... ... ... ...
..................................... WHITEPAPER PEPPERDATA OVERVIEW AND DIFFERENTIATORS INTRODUCTION Prospective customers will often pose the question, How is Pepperdata different from tools like Ganglia,
Distributed File System. MCSN N. Tonellotto Complements of Distributed Enabling Platforms
Distributed File System 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributed File System Don t move data to workers move workers to the data! Store data on the local disks of nodes
YARN Apache Hadoop Next Generation Compute Platform
YARN Apache Hadoop Next Generation Compute Platform Bikas Saha @bikassaha Hortonworks Inc. 2013 Page 1 Apache Hadoop & YARN Apache Hadoop De facto Big Data open source platform Running for about 5 years
lprof : A Non- intrusive Request Flow Pro5iler for Distributed Systems
lprof : A Non- intrusive Request Flow Pro5iler for Distributed Systems Xu Zhao, Yongle Zhang, David Lion, Muhammad FaizanUllah, Yu Luo, Ding Yuan, Michael Stumm Oct. 8th 2014 Performance analysis tools
The Hadoop Distributed File System
The Hadoop Distributed File System Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Yahoo! Sunnyvale, California USA {Shv, Hairong, SRadia, Chansler}@Yahoo-Inc.com Presenter: Alex Hu HDFS
Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware
Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware Created by Doug Cutting and Mike Carafella in 2005. Cutting named the program after
Cloudera Enterprise Reference Architecture for Google Cloud Platform Deployments
Cloudera Enterprise Reference Architecture for Google Cloud Platform Deployments Important Notice 2010-2015 Cloudera, Inc. All rights reserved. Cloudera, the Cloudera logo, Cloudera Impala, Impala, and
Big Data Analytics(Hadoop) Prepared By : Manoj Kumar Joshi & Vikas Sawhney
Big Data Analytics(Hadoop) Prepared By : Manoj Kumar Joshi & Vikas Sawhney General Agenda Understanding Big Data and Big Data Analytics Getting familiar with Hadoop Technology Hadoop release and upgrades
There's Plenty of Room in the Cloud
There's Plenty of Room in the Cloud [Shameless reference to Feynman s talk from 1959] Lecturer: Zoran Dimitrijevic Altiscale, Inc. Spring 2015 CS290B -- Cloud Computing 50 Years of Moore
Operations and Big Data: Hadoop, Hive and Scribe. Zheng Shao @ 铮 9 12/7/2011 Velocity China 2011
Operations and Big Data: Hadoop, Hive and Scribe Zheng Shao @ 铮 9 12/7/2011 Velocity China 2011 Agenda 1 Operations: Challenges and Opportunities 2 Big Data Overview 3 Operations with Big Data 4 Big Data
Hadoop Distributed File System. Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org June 3 rd, 2008
Hadoop Distributed File System Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org June 3 rd, 2008 Who Am I? Hadoop Developer Core contributor since Hadoop s infancy Focussed
Introduction to HDFS. Prasanth Kothuri, CERN
Prasanth Kothuri, CERN 2 What s HDFS HDFS is a distributed file system that is fault tolerant, scalable and extremely easy to expand. HDFS is the primary distributed storage for Hadoop applications. Hadoop
Cloudera Manager Training: Hands-On Exercises
201408 Cloudera Manager Training: Hands-On Exercises General Notes... 2 In- Class Preparation: Accessing Your Cluster... 3 Self- Study Preparation: Creating Your Cluster... 4 Hands- On Exercise: Working
Big Data Trends and HDFS Evolution
Big Data Trends and HDFS Evolution Sanjay Radia Founder & Architect Hortonworks Inc Page 1 Hello Founder, Hortonworks Part of the Hadoop team at Yahoo! since 2007 Chief Architect of Hadoop Core at Yahoo!
Sujee Maniyam, ElephantScale
Hadoop PRESENTATION 2 : New TITLE and GOES Noteworthy HERE Sujee Maniyam, ElephantScale SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member
Case Study : 3 different hadoop cluster deployments
Case Study : 3 different hadoop cluster deployments Lee moon soo moon@nflabs.com HDFS as a Storage Last 4 years, our HDFS clusters, stored Customer 1500 TB+ data safely served 375,000 TB+ data to customer
BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB
BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB Planet Size Data!? Gartner s 10 key IT trends for 2012 unstructured data will grow some 80% over the course of the next
Architecting For Failure Why Cloud Architecture is Different! Michael Stiefel www.reliablesoftware.com development@reliablesoftware.
Architecting For Failure Why Cloud Architecture is Different! Michael Stiefel www.reliablesoftware.com development@reliablesoftware.com Outsource Infrastructure? Traditional Web Application Web Site Virtual
Cloudera Manager Health Checks
Cloudera, Inc. 1001 Page Mill Road Palo Alto, CA 94304-1008 info@cloudera.com US: 1-888-789-1488 Intl: 1-650-362-0488 www.cloudera.com Cloudera Manager Health Checks Important Notice 2010-2013 Cloudera,
HDFS: Hadoop Distributed File System
Istanbul Şehir University Big Data Camp 14 HDFS: Hadoop Distributed File System Aslan Bakirov Kevser Nur Çoğalmış Agenda Distributed File System HDFS Concepts HDFS Interfaces HDFS Full Picture Read Operation
Big Data With Hadoop
With Saurabh Singh singh.903@osu.edu The Ohio State University February 11, 2016 Overview 1 2 3 Requirements Ecosystem Resilient Distributed Datasets (RDDs) Example Code vs Mapreduce 4 5 Source: [Tutorials
Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components
Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components of Hadoop. We will see what types of nodes can exist in a Hadoop
Managing large clusters resources
Managing large clusters resources ID2210 Gautier Berthou (SICS) Big Processing with No Locality Job( /crawler/bot/jd.io/1 ) submi t Workflow Manager Compute Grid Node Job This doesn t scale. Bandwidth
Hadoop Architecture. Part 1
Hadoop Architecture Part 1 Node, Rack and Cluster: A node is simply a computer, typically non-enterprise, commodity hardware for nodes that contain data. Consider we have Node 1.Then we can add more nodes,
Apache Hadoop FileSystem and its Usage in Facebook
Apache Hadoop FileSystem and its Usage in Facebook Dhruba Borthakur Project Lead, Apache Hadoop Distributed File System dhruba@apache.org Presented at Indian Institute of Technology November, 2010 http://www.facebook.com/hadoopfs
Scalable Forensics with TSK and Hadoop. Jon Stewart
Scalable Forensics with TSK and Hadoop Jon Stewart CPU Clock Speed Hard Drive Capacity The Problem CPU clock speed stopped doubling Hard drive capacity kept doubling Multicore CPUs to the rescue!...but
Big Data and Hadoop. Module 1: Introduction to Big Data and Hadoop. Module 2: Hadoop Distributed File System. Module 3: MapReduce
Big Data and Hadoop Module 1: Introduction to Big Data and Hadoop Learn about Big Data and the shortcomings of the prevailing solutions for Big Data issues. You will also get to know, how Hadoop eradicates
Introduction to HDFS. Prasanth Kothuri, CERN
Prasanth Kothuri, CERN 2 What s HDFS HDFS is a distributed file system that is fault tolerant, scalable and extremely easy to expand. HDFS is the primary distributed storage for Hadoop applications. HDFS
Hadoop: Embracing future hardware
Hadoop: Embracing future hardware Suresh Srinivas @suresh_m_s Page 1 About Me Architect & Founder at Hortonworks Long time Apache Hadoop committer and PMC member Designed and developed many key Hadoop
L1: Introduction to Hadoop
L1: Introduction to Hadoop Feng Li feng.li@cufe.edu.cn School of Statistics and Mathematics Central University of Finance and Economics Revision: December 1, 2014 Today we are going to learn... 1 General
Introduction to Hadoop HDFS and Ecosystems. Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data
Introduction to Hadoop HDFS and Ecosystems ANSHUL MITTAL Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data Topics The goal of this presentation is to give
Benchmarking Top NoSQL Databases Apache Cassandra, Couchbase, HBase, and MongoDB Originally Published: April 13, 2015 Revised: May 27, 2015
Benchmarking Top NoSQL Databases Apache Cassandra, Couchbase, HBase, and MongoDB Originally Published: April 13, 2015 Revised: May 27, 2015 http://www.endpoint.com/ Table of Contents Executive Summary...
HDFS Federation. Sanjay Radia Founder and Architect @ Hortonworks. Page 1
HDFS Federation Sanjay Radia Founder and Architect @ Hortonworks Page 1 About Me Apache Hadoop Committer and Member of Hadoop PMC Architect of core-hadoop @ Yahoo - Focusing on HDFS, MapReduce scheduler,
Getting Started with Hadoop. Raanan Dagan Paul Tibaldi
Getting Started with Hadoop Raanan Dagan Paul Tibaldi What is Apache Hadoop? Hadoop is a platform for data storage and processing that is Scalable Fault tolerant Open source CORE HADOOP COMPONENTS Hadoop
Parallel Data Mining and Assurance Service Model Using Hadoop in Cloud
Parallel Data Mining and Assurance Service Model Using Hadoop in Cloud Aditya Jadhav, Mahesh Kukreja E-mail: aditya.jadhav27@gmail.com & mr_mahesh_in@yahoo.co.in Abstract : In the information industry,
Hadoop Distributed File System. Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org dhruba@facebook.com
Hadoop Distributed File System Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org dhruba@facebook.com Hadoop, Why? Need to process huge datasets on large clusters of computers
Hadoop Distributed File System. T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela
Hadoop Distributed File System T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela Agenda Introduction Flesh and bones of HDFS Architecture Accessing data Data replication strategy Fault tolerance
COURSE CONTENT Big Data and Hadoop Training
COURSE CONTENT Big Data and Hadoop Training 1. Meet Hadoop Data! Data Storage and Analysis Comparison with Other Systems RDBMS Grid Computing Volunteer Computing A Brief History of Hadoop Apache Hadoop
Building a SaaS Application. ReddyRaja Annareddy CTO and Founder
Building a SaaS Application ReddyRaja Annareddy CTO and Founder Introduction As cloud becomes more and more prevalent, many ISV s and enterprise are looking forward to move their services and offerings
Experiences with Lustre* and Hadoop*
Experiences with Lustre* and Hadoop* Gabriele Paciucci (Intel) June, 2014 Intel * Some Con fidential name Do Not Forward and brands may be claimed as the property of others. Agenda Overview Intel Enterprise
Lecture 5: GFS & HDFS! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl
Big Data Processing, 2014/15 Lecture 5: GFS & HDFS!! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl 1 Course content Introduction Data streams 1 & 2 The MapReduce paradigm Looking behind
Apache Hadoop FileSystem Internals
Apache Hadoop FileSystem Internals Dhruba Borthakur Project Lead, Apache Hadoop Distributed File System dhruba@apache.org Presented at Storage Developer Conference, San Jose September 22, 2010 http://www.facebook.com/hadoopfs
Realtime Apache Hadoop at Facebook. Jonathan Gray & Dhruba Borthakur June 14, 2011 at SIGMOD, Athens
Realtime Apache Hadoop at Facebook Jonathan Gray & Dhruba Borthakur June 14, 2011 at SIGMOD, Athens Agenda 1 Why Apache Hadoop and HBase? 2 Quick Introduction to Apache HBase 3 Applications of HBase at
Hadoop. Apache Hadoop is an open-source software framework for storage and large scale processing of data-sets on clusters of commodity hardware.
Hadoop Source Alessandro Rezzani, Big Data - Architettura, tecnologie e metodi per l utilizzo di grandi basi di dati, Apogeo Education, ottobre 2013 wikipedia Hadoop Apache Hadoop is an open-source software
Some quick definitions regarding the Tablet states in this state machine:
HBase Terminology Translation Legend: tablet == region Accumulo Master == HBase HMaster Metadata table == META TabletServer (or tserver) == RegionServer Assignment is driven solely by the Master process.
Hadoop Distributed File System. Dhruba Borthakur June, 2007
Hadoop Distributed File System Dhruba Borthakur June, 2007 Goals of HDFS Very Large Distributed File System 10K nodes, 100 million files, 10 PB Assumes Commodity Hardware Files are replicated to handle
Qsoft Inc www.qsoft-inc.com
Big Data & Hadoop Qsoft Inc www.qsoft-inc.com Course Topics 1 2 3 4 5 6 Week 1: Introduction to Big Data, Hadoop Architecture and HDFS Week 2: Setting up Hadoop Cluster Week 3: MapReduce Part 1 Week 4:
How To Scale Out Of A Nosql Database
Firebird meets NoSQL (Apache HBase) Case Study Firebird Conference 2011 Luxembourg 25.11.2011 26.11.2011 Thomas Steinmaurer DI +43 7236 3343 896 thomas.steinmaurer@scch.at www.scch.at Michael Zwick DI
Hadoop & Spark Using Amazon EMR
Hadoop & Spark Using Amazon EMR Michael Hanisch, AWS Solutions Architecture 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda Why did we build Amazon EMR? What is Amazon EMR?
Journal of science STUDY ON REPLICA MANAGEMENT AND HIGH AVAILABILITY IN HADOOP DISTRIBUTED FILE SYSTEM (HDFS)
Journal of science e ISSN 2277-3290 Print ISSN 2277-3282 Information Technology www.journalofscience.net STUDY ON REPLICA MANAGEMENT AND HIGH AVAILABILITY IN HADOOP DISTRIBUTED FILE SYSTEM (HDFS) S. Chandra
Forensic Clusters: Advanced Processing with Open Source Software. Jon Stewart Geoff Black
Forensic Clusters: Advanced Processing with Open Source Software Jon Stewart Geoff Black Who We Are Mac Lightbox Guidance alum Mr. EnScript C++ & Java Developer Fortune 100 Financial NCIS (DDK/ManTech)
Accelerating and Simplifying Apache
Accelerating and Simplifying Apache Hadoop with Panasas ActiveStor White paper NOvember 2012 1.888.PANASAS www.panasas.com Executive Overview The technology requirements for big data vary significantly
Network-Aware Scheduling of MapReduce Framework on Distributed Clusters over High Speed Networks
Network-Aware Scheduling of MapReduce Framework on Distributed Clusters over High Speed Networks Praveenkumar Kondikoppa, Chui-Hui Chiu, Cheng Cui, Lin Xue and Seung-Jong Park Department of Computer Science,
Introduction to Cloud Computing
Introduction to Cloud Computing Cloud Computing I (intro) 15 319, spring 2010 2 nd Lecture, Jan 14 th Majd F. Sakr Lecture Motivation General overview on cloud computing What is cloud computing Services
Introduction to Big data. Why Big data? Case Studies. Introduction to Hadoop. Understanding Features of Hadoop. Hadoop Architecture.
Big Data Hadoop Administration and Developer Course This course is designed to understand and implement the concepts of Big data and Hadoop. This will cover right from setting up Hadoop environment in
Storage Architectures for Big Data in the Cloud
Storage Architectures for Big Data in the Cloud Sam Fineberg HP Storage CT Office/ May 2013 Overview Introduction What is big data? Big Data I/O Hadoop/HDFS SAN Distributed FS Cloud Summary Research Areas
DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing WHAT IS CLOUD COMPUTING? 2
![DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing WHAT IS CLOUD COMPUTING? 2](/thumbs/27/10009660.jpg "DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing WHAT IS CLOUD COMPUTING? 2")
DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing Slide 1 Slide 3 A style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet.
Introduction to Cloud Computing
Discovery 2015: Cloud Computing Workshop June 20-24, 2011 Berkeley, CA Introduction to Cloud Computing Keith R. Jackson Lawrence Berkeley National Lab What is it? NIST Definition Cloud computing is a model
BookKeeper. Flavio Junqueira Yahoo! Research, Barcelona. Hadoop in China 2011
BookKeeper Flavio Junqueira Yahoo! Research, Barcelona Hadoop in China 2011 What s BookKeeper? Shared storage for writing fast sequences of byte arrays Data is replicated Writes are striped Many processes
Practical Performance Understanding the Performance of Your Application
Neil Masson IBM Java Service Technical Lead 25 th September 2012 Practical Performance Understanding the Performance of Your Application 1 WebSphere User Group: Practical Performance Understand the Performance
Pro Apache Hadoop. Second Edition. Sameer Wadkar. Madhu Siddalingaiah
Pro Apache Hadoop Second Edition Sameer Wadkar Madhu Siddalingaiah Contents J About the Authors About the Technical Reviewer Acknowledgments Introduction xix xxi xxiii xxv Chapter 1: Motivation for Big
Multi-Datacenter Replication
www.basho.com Multi-Datacenter Replication A Technical Overview & Use Cases Table of Contents Table of Contents... 1 Introduction... 1 How It Works... 1 Default Mode...1 Advanced Mode...2 Architectural
Matt Benton, Mike Bull, Josh Fyne, Georgie Mackender, Richard Meal, Louise Millard, Bogdan Paunescu, Chencheng Zhang
GROUP 5 Hadoop Revision Report Matt Benton, Mike Bull, Josh Fyne, Georgie Mackender, Richard Meal, Louise Millard, Bogdan Paunescu, Chencheng Zhang 9 Dec 2010 Table of Contents 1. Introduction... 2 1.1.
Datacenter Operating Systems
Datacenter Operating Systems CSE451 Simon Peter With thanks to Timothy Roscoe (ETH Zurich) Autumn 2015 This Lecture What s a datacenter Why datacenters Types of datacenters Hyperscale datacenters Major
Take An Internal Look at Hadoop. Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com
Take An Internal Look at Hadoop Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com What s Hadoop Framework for running applications on large clusters of commodity hardware Scale: petabytes of data
Distributed File Systems
Distributed File Systems Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications.