Realtime Linux Kernel Features

Size: px

Start display at page:

Download "Realtime Linux Kernel Features"

Jonah McKinney
8 years ago
Views:

1 Realtime Linux Kernel Features Tim Burke, Red Hat, Director Emerging Technologies Special guest appearance, Ted Tso of IBM

2 Realtime what does it mean to you?

3 Agenda What? Terminology, Target capabilities How? Highlights of upstream realtime kernel development Performance results IBM realtime Java by Ted Tso

4 Throughput & variability Average times vary considerably No prioritization Favors throughput (traffic volume / time)

5 Predictability, deterministic High speed lane Prioritization Determinism = consistent response time = low latency Downside Average throughput may decrease

6 Example use cases Financial Services time is money, consistency is law Government, command & control Telco & network infrastructure Industrial automation

7 Low latency response time strategies Increase concurrency Remove serialization Split up lengthy operations Downside not everyone needs a 4 person tire changing pit crew

8 What? Deterministic latency Predictability in response time Ensure that highest priority processes run first Deterministic upper bound on latency Not guaranteed response time, but greatly reduced probability of poor response Example: 100,000 transactions in a minute 99,999 at 2ms and 1 at 20ms isn't good enough Avoid non deterministic kernel black holes and serialization points

greatly reduced probability of poor response Example: 100,000 transactions in a minute 99,999

9 What? Timing targets Deterministic upper bound on latency. 10 µs sleep time, vs 2 ms in stock RHEL5 Context switch latency under 25 µs, % under 20 µs (from interrupt to commence running new process) Highly accurate gettimeofday() with nanosecond resolution Average of 38% improvement vs stock RHEL5 Can be tuned to µs accuracy vs ms based system call performance optimization Note: lower bound on timing constraints is hardware dependent

9999% under 20 µs (from interrupt to commence running new process) Highly accurate gettimeofday() with

10 How? Upstream community development rt patchset A working sandbox for a community of developers & users The recognized Linux epicenter for RT active work in progress Comprehensive RT enablement, not just a quick VM/scheduler hack A growing collection of many features. Not just 1 thing. Many of its features & enabling cleanups are already upstream. Large patch set in core kernel codepaths (interrupts, scheduler, locks, etc). 120,000 lines at its height. 80K after Many more in Other parts will go into.22,.23,... x86 & x86 64 (Intel & AMD)

Not just 1 thing. Many of its features & enabling cleanups are already upstream.

11 How? Community Kernel Project Upstream rt developer participants and approximate contribution rate 45% Ingo Molnar lead developer and overall upstream leader. Focus on scheduler, locking, interrupts. Red Hat fulltime employee 35% Thomas Gleixner developer, primarily concentrating on timers. Contractor to Red Hat 10% Steven Rostedt developer. Red Hat fulltime employee 10% all other participants (IBM, Monta Vista, Timesys, Novell, etc) All efforts are ultimately shaped towards long term mainstream inclusion Additional Red Hat crew Testing & test development Fixing Tool development system mgt & performance monitoring

Contractor to Red Hat 10% Steven Rostedt developer.

12 Real time kernel work upstream Items and prior are in RHEL5 Priority inheritance futexes (PI futex) (2.6.18) Work over the last 2+ years. 90% Red Hat, 1.2million lines Mostly maintained in rt tree Generic IRQ layer (2.6.18) Core time re write (2.6.18) Sleepable RCU (2.6.19) Latency Tracer (2.6.19) High res+dynticks (2.6.21) Over the last year, many patches have moved from rt to mainline kernel: BKL preemptable (2.6.8) Mutex patch (2.6.16) Semaphore to Mutex conversion (ongoing ~85% done) Hrtimers subsystem (2.6.16) Robust futexes (2.6.17) Lock validator (2.6.18) Conversion of spin locks to mutex ( ) All Interrupt handling in threads (~ ) Full rt preempt (~ )

6.8) Mutex patch (2.6.16) Semaphore to Mutex conversion (ongoing ~85% done) Hrtimers subsystem (2.6.16) Robust futexes (2.6.17) Lock validator (2.6.18) Conversion of spin locks to mutex (2.

13 Upstream success Improve kernel lock synchronization Improve granularity identify and correct contention points Mutex rather than semaphores Mutexes are lighter weight Lock validator Efficient runtime confirmation of lock ordering Can detect race conditions without actually hitting them Priority Inheritance (PI) Prevents low priority processes blocking higher priority. Problem scenario: Low priority process takes lock High priority process needs lock, but must wait Long running medium priority process preempts low priority process Solution: temporarily boost low pri process to allow completion Required for realtime java 1000's of threads

low priority processes blocking higher priority.

14 Upstream success timer precision & interrupt handling Timer enhancements Infrastructure cleanup factor common code, increase fields to represent nanosecond precision Timer precision utilize high resolution hardware timers at microsecond precision rather than approximate periodic time interrupt millisecond precision Generic timeof day cleanly accommodate diverse clock sources VDSO gettimeofday() performance enhancement for millisecond accuracy Dynamic ticks power savings no need to to timer interrupt 1000 times per second on idle system transition to low power state (great for OLPC) Interrupt handling Generic IRQ mechanism infrastructure cleanup factor common code More fine grained hardware interrupt control

accommodate diverse clock sources VDSO gettimeofday() performance enhancement for millisecond accuracy Dynamic ticks power savings no need to to timer interrupt 1000 times per

15 Realtime kernel performance 01/2007 Testing Results x86_64 em64t, amd64 4 socket dual core AMD Ghz 2 socket dual core Intel Woodcrest 3Ghz Workload Tibco latency test using standard, unmodified application RT kernel rt vs RHEL5 (both tuned) Tibco EMS App Improve spikes and variability by 10x Maintain within 20% overall response time Lmbench Improvements of context swtich rates +20 to 61% Degradation in file system perf 21% due to RT nature of handling file I/O

16 Red Hat Confidential

17 RHEL5 tuned vs RT tuned 05/2007 Red Hat Confidential

18 Number of Samples per 10k messages Histogram of Tibco EMS Response Times RHEL rt <1 ms Tibco App 1 2 ms 2 5ms > 200 Peak 10ms 20ms 50ms 100ms ms ms <1 ms 1 2 ms 2 5ms 5 10ms 10 20ms RHEL rt Red Hat Confidential 20 50ms50 100ms > 200 ms Peak ms

0 5 10 20 50 > 200 Peak 10ms 20ms 50ms 100ms ms ms <1 ms 1 2 ms 2 5ms 5 10ms 10 20ms RHEL5 9997642.5 622.0 8264.0 11482.0 4925.

19 How? performance monitoring tools Existing standard RHEL5 based performance monitoring tools remain relevant Gdb, OProfile Frysk source level debuggers & profiler SystemTap, kprobe kernel event tracing and dynamic data collection kexec/kdump standard kernel dump / savecore capabilities Latency Tracer new feature post RHEL5 Runtime trace capture of longest latency codepaths Peak detector Selectable triggers for threshold tracing Detailed kernel profiles based on latency triggers

kexec/kdump standard kernel dump / savecore capabilities Latency Tracer new feature post RHEL5 Runtime trace capture of

20 How? No application changes required All of the realtime enhancements are in the kernel under the hood from an application perspective. No application changes are required to benefit from realtime enhancements. Applications which are latency bottlenecked due to kernel scheduling and interrupt handling will see benefit. Latencies introduced entirely in userspace (sub optimal application code, unbounded java garbage collection, etc) are not eliminated. Recompilation is not required (same gcc/glibc as standard RHEL5) Applications recompiled on RHEL5 benefit from pi mutex glibc implementation enhancements to avoid syscall overhead on uncontested locks.

Applications which are latency bottlenecked due to kernel scheduling and interrupt handling will see benefit.

21 What? The broader picture Most use cases are not isolated systems Commonly networked pipeline of systems Job Scheduler Load Balancer Clients Computation Storage Archival file/db

22 Realtime & Messaging go hand in hand Most of the projects that drive realtime requirements include messaging Realtime has advantages for any message system, but unique advantages when combined with RHM. Red Hat is a co founder in open standards and implementation of messaging technology Requirements cover: predictability, high speed, reliability, multi vendor interoperability, security and scalability. AMQP Advanced Message Queuing Protocol an open standard for message oriented middleware. In development 3+ years, went public June, End user participation Qpid Apache project implementation of AMQP, Upstream for RHM Upcoming Red Hat Enterprise Messaging product offering coming soon.

23 More info Upstream realtime development Mailing list Kernel source download Test programs Presentations Discuss interest with Red Hat sales & marketing

24 Real Time in the Real World Navy contracted to Raytheon to design the computer systems on the next generation Destroyer, or the DDG 1000 project. technology.com/projects/dd21 Real time Java from IBM running on Real time Linux was key to Raytheon choice to partner with IBM instead of Sun. IBM Blades and real time JVM will be used to build the central data centers.

25 IBM's Websphere Real Time A full J2SE JVM with RTSJ support With an IBM exclusive real time garbage collector 1ms maximum pause time Uses at most 30% CPU utilization over any 10ms window. Available today on a modified 32 bit RHEL4 system from IBM. IBM is working with Red Hat to make this available on a productized real time version of RHEL5

26 The complete enterprise realtime picture

Red Hat Summit 2009 Bryan Che

Red Hat Summit 2009 Bryan Che 1 Red Hat Enterprise MRG Update Bryan Che Product Manager, Red Hat September 2, 2009 2 About Red Hat Enterprise MRG Integrated platform for high performance distributed computing High speed, interoperable,