Frysk The Systems Monitoring and Debugging Tool Andrew Cagney
Agenda Two Use Cases Motivation Comparison with Existing Free Technologies The Frysk Architecture and GUI Command Line Utilities Current Status and Future Road map
What is Frysk Always on system monitoring and debugging tool Lets Users: Observe running threads, processes, and hosts Follow locking and other more complex primitives Debug complete systems Front row seats to system s execution
Use Case #1, K. K. is a compiler developer Spends a lot of time running the test suite, constantly monitoring for compiler crashes For each crash, K. has to go back and reproduce the failing scenario under the debugger so that it is possible to diagnose what is going wrong
K. Uses Frysk K. starts Frysk Sets up a monitored environment Establishes observers for a terminating program
K. runs tests under Frysk K. runs the test suite Entire test run monitored Frysk detects crashing programs K. notified
K. examines crash K. can examine the crashing code
Use Case #2, S. S. is a System Adminstrator Having trouble with the web server Occasionally, but always at 2am, the shopping cart daemon dies S. is not getting much sleep
S. uses Frysk Implements a custom observer to take the following actions when the shopping cart dies: Sends a page to S. Saves the log information Disables the shopping cart
Motivation Customer Driven Need to Leap Frogs 20 year old technology Red Hat Initiative
Project Goals Graphical, fast startup, low overhead Always On Non stop C++ just works Also native and interpreted Java Scalable: O(1) or O(log(N)) algorithms
Project Goals, continued Powerful and complete Thread Model Thread groups, OpenMP and MPI Programmable Core Java, C++ Scriptable Python/Jython Integration with user level SystemTAP
Underlying Technologies Linux 2.6 Kernel Native POSIX Thread Library (NPTL) ELF (libelf) and DWARF (libdwfl) libunwind Implement in C++, Native Java GUI driven design Java GNOME
System Analysis Tools Oprofile SystemTap Frysk Statistical System Sampling Access to Full Model of System
Frysk vs Traditional Debugger Non stop All threads, processes, and hosts Auto attach Assume optimized Debug info on demand; on disk Implement in C++ and Java; Object Oriented; Event Driven; Observer Based Stop Start Single Thread and Process Explicit Attach Assume O0 Debug info loaded immediately; in memory Implemented in C; Procedural; Blocking; Polling
Frysk GUI vs Eclipse CDT Light Weight Integrated Wide band direct memory Java GNOME Code Display based on Text and Debug info Can be built as Eclipse Plug in Heavy Weight Framework Separate debug process Narrow band socket SWT Code Display based on Text files
The Frysk Architecture Common ABI GUI fstack ftrace Eclipse Frysk's Core CLI Code Model Lang Model Process Model Kernel
Frysk's Core Provides Object Models, or Views of System Process Model: Task, Process, and Host Language Model: Method, Stack, Variable Code Model: File, Line, Source code Strictly Event Driven: never blocks on kernel Asynchronous: separate dedicated thread CLI library based on HPDF Specification
Process Model Host tower.redhat.com Proc firefox Proc httpd Proc gnome terminal Task PR_Select Task PR_Select Task main Proc bash Proc bash Proc grep
GUI View
Monitoring, Multiple Observers
Opening Source Window...
... Source View
Language Model Frame Method Frame Method void b() Method void do_something() Method int c() Frame Method Variables
Code Model Implemented with CDTParser Provides Static Information: File and Line Source Code, including variables and functions Combined with Language Model by GUI
Example: In line Code 1
Example: In line Code 3
Example: In line Code 2
Stand Alone Utilities Implemented as Wrappers around Fryk's Core Fstack Works on one or more processes Symbolic back trace produced off line Ftrace Similar to strace Correctly tracks clone and fork
Current Status Tracks threads, and processes Monitors System Calls Command Line Interpreter C++ parser; thread group parser; tab completion Terminal Window Minimal Program Control stop on event, resume
Current Status, continued Running on FC 5, and RHEL4 U4 Using GCC 4.1 Tech preview in RHEL4 U4 Using GNOME 2.8 back ported to RHEL4 RPMs available for: X86 on RHEL4; X86 and x86 64 on FC 5
Road Map PowerPC 64 support Xen Integration Debugging Optimized Code Address Map and Heap view Ongoing C++ Corefile Debugging Watching Variables Distributed Debugging Interaction View Eclipse Integration
Road Map, example
Further Information Project Page: http://sources.redhat.com/frysk/ IRC: irc://gimp.net/frysk Mailing list: frysk@sources.redhat.com
Questions?