Linux Scheduler Analysis and Tuning for Parallel Processing on the Raspberry PI Platform. Ed Spetka Mike Kohler

Transcription

1 Linux Scheduler Analysis and Tuning for Parallel Processing on the Raspberry PI Platform Ed Spetka Mike Kohler

2 Outline Abstract Hardware Overview Completely Fair Scheduler Design Theory Breakdown of the CFS Scheduling Policies Modifications to the CFS Brain F*** Scheduler Design Theory Modifications to the BFS Custom System Calls for the ARM Architecture. Cross Compiling for the ARM Architecture. Testing Approach Testing Results

3 Abstract What is the best approach for tuning the Linux scheduler for parallel computing? Customizations to the current scheduler? Using an alternate scheduler? Just leave things alone?

4 Components 3x Raspberry PI Model B computers Linux Kernel Current Linux Scheduler Alternate Linux Scheduler MPI

5 What is a Raspberry Pi You Say? Ingredients JUST KIDDING!!!!! 2 cups all-purpose flour 1 tablespoon sugar 1/2 teaspoon salt 3/4 cup shortening 1 egg, lightly beaten 3 tablespoons cold water 1 tablespoon white vinegar FILLING: 1-1/3 cups sugar 2 tablespoons quick-cooking tapioca 2 tablespoons cornstarch 5 cups fresh or frozen unsweetened raspberries, thawed 1 tablespoon butter TOPPING: 1 tablespoon 2% milk 1 tablespoon sugar

6 Raspberry PI Model B Broadcom BCM2835 w/ ARM1176JZFS single core processor overclocked to 850MHz 512M Memory Raspbian Wheezy OS

7 Current Linux Scheduler Completely Fair Scheduler (CFS) Written by Red Hat Kernel Developer Ingo Molnar. Aims to maximize overall CPU utilization while maximizing interactive performance. Uses a red/black binary search tree (RBTREE) for process scheduling. CFS is an O(log N) scheduler

8 CFS Design Theory Models an ideal precise multitasking CPU. "Hardware CPU that that can run multiple processes at the same time (in parallel), giving each process an equal share of processor power (not time, but power)".

9 CFS Design Theory - "Ideal" Processor With 1 task 100% of the CPU's power is utilized. With 2 tasks each task receives 50% of the CPU's power. With 4 tasks each task receives 25% of the CPU's power. This concept is considered "Fair" to all of the processes because they run in parallel.

10 CFS Design Theory - "Ideal" Processor

11 CFS Design Theory - Apparent Issues In a single core CPU the ideal processor is non-existent. Even in a multi-core CPU, only one process can run in each core at one given time.

12 CFS Design Theory - Real Processor

13 CFS Design CFS attempts to keep track of the fair share of the CPU for each process. CFS runs a fair clock at a fraction of the real CPU clock speed. CPU time for processes is calculated by: wall time / total waiting processes

14 CFS Design - Waiting Processes As a process waits for the CPU, the scheduler tracks the amount of time it would have used on the ideal processor.

15 CFS Design - Waiting Processes The p->se.vruntime variable, is used to rank processes for scheduling and to determine the amount of time the process is allowed to execute before being preempted. <k_root>/include/linux/sched.h

16 CFS Design - Scheduling Policies SCHED_NORMAL Default CFS scheduling policy. Used for regular tasks. Better suited for interactive environments. <k_root>/kernel/sched/fair.c

17 CFS Design - Scheduling Policies SCHED_FIFO POSIX specified First-in/First-out policy. The highest priority process runs until blocked. <k_root>/kernel/sched/rt.c

18 CFS Design - Scheduling Policies SCHED_RR POSIX specified Round Robin policy. Each task gets a time quantum. The time quantum is dependant on the priority of the task. When the time quantum is up, the process is preempted. <k_root>/kernel/sched/rt.c

19 CFS Design - Scheduling Policies SCHED_BATCH Does not preempt nearly as often as regular tasks would. Tasks to run longer. Make better use of caches. Well suited for batch jobs. <k_root>/kernel/sched/fair.c

20 CFS Design - Waiting Processes The process with the longest wait time is picked by the scheduler and assigned to the CPU.

21 CFS Design - Running Processes CFS runs a task, and when the scheduler tick happens the task's CPU usage time is added to the p->se.vruntime variable. Once p->se.vruntime gets high enough so that another task becomes the "leftmost task" of the time-ordered RBTREE the current task is preempted and the new leftmost task is inserted into the CPU.

22 CFS Design - Running Processes CFS also accounts for a small amount of "granularity" distance relative to the leftmost task. This is done to prevent over scheduling of tasks and avoid thrashing the cache.

23 Customizations Create a custom Linux Kernel with modifications to the scheduler.

24 Customizations Utilized the Raspberry PI Linux Kernel Source. Compiled custom Linux Kernels CFS with no modifications CFS with batch processing as default Completely change the scheduler to BFS Created a system call to force the running process to modify it's scheduler to SCHED_BATCH and nice value to -20.

25 Customizations Cross compiled Linux Kernel on x86_64 using ARM architecture cross compilation library arm-linux-gnueabi. Used specially created tools for the Raspberry PI to create the kernel boot image.

26 Custom Modifications Changed the default scheduling policy in the task entry point from SCHED_NORMAL to SCHED_BATCH to focus primarily on batch processing. Implemented a custom kernel system call to include in any MPI job to always use SCHED_BATCH and modify niceness.

27 Kernel Hacking Modification to the task entry point structure. <k_root>/include/linux/init_task.h Modifications to INIT_TASK(task) macro, changing the default policy to SCHED_BATCH.

28 Kernel Hacking - System Calls System calls on the ARM architecture. <k_root>/arch/arm/kernel/calls.s <k_root>/arch/arm/include/asm/unistd.h Custom system call code <kroot>/kernel/raspberry/sysc_raspb.c

29 Kernel Hacking - Tunables Preemption Latency CPU Scaling SCHED_TUNABLESCALING_NONE - unscaled, always *1 SCHED_TUNABLESCALING_LOG - scaled logarithmical, *1 + ilog(ncpus) SCHED_TUNABLESCALING_LINEAR - scaled linear, *ncpus <k_root>/kernel/sched/fair.c

30 Alternate Linux Scheduler Brain F*** Scheduler (BFS) Written by kernel programmer Con Kolivas The objective of BFS is to provide a scheduler with a simpler algorithm, that does not require adjustment of heuristics or tuning parameters to tailor performance to a specific type of computation workload

31 BFS Design Theory Forward looking only Make the most of lower spec machines Not made to scale to massive hardware less than 16 cores best Desktop oriented scheduler Extremely low latencies for excellent interactivity 2

32 BFS Design - BFS Data Structure When requesting CPU each task given time slice and Vdeadline Virtual Deadline - Longest time that any 2 tasks with the same niceness will have to wait before running on the CPU Virtual because no guarantee task will complete on time

33 BFS Design Theory Patches against the latest version of the kernel like the 0(1) scheduler BFS uses runqueues but only one system wide runqueue containing all non-running tasks no complex heuristics necessary since only one runqueue to determine the next scheduled task Virtual deadlines keep track of the deadline of each task

34 BFS Design Theory - Apparent Issues Task look up O(n) BFS uses single queue for all processors and virtual deadlines are CPU relative so tasks cannot be ordered by their Vdeadline No tree structure can be used Scales poorly with increasing tasks Lock contention increases with one shared structure

35 Kernel Hacking - Patching BFS The patch can be obtained from: 0/3.6/3.5-sched-bfs-425.patch Patching the code into the Linux kernel is simple. $ patch -p1 < 3.5-sched-bfs-425.patch Simply run a make oldconfig, and answer the new questions, grab a cup of coffee, a copy of War and Peace, and by the time you are done reading the compilation should be complete.

36 Cross Compiling for ARM VMware Player Linux Mint 14 Nadia for x86_64 Debian gcc-arm-linux-gnueabi package Cross compilation command on Linux Mint 14: $ make ARCH=arm \ CROSS_COMPILE=/usr/bin/arm-linux-gnueabi- \ <target> -j5 AMD Athlon running CentOS 6.3 Raspberry PI Cross Compilation Tools package Cross compilation command on CentOS 6.3: $ make arch=arm \ CROSS_COMPILE=/home/pi/kernel/tools/arm-bcm2708/\ arm-bcm2708-linux-gnueabi/bin/arm-bcm2708-linux-\ gnueabi- <target>

37 CPU Load Testing Approach Created a batch MPI program that calculates PI. Program was used to test all schedulers and their modifications. Number of calculations tested 10,000,000. Number of iterations tested: 100

38 Test Results Wall Time Average for 10,000,000 calculations

39 Conclusion After analysis, we determined that you must have some idea of the load that will be placed in the parallel environment to make a proper determination for which Linux scheduler to choose.