QoS in the Linux Operating System. Technical Report

Transcription

1 Unverstät Karlsruhe (H) Insttut für elematk QoS n the Lnux Operatng System echncal Report Marc Bechler and Hartmut Rtter Insttut für elematk Fakultät für Informatk Unverstät Karlsruhe (H) E-Mal: [mbechler rtter]@telematk.nformatk.un-karlsruhe.de Fakultät für Informatk Insttut für elematk

2

3 able of Contents QoS n the Lnux Operatng System Introducton he QoS Archtecture Background: Schedulng and Polcy-based Networkng wthn the Lnux Kernel Schedulng n the Lnux Operatng System Bascs: he Archtecture of Lnux Processes, hreads and asks Schedulng Interrupt Handlng Schedulng Algorthm Real-me asks Common asks Intalzaton of Expred asks Dspatchng Example Desgn Issues for an Adaptve Scheduler Varable me Slces Negatve Sde Effects A QoS-based Schedulng Approach Example QoS Support n heory and Practce Further Implementaton Detals Addng a New System Call he Implementaton of modfy_qos() Bblography...39 Appendx A: Overvew of the Modfed Fles...41 Appendx B: Gettng Started...43 Appendx C: he ask Structure task_struct...45 Appendx D: Schedulng Source Code...47 Appendx E: Proof of Maxmum hreshold...53 Appendx F: Dervaton of the Percentage A...55

4

5 1 Introducton hs document descrbes a man part of the mplementaton of our QoS archtecture, whch has been developed n the context of our UNIQuE proect [1]. In order to acheve an end-to-end support for qualty of servce, our approach bases on feedback loops as proposed n [2]. hs techncal report covers the mplementaton detals of the realzaton of an adaptve CPU scheduler, whch s a key element n the QoS archtecture. For the mplementaton we used the Lnux operatng system (verson ) as the QoS archtecture requres modfcatons n the kernel mechansms of the operatng system to acheve the end-to-end QoS support and the sources of Lnux are publc avalable. he document s structured as follows: In the next chapter the basc concept of our QoS archtecture s descrbed. In ths context, t wll become clearer what the term adaptve scheduler denotes and what requrements result from ths approach. In chapter 3, the CPU schedulng n the exstng Lnux kernel s descrbed n detal, whereas chapter 4 presents the modfed adaptve scheduler as well as some theoretcal and practcal results. Chapter 5 fnally gves an overvew mplementaton detals; followed by a bblography and techncal appendces. As wth many research and development especally n the Lnux area, ths work s not ready but work-n-progress, but we thnk t brngs up new aspects of Qualty of Servce support that fts the user s needs and s talored much better to ndvdual user preferences. An electronc verson of ths document can be found on the home ste of our UNIQuE proect as well as other papers related wth ths topc:

6

7 2 he QoS Archtecture he overall pcture of the QoS archtecture s gven n Fgure 2.1. Bascally, t conssts of a set of Montorng&Control enttes, whch are each assocated wth a resource. he resources currently under consderaton n the end system are the processng tme (processor scheduler entty), the avalable data rate shared between dfferent flows and the avalable network lnks n a mult-homed end system. he Montorng&Control enttes montor the resource sharng of the assocated resource,.e. they schedule the resource usage between the demands. he control part realzes the nteracton wth other Montorng&Control enttes. hey accept demands for a modfed resource share, and pass ths demand down to another entty accordng to some adaptaton and coordnaton rules. User adaptve applcaton proactve applcaton Applcatons Process Scheduler Montorng &Control Kernel Data Scheduler Montorng &Control Networksubsystem Network swtchng Network swtchng Montorng &Control Lnk layer Informaton Base Informaton Base Informaton Base Adapter 1 Adapter 2 Adapter 3 Fgure 2.1: Overall Pcture of our QoS Archtecture Many applcaton-orented approaches exst that try to mantan nformaton about the applcaton s requrements and perform resource reservaton n the end system based on ahead calculaton of the need [3, 4]. Opposed to ths, our QoS archtecture s strctly user orented. he resource sharng n the end system and the bas of the resource dstrbuton should be done n favor of those applcatons the user s currently focusng on. hs could be smply the wndow that s actve n the moment, as t s done n the Wndows 2000 operatng system [5]. As ths mght ft

8 4 Marc Bechler and Hartmut Rtter the user s nterest only some tmes, n [6] we proposed a smple user nterface that allows the user to sgnal hs current commtment to one or another applcaton more explctly. hs nterface s also shown n Fgure 2.1: A button labeled wth the letter Q (denotng Qualty of Servce) s ntegrated nto the wndow ttle bar of each applcaton started on the system. hs Q-Button can be ntegrated nto any wndowng system (X-Wndow as well as Wndows 2000) and does not necessarly depend on an applcaton. It s the ob of the operatng system (and, f supported, of the applcaton tself) to determne the meanng of better. he user s not concerned wth techncal detals and parameters detachng hm or her from productve work. Instead, the mappng of a sngle user parameter, gven ust by a clck on a button, must be mapped onto dfferent system parameters lke processng tme, data rate, used network, cost factors etc. (cf. [7]). Dfferent parts of the archtecture have been realzed up to now: he adaptve data scheduler s descrbed n [5], the network swtchng s part of the current work. Whereas applcaton adaptaton s a well-known concept n dstrbuted multmeda systems [8], we frst ntroduced the noton of proactve applcatons [2, d]. In contrast to adaptve 1 applcatons that try to adapt to the current stuaton but cannot actvely control resources, proactve applcatons actvely take effect on the sharng of resources n advance. hus, besde user nteracton by means of the Q-Button applcatons can nteract wth the system on ther own. here are some basc desgn decsons that result from the presented archtecture and follow the goal not to desgn a completely new operatng system kernel: & Insde each Montorng&Control entty, mnor resource varatons should be hdden from the neghbor enttes. hus, short-term and long-term varatons are gong to be decoupled and the system stablty s ncreased. & he Montorng&Control enttes accept and deal wth smple resource requests that are passed over from other Montorng&Control enttes as well as from proactve applcatons. In general, these requests result n a bas of the resource dstrbuton n favor of the flow or process related to the request. Obvously, resources are lmted and can not be ncreased, but they should be gven to the applcaton that s currently n the user s focus. hus a paradgm shft from calculaton and reservaton n advance to adaptaton to the stuaton s performed and complexty s beng reduced. & Especally the CPU schedulng s crucal for both the behavor and the performance of an operatng system. Hence, modfcatons of the used algorthms should be done very carefully, exstng applcatons relyng on some propertes should not be broken. 1 In our context, adaptve applcatons have the same meanng as reactve applcatons as they react on dynamc envronments by adaptng to the current avalablty of resources.

9 he QoS Archtecture 5 Followng these goals, n the next chapter the exstng CPU scheduler n the Lnux operatng system s descrbed and analyzed n more detal. Afterwards, the concept of an adaptve scheduler s presented n chapter 4.

10

11 3 Background: Schedulng and Polcy-based Networkng wthn the Lnux Kernel he basc dea for provdng applcatons wth the qualty of servce they need s to use an adaptve process scheduler combned wth an adaptve networkng subsystem, as descrbed n the prevous secton. Before the mplementaton of the adaptve process scheduler s descrbed n more detal, we gve an ntroducton n the schedulng of tasks wthn the Lnux kernel. hs helps to understand the decsons we made for the mplementaton of our adaptve process scheduler. Accordng to the prevous chapters, the followng general condtons were taken nto account: & Exstng ( legacy ) applcatons should be executable further on wthout any restrctons. Addtonally, those legacy applcatons should also take advantage of the QoS support. & he mechansms for achevng qualty of servce should be smple and effcent. & he user (or the applcaton, respectvely) must be nvolved n the closed loop n an actve way,.e., s/he must be able to affect the qualty of servce for each applcaton. & If possble, encroachments n the mechansms of the operatng system should be mnmzed as modfcatons may result n an napprecable behavor of the system. 3.1 Schedulng n the Lnux Operatng System Wthn the Lnux Kernel, the mechansms for schedulng the avalable CPU resources to the runnng processes are prmarly optmzed for hgh effcency, farness, throughput and low response tmes. However, qualty of servce aspects were not consdered for the development of the scheduler. For a runnng applcaton t s not possble to use any QoS mechansms related to the schedulng of tasks as a specfcaton of the user s preferences s not provded. In the followng sectons, the terms process schedulng, thread schedulng, and task schedulng wll appear several tmes, partcularly n the context of the queston, to whch knd of obects the CPU resources are scheduled. In order to avod obscurty, we use the generc term schedulng for allottng the avalable computaton tme. A detaled descrpton of further components and the mechansms of the Lnux kernel are descrbed n [9, 10] Bascs: he Archtecture of Lnux In contrast to the current evoluton of operatng systems towards a mcro-kernel archtecture, the Lnux kernel bases on a monolthc archtecture. hs basc desgn aspect also affects the

12 8 Marc Bechler and Hartmut Rtter functonng of the process scheduler. In mcro-kernel archtectures, e.g., the Mach Kernel [11] or the Kernel of Mcrosoft Wndows 2000 [5], the operatng system s kernel only provdes a mnmum of functonalty for performng basc operatons. Examples are the nter-process communcaton (IPC), and the memory management (MM). Bult on top of the mcro-kernel, the remanng functonalty of the operatng system s mplemented n ndependent processes or threads runnng n the user space. hey communcate wth the mcro-kernel va a defned nterface. For the schedulng of the avalable computng tme, the mcro-kernel only supports the basc mechansm for swtchng from one process or thread to the next, called a context swtch. he schedule sequence of the processes or tasks s determned separately, e.g., by kernel threads. In monolthc archtectures lke the Lnux operatng system the entre functonalty s concentrated wthn the kernel,.e., all mechansms and nformaton relevant for the schedulng are avalable n the kernel. Processes User Mode asks Scheduler System Mode CPU Hardware Fgure 3.1: he Archtecture of the Lnux Operatng System Fgure 3.1 gves a coarse vew on the Lnux archtecture, focused on the nteracton between applcatons, scheduler, and hardware. Further resource managers lke the memory manager are not shown n Fgure 3.1 as the llustraton mght become too complex. Lke other operatng systems such as Mcrosoft Wndows 2000 [5], a Lnux process resdes n one of two modes; a prvleged system mode, and a non-prvleged user mode. In general, applcatons are runnng as processes wthn the user mode. Each process exsts ndependent from other processes;.e., each process has ts own memory space and s not allowed to access the memory of other processes. When a process enters the Lnux kernel (e.g., n the case of an nterrupton or a system call), the system swtches to the prvleged system mode. In the Lnux operatng system, only one process can be

13 Background: Schedulng and Polcy-based Networkng wthn the Lnux Kernel 9 executed n the system mode, as the kernel performs crtcal operatons whch must be protected from mutual preempton 2. From the Lnux kernel s pont of vew, the processes are noted as tasks and are treated lke common functons. In Fgure 3.1, the tasks are marked by the dotted lnes of each process resumed n the system kernel. Once a process/task enters the prvleged system mode, t has access to the resources of every other task currently runnng on the system, whch s not possble n the user mode. he scheduler s completely mplemented nsde the operatng system s kernel. In mult-processor envronments, each processor has ts own scheduler, but the schedulng algorthm used by each scheduler s always the same Processes, hreads and asks In papers about Lnux, the terms process, thread, and task are n many cases handled wth the same mportance, although they are completely dfferent. A process s generally defned as a sequence of actvtes related to a dstngushed problem. Each process exsts ndependent of other processes. Processes are not able to affect each other, albet they can communcate wth each other usng kernel mechansms for nter-process communcaton. hus, swtchng from one process to the next one requres a complete swtch of the envronment, ncludng address space and CPU regsters. Wthn a process address space, one or more threads could exst n parallel workng on the same problem. hose threads are runnng wthn the same envronment,.e., swtchng from one thread to the next only requres a fast context swtch rather than an extensve swtch of the envronment. hus, threads are also known as lght-weghted processes. Another mportant dfferentaton s that parts of the process memory space (the so called shared memory) are accessble by all threads runnng wthn that process. hs feature allows for an easy and effcent synchronzaton of threads and an effcent data exchange between threads as no communcaton paths have to be establshed. he basc dfference between threads and processes s llustrated n Fgure 3.2. (Note that Fgure 3.2 s a typcal example for a mult-threadng operatng system, such as Solars. It s not typcal for Lnux!) Wthn the envronment of a process, one or more threads are runnng smultaneously. he arrows mark the current poston of the nstructon counter for resumng the computaton. In mult-threaded operatng system desgns, threads are dspatched mmedately to the processor, whereas n Lnux processes are scheduled and threads mght run wthn a process as descrbed below. 2 Note that even n mult-processor envronments only one process could be executed wthn the system mode.

14 10 Marc Bechler and Hartmut Rtter Process 1 Process 2 Process 3 CPU Fgure 3.2: Dfference between Processes and asks From the vewpont of the Lnux kernel, every process runnng on the system s referred to as a task. Compared to processes, tasks are allowed to access every resource (n partcular, tasks are allowed to modfy other tasks), as they are runnng n the prvleged system mode. hus, the schedulng algorthm runnng n the kernel shares the avalable CPU bandwdth between the tasks currently runnng on the system, ndependent whether a task represents a process or a thread outsde the Lnux kernel. he task s structure s defned n nclude/lnux/sched.h (see struct task_struct) and contans parameters that characterze a task, such as memory management nformaton, varous varables for states, flags, and statstcs, ponters to related tasks (such as chld tasks, parent tasks, prevous and next task n the task-lst, etc.), hash tables, process credentals, lmts, fle system nfo, sgnal handlers, etc. he defnton of the complete task structure s prnted n Chapter 0. Orgnally, threads were not supported from the Lnux operatng system; applcatons were bascally mapped to processes. In order to acheve compatblty to the POSIX c standard [12] relevant for UNIX operatng systems, there are two basc mplementaton desgns enablng the use of threads n Lnux. he basc dfference between those approaches s the way how the scheduler treats threads: & he frst approach s the mplementaton of threads wthn a process, mplemented by the Phread lbrary [12]. If a process starts one or more Phreads, t s responsble for schedulng the computaton tme he gets from the operatng system s scheduler to the Phreads currently runnng wthn ts context. hus, Phreads are nvsble for the scheduler as they are completely mplemented wthn the user space. & he alternatve s the mappng of threads to processes,.e., each thread s represented by exactly one process. One example for ths approach s the system call clone() (defned n arch/386/kernel/process.c) Usng ths system call, a new process s forked, that s allowed to access a shared memory allocated from the parent s process. In ths case, the

15 Background: Schedulng and Polcy-based Networkng wthn the Lnux Kernel 11 Lnux scheduler s responsble for sharng the computaton tme between all threads, as those threads are represented as common tasks for the kernel. hus, the more threads created by a process or an applcaton, the more computaton tme s scheduled for ths applcaton. In the remander of ths document, the dfferentaton between processes, threads, and tasks s not consdered as the scheduler of the Lnux operatng system s responsble for schedulng the avalable computaton tme to tasks. Note that the defnton of the task structure (struct task_struct) can be found n nclude/lnux/unstd.h Schedulng he prncple ob of the scheduler s to decde the next executable task that s dspatched to the CPU. A task s marked as executable, f the computaton of the task can be (actvely) contnued and does not depend on any other resources, such as arrvng data from a hard dsk drve. From the mplementaton pont of vew, a task p s executable f ts actual state s p->state = ASK_RUNNING. he schedulng functonalty can be found n kernel/sched.c n the functon schedule(). In Lnux, the schedulng works n a preemptve manner,.e., a task wll run as long as one of the followng events occur: & the task voluntarly releases ts control back to the scheduler, or & the scheduler revokes the CPU resource from the actve task and dspatches the CPU to another task. he schedulng sequence s defned by a round-robn-based tme dvson multplexng mechansm. Each (executable) task s dspatched to the CPU for a short perod of tme. hs perod, the so-called tme slce, has a value of 200 ms n Lnux (defned n the constant varable DEF_PRIORIY n nclude/lnux/sched.h). For each task currently dspatched to a processor, the tme slce wll be counted downwards. If the tme slce expres, the actve task wll be preempted n favor of another task. An expred task wll not be dspatched untl the tme slce of each executable task s expred. After ths tme, all tasks ncludng the non-executable wll be re-ntalzed and the competton for the CPU starts from the begnnng. In Lnux, tme s measured n tcks (specfed n kernel/tmer.c) defned by the constant varable HZ n the fle nclude/asm/param.h (for x86-based CPU archtectures). he temporal resoluton of tme depends on the CPU archtecture. In x86-based processor archtectures, one tck corresponds to a tme perod of 10 ms (HZ = 100), whereas n the Alpha archtecture, the tme s updated 1,024 tmes per seconds 3,.e., one tck takes ms. he tme [n tcks] passed snce the last system startup s stored n the global varable ffes, defned n the fle 3 he reason for ths unusual value s a regster n the Alpha CPU whch s updated 1,024 tmes per second.

16 12 Marc Bechler and Hartmut Rtter nclude/lnux/sched.h. However, n many documents about the Lnux kernel, tcks and ffes are used n the same sense. As mentoned above, the scheduler wll be called n three cases: 1. he tme slce of the actve task expres. 2. A task voluntarly releases the control back to the scheduler (e.g., t has to wat for a resource). 3. An nterrupt or a system call trggers the scheduler. he functonng of the scheduler can be subdvded n three steps. Frst, nterrupts that were not handled completely have to be fnshed. he second step s the selecton of the next task that wll be scheduled to the CPU. Fnally, the new thread wll be dspatched to the CPU. he three steps are explaned n the followng sectons n more detal Interrupt Handlng Interrupts are used for notfyng the operatng system that certan events occurred, e.g., a completed data transfer or an error n a perpheral devce. Interrupts are handled wthn the kernel of an operatng system; therefore, the system swtches to the prvleged system mode. In Lnux, three types of nterrupts can be dstngushed: & Fast nterrupts lke the nput of characters va the keyboard requre no tme-senstve actons and are handled mmedately. For the scheduler, the handlng of fast nterrupts s transparent; t can only be notced by the system that there s less tme avalable for the tasks. & In contrast to fast nterrupts, a slow nterrupt could be ntercepted by other nterrupts. he handlng of slow nterrupts s realzed n two steps. Frst, actons that are essental for handlng the nterrupt are performed. he remanng operatons for ts completon, called bottom halfs, are carred out when the scheduler s called the next tme. he completon of the nterrupt routne s performed before the schedulng algorthm starts; on account of the 20 tcks for the ntal tme slce, the maxmum amount of tme for the completon s 200 ms. he system tmer s a typcal example for a slow nterrupt, whch s released every 10 ms by the processor (on x86-based archtectures). he actons n the frst step are generally to update the system tme,.e., by ncrementng the ffes. Later, the bottom half takes care on executng expred tmers and updatng statstcs for the currently scheduled task.

17 Background: Schedulng and Polcy-based Networkng wthn the Lnux Kernel 13 & he thrd type of nterrupt s a system call. A system call requres to enter the system mode, whch s realzed by a soft nterrupt 4. he scheduler s never called drectly; nstead, the need_resched flag (defned n the task structure n nclude/lnux/sched.h) of an executable task s set to 1. hs flag s automatcally checked after the completon of a system call and, when approprate, the scheduler wll be trggered. If the scheduler s executed, the frst step s to complete the bottom halfs. In the Lnux kernel, the exstence of bottom halfs are ndcated by the varable tq_scheduler, defned n nclude/lnux/tqueue.h, as can be seen by the followng lnes of code. 444 f (tq_scheduler) 445 goto handle_tq_scheduler; If the tq_scheduler flag s set, the functon run_task_queue() (also defned and mplemented n nclude/lnux/tqueue.h) wll be called whch works out all bottom halfs that were marked by nterrupt handlng routnes. Bascally, the same procedure s performed n the next step for handlng the bottom halfs of software nterrupts. 457 f (softrq_state[ths_cpu].actve & softrq_state[ths_cpu].mask) 458 goto handle_softrq; he treatment of software nterrupts s performed by the functon do_softrq(), whch s mplemented n kernel/softrq.c and bascally works n the same way as descrbed n the prevous case Schedulng Algorthm he schedulng algorthm n Lnux s responsble for decdng whch (executable) task s next dspatched to the processor. Accordng to the POSIX b standard [13] for UNIX operatng systems, Lnux dstngushes two classes of tasks: real-tme tasks and common tasks. Compared to common tasks, real-tme tasks are prncpally favored by the scheduler. hey only have to compete wth other real-tme tasks for the avalable computaton tme. For schedulng real-tme tasks, the POSIX b specfcaton defnes two operatons: Perodc real-tme tasks are scheduled by a round-robn-based algorthm, whereas a sporadc real-tme task s executed untl t voluntarly releases the control over the CPU or untl a real-tme task wth a hgher prorty preempts t, whereas perodc real-tme tasks are preempted by the expraton of ther tme slce. In the Lnux kernel, the 4 A soft nterrupt s ntated by software. In Lnux, soft nterrupts for swtchng to the system mode are realzed by callng the nterrupt 0x80.

18 14 Marc Bechler and Hartmut Rtter dfferent types of tasks are defned n nclude/lnux/sched.h. Perodc real-tme tasks are represented by SCHED_RR, sporadc real-tme tasks by SCHED_FIFO, and common tasks by SCHED_OHER. Before the core schedulng algorthm starts, some actons need to be performed that guarantee a correct and far schedulng of the avalable CPU resources. he actve perodc realtme task s appended to the end of the queue of real-tme tasks, so other equal real-tme tasks wll also be scheduled. Furthermore, tasks watng for specfc events must be marked as executable f the correspondng sgnals arrve [10]. Further modfcatons of a task s state are descrbed n [14]. However, those modfcatons are only relevant for schedulng n a way that the set (realzed as a lst) of executable tasks s updated. For the decson of the next task that wll be dspatched to the CPU, the schedulng algorthm computes a weght for each executable task. hs task s mplemented n the functon goodness() (n kernel/sched.c) and wll be descrbed n more detal. As real-tme tasks are favored, t seems that there are dfferent schedulers for each knd of tasks. Indeed, both common tasks and real-tme tasks are scheduled by the same schedulng functon. Only the weght functon dffers for each class of tasks Real-me asks he prorty of real-tme tasks potentally runnng n the system s always hgher compared to common tasks. hus, a real-tme task only has to compete wth other real-tme tasks for the avalable computaton tme. herefore, a fxed prorty scheme s used, where the decson for the next task dspatched to the CPU s exclusvely determned by the real-tme prorty of a task, defned n the varable rt_prorty (defned for each task). Other factors such as the current sze of the consumed computaton tme are not taken nto account. Generally, the task wth the hghest realtme prorty wll be executed; t has to share the CPU bandwdth wth other real-tme tasks havng the same real-tme prorty. If a task wth a lower prorty currently runs on the system, t wll be preempted. A dynamc modfcaton of the real-tme prortes s n general possble by external programs, however, the scheduler does not alter ther values. hus the weght calculated for each real-tme task s constant (f one of those external programs does not modfy the real-tme prorty). An expred tme slce causes the preempton of a perodc real-tme task, whch wll be re-ntalzed mmedately. he calculaton of a real-tme s weght s very smple, as can be seen by the followng two lnes of code n the goodness() functon: 120 f (p->polcy!= SCHED_OHER) { 121 weght = p->rt_prorty;

19 Background: Schedulng and Polcy-based Networkng wthn the Lnux Kernel 15 A real-tme task that s not marked wth the SCHED_OHER flag gets a weght of ts (constant) real-tme prorty plus 1,000. hs weght could not be reached by any common tasks; thus, executable real-tme tasks are always treated before any common tasks. he handlng of real-tme tasks reveals that the term real-tme s not approprate n ths context. ypcally, a real-tme task has to delver a correct value wthn a pre-determned tme. Although a Lnux-based applcaton can affect the real-tme prorty of ts process, there s no correlaton to the support of a guaranteed qualty of servce, as nether a reservaton nor any admsson control s performed. Real-tme n the context of Lnux only means that real-tme tasks have a hgher prorty as common tasks. he scheduler handles real-tme tasks accordng to the POSIX b defntons [12]. Nevertheless, the term real tme wll be used further on n ths paper as both the lterature on Lnux as well as the source code of the Lnux kernel use ths term. (Note that wth R-Lnux [15] there s an extenson to Lnux supportng hard real-tme capabltes!) Common asks Common tasks are only scheduled f no executable real-tme task currently runs on the system. he schedulng mechansm uses a round-robn-based algorthm, where the schedulng sequence can change dynamcally. In contrast to real-tme tasks, other parameters are used for the calculaton of a task s weght. For common tasks, the weght functon guarantees a far schedulng, whch s not necessarly gven for real-tme tasks. All parameters used by the weght functon are defned for each task n the task s structure (struct task_struct, see Appendx C). & me slce: he current sze [n tcks] of the tme slce. hs value s defned n the varable counter and s decreased when the task s scheduled to the CPU. hus, f a task s tme slce expres, counter s 0. he tme slce s the bass for the weght functon. If the tme slce expres, the scheduler has to re-ntalze the counter varable (whch s done f the tme slce of each executable task s expred). & Processor swtch: hs parameter s only used f symmetrc mult-processng s enabled. In SMP envronments, each CPU has ts own scheduler. herefore, a slght advantage s gven to the task s weght f the task was last dspatched to ths CPU, as the processor s cache mght stll contan vald entres for the envronment of ths process and mght not to reload the complete envronment from memory. Addtonally, some vald address mappngs could have remaned n the CPU s LB (translaton look-asde buffer). hs advantage gets lost when the task swtches to another processor. he processor swtch parameter s mplemented by addng PROC_CHANGE_PENALY to the weght. he schedule penalty s defned n nclude/asm/smp.h and s set to 15.

20 16 Marc Bechler and Hartmut Rtter & Envronment swtch: asks runnng n the same memory as the current task (whch could be realzed by forkng threads wth the clone() system call, see secton 3.1.2) are addtonally preferred by ncreasng ther weght. In ths case a more effcent context swtch (restorng the processor regsters) s suffcent for swtchng to the next task nstead of swtchng to a completely new envronment. hs slght advantage (whch s mplemented by ncrementng the weght) results n an mproved effcency, albet t contradcts to the prncple of farness for common tasks. & Prorty: In the Lnux operatng system, the prorty has two meanngs. Frst, t defnes the (mnmum) sze used for the ntalzaton of the tme slces (n the counter varable) f all executable tasks are expred and need to be re-ntalzed. he prorty s mplemented n the task_struct s varable prorty and has a pre-defned value of 20 tcks for Intel archtectures. hat s why the default value for a tme slce s 200 ms; t guarantees a far sharng of the avalable computaton tme between the executable tasks. prorty s not a statc varable; e.g., the nce command (only avalable for root) manpulates the prorty of a task, and thus ts share on the CPU resources. he second meanng of the prorty arses n ts use for the weght functon, as t s added to the parameters lsted above. herefore, processes wth a lower prorty are ndrectly penalzed as ther tme slce wll not be consumed up as fast as the tme slce of tasks wth a hgher prorty. Note that the prorty used for common tasks s dfferent from the prorty used for weghtng real-tme tasks (rt_prorty), whch s of no relevance for common tasks. Wth the descrpton of the parameters nvolved n the calculaton of the weght for common tasks, the followng lnes of code whch mplement the core functonalty of the goodness () functon summarze those facts and are easy to understand: 132 weght = p->counter; 133 f (!weght) 134 goto out; #fdef SMP 137 /* Gve a largsh advantage to the same processor... */ 138 /* (ths s equvalent to penalzng other processors) */ 139 f (p->processor == ths_cpu) 140 weght += PROC_CHANGE_PENALY; 141 #endf /*.. and a slght advantage to the current MM */ 144 f (p->mm == ths_mm!p->mm) 145 weght += 1; 146 weght += p->prorty;

21 Background: Schedulng and Polcy-based Networkng wthn the Lnux Kernel out: 149 return weght; As can be seen n lne 133, the weght wll not be calculated for expred tasks (.e., counter must be greater than 0). he bass s the currently remanng tme slce (counter), maybe ncreased by the PROC_CHANGE_PENALY f SMP s enabled n the kernel confguraton (see above), maybe ncremented f an envronment swtch can be avoded (lne 145), and fnally ncreased by the prorty n lne 146. Note that all the nformaton used for the calculaton of the weght functon s stored n each task s structure. he only sense of the goodness() functon s bascally to determne the next task that s dspatched to the CPU. It has no effects on the share a task gets on the avalable computaton tme ths share s defned only by the remanng tme slce, represented n the counter parameter (as descrbed above). Compared to the calculaton of a task s weght, the schedulng algorthm works qute easy, as can be seen n the code prnted below. he schedulng algorthm uses the goodness() functonalty. However, t mght be possble that the actve task gves up ts control voluntarly. hs s marked by settng the task s state to SCHED_YIELD. hus, for the actve task, the prev_goodness() routne wll be called, whch frst checks f the SCHED_YIELD flag s set. In ths case, the task becomes no weght (weght = 0), otherwse the weght wll be computed accordng to the goodness() routne. 486 /* 487 * ths s the scheduler proper: 488 */ repeat_schedule: 491 /* 492 * Default process to select */ 494 next = dle_task(ths_cpu); 495 c = -1000; 496 f (prev->state == ASK_RUNNING) 497 goto stll_runnng; stll_runnng_back: 500 lst_for_each(tmp, &runqueue_head) { 501 p = lst_entry(tmp, struct task_struct, run_lst); 502 f (can_schedule(p)) { 503 nt weght = goodness(p, ths_cpu, prev->actve_mm); 504 f (weght > c) 505 c = weght, next = p;

22 18 Marc Bechler and Hartmut Rtter 506 } Startng wth the dle task (lne 494; whch s always on the frst poston n the run queue), for each (executable) task n the run queue t s checked whether the task could be scheduled or not (lne 502). hs check s only of relevance for SMP systems; n sngle CPU envronments, the can_schedule() routne s always true. herefore, for each executable task the goodness() functon s called, whereby the helper varable c (lne 495) stores the hghest value of weght. hs process shows that there s a second aspect that nfluences the determnaton of the next task f two tasks have the same (hghest) weght, the one that s postoned n front of the other tasks n the run queue wns the competton for the CPU resources. Remember that tasks wth an expred tme slce have a weght of zero. If the tme slce of all executable tasks s expred, ther tme slces wll be re-ntalzed, as descrbed n the followng secton Intalzaton of Expred asks If the tme-slce of a perodc real-tme task expres, the scheduler re-ntalzes ts tme slce mmedately by assgnng the value stored n the prorty parameter,.e., task->counter = task->prorty, as can be seen n lnes (Appendx D). In contrast to real-tme tasks, common tasks are only re-ntalzed f the tme slce of each executable task s expred. In ths case, all tasks wll be re-ntalzed ncludng the tasks that are currently not executable. hereby, the response tme for tasks watng for some resources s mproved to acheve a faster reacton to the sgnal(s) the task s watng for. he code for the re-calculaton of the expred tmers s also qute easy to understand. In order to avod undefned states whle modfyng the task s parameter, the task lst (tasklst) has to be locked; other routnes are allowed to read the parameters, but not to modfy them (lne 603). After the re-ntalzaton, the run queue s unlocked (n lne 607). 599 recalculate: 600 { 601 struct task_struct *p; 602 spn_unlock_rq(&runqueue_lock); 603 read_lock(&tasklst_lock); 604 for_each_task(p) 605 p->counter = (p->counter >> 1) + p->prorty; 606 read_unlock(&tasklst_lock); 607 spn_lock_rq(&runqueue_lock); 608 }

23 Background: Schedulng and Polcy-based Networkng wthn the Lnux Kernel 19 he re-ntalzaton of the tme slces s done n lne 605. Addtonally to the prorty parameter, the half of the orgnal sze of the remaned tme slce s added. herefore, the new sze of the tme slce wll be calculated accordng to the followng equaton: old tmeslce new tmeslce = prorty + 2 hs guarantees that tasks watng for resources (and, thus, do not completely utlze ther tme slce) can react more spontaneous f the resource wll become avalable. Dvdng the tme slce s realzed as a smple and effcent bt shftng. Note that tasks wth an expred tme slce are re-ntalzed only by the prorty (whch s equal to 200 ms) as ther counter s zero. If, as an example, a re-ntalzaton s nvoked and a (watng) task has a remanng tme slce of 17 tcks, t wll be rentalzed wth ( /2 ) tcks = 28 tcks. If the scheduler s called the next tme, ths task wll be preferred as the tme slce s the bass for calculatng a task s weght. Addtonally, the CPU s bandwdth wll be shared more equally between the tasks as the computaton tme currently not needed wll be partcularly avalable later. Albet the tme slce for a watng task seems to be ncreasng wth every re-ntalzaton, t has a maxmum threshold of the double prorty (.e., 40 tcks), whch can be easly proved by nducton (see Appendx E for the proof). In the case that no executable task currently exsts, Lnux provdes a specal task that s always executable. hs task s called the dle task and s scheduled n ths stuaton. he dle task s mplemented as a smple endless loop, whch s defned as a pre-processor macro n the fle kernel/sched.c. Accordngly, the operatng system provdes an dle task for each CPU n SMP envronments Dspatchng he task wth the hghest weght wns the competton for the resources of the CPU and has to be dspatched n the last step of the schedulng to the correspondng processor. Before ths step s done, some statstcal operatons are performed. For the current task, the kstat.context_swtch varable (lne 558) s ncremented whch stores the current number of preemptons. Addtonally, n mult-processor envronments, the average utlzaton of the tme slce s computed (lnes ). For dspatchng, the context (whch are the regster values of the processor) of the precedng task must be saved, before the new context could be loaded from memory. he components of a context are descrbed n [16] n more detal, as archtecture specfc characterstcs are taken nto account. Note that the scheduler tself s not related to any task; untl ths pont of tme the scheduler runs n the context of the prevous task that should be preempted. After dspatchng, the schedulng s fnshed.

24 20 Marc Bechler and Hartmut Rtter If the tme slces of all executable tasks are expred, the tme slces wll be re-ntalzed (as descrbed above) and the weght functon for each executable task wll be repeated. For dspatchng the next task, the scheduler frst swtches the memory context (lnes ), before t loads the context of the new task n the CPU regsters usng the swtch_to() functon n lne 592 (note that ths routne s completely wrtten n assembler code, see nclude/asm/system.h). he followng secton gves a smple example of the schedulng Example Fgure 3.3 (a) demonstrates the schedulng n an example usng three executable (common) tasks. At the tme pont t 0 = 0 tcks, all tasks have a remanng tme slce of 20 tcks (whch means that they were ust re-ntalzed) and the task runnng before t 0 has to wat for any resource. We also assume a system runnng wth one CPU. (a) ask 1 ask 2 ask t [cks] (b) ask 1 ask 2 ask 3 t 0 = 0 tcks = = = 40 t 1 = 10 tcks = = 40 t 2 = 30 tcks = = 40 t 3 = 40 tcks = = 31 t 4 = 50 tcks = t 5 = 60 tcks = = = 40 Fgure 3.3: Example for Schedulng of dfferent asks he dotted lnes n Fgure 3.3 mark the tme ponts of the scheduler s actvaton. Whle runnng, the weght wll be calculated for each of the three tasks accordng to e results of the goodness() functon are lsted n Fgure 3.3 (b), and base on the followng calculatons:

25 Background: Schedulng and Polcy-based Networkng wthn the Lnux Kernel 21 & t 0 = 0 tcks. At the begnnng of the example, no task has consumed anythng of ts tme slce. hus, the weght of 40 results by addng the tme slce (counter, 20 tcks) to the prorty (20 tcks). As ask 1 s the frst n the run queue, t wll be dspatched. & t 1 = 10 tcks. After 100 ms, ask 1 voluntarly gves the control over the CPU back to the scheduler as t has to wat for an external resource. At ths pont, the weght for ask 2 and ask 3 are stll 40 as descrbed above. hus, ask 2 s the next task n the run queue and wll be dspatched. & t 2 = 30 tcks. he tme slce from ask 2 expres resultng n the actvaton of the scheduler. he weght functon wll weght ask 2 wth 0 as ts tme slce has expred. hus, ask 3 wll be scheduled to the CPU. & t 3 = 40 tcks. ask 1 s sgnaled that the needed resource s now avalable and, the scheduler wll be called. hus, ask 1 becomes executable and wll be weghted wth 30 as ts remanng tme slce s 10 tcks. ask 3, whch has also a remanng tme slce of 10 tcks, wll be kept dspatched, as t s weght s 31 due to the advantage that no swtchng of the memory context s needed. & t 4 = 50 tcks. he weght for ask 2 and ask 3 s 0 as the tme slces of both tasks are expred. & t 5 = 60 tcks. At ths pont, all tme slces are expred and the scheduler ntates a rentalzaton, whch results n a weght of 0 for all tasks. hus, the tme slce of all tasks wll be re-ntalzed and the weght wll be computed a second tme for each task. Note that after the re-ntalzaton, ask 1 wll keep the control over the CPU as no memory context swtch s needed and, thus, the new weght wll be 41 compared to 40 for ask 2 and ask 3. hs bref and smple example demonstrates the rather nflexble but effcent work of the scheduler. However, the support for qualty of servce s not possble as the Lnux kernel does not support possbltes to affect the schedulng of the tasks. he followng secton descrbes the problems and the realzaton of an adaptve task scheduler for supportng qualty of servce to applcatons.

26

27 4 Desgn Issues for an Adaptve Scheduler An adaptve schedulng mechansm s the bass for supportng qualty of servce on operatng system s level. Our approach s to realze an adaptve scheduler wth a mnmum of modfcatons at the fundamental schedulng mechansms n Lnux. herefore, two basc possbltes can be dentfed: he use of perodc real-tme tasks wth dynamc prortes, or a flexble tme slce scheme for common tasks. We decded to realze the second approach, as the use of dynamc realtme prortes has several dsadvantages: & A problem we dentfed n many tests s the favored treatment of real-tme tasks. In general, an executable real-tme task s always scheduled before all other common tasks and must not compete wth them for the CPU resources. If a complex multmeda applcaton bases on one or more real-tme tasks, the performance of all other applcatons runnng on the system wll break down. & Dynamc real-tme prortes requre an extensve and complex prorty management. A Realtme QoS Manager (RQM) would be necessary to handle the schedulng polcy of QoSbased applcatons. As an example, the RQM has to perform a contnuous adaptaton of the real-tme prortes for all tasks n order to guarantee the desred sharng, as the Lnux scheduler only consders executable tasks wth the hghest real-tme prorty. & he Real-tme QoS Manager also has to handle the prorty nverson problem, whch occurs by the use of flexble prortes for real-tme tasks. In the prorty nverson problem, a low prorty task that currently uses a resource blocks a (runnng) task wth a hgher real-tme prorty that tres to access ths resource. herefore, the RQM has to perform addtonal operatons for (real-tme) prorty nverson or the temporary ncreasng of (real-tme) prorty. & he last aspect s the real-tme prorty tself, whch has no relaton to the scheduled CPU resources. herefore, no statement about the effectve qualty of servce support wll be possble. Partcularly, t wll become very hard to map the applcaton s QoS requrements to real-tme prortes. 4.1 Varable me Slces Instead of usng dynamc prortes for real-tme tasks for supportng qualty of servce on operatng system s level, we use a flexble tme slce scheme for all tasks. hereby, the component s of specal nterest that defnes the prorty of a (common) task and smultaneously defnes the (mnmum) tme slce for ts re-ntalzaton. hus, the basc dea of our approach s to use the prorty varable as a representatve for qualty of servce. As can be seen n Fgure 4.1, we also

28 24 Marc Bechler and Hartmut Rtter realzed a QoS Manager that s responsble for nteractng wth the applcatons (see also chapter 2, note that QoS Manager and RQM are dfferent enttes!). If a process requests an mproved QoS support from the QoS Manager, t gets more CPU resources as ts tme slce wll be ncreased. Note that the QoS Manager has to take nto account whether schedulng more CPU resources to a task ndeed mproves the applcatons qualty of servce, or f the CPU resources are not the bottleneck. he mappng of dfferent tme slces to the correspondng amounts of CPU resources s mplctly performed by the scheduler. hs approach addtonally has the advantage that other tasks wll not starve (whch mght be possble n the real-tme approach), as the Lnux scheduler wll be called at the latest after all executable tasks are expred and, thus, were dspatched durng ths tme perod. Conversely, t has to be consdered that the prorty s at least one tck, as tasks wth a prorty of 0 are nterpreted as expred and are not consdered by the scheduler. User Mode Scheduler QoS Manager System Mode CPU Hardware Fgure 4.1: A QoS-based Scheduler for Lnux he QoS support specfed by the user or an applcaton can be easly mapped on the sze of a task s tme slce. On account of the coarse resoluton of tme wthn the Lnux kernel, we chose the dentty functon for the mappng. A modfcaton of the qualty of servce support by +1 results n an ncrease of the tme slce by 1 tck. A formal descrpton of the percental amendment on the share can be found n secton 4.3. Note that the mappng of the specfed qualty of servce covers only the scheduled computaton tme. Predctons about the tme ponts for the actvaton of the scheduler are not possble as ths manly depends on the unpredctable occurrence of nterrupts and other system events. hus, t s not possble to specfy the trggerng of the scheduler. It s only possble to determne the maxmum tme perod between two calls of the scheduler, whch s the double of the

29 Desgn Issues for an Adaptve Scheduler 25 maxmum tme slce (see secton and Appendx E). However, ths also means that t s not possble to guarantee the desred qualty of servce to an applcaton, as ths bascally depends on the behavor of the other tasks, nterrupts and system events. hat s, our approach only enables a statstcal guarantee of the QoS support, whch we call a soft qualty of servce. But, we beleve that ths knd of guarantee s suffcent for multmeda applcatons, as can be seen on the results descrbed n [2]. Note that our approach descrbed so far does not depend on explct nterventon n the schedulng mechansms of the Lnux kernel. However, there are some negatve sde effects, whch requre slght modfcatons of the scheduler Negatve Sde Effects As some mechansms of the Lnux operatng system are not desgned for supportng qualty of servce wthn the end system, the use of varable tme slces results n several negatve sde effects. As an example, the use of real-tme tasks s n contrast to the QoS support, as those tasks are bascally preferred and the user as well as the QoS Manager has rather any possbltes to control real-tme tasks dynamcally. One approach for avodng ths problem mght be to treat real-tme tasks as common tasks. In ths case, only slght modfcatons of the scheduler are requred. However, ths encroachment contravenes aganst the POSIX b specfcaton [13], whch explctly postulates the exstence of real-tme tasks and ther schedulng. Addtonally, the modfcatons could result n an undesrable behavor of exstng real-tme applcatons. hus, our approach treats real-tme tasks as before. he user mght remedy ths dsadvantage ether by closng real-tme applcatons, or (f s/he has root access) by usng the system call sched_setscheduler(), whch resets a real-tme task to a common task. A second obvous problem results from the demand-drven actvaton of the scheduler. A task that s currently dspatched to the CPU wll only be preempted f t voluntarly releases the control over the CPU, f ts tme slce expres, or f a system call explctly trggers the scheduler. Although ths mechansm s very effcent for small tme slces as the scheduler wll only be called f t s necessary, t s dsadvantageous for large tme slces. If, e.g., two MPEG-2 vdeos are runnng wth a tme slce of one second, the CPU resources are shared equally between both applcatons. However, whle runnng, the frst applcaton gets the complete CPU resources for 1 s and dsplays the vdeo sequence wth the maxmum frame rate, whle the other applcaton s suspended. After 1 s, the tme slce of the frst vdeo expres and the second vdeo runs for 1 s, whle the frst applcatons s suspended. Wth a further ncreasng of the tme slce, ths effect s renforced. However, ths qualty of servce support s not n terms of the user s satsfacton t would be more convenent f both vdeo sequences would be played smultaneously n a smoother way.

30 26 Marc Bechler and Hartmut Rtter Another weak pont that occurs from large tme slces s the calculaton of the weght for each task. he goodness() functon uses the a task s (re-)ntalzaton value for the tme slce (stored n the prorty varable), resultng n a hgh weght for tasks wth a large tme slce. hus, ths task frst utlzes ts complete tme slce, before other tasks are scheduled even f the scheduler wll be called multple durng ths tme perod. hs effect s clarfed by the example shown n Fgure 4.2. In ths example, the tme slce for ask 1 s 600 ms. hus, ts mnmal weght wll always exceed a value of 60 (untl ts tme slce expres), whch wll never be reached by common tasks wth a tme slce of 20 ms. For common tasks, the maxmum weght s 41 tcks (20 tcks remanng tme slce + 20 tcks prorty + 1 tck for avodng a context swtch). Note that the weghts for each tasks at t 5 n Fgure 4.2 are the values for the second call of the functon goodness(), after all tasks were re-ntalzed. (a) ask 1 ask 2 ask 3 ask t [cks] (b) ask 1 ask 2 ask 3 ask 4 t 0 = 0 tcks = = = = 40 t 1 = 40 tcks = = = = 40 t 2 = 60 tcks = = = 40 t 3 = 80 tcks = = 40 t 4 = 100 tcks = 40 t 5 = 120 tcks = = = = 41 Fgure 4.2: Example for asks wth large me Slces In the context of large tme slces, the re-ntalzaton after the expry of a task s tme slce s also crucal for the QoS support. he specfcaton of qualty of servce on the level of the operatng system has no drect correlaton to the tme slce of a task. Addtonally, a re-ntalzed tme slce also depends on the current tme slce (see lne 605) and, thus, results n an unpredctable qualty of servce support, f the task has to wat for any resources. If, e.g., an adaptve vdeo applcaton wth a prorty of 100 tcks has to wat for ncomng data, ts tme slce ncreases up to 2 s. If the data s avalable, the applcaton gets the complete CPU resources for 2 s. hereupon,