IO latency tracking. Daniel Lezcano (dlezcano) Linaro Power Management Team

Transcription

1 IO latency tracking Daniel Lezcano (dlezcano) Linaro Power Management Team

2 CPUIDLE CPUidle is a framework divided into three parts: The generic framework The back-end driver: low level calls or hidden by firmware (ACPI, PSCI, MWAIT) The governor Select idle state Idle task Enter idle state CPUidle Generic framework Governor PM Back end driver

3 CPUIDLE : the generic framework Provides an API: To register a driver and a governor To ask for the governor suggestion To enter idle with the state abstraction No algorithm

4 CPUIDLE : the back end drivers It gives the abstraction for the complex PM code Very generic for some hardware based on firmware abstraction (eg. ACPI, PSCI,...) Very hardware dependent and complex if directly handled in the kernel (eg. ARM SoC specific drivers)

5 CPUIDLE : the governors Two governors used today: Ladder with periodic tick configuration: for server focused on performance Menu with nohz configuration: for most of the platforms trying to provide the best trade-off between performance and energy saving

6 CPUIDLE : the menu governor Tries to predict the next event on the system Does statistics on each CPU for the sleep duration Weighting the idle states regarding the pending IOs

7 CPUIDLE : the menu governor Source of wakeups: IRQ Timer, IOs, keyboard, IPI Mostly generated by the scheduler But most of the interrupts are coming from: Timers, IOs and rescheduling IPIs

8 CPUIDLE : the menu governor It takes all the source of interrupts without distinction: Is it up to the scheduler to predict the scheduler behavior (IPI reschedule)? What about if a task is moved to another CPU? Could stay to a very shallow state for seconds because the statistics are different Why not focus on event which are predictable? Timers and IOs and consider the rest as noise preventing us to be accurate

9 Why an IO could be predictable? The duration of an IO is inside a reasonable interval, with repeating pattern The tasks are blocked on IO, hence these latencies information can be tied with it The information can follow the task when this one is migrated

10 Experimentation Latency measurements for: SSD 6Gb/s HDD 6Gb/s with different block sizes From 4KB to 512KB

11 SSD behavior

12 SSD behavior

13 HDD behavior

14 HDD behavior

15 HDD behavior

16 HDD behavior

17 Observations Let's group the different latencies into buckets Experiment with different bucket size : 50us, 100us, 200us and 500us And observe the distribution : how many times a bucket is hit?

18 SSD 4KB Random RW

19 SSD 8KB Random RW

20 SSD 16KB Random RW

21 SSD 32KB Random RW

22 SSD 64KB Random RW

23 SSD 128KB Randow RW

24 SSD 256KB Random RW

25 SSD 512KB Randow RW

26 HDD 4KB Random RW

27 HDD 8KB Random RW

28 HDD 16KB Random RW

29 HDD 32KB Random RW

30 HDD 64KB Randow RW

31 HDD 128KB Randow RW

32 HDD 256KB Randow RW

33 HDD 512KB Random RW

34 Observations - Conclusions The smaller the bucket is, the more there are buckets Reduce probability to estimate the right bucket On slow media, there is an important deviation from the average latency The 200us bucket size shows a interesting trade-off between the number of buckets and the accuracy We can rely on these observations to build a model to predict the next IO

35 IO latency tracking infrastructure Each time a task is unblocked after an IO completion, measure the duration Add this latency to the IO latency tracking infrastructure Next time the task is blocked on a IO, ask the IO latency tracking infrastructure the guessed blocking time When going to idle, take the remaining time to be blocked on an IO as part of the equation to compute the sleep duration

36 IO latency tracking infrastructure Task Scheduler IO latency tracking io_schedule io_latency_begin io_latency_guess + io_latency_insert_node io_latency_end wakeup io_latency_delete_node + io_latency_add

37 IO latency tracking infrastructure Group the latencies into buckets, representing an interval (200us) Each bucket has an index Each index gives the bucket's interval index : 0 => [0, 199] index : 10 => [2000, 2799] Index : 5 => [1000, 1199]

38 IO latency tracking infrastructure io_latency_add(413us) find_bucket 413%200 = 2 bucket2 bucket3 bucket6 bucket12 update_history(bucket2,413us) bucket2.hits++ bucket2.suchits++ bucket2.avg =...

39 IO latency tracking infrastructure How do we guess the next blocking time? Buckets are sorted Position in the list shows the history (first happening more, last happening less) Buckets have the number of hits and the position in the list weight these numbers Compute a score with the position in the list and the number of hits with a decaying function

40 IO latency tracking infrastructure Score = nrhits / (pos + 1)² first bucket2.hits = 123 Score = 123 / (0+1)² = 123 bucket3.hits = 321 Score = 321 / (1+1)² = 80 bucket6.hits = 12 Score = 12 / (2+1)² = 1 last bucket12.hits = 32 Score = 12 / (3+1)² = 0 Bucket 2 is the next expected IO latency

41 IO latency tracking infrastructure What happens when there are several tasks blocked on an IO? A red-black tree per cpu The number of elements of this tree is the number of tasks blocked on a IO Each node is the guessed IO duration for the corresponding task Left most node is the smaller guessed IO duration

42 IO latency tracking infrastructure + CPUidle When entering idle we have now: The next timer event which is reliable (next_timer_event) The next IO completion (next_io_event) The next event is: next_event = min(next_timer_event, next_io_event)

43 IO latency tracking + CPUidle Choosing the idle state is straighforward: for (i = 0; i < drv->state_count; i++) { struct cpuidle_state *s = &drv->states[i]; struct cpuidle_state_usage *su = &dev->states_usage[i]; if (s->disabled su->disable) continue; if (s->target_residency > next_event) continue; if (s->exit_latency > latency_req) continue; } idle_state = i;

44 The big picture Task waiting for an IO Task waiting for a timer Idle task IO latency tracking Timer framework idle_for(min(t1, tn)) t1 tn+5 tn+4 tn+3 tn+2 tn+1 tn CPUIDLE Choose idle state Basic selection loop sleep Backend driver

45 Some results Tools used to measure: Some extra infos added to sysfs for each cpu Over predicted : the sleep duration was shorter Under predicted : the sleep duration was longer Idlestat : a tool computing the idle state statistics Iolatsimu: a tool implementing the prediction algorithm and randomly read/write a file

46 Some results Tests done under certain circumstances Process pinned on one cpu in order to prevent migration Tried on the media described at the beginning of the documentation For each block size measure the prediction regarding the idle state which has been choose

47 Some results IO latency framework + timer give an expected sleep time of 30us 1. Effective sleep time = Effective sleep time = Effective sleep time = 45 residency = 10 residency = 20 residency = 40 residency = 70 residency = 150 Shallow Deep 1. and 3. are failed predictions, 2. is a right prediction

48 SDD results - 4KB

49 SDD results 8KB

50 SSD results - 16KB

51 SSD results - 32KB

52 SDD results - 64KB

53 SDD results - 128KB

54 SDD results - 256KB

55 SDD results - 512KB

56 HDD results - 4KB

57 HDD results - 8KB

58 HDD results - 16KB

59 HDD results - 32KB

60 HDD results - 64KB

61 HDD results - 128KB

62 HDD results - 256KB

63 HDD results - 512KB

64 Conclusion A noticeable improvement in terms of prediction, more than 30% under some circumstances The resulting design makes the idle state decision within the scheduler Idling decision integrated Smarter decision could be made in the future

65 Next steps Improve the IO latency tracking framework code Investigate pointless wakeup resulting from an IPI when an IO completes Fix the IO completions measurement probes Make the latency per device Big kernel impact Improve the scheduler to take smarter decision regarding idle

66 THANKS!