Deadlock Victim. dimanche 6 mai 12

Transcription

1 Deadlock Victim by Dr Heinz Kabutz && Olivier Croisier The Java Specialists Newsletter && The Coder's Breakfast && 1

2 You discover a race condition 2

3 Data is being corrupted 3

4 What ya gonna do? 4

5 Who ya gonna call? 5

6 Ghostbusters? 6

7 you do what we all do 7

8 you use synchronized 8

9 synchronized 9

10 synchronized 9

11 synchronized can be bad medicine 10

12 deadlock awaits you! 11

13 12

14 Deadlock Victim 13

15 Correctness fix can lead to liveness fault 14

16 lock ordering deadlock 15

17 at least two locks 16

18 and at least two threads 17

19 locked in different orders 18

20 can cause a deadly embrace 19

21 requires global analysis to fix 20

22 resource deadlocks 21

23 bounded queues 22

24 bounded thread pools 23

25 semaphores 24

26 class loaders 25

27 lost signals 26

28 The Drinking Philosophers 27

29 @ the Java Specialist Symposium in Crete 28

30 Symposium == sym + poto Literally - to drink together 29

31 our philosophers need two cups 30

32 they either want to drink or think 31

33 Table is ready, all philosophers are thinking

34 philosophers 5 wants to drink, takes right cup

35 Philosopher 5 is now drinking with both cups

39 Philosopher 3 finished drinking, returns right cup

41 Philosopher 3 returns Left cup

42 philosophers can get stuck 41

43 if they all pick up the right cup 42

44 A deadlock can easily happen with this design

45 Philosopher 5 wants to drink, takes right cup

50 Deadlock!

51 they will all stay thirsty

52 this is a deadly embrace

53 this can be detected automatically 52

54 search the graph of call stacks 53

55 and look for circular dependencies 54

56 ThreadMXBean does that for us 55

57 simple deadlocks are quickly found 56

58 but what do you DO with them? 57

59 what will the dog do with the car if he ever catches it? 58

60 databases choose deadlock victim 59

61 but what does Java do? 60

62 it should probably throw an Error 61

63 but the threads simply hang up 62

64 it is best to avoid causing them! 63

65 global order of locks in program 64

66 public class LeftRightDeadlock { private final Object left = new Object(); private final Object right = new Object(); public void leftright() { synchronized (left) { synchronized (right) { dosomething(); } } } public void rightleft() { synchronized (right) { synchronized (left) { dosomethingelse(); } } } } 65

67 Interleaving causes deadlock lock left trying to lock right A leftright() BLOCKED lock right trying to lock left B rightleft() BLOCKED 66

68 we define a fixed global order 67

69 e.g. left before right 68

70 deadlock is solved! public void rightleft() { synchronized (left) { synchronized (right) { dosomethingelse(); } } } 69

71 our drinking philosophers 70

72 each cup gets a unique number 71

73 lock the highest number cup first 72

74 then lock the lower number cup 73

75 drink 74

76 unlock lower order cup 75

77 unlock higher order cup 76

78 think 77

79 Global Lock ordering We start with all the philosophers thinking

80 Philosopher 5 takes cup 5 Cup 5 has higher number 1 Remember our rule!

81 Philosopher 1 takes cup 2 Must take the cup with the higher number first 1 2 In this case cup

82 Philosopher 2 takes cup

84 Philosopher 1 takes cup 1 - Drinking

85 Philosopher 1 returns Cup 1 Cups are returned in the opposite order to what they are acquired

87 Philosopher 5 returns cup

95 Philosopher 3 Returns cup

100 sorting locks by property is easy 99

101 e.g. account IBAN number 100

102 or employee number 101

103 or some other characteristic 102

104 maybe even System.identityHashCode() 103

105 use "open calls" 104

106 Vector synchronizes writeobject() private synchronized void writeobject(objectoutputstream s) throws IOException { s.defaultwriteobject(); } 105

107 synchronized writeobject() is invoking s.defaultwriteobject() private synchronized void writeobject(objectoutputstream s) throws IOException { s.defaultwriteobject(); } 106

108 this is not an "open call" private synchronized void writeobject(objectoutputstream s) throws IOException { s.defaultwriteobject(); } 107

109 but can it cause problems? private synchronized void writeobject(objectoutputstream s) throws IOException { s.defaultwriteobject(); } 108

110 yes it can! private synchronized void writeobject(objectoutputstream s) throws IOException { s.defaultwriteobject(); } 109

111 serialize both vectors at same time Vector v1 = new Vector(); Vector v2 = new Vector(); v1.add(v2); v2.add(v1); 110

112 this could happen with nested data structures 111

113 it happened to an IBM client 112

114 private void writeobject(objectoutputstream stream) throws IOException { Vector<E> cloned = null; // this specially fix is for a special dead-lock in customer // program: two vectors refer each other may meet dead-lock in // synchronized serialization. Refer CMVC synchronized (this) { try { cloned = (Vector<E>) super.clone(); cloned.elementdata = elementdata.clone(); } catch (CloneNotSupportedException e) { // no deep clone, ignore the exception } } cloned.writeobjectimpl(stream); } private void writeobjectimpl(objectoutputstream stream) throws IOException { stream.defaultwriteobject(); } 113

115 Java 7 uses an "open call" private void writeobject(objectoutputstream s) throws IOException { final ObjectOutputStream.PutField fields = s.putfields(); final Object[] data; synchronized (this) { fields.put("capacityincrement", capacityincrement); fields.put("elementcount", elementcount); data = elementdata.clone(); } fields.put("elementdata", data); s.writefields(); } 114

116 More synchronized writeobject()s File StringBuffer Throwable URL Hashtable and others 115

117 we have seen the deadly embrace 116

118 are there other deadlocks? 117

119 semaphore deadlocks 118

120 Bounded DB Connection Pools For example, say you have two DB connection pools Some tasks might require connections to both databases Thus thread A might hold semaphore for D1 and wait for D2, whereas thread B might hold semaphore for D2 and be waiting for D1 119

121 public class DatabasePool { private final Semaphore connections; public DatabasePool(int connections) { this.connections = new Semaphore(connections); } public void connect() { connections.acquireuninterruptibly(); System.out.println("DatabasePool.connect"); } public void disconnect() { System.out.println("DatabasePool.disconnect"); connections.release(); } } 120

122 ThreadMXBean does not show this as a deadlock! 121

123 JVM does not detect this deadlock 122

124 JVM does not detect this deadlock 122

125 Avoiding and diagnosing deadlocks Avoiding Liveness Hazards 123

126 Avoiding and diagnosing deadlocks You cannot get a lock-ordering deadlock with a single lock Easiest way to avoid deadlocks But not always practical 124

127 Avoiding and diagnosing deadlocks For multiple locks we need to define a lock order specify and document possible lock sequences Identify where multiple locks could be acquired Do a global analysis to ensure that lock ordering is consistent This can be extremely difficult in large programs 125

128 Use open calls whenever possible 126

129 Unit Testing for lock ordering deadlocks Code typically has to be called many times before a deadlock happens How many times do you need to call it to prove that there is no deadlock? Nondeterministic unit tests are bad - they should either always pass or always fail 127

130 Testing the Bank In the transfermoney() method, a deadlock occurs if after the first lock is granted, the first thread is swapped out and another thread requests the second lock public class Bank { public boolean transfermoney(account from, Account to, Amount amount) { synchronized (from) { // if thread is swapped out here, we can end up with a deadlock synchronized (to) { return doactualtransfer(from, to, amount); } } } } 128

131 We can force the context switch 129

132 simply add a sleep after first lock 130

133 Adding a sleep to cause deadlocks public class Bank { public boolean transfermoney(account from, Account to, Amount amount) { synchronized (from) { sleepawhilefortesting(); synchronized (to) { return doactualtransfer(from, to, amount); } } } protected void sleepawhilefortesting() {} } 131

134 In our unit test we use a SlowBank public class SlowBank extends Bank { private final long timeout; private final TimeUnit unit; public SlowBank(long timeout, TimeUnit unit) { this.timeout = timeout; this.unit = unit; } protected void sleepawhilefortesting() { try { unit.sleep(timeout); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } } } 132

135 Won't this cause problems in production? 133

136 In production we only use Bank 134

137 HotSpot will optimize away empty sleepawhilefortesting() 135

138 it is thus a zero cost 136

139 as long as SlowBank is never used 137

140 How can you verify the lockordering deadlock? 138

141 CTRL+Break, jstack, kill -3, jconsole, etc. 139

142 but that won't be a good unit test 140

143 Better via Java's ThreadMXBean 141

144 ThreadMXBean Get instance with ManagementFactory.getThreadMXBean() findmonitordeadlockedthreads() Includes only "monitor" locks, i.e. synchronized Only way to find deadlocks in Java 5 142

145 ThreadMXBean Better is finddeadlockedthreads() Includes "monitor" and "owned" (Java 5) locks Preferred method to test for deadlocks But, does not find deadlocks between semaphores See 143

146 public class BankDeadlockTest { private final static ThreadMXBean tmb = ManagementFactory.getThreadMXBean(); private boolean isthreaddeadlocked(long tid) { long[] ids = tmb.finddeadlockedthreads(); if (ids == null) return false; for (long id : ids) { if (id == tid) return true; } return false; } 144

147 private void checkthatthreadterminates(thread thread) throws InterruptedException { for (int i = 0; i < 2000; i++) { thread.join(50); if (!thread.isalive()) return; if (isthreaddeadlocked(thread.getid())) { fail("deadlock detected!"); } } fail(thread + " did not terminate in time"); } 145

148 @Test public void testfortransferdeadlock() throws InterruptedException { final Account alpha = new Account(new Amount(1000)); final Account ubs = new Account(new Amount( )); final Bank bank = new SlowBank(100, TimeUnit.MILLISECONDS); Thread alphatoubs = new Thread("alphaToUbs") { public void run() { bank.transfermoney(alpha, ubs, new Amount(100)); } }; Thread ubstoalpha = new Thread("ubsToAlpha") { public void run() { bank.transfermoney(ubs, alpha, new Amount(100)); } }; alphatoubs.start(); ubstoalpha.start(); checkthatthreadterminates(alphatoubs); } } 146

149 Output with broken transfermoney() method We see the deadlock within about 100 milliseconds junit.framework.assertionfailederror: Deadlock detected! at BankDeadlockTest.checkThatThreadTerminates(BankDeadlockTest.java:20) at BankDeadlockTest.testForTransferDeadlock(BankDeadlockTest.java:55) 147

150 Output with broken transfermoney() method If we fix the transfermoney() method, it also completes within about 100 milliseconds This is the time that we are sleeping for testing purposes Remember that the empty sleepawhilefortesting() method will be optimized away by HotSpot 148

151 Timed lock attempts Another technique for solving deadlocks is to use the timed trylock() method of explicit Java locks 149

152 However, if trylock() fails it may be 150

153 a deadlock 151

154 or some thread taking a long time 152

155 or a thread in an infinite loop 153

156 But ThreadMXBean shows it as deadlocked 154

157 even if it is a temporary condition 155

158 The JVM has support for trylock with synchronized 156

159 but it is hidden away 157

160 So here it is 158

161 159

162 Told you it was hidden! 160

163 It is hidden for a good reason 161

164 if you need trylock functionality 162

165 rather use explicit locks (ReentrantLock) 163

166 trylock with synchronized Technically possible in Sun JVM using Unsafe.getUnsafe().tryMonitorEnter(monitor) Unsafe.getUnsafe().monitorExit(monitor) 164

167 Deadlock analysis with thread dumps 165

168 Deadlock analysis with thread dumps We can also cause a thread dump in many ways: Ctrl+Break on Windows or Ctrl-\ on Unix Invoking "kill -3" on the process id Calling jstack on the process id Only shows deadlocks since Java 6 Intrinsic locks show more information of where they were acquired 166

169 Can a deadlock be resolved? 167

170 NO with synchronized 168

171 YES with ReentrantLock 169

172 synchronized() goes into BLOCKED state 170

173 ReentrantLock.lock() goes into WAITING state 171

174 We can write a DealockArbitrator 172

175 This stops one of the threads with a DeadlockVictimError 173

176 Our special Error for deadlocks public class DeadlockVictimError extends Error { private final Thread victim; public DeadlockVictimError(Thread victim) { super("deadlock victim: " + victim); this.victim = victim; } } public Thread getvictim() { return victim; } 174

177 public class DeadlockArbitrator { private static final ThreadMXBean tmb = ManagementFactory.getThreadMXBean(); public boolean tryresolvedeadlock( int attempts, long timeout, TimeUnit unit) throws InterruptedException { for (int i = 0; i < attempts; i++) { long[] ids = tmb.finddeadlockedthreads(); if (ids == null) return true; Thread t = findthread(ids[i % ids.length]); if (t == null) throw new IllegalStateException("Could not find thread"); t.stop(new DeadlockVictimError(t)); unit.sleep(timeout); } return false; } 175

178 finding threads by id is tricky public boolean tryresolvedeadlock() throws InterruptedException { return tryresolvedeadlock(3, 1, TimeUnit.SECONDS); } } private Thread findthread(long id) { for (Thread thread : Thread.getAllStackTraces().keySet()) { if (thread.getid() == id) return thread; } return null; } 176

179 177

180 Applicability of DeadlockArbitrator Only use in extreme circumstances Code that is outside your control and that deadlocks Where you cannot prevent the deadlock Remember, it only works with ReentrantLock 178

181 Deadlock Victim Questions? by Dr Heinz Kabutz && Olivier Croisier The Java Specialists Newsletter && The Coder's Breakfast && 179