Successful DB2 NETLOGON in LAB (Sniffer on LAB HUB)

Troubleshooting Analysis for Windows 2000 Active Directory Authentication Problem Overview Servers DB1 & DB2 are configured with clustering (DB1 is active and DB2 is backup). The PDC (server NT9) is connected to a different subnet, which is separated by two firewalls. The DB server clustering services depend on Active Directory Authentication. When the servers cannot authenticate, the clustering service cannot start. To initially start the clustering services on the DB servers, the work-around is to connect to the DB server via terminal services and to manually map a network drive from the DB server to the PDC server using a domain USERID. Once the drive is mapped, the clustering service can be successfully started and then the network drive map can be disconnected. Once the cluster service is up and running it continues to work fine even if the Active Directory authentication fails (until the clustering service has to be restarted again). Connectivity for the Production Environment DB1 backend ----> client-fw1 ----> utilfw2 ----> NT9 Connectivity for the LAB Environment DB2 backend ----> NT9 Source Trace Files Trace file "Filter for NT09 IP (Mar31pm-apr01am).cap" was obtained with port monitor configured for the DB1 server and shows all traffic between the production servers DB1 & NT9 from 2003-03- 31 15:01 to 2003-04-01 08:13. Trace file " lab capture 02 DB2 communicating with NT9 ok.cap " shows all traffic on the LAB segment hub, including traffic between replica servers DB2 & NT9. Observations?? Switches and firewalls are not logging any drops between NT9 and the DB servers.?? Both DB servers are logging NETLOGON system authentication errors because they do not receive the responses to their RPC NETLOGON request packets submitted to TCP port 1026 on NT9 (NETLOGON UUID = 12345678-1234-abcd-ef00-01234567cffb).?? The NETLOGON request packets seen in the LAB appear identical to those sent on the production network.?? DB1 can successfully communicate with the Directory Replication Interface via the same port 1026 on NT9 (NTDS UUID = e3514235-4b06-11d1-ab04-00c04fc2dcd2).?? The NETLOGON response packets are being intercepted and RESET by utilfw2.?? utilfw2 runs Firewall-1 software on a Nokia platform. Revised on 14/05/2003 by Daniel Cayer Page 1 of 9

No. Time Successful DB2 NETLOGON in LAB (Sniffer on LAB HUB) NOTE: DB server and PDC are on the same local subnet. The traffic in the production network is identical to that in the LAB until the NETLOGON request, which is successfully acknowledged and answered in the LAB. This trace files contains all traffic from both servers since they were powered on. 1 Search for Domain Controller 1.1 DNS query for DC SRC DST Pro Info 173 2003-02-26 10:27:51 DB2 NT9 DNS Standard query SRV _ldap._tcp.default-first-site-name._sites.dc._msdcs.cwh- OTTAWA.COM 1.2 DNS response: DC found 174 2003-02-26 10:27:51 NT9 DB2 DNS Standard query response SRV 0 100 389 cwh-ott-nt-009.cwh-ottawa.com 2 Determine if DC is closest one available 2.1 LDAP search request for matching host name, domain name, SID & GUID 175 2003-02-26 10:27:51 DB2 NT9 LDAP MsgId=1 MsgType=Search Request 2.2 Successful LDAP response 176 2003-02-26 10:27:51 NT9 DB2 LDAP MsgId=1 MsgType=Search Entry 3 Establishment of secured channel between DB2 & DC (NT09) 3.1 PORTMAPPER (EPM) request via RPC for Active Directory Logon 177 2003-02-26 10:27:51 DB2 NT9 TCP 1103 > 135 [SYN] Seq=1560904577 Ack=0 Win=16384 Len=0 178 2003-02-26 10:27:51 NT9 DB2 TCP 135 > 1103 [SYN, ACK] Seq=4114774075 Ack=1560904578 Win=17520 Len=0 179 2003-02-26 10:27:51 DB2 NT9 TCP 1103 > 135 [ACK] Seq=1560904578 Ack=4114774076 Win=17520 Len=0 180 2003-02-26 10:27:51 DB2 NT9 DCERPC Bind: call_id: 1 UUID: EPM 181 2003-02-26 10:27:51 NT9 DB2 DCERPC Bind_ack: call_id: 1 accept max_xmit: 5840 max_recv: 5840 182 2003-02-26 10:27:51 DB2 NT9 EPM Map request 3.2 PORTMAPPER response (port = 1026) 183 2003-02-26 10:27:51 NT9 DB2 EPM Map reply 184 2003-02-26 10:27:51 DB2 NT9 TCP 1103 > 135 [FIN, ACK] Seq=1560904806 Ack=4114774464 Win=17132 Len=0 3.3 NETLOGON request 185 2003-02-26 10:27:51 DB2 NT9 TCP 1104 > 1026 [SYN] Seq=1560942590 Ack=0 Win=16384 Len=0 186 2003-02-26 10:27:51 NT9 DB2 TCP 1026 > 1104 [SYN, ACK] Seq=4114826958 Ack=1560942591 Win=17520 Len=0 187 2003-02-26 10:27:51 NT9 DB2 TCP 135 > 1103 [ACK] Seq=4114774464 Ack=1560904807 Win=17292 Len=0 188 2003-02-26 10:27:51 NT9 DB2 TCP 135 > 1103 [FIN, ACK] Seq=4114774464 Ack=1560904807 Win=17292 Len=0 189 2003-02-26 10:27:51 DB2 NT9 TCP 1104 > 1026 [ACK] Seq=1560942591 Ack=4114826959 Win=17520 Len=0 190 2003-02-26 10:27:51 DB2 NT9 DCERPC Bind: call_id: 1 UUID: RPC_NETLOGON 191 2003-02-26 10:27:51 DB2 NT9 TCP 1103 > 135 [ACK] Seq=1560904807 Ack=4114774465 Win=17132 Len=0 3.4 NETLOGON request acknowledgement 192 2003-02-26 10:27:51 NT9 DB2 DCERPC Bind_ack: call_id: 1 accept max_xmit: 5840 max_recv: 5840 3.5 NETLOGON server challenge request 193 2003-02-26 10:27:51 DB2 NT9 RPC_NETLOGON ServerReqChallenge request, REBIZX-DB2 3.6 NETLOGON server challenge response 194 2003-02-26 10:27:51 NT9 DB2 RPC_NETLOGON ServerReqChallenge reply 3.7 NETLOGON server authentication request 195 2003-02-26 10:27:51 DB2 NT9 RPC_NETLOGON ServerAuthenticate3 request 3.8 NETLOGON server authentication response 196 2003-02-26 10:27:51 NT9 DB2 RPC_NETLOGON ServerAuthenticate3 reply 3.9 New NETLOGON connection for Domain Info lookup 197 2003-02-26 10:27:51 DB2 NT9 TCP 1105 > 1026 [SYN] Seq=1561007361 Ack=0 Win=16384 Len=0 198 2003-02-26 10:27:51 NT9 DB2 TCP 1026 > 1105 [SYN, ACK] Seq=4114884240 Ack=1561007362 Win=17520 Len=0 199 2003-02-26 10:27:51 DB2 NT9 TCP 1105 > 1026 [ACK] Seq=1561007362 Ack=4114884241 Win=17520 Len=0 200 2003-02-26 10:27:51 DB2 NT9 DCERPC Bind: call_id: 3 UUID: RPC_NETLOGON 201 2003-02-26 10:27:51 NT9 DB2 DCERPC Bind_ack: call_id: 3 accept max_xmit: 5840 max_recv: 5840 202 2003-02-26 10:27:51 DB2 NT9 RPC_NETLOGON NetrLogonGetDomainInfo request Revised on 14/05/2003 by Daniel Cayer Page 2 of 9

3.10 Domain Info response (encrypted payload) 203 2003-02-26 10:27:51 NT9 DB2 RPC_NETLOGON NetrLogonGetDomainInfo reply 3.11 Establish SMB connection, authenticate with Kerberos, etc 204 2003-02-26 10:27:51 DB2 NT9 ICMP Echo (ping) request 205 2003-02-26 10:27:51 NT9 DB2 ICMP Echo (ping) reply 206 2003-02-26 10:27:51 DB2 NT9 TCP 1106 > 445 [SYN] Seq=1561041120 Ack=0 Win=16384 Len=0 207 2003-02-26 10:27:51 NT9 DB2 TCP 445 > 1106 [SYN, ACK] Seq=4114942432 Ack=1561041121 Win=17520 Len=0 208 2003-02-26 10:27:51 DB2 NT9 TCP 1106 > 445 [ACK] Seq=1561041121 Ack=4114942433 Win=17520 Len=0 209 2003-02-26 10:27:51 DB2 NT9 ICMP Echo (ping) request 210 2003-02-26 10:27:51 NT9 DB2 ICMP Echo (ping) reply 211 2003-02-26 10:27:51 DB2 NT9 SMB Negotiate Protocol Request 212 2003-02-26 10:27:51 NT9 DB2 SMB Negotiate Protocol Response 213 2003-02-26 10:27:51 DB2 NT9 KRB5 AS-REQ 214 2003-02-26 10:27:51 NT9 DB2 KRB5 KRB-ERROR 215 2003-02-26 10:27:51 DB2 NT9 KRB5 AS-REQ 216 2003-02-26 10:27:51 NT9 DB2 KRB5 AS-REP 217 2003-02-26 10:27:51 DB2 NT9 KRB5 TGS-REQ 218 2003-02-26 10:27:51 NT9 DB2 KRB5 TGS-REP 219 2003-02-26 10:27:51 DB2 NT9 KRB5 TGS-REQ 220 2003-02-26 10:27:51 NT9 DB2 KRB5 TGS-REP 221 2003-02-26 10:27:51 DB2 NT9 SMB Session Setup AndX Request[Unreassembled Packet] 222 2003-02-26 10:27:51 DB2 NT9 NBSS NBSS Continuation Message 223 2003-02-26 10:27:51 NT9 DB2 TCP 445 > 1106 [ACK] Seq=4114942624 Ack=1561043946 Win=17520 Len=0 224 2003-02-26 10:27:51 NT9 DB2 SMB Session Setup AndX Response, Error: STATUS_MORE_PROCESSING_REQUIRED 225 2003-02-26 10:27:51 DB2 NT9 SMB Session Setup AndX Request[Unreassembled Packet] 226 2003-02-26 10:27:51 DB2 NT9 NBSS NBSS Continuation Message 227 2003-02-26 10:27:51 NT9 DB2 TCP 445 > 1106 [ACK] Seq=4114943017 Ack=1561046572 Win=17520 Len=0 228 2003-02-26 10:27:51 NT9 DB2 SMB Session Setup AndX Response 229 2003-02-26 10:27:51 DB2 NT9 SMB Tree Connect AndX Request,Path: \\CWH-OTT-NT-009.CWH-OTTAWA.COM\IPC$ 230 2003-02-26 10:27:51 NT9 DB2 SMB Tree Connect AndX Response 231 2003-02-26 10:27:51 DB2 NT9 SMB NT Create AndX Request, Path: \lsarpc 232 2003-02-26 10:27:51 NT9 DB2 SMB NT Create AndX Response, FID: 0x4000 233 2003-02-26 10:27:51 DB2 NT9 DCERPC Bind: call_id: 1 UUID: LSA 234 2003-02-26 10:27:51 NT9 DB2 DCERPC Bind_ack: call_id: 1 accept max_xmit: 4280 max_recv: 4280 235 2003-02-26 10:27:51 DB2 NT9 LSA OpenPolicy2 request, \\cwh-ott-nt-009.cwh-ottawa.com 236 2003-02-26 10:27:51 NT9 DB2 SMB Write AndX Response, FID: 0x4000, 140 bytes 237 2003-02-26 10:27:51 DB2 NT9 SMB Read AndX Request, FID: 0x4000, 1024 bytes at offset 0 238 2003-02-26 10:27:51 NT9 DB2 LSA OpenPolicy2 reply 239 2003-02-26 10:27:51 DB2 NT9 SMB NT Create AndX Request, Path: \lsarpc 240 2003-02-26 10:27:51 NT9 DB2 SMB NT Create AndX Response, FID: 0x4001 241 2003-02-26 10:27:51 DB2 NT9 DCERPC Bind: call_id: 2 UUID: LSA 242 2003-02-26 10:27:51 NT9 DB2 DCERPC Bind_ack: call_id: 2 accept max_xmit: 4280 max_recv: 4280 243 2003-02-26 10:27:51 DB2 NT9 LSA QueryInfoPolicy request, Primary Domain Information 244 2003-02-26 10:27:51 NT9 DB2 SMB Write AndX Response, FID: 0x4001, 96 bytes 245 2003-02-26 10:27:51 DB2 NT9 SMB Read AndX Request, FID: 0x4001, 1024 bytes at offset 0 246 2003-02-26 10:27:51 NT9 DB2 LSA QueryInfoPolicy reply 247 2003-02-26 10:27:51 DB2 NT9 LSA QueryInfoPolicy request, Account Domain Information 248 2003-02-26 10:27:51 NT9 DB2 SMB Write AndX Response, FID: 0x4001, 96 bytes 249 2003-02-26 10:27:51 DB2 NT9 SMB Read AndX Request, FID: 0x4001, 1024 bytes at offset 0 250 2003-02-26 10:27:51 NT9 DB2 LSA QueryInfoPolicy reply 251 2003-02-26 10:27:51 DB2 NT9 LSA LookupSIDs2 request 252 2003-02-26 10:27:51 NT9 DB2 SMB Write AndX Response, FID: 0x4001, 240 bytes 253 2003-02-26 10:27:51 DB2 NT9 SMB Read AndX Request, FID: 0x4001, 1024 bytes at offset 0 254 2003-02-26 10:27:51 NT9 DB2 LSA LookupSIDs2 reply 255 2003-02-26 10:27:51 DB2 NT9 LSA Close request 256 2003-02-26 10:27:51 NT9 DB2 SMB Write AndX Response, FID: 0x4001, 96 bytes 257 2003-02-26 10:27:51 DB2 NT9 SMB Read AndX Request, FID: 0x4001, 1024 bytes at offset 0 258 2003-02-26 10:27:51 NT9 DB2 LSA Close reply 259 2003-02-26 10:27:51 DB2 NT9 SMB Close Request, FID: 0x4000 260 2003-02-26 10:27:51 NT9 DB2 SMB Close Response 261 2003-02-26 10:27:51 DB2 NT9 SMB Close Request, FID: 0x4001 262 2003-02-26 10:27:51 NT9 DB2 SMB Close Response 263 2003-02-26 10:27:51 DB2 NT9 TCP 1105 > 1026 [ACK] Seq=1561008268 Ack=4114885009 Win=16752 Len=0 264 2003-02-26 10:27:51 DB2 NT9 TCP 1104 > 1026 [ACK] Seq=1560943001 Ack=4114827099 Win=17380 Len=0 265 2003-02-26 10:27:51 DB2 NT9 TCP 1106 > 445 [ACK] Seq=1561048707 Ack=4114945372 Win=16066 Len=0 Revised on 14/05/2003 by Daniel Cayer Page 3 of 9

No. Time Failed DB1 NETLOGON on Production LAN (Sniffer next to DB1) 1 Search for Domain Controller SRC DST Proto Info 4571 2003-03-31 22:14:24 DB1 NT09 DNS Standard query SRV _ldap._tcp.pdc._msdcs.rebizx-db1 4572 2003-03-31 22:14:24 NT09 DB1 DNS Standard query response, No such name 4573 2003-03-31 22:14:26 DB1 NT09 DNS Standard query SRV _ldap._tcp.default-first-site- Name._sites.dc._msdcs.rebizx-db1 4574 2003-03-31 22:14:26 NT09 DB1 DNS Standard query response, No such name 4575 2003-03-31 22:14:26 DB1 NT09 DNS Standard query SRV _ldap._tcp.dc._msdcs.rebizx-db1 4576 2003-03-31 22:14:26 NT09 DB1 DNS Standard query response, No such name 4577 2003-03-31 22:17:22 DB1 NT09 DNS Standard query SRV _ldap._tcp.default-first-site- Name._sites.dc._msdcs.CWH-OTTAWA.COM 4578 2003-03-31 22:17:22 NT09 DB1 DNS Standard query response SRV 0 100 389 cwh-ott-nt-009.cwh-ottawa.com 2 Determine if DC is closest one available 4579 2003-03-31 22:17:22 DB1 NT09 LDAP MsgId=3743 MsgType=Search Request 4580 2003-03-31 22:17:22 NT09 DB1 LDAP MsgId=3743 MsgType=Search Entry 3 Establishment of secured channel between DB1 & DC (NT09) 3.1 PORTMAPPER (EPM) request via RPC for Active Directory Logon 4581 2003-03-31 22:17:22 DB1 NT09 TCP 1673 > epmap [SYN] Seq=192046899 Ack=0 Win=16384 Len=0 4582 2003-03-31 22:17:22 NT09 DB1 TCP epmap > 1673 [SYN, ACK] Seq=4024295118 Ack=192046900 Win=17520 Len=0 4583 2003-03-31 22:17:22 DB1 NT09 TCP 1673 > epmap [ACK] Seq=192046900 Ack=4024295119 Win=17520 Len=0 4584 2003-03-31 22:17:22 DB1 NT09 DCERPC Bind: call_id: 1 UUID: EPM 4585 2003-03-31 22:17:22 NT09 DB1 DCERPC Bind_ack: call_id: 1 accept max_xmit: 5840 max_recv: 5840 4586 2003-03-31 22:17:22 DB1 NT09 EPM Map request 3.2 PORTMAPPER response (port = 1026) 4587 2003-03-31 22:17:22 NT09 DB1 EPM Map reply 4588 2003-03-31 22:17:22 DB1 NT09 TCP 1673 > epmap [FIN, ACK] Seq=192047128 Ack=4024295331 Win=17308 Len=0 4589 2003-03-31 22:17:22 DB1 NT09 TCP 1674 > 1026 [SYN] Seq=192089439 Ack=0 Win=16384 Len=0 4590 2003-03-31 22:17:22 NT09 DB1 TCP epmap > 1673 [ACK] Seq=4024295331 Ack=192047129 Win=17292 Len=0 4591 2003-03-31 22:17:22 NT09 DB1 TCP epmap > 1673 [FIN, ACK] Seq=4024295331 Ack=192047129 Win=17292 Len=0 4592 2003-03-31 22:17:22 DB1 NT09 TCP 1673 > epmap [ACK] Seq=192047129 Ack=4024295332 Win=17308 Len=0 3.3 NETLOGON request 4593 2003-03-31 22:17:22 NT09 DB1 TCP 1026 > 1674 [SYN, ACK] Seq=4024345794 Ack=192089440 Win=17520 Len=0 4594 2003-03-31 22:17:22 DB1 NT09 TCP 1674 > 1026 [ACK] Seq=192089440 Ack=4024345795 Win=17520 Len=0 4595 2003-03-31 22:17:22 DB1 NT09 DCERPC Bind: call_id: 1 UUID: RPC_NETLOGON 3.3.1 Retransmission of NETLOGON request (3-second timeoute) 4596 2003-03-31 22:17:26 DB1 NT09 DCERPC Bind: call_id: 1 UUID: RPC_NETLOGON 3.3.2 NETLOGON error (NT09 reset of the TCP connection) 4597 2003-03-31 22:17:26 NT09 DB1 TCP 1026 > 1674 [RST] Seq=4024345795 Ack=4024345795 Win=0 Len=0 3.3.3 RETRY NETLOGON (2nd attempt, from a different source port) 4598 2003-03-31 22:17:26 DB1 NT09 TCP 1675 > 1026 [SYN] Seq=193074352 Ack=0 Win=16384 Len=0 4599 2003-03-31 22:17:26 NT09 DB1 TCP 1026 > 1675 [SYN, ACK] Seq=4025232633 Ack=193074353 Win=17520 Len=0 4600 2003-03-31 22:17:26 DB1 NT09 TCP 1675 > 1026 [ACK] Seq=193074353 Ack=4025232634 Win=17520 Len=0 4601 2003-03-31 22:17:26 DB1 NT09 DCERPC Bind: call_id: 1 UUID: RPC_NETLOGON 3.3.4 Retransmission of NETLOGON request (3-second timeoute) 4604 2003-03-31 22:17:29 DB1 NT09 DCERPC Bind: call_id: 1 UUID: RPC_NETLOGON 3.3.5 NETLOGON error (NT09 reset of the TCP connection) 4605 2003-03-31 22:17:29 NT09 DB1 TCP 1026 > 1675 [RST] Seq=4025232634 Ack=4025232634 Win=0 Len=0 Revised on 14/05/2003 by Daniel Cayer Page 4 of 9

NOTE: Other successful TCP connections are seen on NT09 port 1026 for another UUID: No. Time SRC DST Proto Info 800 2003-03-31 16:25:57 DB1 NT09 TCP 3927 > 1026 [SYN] Seq=1126453154 Ack=0 Win=16384 Len=0 804 2003-03-31 16:25:57 NT09 DB1 TCP 1026 > 3927 [SYN, ACK] Seq=3006786322 Ack=1126453155 Win=17520 Len=0 805 2003-03-31 16:25:57 DB1 NT09 TCP 3927 > 1026 [ACK] Seq=1126453155 Ack=3006786323 Win=17520 Len=0 812 2003-03-31 16:25:57 DB1 NT09 DCERPC Bind: call_id: 1 UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 ver 4.0 813 2003-03-31 16:25:57 DB1 NT09 TCP 3927 > 1026 [PSH,ACK] Seq=1126454615 Ack=3006786323 Win=17520 Len=1156 814 2003-03-31 16:25:57 NT09 DB1 TCP 1026 > 3927 [ACK] Seq=3006786323 Ack=1126455771 Win=17520 Len=0 815 2003-03-31 16:25:57 NT09 DB1 DCERPC Bind_ack: call_id: 1 accept max_xmit: 5840 max_recv: 5840 816 2003-03-31 16:25:57 DB1 NT09 DCERPC Alter_context: call_id: 1 UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 ver 4.0 817 2003-03-31 16:25:57 NT09 DB1 DCERPC Alter_context_resp: call_id: 1 accept max_xmit: 5840 max_recv: 5840 818 2003-03-31 16:25:57 DB1 NT09 DCERPC Request: call_id: 1 opnum: 0 ctx_id: 0 819 2003-03-31 16:25:57 NT09 DB1 DCERPC Response: call_id: 1 ctx_id: 0 820 2003-03-31 16:25:57 DB1 NT09 DCERPC Request: call_id: 2 opnum: 12 ctx_id: 0 821 2003-03-31 16:25:57 NT09 DB1 DCERPC Response: call_id: 2 ctx_id: 0 822 2003-03-31 16:25:57 DB1 NT09 DCERPC Request: call_id: 3 opnum: 12 ctx_id: 0 823 2003-03-31 16:25:57 NT09 DB1 DCERPC Response: call_id: 3 ctx_id: 0 824 2003-03-31 16:25:57 DB1 NT09 DCERPC Request: call_id: 4 opnum: 1 ctx_id: 0 825 2003-03-31 16:25:57 NT09 DB1 DCERPC Response: call_id: 4 ctx_id: 0 826 2003-03-31 16:25:57 DB1 NT09 TCP 3927 > 1026 [FIN, ACK] Seq=1126456608 Ack=3006787506 Win=16337 Len=0 827 2003-03-31 16:25:57 NT09 DB1 TCP 1026 > 3927 [ACK] Seq=3006787506 Ack=1126456609 Win=16683 Len=0 828 2003-03-31 16:25:57 NT09 DB1 TCP 1026 > 3927 [FIN, ACK] Seq=3006787506 Ack=1126456609 Win=16683 Len=0 829 2003-03-31 16:25:57 DB1 NT09 TCP 3927 > 1026 [ACK] Seq=1126456609 Ack=3006787507 Win=16337 Len=0 Kerberos Errors From DB1 to NT9 (Sniffer next to DB1) Trace file " DB1 kereberos failed to NT9.cap " shows that DB1 is using the wrong name to authenticate with Kerberos. In fact DB1 uses LDAP1's IP address instead of its own FQDN!!! These Kerberos errors are occurring at a regular 40-minute interval. NOTE: Sniffer does not decode Kerberos Use Ethereal instead! Frame 5 (1373 bytes on wire, 1373 bytes captured) Arrival Time: Mar 28, 2003 10:02:48.124115000 Ethernet II, Src: 00:02:a5:6b:8d:96, Dst: 00:00:5e:00:01:04 Internet Protocol, Src Addr: 172.20.6.39 (172.20.6.39), Dst Addr: 172.20.0.193 (172.20.0.193) User Datagram Protocol, Src Port: 2729 (2729), Dst Port: 88 (88) Kerberos Version: 5 MSG Type: TGS-REQ Pre-Authentication Type: PA-TGS-REQ Value: 6E82048730820483A003020105A10302... Request Options: 0040810010 Realm: CWH-OTTAWA.COM Server Name: HOST Type: Service and Instance Name: HOST Name: 172.20.6.37 End Time: 2037-09-13 02:48:05 (Z) Random Number: 1281252417 Encryption Types Type: rc4-hmac This should be a qualified domain such as "cwh-ott- nt-009.cwh- OTTAWA.COM"!!! Revised on 14/05/2003 by Daniel Cayer Page 5 of 9

Type: Unknown encryption type 0xff7b Type: Unknown encryption type 0x80 Type: des-cbc-md5 Type: des-cbc-crc Type: rc4-hmac-exp Type: Unknown encryption type 0xff79 Frame 6 (150 bytes on wire, 150 bytes captured) Arrival Time: Mar 28, 2003 10:02:48.127646000 Ethernet II, Src: 00:a0:8e:32:ba:53, Dst: 00:02:a5:6b:8d:96 Internet Protocol, Src Addr: 172.20.0.193 (172.20.0.193), Dst Addr: 172.20.6.39 (172.20.6.39) User Datagram Protocol, Src Port: 88 (88), Dst Port: 2729 (2729) Kerberos Version: 5 MSG Type: KRB-ERROR stime: 2003-03-28 15:01:20 (Z) susec: 982004 Error Code: KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN realm: CWH-OTTAWA.COM sname: krbtgt Type: Service and Instance Name: krbtgt Name: CWH-OTTAWA.COM Failed NETLOGON on Production LAN (Sniffer next to NT9) NOTE: Span a port on switch connected to NT9 for the Sniffer. Trace file shows NETLOGON request from DB1 is reaching NT9 and NT9 is acknowledging!!! No. Time SRC DST Pro Info 7901 2003-04-02 14:17:54 DB1 NT9 TCP 1280 > 1026 [SYN] Seq=2306675940 Ack=0 Win=16384 Len=0 7902 2003-04-02 14:17:54 NT9 DB1 TCP 1026 > 1280 [SYN, ACK] Seq=1956349167 Ack=2306675941 Win=17520 Len=0 7904 2003-04-02 14:17:54 DB1 NT9 TCP 1280 > 1026 [ACK] Seq=2306675941 Ack=1956349168 Win=17520 Len=0 7905 2003-04-02 14:17:54 DB1 NT9 DCERPC Bind: call_id: 1 UUID: RPC_NETLOGON 7906 2003-04-02 14:17:54 NT9 DB1 DCERPC Bind_ack: call_id: 1 accept max_xmit: 5840 max_recv: 5840 7907 2003-04-02 14:17:54 DB1 NT9 TCP 1280 > 1026 [RST] Seq=2306676013 Ack=0 Win=0 Len=0 7912 2003-04-02 14:17:58 DB1 NT9 DCERPC Bind: call_id: 1 UUID: RPC_NETLOGON 7913 2003-04-02 14:17:58 NT9 DB1 TCP 1026 > 1280 [RST] Seq=1956349168 Ack=1956349168 Win=0 Len=0 Where are these packets dropped and who is sending the RST on behalf of the DB servers??? obtain trace file from segment between 2 firewalls Failed NETLOGON on Production LAN (Trace from utilfw2 interface facing the other firewall) Packet 7906 above (NETLOGON response) is intercepted by utilfw2 and packet 7907 is originated from this same utilfw2 because both packets do not show up on the other side of this firewall: No. Time SRC DST Pro Info 1 2003-04-03 00:45:00 DB1 NT9 TCP 4447 > 1026 [SYN] Seq=3797945959 Ack=0 Win=16384 Len=0 2 2003-04-03 00:45:00 NT9 DB1 TCP 1026 > 4447 [SYN, ACK] Seq=2862503320 Ack=3797945960 Win=17520 Len=0 Revised on 14/05/2003 by Daniel Cayer Page 6 of 9

3 2003-04-03 00:45:00 DB1 NT9 TCP 4447 > 1026 [ACK] Seq=3797945960 Ack=2862503321 Win=17520 Len=0 4 2003-04-03 00:45:00 DB1 NT9 DCERPC Bind: call_id: 1 UUID: RPC_NETLOGON 5 2003-04-03 00:45:03 DB1 NT9 DCERPC Bind: call_id: 1 UUID: RPC_NETLOGON 6 2003-04-03 00:45:03 NT9 DB1 TCP 1026 > 4447 [RST] Seq=2862503321 Ack=2862503321 Win=0 Len=0 Revised on 14/05/2003 by Daniel Cayer Page 7 of 9

Questions 1. Is there any way to determine the correctness of the NETLOGON requests from DB1 (i.e.: is DB1 attempting to logon to NT09 correctly)? ANSWER: YES! The packets in the LAB are identical! 2. Is there any way to determine why NT09 does not even acknowledge these NETLOGON requests at the TCP layer? ANSWER: YES! NT09 DOES acknowledge the packets. The firewall is intercepting and dropping the response, which includes the TCP ACK!!! 3. Is NT09 supposed to be listening on port 1026 for both NETLOGON and NTDS UUIDs (e3514235-4b06-11d1-ab04-00c04fc2dcd2 & 12345678-1234-abcd-ef00-01234567cffb)? ANSWER: YES! This is a normal behavior for Win2K. 4. Are the Kerberos errors the cause of the NETLOGON failures? ANSWER: NO! NETLOGON fails because of the firewall. 5. What is the root cause of the Kerberos errors? 6. What are the dependencies between Kerberos and Active Directory Authentication? Suggestions 1. Fix Firewall 2. Fix Kerberos problem (Windows patch???) Microsoft Knowledge Base Articles: 260371 - Troubleshooting Common Active Directory Setup Issues in Windows 2000 220940 - How to Enable Diagnostic Event Logging for Active Directory Services 248807 - Using Uppercase Letters for Kerberos Realm Names 262177 - HOW TO: Enable Kerberos Event Logging 235529 - Kerberos Support on Windows 2000-Based Server Clusters 280132 - XCCC: Exchange 2000 Windows 2000 Connectivity Through Firewalls 308111 - A Missing Service Principal Name May Prevent Domain Controllers from Replicating Revised on 14/05/2003 by Daniel Cayer Page 8 of 9

Conclusion Once we were able to identify the failed NETLOGON requests in the trace files (corresponding to the NETLOGON errors on the DB servers), we then moved the Sniffer next to the PDC server and confirmed that the NETLOGON requests were indeed being answered. Additional traces from the firewalls allowed us to determine that the NETLOGON responses from the PDC were being blocked by the utilfw2 firewall. Upon reception of the NETLOGON response packet from NT9, utilfw2 would immediately send back a TCP RST to NT9. A support call was made to the firewall vendor (Check-Point) who confirmed that they did not support Microsoft Active Directory on this version of the firewall-1 software (version 4.x). Their recommendation was to upgrade the firewall software to a more recent version. Lessons Learned?? How Windows 2000 Active Directory Authentication works?? What Windows 2000 Active Directory Authentication looks like "on-the-wire". Revised on 14/05/2003 by Daniel Cayer Page 9 of 9