Keberos troubleshooting

First and foremost, in my opinion the absolute BEST way to troubleshoot a Kerberos problem is with WireShark. It has the filters built right in to show you exactly what the Kerberos payloads look like. You'll just need to plant it somewhere in the line of fire, preferably in this case the client or the KDC.

• You can enable debug logging for client side Kerberos with a TMSH command:

  tmsh modify sys db log.rba.level value debug

  But make sure you turn this off when you're done.

  tmsh modify sys db log.rba.level value notice

  This log data will show up in /var/log/apm

• On any Windows machine, the default security posture for IE is to only pass Kerberos credentials to a site in its Local Intranet sites list

  Tools - Internet Options - Security tab - Local Intranet - Sites button - Advanced

  You'll want to add this site URL to that list. To do the same thing in Firefox (on a domain-joined Windows machine), type about:config in the address bar and look for:

  network.negotiate-auth.trusted-uris

• Time is critical to Kerberos. You usually have a little bit of fudge room, but it's always best for all parties to be in step. That means the client, KDC, and BIG-IP should all be synchronized.

• Kerberos also relies very heavily on DNS. Make sure the hostname/SPN of the APM VIP has both an A and PTR record in AD DNS. You cannot use IP addresses with Kerberos.

• Admittedly some of the documentation out there for setting this stuff up is, at times, vague. Assuming you're using a Windows 2003 or 2008 AD, you'll want to create your keytab file with this syntax:

  ktpass /princ HTTP/vip.example.com@EXAMPLE.COM /mapuser example\krbsvc /ptype KRB5_NT_PRINCIPAL /pass password /out c:\file.keytab

  The most important parts being the SPN format - try to use an uppercase SPN service (HTTP/), lowercase server name (vip.example.com), and uppercase realm (@EXAMPLE.COM). You'll also want to use the KRB5_NT_PRINCIPAL ptype. Some of the other ptypes will work, but this is more semantically correct. And finally, though this isn't a hard and fast rule, I've had difficulties in the past when the username was not lowercase.

• After this command runs, you'll notice in the AD service account's properties that a) the account's User Logon Name is set to the SPN value, and B) a servicePrincipalName attribute has also been created for the account (with this same SPN value). Make sure that is true in your case.

• Unless you need it for something else, try removing the DNS settings from the BIG-IP and test. I know that may sound odd, but I've seen that cause a problem too. You may also want to fire up WireShark and look for any anomalous DNS traffic.

• Between tests you'll want to clear any residual Kerberos caches. For APM:

  bigstart restart rba

  For Windows (the client):

  klist purge     

• This one is a long shot, but it's also possible that password changes and other account activities can update the KVNO number and put it out of sync with the keytab. To verify, from the KDC:

  ldifde -f c:\spn_out.txt -d "DC=example.com,DC=com" -l *,msDS-KeyVersionNumber -r "(serviceprincipalname=HTTP/vip*)" -p subtree

  Look for the "msDS-KeyVersionNumber" string in the output file

  Then from the BIG-IP:

  klist -ekt  

  If the KVNO numbers don't match, you'll want to completely delete and recreate the AD service account and re-export a keytab.

• If all else fails, please respond back with the Kerberos error information you see in the WireShark capture.