Workstation trust relationship issues and iPhone login prompt issues: RESOLVED

Just recently I worked on resolving an authentication issue that took a month long to solve with Microsoft Premier Support Services on a Sev-A case. The issue ended up just needing a hotfix applied to all the remaining 2003 DCs in the domain.

The symptoms were the following:

1) iPhones were randomly prompting for logon and sometimes would not allow authentication for up to 30 minutes. This was also logged in the UAG 2010 servers as Event ID 14 mentioning the "trust relationship" has failed. Android phones also had the same issue however they never prompted on error so it was quiet on the android front.


 
2)  Workstations would randomly fall off the domain and get errors like Workstation trust relationship between this workstation and primary domain failed, or no logon servers available to process this logon request. The workaround was to logon with your username with the SPN like user@user.com or rejoin the machine to the domain.

Environment for the affected AD site:

5 2003 R2 DCs, 2 2008 R2 DCs, Exchange 2010 in the site (3 CAS) with iPhones/Android phones coming in through 2 UAG 2010 array.

Problem and Solution:

We found out that the KRBTGT object had been authoritatively restored causing it increase it's version number basically making it unreadable by the 2003 DCs. We also found that the KRBTGT object had been moved to a different OU folder (possibly by a disabled users script moving disabled accounts to the "Disabled Users" OU). Simply applying the hotfix from MSFT to all the 2003 DCs solved the problem.Another solution would be to upgrade all the remaining 2003 DCs.


Takeaways:
Solving kerberos authentication issues is sometimes very hard to do. We ended up working on many fronts to solve this problem (UAG, Exchange, AD). Some good debugging techniques are to Enable Kerberos Logging, Enable netlogon.log logging on all the effected DC's and try to actively start up netmon traces as well when you notice the issue occuring.

Microsoft KB with hotfix: http://support.microsoft.com/kb/939820