What happens when deleting the OCS computer object?

Sorry sorry I've been very busy the last three weeks. First the kick-off for The New Way of Work event (second) at e-office together with Macaw. Later on I had a big issue on the customer side. The last 2 weeks I was working on a big OCS issue. The entire environment was corrupted and each MCU failed to start. Hundreds of users failed to sign in. Lots of "server unavailable messages"!

Customer case:

Customer was implementing new GPO's into their Active Directory. While doing this they use Group Policy Managed Console to edit those GPO's. Accidentally the Member Server OU (top level OU with 150 production computer objects beneath) was removed without any notification. After doing this they had a big problem with member servers joined in the domain infrastructure. The members server OU also contained the OCS Servers OU. So after that we had a very clear case. How can we restore the entire computer object of OCS and its configuration?

Options:

• AdRestore v1.1 By Mark Russinovich, Windows Server 2003 introduces the ability to restore deleted ("tombstoned") objects. This simple command-line utility enumerates the deleted objects in a domain and gives you the option of restoring each one.
• Windows Server 2003 - Authoritative Restore, an Authoritative Restore of Active Directory is one of the hardest tasks in Windows Server 2003.  To succeed, you need to understand how Active Directory Replication works, be an expert with NTDSutil, find the backup tapes and above all, a sound written plan.

Because no DC can be taken offline the ADRestore was the best option.Tombstone reanimation (which has nothing to do with zombies) provides the only way to recover deleted objects without taking a DC offline, and it's the only way to recover a deleted object's identity information, such as its objectGUID and objectSid attributes. Source

Our briefed investigation to fix this issue:

Error messages (1):

Office Communications Server ACP MCU Service could not be started.

Message: No corresponding MSFT_SIPMCUSetting for service DN 'CN=LS ACP MCU,CN=Microsoft,CN=OCS01,OU=OCS Servers,OU=Member Servers,DC=nl,DC=contoso,DC=local'. Stack: at Microsoft.Rtc.Internal.Wmi.WmiConsumer.get_Msft_SipMcuSetting() at Microsoft.Rtc.Internal.Wmi.WmiConsumer.get_Msft_SipMcuFactorySetting()at Microsoft.Rtc.Internal.Wmi.WmiConsumer.get_PoolDn()at Microsoft.LiveServer.AcpMcu.Config.ReadPoolConfig()at Microsoft.LiveServer.AcpMcu.Config.InitializeWmiConsumer()at Microsoft.LiveServer.AcpMcu.Config..ctor()at Microsoft.LiveServer.AcpMcu.ConferenceManager..ctor(IServiceFeedback service) at Microsoft.LiveServer.AcpMcu.AcpMcuService.CreateConferenceManager() at Microsoft.Rtc.Server.McuInfrastructure.RetryLogicHandler`1.ExecuteMethod(RetryableMethodDelegate retryableMethod, TimeSpan retryPeriod, TimeSpan maxDuration) at Microsoft.LiveServer.AcpMcu.AcpMcuService.OnStart(String[] args)

Error messages (2):

Failed to start service for the following reasons No corresponding MSFT_SIPMCUSetting for service DN 'CN=LS AV MCU,CN=Microsoft,CN=OCS01,OU=OCS Servers,OU=Member Servers,DC=nl,DC=contoso,DC=local'.

After some investigation (with ADRestore) and looking for other deleted objects (not only computer objects) I found (in the lost and found container) the following entries:

Enumerating domain deleted objects:

cn: LS ACP MCU DEL:54eaa9df-0401-4ff5-b31d-bcdd4ae05afe distinguishedName: CN=LS ACP MCU\0ADEL:54eaa9df-0401-4ff5-b31d-bcdd4ae05afe,CN=Deleted Objects,DC=nl,DC=contoso,DC=local lastKnownParent: CN=Microsoft\0ADEL:90419c61-6339-47f1-975a-add0cedfdd52,CN=Deleted Objects,DC=nl,DC=contoso,DC=local

and serveral MCU's with CN: cn: LS AV MCU, cn: LS Data MCU, cn: LS IM MCU

hmmm. Looks like the OCS computer objects stores configuration information right under the computer object in AD. Lets check into our own environment. Open dsa.msc (open in Advanced Features and Users, Groups, and Computer and Containers).

Browse to your OCS computer object:

 

Well probably the deleted objects matches the configuration of this customers computer object. The problem we had was that each MCU was listed twice (old unprep of previous OCS installation in PoC). Another question was rising which object to we need to restore?

Tip: The whenChanged attribute gets updated during the tombstone process, and should reflect when the actual delete took place.After having that information (thanks, Scott Oseychik/Senior Escalation Engineer/Microsoft, Unified Communications Team)

We restored the last objects (newest timestamp) and try again to start the OCS MCU services. So you probably think that the issue was fixed... not! ;-( the corresponding services were still in pending state.

Again after some investigation we decided to activate the MCU's again:

1. First we deactivated the MCU's by using lcscmd.exe;
2. after that we broke the domain membership of this OCS server (back in workgroup);
3. removed the OCS Servers OU and the corresponding computer account (OCS01;
4. re-create the OCS Server OU under Member Servers OU;
5. OCS01 rollback into domain (contoso);
6. move computer account to OCS Servers OU;
7. activate MCU again;
8. after all these steps the OCS infrastructure was fixed and started ;-)

Conclusion:

ADRestore is really a powerful tool but probably not the best tool to restore configuration objects of OCS. We do some restores by using the ADRestore but we saw that some value were stripped off. ;-( example: MCUFactory URL was not restored. Well just a brainwave at my side.