Category Archives: AD DS

ADDSCHK – Quick check on domain size

Sometimes while looking at somebody’s else ADDS environment you may want to know some basics about it – things such as total number of users, or in which OU this specific server is hiding. What surprises me a lot is that how frequently you can see people telling you that they don’t have right consoles here on this server (while their just in one PoSh line from all they need), or they not sure if they have permissions (which they usually do have). If you are lucky you just spend some time waiting for a person switching over to some other machine or directly to DC (yes to DC, just because ADUC console lives there 🙂 ), or in some other cases you will be dragged through multiple redirects / additions of people to the call only to end up explaining final person in that chain exact steps to be performed to get your questions answered (which you were perfectly able to do without switching servers and involving other people, in the first place).

Unless you already got it, it is more preferable and faster just to do yourself a favor of comfortably staying on the server where you working and issue Install-WindowsFeature RSAT-AD-PowerShell to solve missing tools problem in 20 seconds, and then, use PoSh to get your questions answered. Here is sample PS function, which I named similarly to  CHKDSK (thing of which I have very fond memories ever since I use it to help my classmate to repair his HDD at the time of 1-2 GB hard drives and Windows 95) – ADDSCHK:

In the world where increasing number of people does not hone their “I can do this in N ways” skills (and sometimes even “I understand how it works” too), you frequently better off speaking PoSh with infrastructure directly than with those who entrusted to keep it up and running 🙂

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Configuring Windows Server 2016 Core Domain Controller

In Windows Server 2016 you no longer have an opportunity to switch back and forth between core and GUI installation, hence you cannot do install and configure AD DS in a lazy way (using full GUI) and then convert it to core. That was something I discovered hard way long time ago – so I already have separate VHDX templates for Server 2016 core and full GUI VMs.

But it has been quite a while since I was playing with Server Core so when I starting provisioning my new Server 2016 core domain controller VMs today I realized that I need to remember quite a few commands to fully install AD DS on Server Core. I was about to create a blog post listing essential commands, but actually found very well written blog post on TechNet covering exactly that: Chad’s Quick Notes – Installing a Domain Controller with Server 2016 Core. So just sharing it here, instead of writing the same myself 🙂

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to force AD DS replication

Just a quick note on how you can force AD DS replication. You ca do this issuing the following command:

To decipher parameters: /A(ll partitions) P(ush) e(nterprise, cross sites) d(istinguished names). Essentially with this command you can do the same as Replmon used to do in Windows 2003 but in in one step. Don’t forget to replace DC_NAME with name of one of your domain controllers. Of course there are other methods to do that, including using Active Directory Sites and Services console (dssite.msc) like that (from/to selected DC):

Or like that:

When to use? When you made some changes in AD DS partitions and don’t want to wait or when you want to have a quick test of AD DS replication.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Curious case of stubborn Workgroup flag

I’ve just recently got a support case from the client where no matter that we tried MSMQ won’t work in Directory Service Integration mode, resulting in the following warning from K2 blackpearl Setup Manager:

MSMQ component is a prerequisite for K2 and it seems that all you need to do covered in K2 documentation: Installation and Configuration Guide > Prepare > Supporting Components Configuration > Message Queuing (MSMQ). As an example of functionality which will be impacted by this issue I can mention task notification emails as those have to be queued in MSMQ before being picked up by the Eventbus.

Yet information there was not sufficient, and required me to do some prolonged troubleshooting and googling. But the thing is that MSFT documentation does not highlight root cause which I found either and only some old obscure blog post lead me in the right direction.

So according to K2 documentation we have at the moment we supposed to set permissions and instead of outlining what permissions to set and where, KB strangely starts from where to set them, omitting WHAT & WHY parts 🙂 Moreover it suggests to set permissions on what its called “root object” and according to KB documentation means MSMQ container itself:

But this is not possible at all – this Root Object does not exist before you installed MSMQ Directory Integration Service, and if was installed correctly there is no need for you to go and set any permissions. But if it was installed but not in Directory Integration mode it won’t have its properties exposed in AD:

For WHAT & WHY parts K2 documentation elegantly refers you to MSFT TechNet. There you have to navigate and do a bit of careful reading to find out that setting permissions in AD DS required before installing Directory Service Integration Features of Message Queuing (why not start with this in K2 documentation?).

MSFT KB says that special permissions have to be set IF you installing Directory Service Integration feature of Message Queuing on a domain controller (we can safely ignore this as we need to install feature on K2 server and we won’t be installing K2 on domain controller, except for “all-in-one” test box scenario). Next MSFT KB says that you have to grant the Network Service account the Create MSMQ Configuration Objects permission to the computer object in AD DS before installing the Directory Services Integration feature on a computer that is a domain controller. So all in all according to MSFT KB you only need to set permissions in case you installing MSMQ Directory integration on DC. So no need for extra permissions, right? There are some required permissions but normally they are in place by default.

To save you pain making sense of all this documentation you have to mess with permissions only when installing MSMQ on domain controller, period. But recently I had a support case where this just does not want to work and after unsuccessful troubleshooting attempts with client I revert to my test lab and run into the same issue there. The fact that I run into it only on one server tells me that it only happens if your computer ACL was somehow customized or locked down either intentionally or not. If that pesky Workgroup flag keeps reverting to 1 despite you installed Directory Service Integration Feature, please make sure that SELF identity has the following rights on your K2/MSMQ server in AD:

So in short K2 KB should be structured like that:

1. K2 requires MSMQ Server and MSMQ Directory Service Integration components to be installed. DS integration improves security as it enables publishing queue properties to AD, authentication and encryption of messages using certificates registered in the directory.

2. Before installing MSMQ on domain controller machine you may need to grant additional permissions as described in Microsoft documentation: https://technet.microsoft.com/en-us/library/cc730960(v=ws.11).aspx

NOTE: this is only applicable for “all-in-one” test servers where you may want K2 to coexist on the same machine with DC. This is not applicable for production deployments where K2 runs on the dedicated AD DS member server.

3. In case you doing normal installation on AD DS member server you usually do not have to grant any special permissions except for if you security ACL for K2 server is customized. If you installed MSMQ Directory Service integration on domain member server and Workgroup flag reverting to 1 all the time then check that SELF identity has Create MSMQ Configuration objects and Delete MSMQ Configuration objects rights granted over your K2 server computer object:

NOTE: (1) My tests show me that it is sufficient to apply these permissions to “This objects only”, but you should understand that enabling disabling Directory Service Integration feature requires restart of the machine, so it seems it is all about rights at the point of installation and first reboot after it to create MSMQ objects correctly – I noticed that if I delete MSMQ objects, revoke rights and even reboot the machine Workgroup flag keep staying set to 0, but reinstalling MSMQ feature reveals this problem again. Granting rights sometimes work on the fly without reinstalling MSMQ – just restart the service.

(2) These advanced permissions may become messy way to quickly as you inherit them from your domain and then each time you click on Add button separate ACL entry is being created for your computer object, so watch out for explicit Deny settings overriding your Allow grants.

(3) As usual monitor Administrative Events view in Event Viewer and watch out for MSMQ related errors – it should be able to tell you what’s wrong.

4. When installing K2 on standalone servers (not joined to domain) or in cases when you unable to make Directory Service Integration work you can consider using workgroup mode and private queues. Though if latter is the case it is much better investigate and resolve your Directory Service Integration issue.

Apart from possibility of better structured related information in K2 documentation I’m wondering why MSFT does not have full set of permissions required for Directory Service Integration mode published somewhere? It seems this issue with lacking rights for SELF identity does not happen as by default in Windows Server 2012 R2 domain on newly provisioned server its ACL entry looks as follows:

Hope this article will save someone some time which may be wasted on troubleshooting this.

UPDATE: And as for original support case, there issue persisted even after we granted all required rights to SELF object. At this point we identified that MSMQ logs the following error with code 0xc00e050f, which indicated that MsmqServices object was missing in client’s AD DS domain Configuration partition (though to find this out it was necessary to read some forum t thread created 13 years ago and relevant for Windows Server 2003 🙂 ). Here is the steps how to verify its presence and correct this if necessary:

1.Run ADSIEdit

2. Expand the the configuration container, then expand Services. Check whether MsmqServices object and the regular MSMQ object are there

3. If the MsmqServices object is missing, right-click on Services and select “New Object”. For the object type, select mSMQEnterpriseSettings, for the object name use MsmqServices

4. Save the changes and close ADSIEdit, and force AD DS replication to avoid situation when machine where MSMQ is being installed queries some other DC than the one where you made this modification

5. Retry the MSMQ installation, it should now succeed.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to: prevent password expiration domain wide

First of all what this post suggest is absolute no go for production environments, but at the same absolute must have for some training/test environments where you want to use the same static passwords all the time without pesky password expiration/change requests. This is one of the first things I do in my test environments (OK, maybe not first, but usually I do this right after first password change prompt 🙂 ). So to get rid of this in test environment just create new GPO and link it to domain (also not the best practice for production/real world, but were fitting for test lab use case), and set settings as depicted below:

That’s it – execute gpupdate and revert to your default password which won’t expire anymore with these group policies in place.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to: enable GC on domain controller (2 ways)

There are two ways of making your DC a GC and you can read on to learn how.

But before we launch into it, just look at this “making your DC a GC” sentence for a moment. It makes me think that it is a good example of what not to do in writing for non-technical audience 🙂 I recently started to watch a very useful course on CBT Nuggets – “Essential Soft Skills for the IT Professional” by Steve Richards, and there you may learn that key things in writing tech reports to non IT audience are: avoid JATB, give MWLH and don’t SUCK 🙂

CBT Nuggets Tech Reports for Non-tech audience

Which of course means avoid Jargon, Acronyms, Techspeak, Buzzwords (JATB), give More Why Less How (MWLH) and don’t Suffer from Using Computer Knowledge (SUCK) 🙂

OK, getting back to the main topic and switching to tech writing again. First it would be nice to check which DCs are already GC-enabled, and you can do this by issuing the following PS cmdlet:

Now how to enable/disable GC:

1) PS way of enabling GC:

And you can use the same cmdlet to disable it as shown on screenshot below:

Enable or disable GC with PS

2) GUI way. Access Active Directory Sites and Services (dssite.msc), locate domain controller you need to make a GC and access General tab of its NTDS Settings Properties:

NTDS Settings - Global Catalog

By the way there is an interesting connection between GC and group scopes. You can only convert to a universal group from any other group scope on a domain controller that has the global catalog. This is somewhat obvious, as universal groups, which combine the best of two worlds (i.e. domain local and global groups) can have members from domains other than the domain where the group object is stored and can be used to provide access to resources in any domain, only a global catalog server is guaranteed to have all universal group memberships that are required for authentication.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to: Add new DC to existing domain with PowerShell

First of all you have to install AD DS role binaries on sever using either Server Manager GUI or PowerShell:

Note that if you are executing above command on real server Core installation you may get errors when using -IncludeManagementTools parameter as some of management tools can’t be installed on Core installation.

Once role is installed you may use script similar to this one to add DC to existing domain:

This script was tested with Windows Server 2012 R2.

Also script above is a great example of using tick (“`”) symbol which greatly improves readability of your PS scripts saved into a file. Just to highlight the difference most of this script supposed to be one line (starting from “Install-ADDSDomainController” cmdlet). Here is this line:

It is very easy to see how inconvenient it is to read/scroll through this line. Using tick symbol you can make your script far more readable.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Domain controller without NTDS settings in Active Directory Sites and Services

Just a quick note as I go through 70-410 training from CBT Nuggets. Normally whenever you access Active Directory Sites and Services you may see list of domain controllers within your site/sites:

NTDS Settings

And each of them has NTDS Settings. But what if you see domain controller without NTDS settings here? It means that some of your domain controllers was demoted/removed from environment, but whenever you do this you have to remove domain controller from your site manually so that it doesn’t presented to you in this location/console.

And, yes, to configure existing DC as Global Catalog you just go to NTDS Settings Properties and select respective checkbox:

NTDS Settings - Global Catalog

Facebooktwittergoogle_plusredditpinterestlinkedinmail

AD DS infrastructure failures and K2

I recently worked on a number of cases where clients complained about errors on K2 side caused by failures on AD DS side. Specifically there were some suggestions that K2 was unable to handle partial outage of AD DS, namely failure of single DC while there are other DCs available. So based on recent cases I saw I did some research and you may find the results of this research below. It is rather a long form write up which may require some updates/edits afterwards but I decided to post it to share this information with wider community as well as to keep all these notes for my own reference in case it will be necessary for me to revisit these findings.

DISCLAIMER: Some flaws in interpretation of system behavior described below are possible, those will be edited/corrected when/if necessary.

Symtoms/What you may see in your K2 environment when there are some issues with your AD DS infrastructure

Most of the applications being used in Microsoft Active Directory networks have certain degree of dependency on availability of Active Directory Directory Services (AD DS) for purposes of authentication and for obtaining required data about directory objects (users, groups etc.).

In case you have failures or other availability issues with your AD DS infrastructure you may observe symptoms/problems on K2 side similar to those described below.

Example scenario 1 (WAN link outage/no DCs are reachable to serve queries against remote domain)

You may observe growing queue of AD SmO queries on IIS side to the point at which all of the queries sent from K2 smarforms to AD DS fail/no longer returning any information and after a long delay the following error message is being thrown:

A referral was returned from the server.

This error comes from AD DS (more specifically from DC which serves K2 app/K2 server queries to specific domain) and most likely caused by the fact that there is no domain available to serve this query at all.

Example scenario 2 (single DC failed, other DCs are available)

You receiving the following error on K2 server:

System.DirectoryServices.ActiveDirectory.ActiveDirectoryServerDownException: The server is not operational. Name: “DC-XYZ.domain.com” —> System.Runtime.InteropServices.COMException: The server is not operational.

You confirmed that DC mentioned in the error message is down but there are other DCs up and running in this domain.

Example scenario 3 (it could be 2b, but you see that only K2 smartforms are affected)

You may see the same error message as in scenario 2, i.e.:

System.DirectoryServices.ActiveDirectory.ActiveDirectoryServerDownException: The server is not operational. Name: “DC-XYZ.domain.com” —> System.Runtime.InteropServices.COMException: The server is not operational.

But you also see that both K2 workspace and base OS working just fine and using alternate DC, but K2 smartforms keep throwing an error which mentions failed DC (which is indeed failed).

All described scenarios are slightly different but in all of these cases it may seem that one K2 didn’t switch to alternative available DC for specific domain. Key question/requirement here is to switch to another available DC without any downtime or with a minimum downtime (no K2 server or K2 service restart).

Research and general recommendations

First of all it is necessary to understand what kind of dependency on AD DS we have from K2 side. Most obvious things are  AD Service SmOs and User Role Manager Service (URM) – both of them dependent on AD DS availability but in a different ways. AD Services queries AD DS directly (so it is a good test to check whether AD DS queries can be served without issues) whereas URM service relies on K2 identity cache and returns you cached data from K2 database. URM service return data from multiple security providers registered in K2 and it stores cached data in Identity.Identity table in K2 database. URM service is dependent on AD DS only at the time of cached data refresh thus it will allow you not to notice AD DS failure if your AD DS cache is not expired yet.

In the beginning of this blog post we mentioned two major scenarios for AD DS failure (with third type which can be qualified as a sub-case of (2)):

1) WAN link failure when no domains are available to serve K2 request to specific domain because all DCs are behind the WAN link. This is applicable to multi domain environments with remote domains.

2) Failure of specific DC to which K2 server is connected for querying specific domain.

Given AD DS design best practices none of those scenarios should present any problems for applications dependent on AD DS:

(1) It is best practice to have extra DCs placed on remote sites so that there is no dependency on WAN link to preserve link bandwidth and safeguard against availability issues. At the very least RODC should be present locally on site for any remote domain if for some reason you can not place RWDC locally on each remote site.

NOTE: in a link failure scenario when there are no locally available DCs there is nothing that can be done from K2 side, it is a question of restoring WAN link or placing locally available RWDC/RODC to mitigate against this scenario.

(2) Golden rule and requirement for any production AD DS deployment is to have not less than two DCs per domain. So failure of one domain controller should not present any issues.

Now separately on scenario (3) when you getting the same error as in scenario (2): “System.DirectoryServices.ActiveDirectory.ActiveDirectoryServerDownException: The server is not operational. Name: “DC-XYZ.domain.com” but you clearly see that your base OS and K2 workspace using alternate available DC where as K2 smartforms keep throwing an error which mentions failed DC. With high probability you may see this error with K2 4.6.8/4.6.9.

In this specific scenario you clearly see that K2 workspace works fine at the time when you have this issue with K2 smartforms. This is because Designer, Runtime and ViewFlow web applications in K2 are using the newer WindowsSTS redirect implementation (http://k2.denalilx.com/Identity/STS/Windows) which was introduced in 4.6.8 whereas K2 Workspace still uses “Windows Authentication”.

I.e. you may see that K2 workspace uses windows authentication and in its web.config file ADConnectionString is configured as “LDAP://domain.com”, for WindowsSTS K2 label is being used, i.e. “LDAP://dc=domain,dc=com”

You may see aforementioned error occurring on the redirect to “http://k2.domain.com/Identity/STS/Windows/”

There is also a known issue with Windows STS implementation in K2 when exception on GetGroups causes user authentification to fail on Windows STS which was fixed in 4.6.10 but there is still open request to improve error handling with the aim to catch exceptions caused by temporary unavailability of DC and then have STS retry again. So that in cases where the DC is inaccessible for a short interval for unknown reasons the retry will then connect successfully.

So in scenario (3) you will likely see that DC locator is switched to alternate DC but Windows STS not performing switch/retry after temporary DC failure. It is something I need to research more, but it seems that in this case you have to restart K2 service to get back to normal operation of K2 smartforms.

Irrespective of scenario (maybe apart from scenario (3)) first point to check when you see any such issues is to work with your AD DS team to clarify which specific issue do you have on AD DS side and whether it is fixed/addressed or not. There is no use to perform any attempts of fixing things from K2 side if AD DS issue is not addressed unless this is an issue with specific DC and there are other locally available DCs. The only possible thing is to temporarily remove connection string to some extra domain if you can afford this (and if it is a less important/additional domain which has an issue).

You may get a confirmation from AD DS support team that the they have issue with one specific DC which is failed or down for maintenance (the latter should be very rare/exceptional case of planned maintenance during business hours) and there are others locally available DCs to serve requests from K2 server. If this is the case you can try to do the following things:

1) Use AD Service SmO to check that you can query affected domain – if it works you should not have any issues in K2, if not proceed with further checks.

2) Use the following command to verify which DC is currently being used by K2 server for specific domain:

If this command returns failed DC then this is an issue with your DC locator service/AD DS infrastructure, or to put it another way problem external to K2.

In general AD DS as a technology with decades of evolution and high adoption rate is very stable and there are no well known cases where DC locator fails to switch to alternative available DC. But depending on configuration and issues of specific environments as well as implementations of application of code which interacts with AD DS there can be some cases when DC locator switching does not work properly.

3) If on the 2nd step you getting failed/unavailable DC try to use the following command:

This will force DC locator cache refresh and may help you to switch to another DC. Note sometimes it is necessary to run this a few times till another DC is selected.

4) If step 3 does not help you to switch to another available DC you may try to restart the netlogon service as DC locator cache is implemented as a part of this service. Here is an example of how to do it with PowerShell:

Once this is done verify whether you are switched to available DC with use of the following command:

5) If you see that after switching of DC locator to available DC K2 AD Service SmOs are still does not work consider K2 service restart/or server reboot. This is most likely could be scenario (3) when K2 workspace/base OS works well but K2 smartforms “stuck” with server down exception.

Note the only valid test here is use of AD Service SmOs to query domain – if it works then no need to do something else from K2 side. In case you see issue in the areas depending on URM User service it may simply be the case that cached data is expired and new data is still builds up. Sometimes it may be necessary to force identity cache refresh and wait till cache builds up completely (this can take very long time in large scale production environments).

Additional details and recommendations

K2 performs bind with the DirectoryEntry class e.g:

This process relies on Domain Controller Locator which an algorithm that runs in the context of the Net Logon service. Essentially Domain Controller Locator is a sort of AD DS client part which is responsible for selecting specific DC for specific domain. Domain Controller Locator has its own cache. The Net Logon service caches the domain controller information so that it is not necessary to repeat the discovery process for subsequent requests. Caching this information encourages the consistent use of the same domain controller and, thus, a consistent view of Active Directory.

NOTE: as you may notice in explanations for scenario 3 K2 Workspace and K2 smartforms perform bind to AD differently, at least connection string they use are different.

Refer to the Microsoft documentation for details:

Domain Controller Location Process

Domain Controller Locator

Recommendations

1) Reconfigure K2 to use GC instead of LDAP.

The global catalog is a distributed data repository that contains a searchable, partial representation of every object in every domain in a multidomain Active Directory Domain Services (AD DS) forest. So essentially your GC placed in local domain can serve part of the queries which otherwise should go to DCs in another domain, potentially over WAN link.

From purely AD DS side GC has the following benefits:

– Forest-wide searches. The global catalog provides a resource for searching an AD DS forest.

– User logon. In a forest that has more than one domain GC can be used during logon for universal group membership group enumeration (Windows 2000 native DFL or higher) and for resolving UPN name when UPN is used at logon.

– Universal Group Membership Caching: In a forest that has more than one domain, in sites that have domain users but no global catalog server, Universal Group Membership Caching can be used to enable caching of logon credentials so that the global catalog does not have to be contacted for subsequent user logons. This feature eliminates the need to retrieve universal group memberships across a WAN link from a global catalog server in a different site. Essentially you may enable this feature to make use of GC even more efficient.

To reconfigure K2 to use GC you have to edit RoleInit XML field of HostServer.SecurityLabel table and replace “LDAP://” to “GC://” with subsequent restart of K2 service.

From K2 prospective it should improve responsiveness of AD SmartObjects as well as slightly decrease reliance on WAN link/number of queries to DCs outside of local domain.

2) Try to use Domain Locator cache refresh clear up for example scenario 2 (see details above, nltest /dsgetdc:DomainName /force) and verify whether it is viable workaround. Use “nltest /dsgetdc:DomainName” to confirm which specific DC is being used by K2 server and verify status and availability of this specific DC with your infrastructure team.

3) In scenario 3 try to restart K2 service but first confirm that DC locator uses working DC.

4) There is also an existing feature request to investigate possibility to built in some DC failure detection/switching capabilities into K2 code in the future versions of the product.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to control Local Administrators group membership on a DC

Reasons and security ramifications aside this is just a quick note on how to control Local Administrators group membership on a DC. So if you try to use lusrmgr.msc on a DC you it will open you Local Users and Groups console with a red cross and warning in central pane:

“The computer %DC_NAME% is a domain controller. This snap-in cannot be used on a domain controller. Domain accounts are managed with the Active Directory Users and Computers snap-in.”

Well thanks for letting me know, but I need to manage this anyway 🙂 So to command prompt.

Issuing this:

net localgroup Administrators

will list current local Administrators group membership.

And using following syntax we can add members to local Administrators group:

net localgroup Administrators /add domainuser_name

There used to be nice video on YouTube showing all this, but this one seems to be no longer available.

Facebooktwittergoogle_plusredditpinterestlinkedinmail