Tag Archives: MSMQ

K2 MSMQ thread & MSMQ abort exception

There is one interesting fix included into K2 4.7 May CU which, as far as I know, is not mentioned in related KBs for some reason (KB001888 and  FP KBs). I just wanted to share some information about it as it may be useful for those who run older versions of K2 preceding to 4.7 May CU and additionally mention how to take K2 process memory dump conveniently.
First things first – the issue and related background information. Within K2 process there is a single thread which checks message queue, it is single thread and it process messages one by one (this is by design, as doing in a multi thread fashion is not necessarily best idea). Exception may occur in the process of message bus message processing and this dedicated K2 thread tries to abort this transaction to perform a retry which normally works just fine, but in rare scenarios MSMQ can throw another exception which causes K2 MSMQ processing thread to stop completely.
On the surface symptoms of this issue: suddenly your K2 users no longer receive any K2 task notifications at all, despite the fact that all relevant settings had not been changed and absolutely correct. Brand new test process confirms that task notifications do not work indeed for all usres while Email Events keep working fine. Quick fix/corrective action: restart K2 service – and all delayed task notifications should be gradually dispatched (depending on how long time ago MSMQ thread stopped, your delay for processing the piled queue may be quite big). Service restart resolves this because K2 starts MSMQ thread on service start up.
Good news is that 4.7 May CU has fix for this built in and when transaction abort is necessary MSMQ thread first checks if transaction in question is not already rollback/committed or completed status and only if this is not the case attempts to abort it. If even then K2 receives an exception then message moved into error queue.
I can imagine that when you run K2 version without this fix in production you may be reluctant to restart K2 service not being 99% sure that you deal with this specific issue, and there is a way to do that. To verify that your MSMQ thread within K2 process is still running you may follow these steps:
1) Take full memory dump of K2 process – for quick check on K2 threads either dump taken from Task Manager or procdump.exe will suffice.

In task manager you just have to locate K2HostSerever.exe from the list of processes, right click on it and select “Create dump file”, like that:

It will show you pop up indicated where dump file has been created once done, something like that:

Things gets more tricky with procdump.exe as you now need to obtain process PID, but it is way more configurable and allows you do more things when it comes to taking process memory dumps. As I work in support I really don’t like to repeat explanations on how to use it and where do I find PID and why PID is not displayed in my Task Manager and so on. So I created this tiny script for that (get it on GitHub):

2) Once dump is taken open it using DebugDiag and search for MSMQ, if MSMQ thread is running you should be able to find something like this:

If you unable to find this in memory dump of your running K2 process it most likely means that MSMQ thread had been stopped due to exception, and if you look under the Previous .NET Exceptions Report (Exceptions in all .NET Heaps) section, you most likely may see the aborted exception with MSMQ stack there.

On a side note DebugDiag also allows you to see if call which failed was made from K2 WorfkflowServer namespace or from something custom/external – that is also very useful to check in the very beginning of troubleshooting process.

I hope you may find this information useful and interesting for one reason or another 🙂 Stay tuned for new posts.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Curious case of stubborn Workgroup flag

I’ve just recently got a support case from the client where no matter that we tried MSMQ won’t work in Directory Service Integration mode, resulting in the following warning from K2 blackpearl Setup Manager:

MSMQ component is a prerequisite for K2 and it seems that all you need to do covered in K2 documentation: Installation and Configuration Guide > Prepare > Supporting Components Configuration > Message Queuing (MSMQ)As an example of functionality which will be impacted by this issue I can mention task notification emails as those have to be queued in MSMQ before being picked up by the Eventbus.

Yet information there was not sufficient, and required me to do some prolonged troubleshooting and googling. But the thing is that MSFT documentation does not highlight root cause which I found either and only some old obscure blog post lead me in the right direction.

So according to K2 documentation we have at the moment we supposed to set permissions and instead of outlining what permissions to set and where, KB strangely starts from where to set them, omitting WHAT & WHY parts 🙂 Moreover it suggests to set permissions on what its called “root object” and according to KB documentation means MSMQ container itself:

But this is not possible at all – this Root Object does not exist before you installed MSMQ Directory Integration Service, and if was installed correctly there is no need for you to go and set any permissions. But if it was installed but not in Directory Integration mode it won’t have its properties exposed in AD:

For WHAT & WHY parts K2 documentation elegantly refers you to MSFT TechNet. There you have to navigate and do a bit of careful reading to find out that setting permissions in AD DS required before installing Directory Service Integration Features of Message Queuing (why not start with this in K2 documentation?).

MSFT KB says that special permissions have to be set IF you installing Directory Service Integration feature of Message Queuing on a domain controller (we can safely ignore this as we need to install feature on K2 server and we won’t be installing K2 on domain controller, except for “all-in-one” test box scenario). Next MSFT KB says that you have to grant the Network Service account the Create MSMQ Configuration Objects permission to the computer object in AD DS before installing the Directory Services Integration feature on a computer that is a domain controller. So all in all according to MSFT KB you only need to set permissions in case you installing MSMQ Directory integration on DC. So no need for extra permissions, right? There are some required permissions but normally they are in place by default.

To save you pain making sense of all this documentation you have to mess with permissions only when installing MSMQ on domain controller, period. But recently I had a support case where this just does not want to work and after unsuccessful troubleshooting attempts with client I revert to my test lab and run into the same issue there. The fact that I run into it only on one server tells me that it only happens if your computer ACL was somehow customized or locked down either intentionally or not. If that pesky Workgroup flag keeps reverting to 1 despite you installed Directory Service Integration Feature, please make sure that SELF identity has the following rights on your K2/MSMQ server in AD:

So in short K2 KB should be structured like that:

1. K2 requires MSMQ Server and MSMQ Directory Service Integration components to be installed. DS integration improves security as it enables publishing queue properties to AD, authentication and encryption of messages using certificates registered in the directory.

2. Before installing MSMQ on domain controller machine you may need to grant additional permissions as described in Microsoft documentation: https://technet.microsoft.com/en-us/library/cc730960(v=ws.11).aspx

NOTE: this is only applicable for “all-in-one” test servers where you may want K2 to coexist on the same machine with DC. This is not applicable for production deployments where K2 runs on the dedicated AD DS member server.

3. In case you doing normal installation on AD DS member server you usually do not have to grant any special permissions except for if you security ACL for K2 server is customized. If you installed MSMQ Directory Service integration on domain member server and Workgroup flag reverting to 1 all the time then check that SELF identity has Create MSMQ Configuration objects and Delete MSMQ Configuration objects rights granted over your K2 server computer object:

NOTE: (1) My tests show me that it is sufficient to apply these permissions to “This objects only”, but you should understand that enabling disabling Directory Service Integration feature requires restart of the machine, so it seems it is all about rights at the point of installation and first reboot after it to create MSMQ objects correctly – I noticed that if I delete MSMQ objects, revoke rights and even reboot the machine Workgroup flag keep staying set to 0, but reinstalling MSMQ feature reveals this problem again. Granting rights sometimes work on the fly without reinstalling MSMQ – just restart the service.

(2) These advanced permissions may become messy way to quickly as you inherit them from your domain and then each time you click on Add button separate ACL entry is being created for your computer object, so watch out for explicit Deny settings overriding your Allow grants.

(3) As usual monitor Administrative Events view in Event Viewer and watch out for MSMQ related errors – it should be able to tell you what’s wrong.

4. When installing K2 on standalone servers (not joined to domain) or in cases when you unable to make Directory Service Integration work you can consider using workgroup mode and private queues. Though if latter is the case it is much better investigate and resolve your Directory Service Integration issue.

Apart from possibility of better structured related information in K2 documentation I’m wondering why MSFT does not have full set of permissions required for Directory Service Integration mode published somewhere? It seems this issue with lacking rights for SELF identity does not happen as by default in Windows Server 2012 R2 domain on newly provisioned server its ACL entry looks as follows:

Hope this article will save someone some time which may be wasted on troubleshooting this.

UPDATE: And as for original support case, there issue persisted even after we granted all required rights to SELF object. At this point we identified that MSMQ logs the following error with code 0xc00e050f, which indicated that MsmqServices object was missing in client’s AD DS domain Configuration partition (though to find this out it was necessary to read some forum t thread created 13 years ago and relevant for Windows Server 2003 🙂 ). Here is the steps how to verify its presence and correct this if necessary:

1.Run ADSIEdit

2. Expand the the configuration container, then expand Services. Check whether MsmqServices object and the regular MSMQ object are there

3. If the MsmqServices object is missing, right-click on Services and select “New Object”. For the object type, select mSMQEnterpriseSettings, for the object name use MsmqServices

4. Save the changes and close ADSIEdit, and force AD DS replication to avoid situation when machine where MSMQ is being installed queries some other DC than the one where you made this modification

5. Retry the MSMQ installation, it should now succeed.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Unable to start Message Queuing service: “The Message Queuing service terminated with service-specific error %%-1072823311”

As you may know MSMQ (see WikipediA or MSDN for details) is one of prerequisites for K2 Server installation, to be more specific it is necessary for you to have the following components installed:

Microsoft Message Queuing (MSMQ) Services

  • Message Queuing Server
  • Directory Service Integration

Microsoft Message Queuing or MSMQ is a message queue implementation developed by Microsoft and deployed in its Windows Server operating systems since Windows NT 4 and Windows 95. MSMQ is essentially a messaging protocol that allows applications running on separate servers/processes to communicate in a failsafe manner. A queue is a temporary storage location from which messages can be sent and received reliably, as and when conditions permit. This enables communication across networks and between computers, running Windows, which may not always be connected. By contrast, sockets and other network protocols assume that direct connections always exist.

Surprisingly enough you not only have to have required MSMQ components installed but also you have to have them in a working state. 🙂 For example MSMQ is a requirement for working Notification Events which provide the functionality to notify via e-mail when specific events are executed on servers, implementing a custom event record. The queuing of events is processed using MSMQ. Transactional queue’s  are received from the client recorder to be persisted to the Event database.  The Event database receives events from the Queuing System (MSMQ) and saves event mappings to the database for processing. (see details here). Both Notification Events and E-mail Events in K2 are depending on MSMQ.

The other day I had a case when K2 server was restored from backup and it was noticed that notification events does not work, K2 Setup Manager was run to double check email related settings and it raised a warning about MSMQ. First of all I confirmed that MSMQ components are installed and they were in place, so I attempted to start MSMQ service but it wailed to start with the following error message:

The Message Queuing service terminated with service-specific error %%-1072823311

As a message text suggests it is worth checking application specific logs, which in MSMQ case can be found in Windows Event viewer. In this specific case I was able to see the following event logged upon each attempt to start MSMQ service:

Event ID 2078 — Message Queuing Logging and Checkpoint Events

The Message Queuing service cannot start. The checkpoint files cannot be recovered. Error %1: %2

The rest was easy as MSFT provides guidance/details on this, see Message Queuing Logging and Checkpoint Events.  Event ID 2078 normally occurs when there is a failure between the time that the checkpoint file was created and when the QMLog was updated with the new version, the QMLog file refers to the earliest checkpoint file version and recovery fails. So this is something you may run into after system restore. Resolution to this is the following:

1) Delete all the checkpoint files, as well as the QMLog file in the Message Queuing storage directory (%windir%\system32\msmq\storage, see Message Queuing Message and Data Files for details). This can result in some messages being duplicated. However, this resolution will get the service running as soon as possible and usually without data loss.

2) Open registry editor and navigate to:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSMQ\Parameters

registry hive and there locate LogDataCreated parameter and ensure that its value is set to 0.

3) Try to start Message Queuing service – it should work now.

Link to related MSFT documentation with more detailed steps:

Event ID 2078 — Message Queuing Logging and Checkpoint Events

While working on this I also noticed that strange/confusing issue when MSMQ is installed on Windows Server 2008 R2 box but no relevant management node/snap-in was available in Server Manager but this is a separate issue worth separate investigation (some relevant reading is here and here), but probably just re-registering the MQSNAP.DLL can fix this.

Facebooktwittergoogle_plusredditpinterestlinkedinmail