K2 MSMQ thread & MSMQ abort exception

There is one interesting fix included into K2 4.7 May CU which, as far as I know, is not mentioned in related KBs for some reason. I just wanted to share some information about it as it may be useful for those who run older versions of K2 preceding to 4.7 May CU and additionally mention how to take K2 process memory dump conveniently.
First things first – the issue and related background information. Within K2 process there is a single thread which checks message queue, it is single thread and it process messages one by one (this is by design, as doing in a multi thread fashion is not necessarily best idea). Exception may occur in the process of message bus message processing and this dedicated K2 thread tries to abort this transaction to perform a retry which normally works just fine, but in rare scenarios MSMQ can throw another exception which causes K2 MSMQ processing thread to stop completely.
On the surface symptoms of this issue is that suddenly your K2 users no longer receive any K2 task notifications at all, despite relevant settings has not been changed and correct. Brand new test process confirms that task notification does not work indeed. Quick fix/corrective action: restart K2 service – and all delayed task notifications should be gradually dispatched (depending on how long time ago MSMQ thread stopped delay for processing the piled queue may be quite big). Service restart resolves this because K2 starts MSMQ thread on service start up.
Good news is that 4.7 May CU has fix for this built in and when transaction abort is necessary MSMQ thread first checks if transaction in question is not already rollback/committed or completed status and only if this is not the case attempts to abort it. If even then K2 receives an exception then message moved into error queue.
I can imagine that when you run K2 version without this fix in production you may be reluctant to restart K2 service not being 99% sure that you deal with this specific issue, and there is a way to do that. To verify that your MSMQ thread within K2 process is still running you may follow these steps:
1) Take full memory dump of K2 process – for quick check on K2 threads either dump taken from Task Manager or procdump.exe will suffice.

In task manager you just have to locate K2HostSerever.exe from the list of processes, right click on it and select “Create dump file”, like that:

It will show you pop up indicated where dump file has been created once done, something like that:

Things gets more tricky with procdump.exe as you now need to obtain process PID, but it is way more configurable and allows you do more things when it comes to taking process memory dumps. As I work in support I really don’t like to repeat explanations on how to use it and where do I find PID and why PID is not displayed in my Task Manager and so on. So I created this tiny script for that (get it on GitHub):

2) Once dump is taken open it using DebugDiag and search for MSMQ, if MSMQ thread is running you should be able to find something like this:

If you unable to find this in memory dump of your running K2 process it most likely means that MSMQ thread had been stopped due to exception, and if you look under the Previous .NET Exceptions Report (Exceptions in all .NET Heaps) section, you most likely may see the aborted exception with MSMQ stack there.

I hope you may find this information useful and interesting for one reason or another 🙂 Stay tuned for new posts.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

SharePoint 2010: Unable to edit user properties

The other day I received a support case where customer complained that they “suddenly” lost ability to edit user properties in SharePoint 2010. As usual picture worth thousand works so the problem was that if you navigate to Site Actions > Site Permissions, locate some user and click on your user name you will be presented with user information page and if you click edit you will see this:

Sort of interesting Edit Personal Settings dialog where it is not possible to edit anything…

But it used to look like that:

You see more properties listed and they are editable. Now having that description you have to be really skillful with art of googling to get to the bottom of this in the form of some blog post similar to this one. Because to do a proper Google search (one which yields a solution) you have to employ the right terms/know the nature of this issue. But if you noticed word suddenly in quotes in the very beginning I had a background information on preceding changes which caused this “sudden” issue hence was able to fix it. I will go into detailed explanation below for the sake of knowledge sharing.

First of all 1st screenshot is normal for environments where UPS is up and running – as soon as you configure it, SharePoint assumes that all property modifications are being performed on AD DS side and synced from there on a regular basis by UPS. Once UPS is provisioned SP 2010 hides/modifies default forms (layouts/userdisp.aspx – the one where you may click Edit Item to change your properties on the _layouts/useredit.aspx form) and instead of them for users whose profiles are synced you supposed to see specific user profile page instead, which will look approximately like this:

If it does not shown and you see the same uneditable edit form as on the first screenshot, then it either means that UPA is not configured completely or user profile is not synced yet (are you sure that user in question is in right OU?)

So essentially first screenshot/problem shows us that UPS was partially configured/profile(s) not synced (as we still not getting user profile page) but default forms already modified in the process of UPS installation, because when you provision a UPA it will set the fields in hidden site user info list to read only and hidden. With this knowledge about the nature of the problem you may google for the right scripts/information.

In my case we worked on environment where UPS was failing to start/work properly (one of these cases where you need delve deeper into configuration of UPS and peruse something like this blog post) hence it was just mandatory to restore ability to edit user properties. And for that you just need to use the script below against your SP web app (grab this script on GitHub).

Once this script completes you will get your editable user properties back. Two potential problems you may have with this:
1) If Get-SPWeb part of the script complains about incompatibility saying something like “Microsoft SharePoint is not supported with version 4.0.30319.34014” just run new PS instance using -version 2.0 switch;
2) If script completes successfully but you getting user profile page instead of your old beloved form – delete user profile and try again (in case you are not keen or fully removing UPA).

And one last note with regards how we run into this in my specific support case. Client was using K2 and they lived quite happily without UPS provisioned in their SharePoint environment which is a bit strange as a UPS is a requirement for K2. But after upgrading to 4.6.11 the very strange issue crop up which had a bit obscure symptoms on the surface, but in the end was isolated to the fact that each time GetUserGroups URM call was performed to the SharePoint provider no SharePoint groups of which this user is direct member were returned. On the surface it looked like random losing of user’s group membership information and randomly failed K2 tasks which were assigned to SharePoint groups. Randomness stemmed from the fact that GetGroupUsers URM call returned all users for the same group just fine.

And knowing that sometimes it is difficult to find where it was told that XYZ is requirement for K2, I’ll clarify this for UPS specifically: you can find it in K2 blackpearl Installation and Configuration Guide > Prerequisites > Environment Configuration > SharePoint Server 2010 User Profile Service set up

“The SharePoint User Profile Service on any non SharePoint Foundation version must be set up correctly for the Identity Services Group Providers to function correctly with regards to User, Group and Membership resolution. It must be correctly populated with the user’s information and the service must be started.”

Posting rather a lot about SharePoint 2010 at the time of 2016 and cloud stuff I rather feel like author of The Old New Thing blog who at some point decided to blog about old stuff as there is less competition there – everybody busy blogging about new and shiny things (though I should admit I don’t go into really low level details as that blog does) 🙂 But I hope these posts may still be useful for someone.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to: configure K2 NLB port rules with PowerShell

Long time ago I did a video and blog post on configuring Windows NLB K2 cluster. I know that those materials are not perfect, but thanks to blogging you not only can be subject of mockeries for mistakes and naivety in your old posts, but also you can review and improve on them over time 🙂

Anyhow my old blog post on creating K2 NLB cluster contained this neat picture of required port rules:

As I tread my test K2 environments as “wipe and load”-ready and subject them to all sort of experiments leading to wipe and load and rebuilds I grow tired of creating this rules via GUI. Thanks to PowerShell and Microsoft Community it is not a problem to find a sample script to create Windows NLB cluster. I actually wanted to rewrite it with minor improvements and K2 specifics to spin off K2 NLB cluster quicker but due to endless lack of time this idea moved on the back-burner. What I did instead though is prepared PS script to create port rules:

That’s help a bit when I rebuilding my test environment. You can grab this script from GitHub too.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to: Install SharePoint 2010 August 2015 CU

On a day to day basis I keep repeating people to always check on with K2 compatibility matrix before installing or upgrading their K2 environments. Very frequently people try to mix K2 with too new Microsoft components which weren’t tested with their version of K2. But there is an opposite issue when Microsoft infrastructure components lag behind in terms of patches/versions fully supported by their release of K2. I know quite a few people still using SharePoint 2010 with K2 4.6.11. So with SharePoint 2010 being an old thing in itself people often skip CUs for this product for some reason which is unknown to me. In this article I want to discuss what is the latest CU for SP 2010 supported by K2 4.6.11 (note that with 4.7 K2 dropped support for SP 2010) and how to install it.

What is the latest CU for SP 2010 supported by K2 4.6.11? As usual you have to check compatibility matrix, but you have to find old one, which will show you this:

And this:

Does it mean that newer CUs will break something/won’t work with K2? Not necessarily, it only means that it won’t be supported because it has not been tested. At the time of release of 4.6.11 latest SP 2010 CU available was August 2015 CU and hence all testing and QA was performed against this CU – K2 cannot guarantee that all will work with newer CUs.

With that knowledge if you are still on SP 2010 it makes sense to make sure you running “newest” supported CU for it. Easiest way to do that is fire off SharePoint Management Shell on your SharePoint server and execute the following command:

This will give you your current build:

Having this information look up in SP builds list @ Todd Klindt’s SharePoint Admin Blog to translate this into CU and SP levels, for example 15.0.7106.5000 translates into August 2013 CU:

Note that there may be minor last digit discrepancies depending on how you look up for build number. So now I know that my SP 2010 is August 2013 CU and for K2 4.6.11 I can go up to August 2015 CU (download link) from that – let’s try to do it.

First things should go first – backup your SharePoint environment. Navigate to Central Administration > Backup and Restore and click Perform a backup – just go through the wizard and create full farm backup. It can be good idea to test that your backup can be restored.

Once backup is done an CU file is downloaded launch it, accept license terms and hit “Continue”:

CU installer will check for installed updates and proceed with extracting files and installation of update after that:

Once done it will ask for reboot:

Most frequent mistake in all this process it assuming that after reboot of your SharePoint server you will be running updated SharePoint version. Quick check with (Get-SPFarm).BuilVersion will show you the same build as it was before you started CU installation process. So update is not finished just yet and to complete it you have to locate SharePoint 2010 Products Configuration Wizard in Start Menu:

and run it. Next you just go through the wizard’s steps to complete upgrade:

Warning you got in the very beginning should be taken seriously if you do this on a server used by other people where IIS reset may have undesired impact:

Once all configuration tasks completed you should get confirmation of successful configuration and click Finish button:

After clicking on Finish it will take you to CA main page automatically. In CA you can navigate to Upgrade and Migration > Upgrade Status where you can see confirmation of successful upgrade:

And now it is time to issue (Get-SPFarm).BuildVersion command once again:

First execution of the command on the screenshot was made before running of SharePoint 2010 Products Configuration Wizard and the second after wizard completion – now we run build 14.0.7155.5000 which is the latest SharePoint 2010 build officially supported by K2 4.6.11.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Freshly installed K2 – unable to access K2 workspace with HTTP Error 401.2

I’ve recently did some quick and dirty installation of K2 4.6.11 on Server 2008 (non R2) hence all you have there is PoSh 2.0 it is not possible to use amazing K2Field.PreReq script to take care about all the prerequisites (it will work for server 2012-2016). So I just went ahead and tried to satisfy complaints from K2 blakcpearl setup manager as I go adding IIS along with KB980368 as indicated by Setup Manager. Alas after installing I was not able to access K2 Workspace with the following error:

“You are not authorized to view this page due to invalid authentication headers.” Why? Quick check of K2 Workspace site authentication showed that Windows Authentication is missing, while according to K2 documentation (source) K2 Workspace does not function if IIS is not configured correctly. Configure the IIS application pool Managed Pipeline mode setting to Classic and ensure that:  

  • Windows Authentication is enabled
  • Anonymous authentication is disabled

So I just went ahead and added Windows Authentication role service and all started work correctly after this. It tells us that: A) K2 setup manager created site config file correctly specifying required authentication method, it just was not installed/available B) Not sure why K2 Setup Manager does not have built-in check to flag this at the installation stage. C) Read up documentation carefully, in this case this page.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to: enable Document Set in Document Library

I just recently bumped in into this thing – you have to enable Document Sets on a library level before you can actually use it. This little point baffled me a bit when I tried to use Create Document Set SmartWizard in K2 Studio:

It took me a while to switch over too my SharePoint document library to realize that I just can’t create document set there:

Normally the same New Document menu contains an option to create Document Set. Well now it’s clear that “Required field is empty” error is just a clumsy way employed by K2 Studio to tell you that Document Sets disabled for your library.

To enable them you just go to Library Settings > Advanced Settings > Allow management of content types as on screenshots below.

Click on Library Settings button on ribbon:

Click on Advanced Settings:

In advanced settings select Yes for Allow management of content types and hit OK:

Now you also need add Document Set content type using Add from existing site content types link in library settings:

At this point you should be able create document sets in SharePoint just fine:

Yet you still going to have problem with K2 SmarWizard I mentioned initially. Why? Just go to K2 App settings for your library to see the answer:

Just click on Regenerate SmartObjects, go back to K2 Studio and Create Document Set SmartWizard to see things working there:

Voila, we can now see our Document Set enabled library and use SmartWizard to create document sets in it.

Just to recap: To be able to create document sets within SharePoint libraries you have to Allow management of content types in Advanced Settings of this library and add Document Site content type using “Add from existing site content types” link in library settings.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to: control Fast Startup setting via registry

If you navigate to Control Panel > All Control Panel Items > Power Options > System Settings you can find GUI option to disable/enable so called Fast Startup option (formerly known as Fast Boot):

This is sort of light-weight/hybrid hibernation mode – with this option enabled on shut down system log offs all users and closes programs yet leaves Windows kernel loaded and system session running, next it notifies drivers (those which have support for this) to prepare for hibernation and saves this system state into hibernation file. This allows for faster boot by means of loading this state into RAM on startup (you can read up more on this here).

Anyhow I just wanted to take a note of corresponding registry key which enables/disables this setting – this is registry key named HiberbootEnabled which can be found under HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Power.

This is especially important as there is no GPO to control this feature, so you can use some scripting to control/verify state of this setting via registry key.

And in case you need to check for supported sleep states on your system (including Fast Startup) you can leverage  powercfg /a command.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

CBT Nuggets Microsoft Windows Server 2012 70-410 with R2 Updates

I’m currently doing a bit of revision of 70-410 content going through “Microsoft Windows Server 2012 70-410 with R2 Updates” training by Garth Schulte. First of all I already passed 70-410 exam and did 70-410 course by James Conrad, but just to take a break before 70-411 I decided to review 70-410 content + go through 70-698 CBT Nuggets course and take an exams on Windows 10 (yes it counts as a pouse before 70-411).

Few words about updated 70-410 training by Garth. First of all it is fully designed with R2 in mind (James Conrad’s course was pre-R2 + some R2 modules added later) so you can’t find there gotchas and detours related with hiccups related with recent release of a product, instead as it covers stable and current release you will find there well structured up-to-date content and as one expect from Garth well covered PowerShell side 🙂 I also like very good slides summarizing key facts you have to memorize before exam – they provide you with compressed knowledge (I guess I stole getAbsract slogan here 🙂 ) you need before taking your exam. Some examples:

Those slides are just great to review before exam (so it could be a good idea to save some screenshots as you go through the course).

Good job Gath 🙂 Once I done with this training and my Win 10 exam I will be focusing on 70-411. And I’m just wondering how do I inject TCF exam and preparation for it in my schedule…

Facebooktwittergoogle_plusredditpinterestlinkedinmail

PS script to get SQL version from BAK file

Quite unpleasant thing about MS SQL Server database backup files it that you can’t restore them on an older version of SQL Server (it seems that even if you really want to create such BAK file it is not possible), moreover this is valid not only for major versions but also for things like R2, meaning you can’t restore BAK file created in SQL Server 2008 R2 on SQL Server 2008 Server (non-R2).

Just to save my time trying to restore BAK files against wrong versions of SQL Server I created this script which allows you to retrieve SQL version from BAK file headers and compare it with your server SQL version. Specify path to your BAK file and check the output – if BAK file SQL version is newer that your server version then BAK file can’t be restored on this server. Sample script output can be found below:

It tells you that backup was taken on SQL Server 2012 SP3, while you run SQL Server 2012 SP2. Once you install SP3 for SQL Server script output will change to this:

Once two numbers match you are ready for backup restore 🙂

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Unable to start SSRS sevice after upgrading SQL Server instance

I recently spent quite a few time trying to figure out why my SSRS service fails to start after performing upgrade from SQL Server 2012 to SQL Server 2016. Service was failing to start with quite generic error “System error 5 has occurred. Access is denied” which is a bit to broad, but after a while I looked into instance bin directory which contains log file where I seen some error indicating that configuration file is garbled somehow. At this point I revert to reinstall once again solution, and this time I noticed warning which I probably missed during initial instance upgrade attempt:

TITLE: Microsoft SQL Server 2016 Setup

——————————

The following error has occurred:

Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index

For help, click: http://go.microsoft.com/fwlink?LinkID=20476&ProdName=Microsoft%20SQL%20Server&EvtSrc=setup.rll&EvtID=50000&ProdVer=13.0.1711.0&EvtType=0x35C10BED%25400xF538E98B

I proceed with upgrade I seen exactly the same problem – I can’t start SSRS service after upgrade completion. This warning essentially lead me to the thread on TechNet containing right solution. It seems that upgrade process somehow can’t handle SSL bindings for SSRS service or handles them differently (produces malformed XML configuration for them?) and what you need to do is remove SSL bindings for SSRS in Reporting Services Configuration Manager (see screenshot below).

Just to save some time which may be wasted on troubleshooting just remove your SSL bindings (I did it for Web Service URL & Web Portal URL) before starting upgade of your instance. You may add these bindings back after upgrade.

Facebooktwittergoogle_plusredditpinterestlinkedinmail