Tag Archives: SQL

Microsoft Azure 70-473 Design and Implement Cloud Data Platform Solutions – Design and Implement Security

Earlier I tried to compile list of Microsoft documentation and other resources relevant for 70-473 exam preparation but I quickly realized that exam scope makes this list too huge and unwieldy. I now decided that I would rather split this into smaller resources lists following specific exam sections. Below you can see the list of reading resources relevant for Design and Implement Security section of 70-473 exam. You will mainly find links to Microsoft documentation in this list.

Here you can see Design and Implement Security exam section topics as described on official exam page:

  • Design and implement SQL Server Database security
    • Configure firewalls; manage logins, users, and roles; assign permissions; configure auditing; configure Transparent Database Encryption (TDE); configure row-level security; configure data encryption; configure data masking; configure Always Encrypted
  • Design and implement Azure SQL Database security
    • Configure firewalls; manage logins, users, and roles; assign permissions; configure auditing; configure row-level security; configure data encryption; configure data masking; configure Always Encrypted, configure Automatic Threat Detection

I tried to structure list of links below based on sub-objectives.

Configure Firewalls

Azure SQL Database and SQL Data Warehouse firewall rules

sp_set_database_firewall_rule (Azure SQL Database)

Azure SQL Database: Firewall security

Configure a Windows Firewall for Database Engine Access

Configure a Server to Listen on a Specific TCP Port
Configure the Windows Firewall to Allow SQL Server Access

TCP/IP Properties (IP Addresses Tab)

SQL Server: Frequently Used Ports

Security Considerations for SQL Server in Azure Virtual Machines

Manage logins, users and roles

Server and Database Roles in SQL Server

Managing Users, Roles, and Logins

Getting Started with Database Engine

Database-Level Roles

Server-Level Roles

CREATE CREDENTIAL (Transact-SQL)

SQL Server Separation of Duties (Word document download)

Assign Permissions

Getting Started with Database Engine Permissions

New Permissions in SQL Server 2014: IMPERSONATE ANY LOGIN, SELECT ALL USER SECURABLES, CONNECT ANY DATABASE and the old CONTROL SERVER

GRANT Server Permissions (Transact-SQL)

SQL Server Best Practices – Implementation of Database Object Schemas
+ see also: What are some best practices for using schemas in SQL Server?

Azure SQL Database and SQL Data Warehouse access control

Principals (Database Engine)

Configure Auditing

Get started with SQL database auditing

Set-AzureRmSqlServerAuditingPolicy

Use-AzureRmSqlServerAuditingPolicy

Configure Transparent Database Encryption (TDE)

Transparent Data Encryption (TDE)

Transparent data encryption for SQL Database and Data Warehouse 

Enable TDE on SQL Server Using EKM

ALTER DATABASE (Azure SQL Database)

Configure Row-Level Security (RLS)

SQL Server 2016 : Implement Row Level Security using Predicate Function and Security Policy

Row-Level Security

SQL Server Security Blog – Row-Level Security block predicates are generally available on Azure SQL DatabaseCREATE SECURITY POLICY (Transact-SQL)

Configure Data Encryption

SQL Server 2016 New Features: Security and Encryption

Encrypt a Column of Data (column/cell level encryption)

Extensible Key Management Using Azure Key Vault (SQL Server)

CREATE ASYMMETRIC KEY (Transact-SQL)

SQL Server Certificates and Asymmetric Keys
OPEN SYMMETRIC KEY (Transact-SQL)

Get started with Azure Key Vault
About keys, secrets, and certificates

Configure Data Masking

SQL Database dynamic data masking

ALTER TABLE (Transact-SQL)

Use Dynamic Data Masking to obfuscate your sensitive data

Configure Always Encrypted

Always Encrypted (Database Engine)

Always Encrypted (client development)

Develop using Always Encrypted with .NET Framework Data Provider

Always Encrypted: Protect sensitive data in SQL Database and store your encryption keys in Azure Key Vault

Microsoft Azure SQL Database provides unparalleled data security in the cloud with Always Encrypted

Configure Always Encrypted using SQL Server Management Studio

SqlConnection.ConnectionString Property
Use .NET (C#) with Visual Studio to connect and query an Azure SQL database

Configure Automatic Threat Detection

Use PowerShell to configure SQL Database auditing and threat detection

Azure SQL Database Threat Detection

Azure Security Center Documentation

Other/General

Controlling and granting database access to SQL Database and SQL Data Warehouse – Secure your Azure SQL Database

Azure Cosmos DB: SQL API getting started tutorial

Get started with Azure Table storage and the Azure Cosmos DB Table API using .NET

ADO.NET Overview

Securing your SQL Database

Azure Storage replication

Collation and Unicode Support

AzureRM.Sql

Management Data Warehouse

SQL Server Profiler

Monitoring SQL Server Performance

Monitor Resource Usage (System Monitor)

Next time I will try to compile similar list for Design and implement high availability, disaster recovery, and scalability section of the exam.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to enable TDE (SQL 2017/Azure SQL Database)

TDE is a SQL Server feature which encrypts your data at rest, i.e. your database files. When TDE is enabled encryption of the database file is performed at the page level. The pages in an encrypted database are encrypted before they are written to disk and decrypted when read into memory. TDE does not increase the size of the encrypted database. Here is TDE architecture schema from MSFT documentation:

Transparent Data Encryption Architecture

Transparent Data Encryption Architecture

This blog post explains how to enable Transparent Data Encryption (TDE) for SQL Database (on-premise/Azure).

Scenario 1. On-premise SQL Server 2017 (this will also work for SQL Server in a Azure VM). You can use the following SQL script to enable TDE:

Be sure to replace ‘K2’ with your target database name and adjust password value. Script uses IF clauses to avid creating things which already exist (which are missing in the sample script you can find in MSFT documentation). Once TDE is enabled you can confirm this in the database properties using SSMS GUI:

Scenario 2. Azure SQL Database. Script mentioned above won’t work here. Easiest/default approach to enable TDE for Azure SQL Database is to do so from Azure Portal:

This approach called service-managed transparent data encryption and by default database encryption key is protected by a built-in server certificate. All newly created SQL databases are encrypted by default by using service-managed transparent data encryption.

Other approach called Bring Your Own Key and requires use of Azure Key Vault.

TDE can also be managed with PowerShell, Transact-SQL and REST API. PowerShell contains number of cmdlets for that:

 

And using T-SQL you can use ALTER DATABASE (Azure SQL Database) SET ENCRYPTION ON/OFF command (encrypts or decrypts a database) and two dynamic management views:

  • databasesys.dm_database_encryption_keys which returns information about the encryption state of a database and its associated database encryption keys
  • sys.dm_pdw_nodes_database_encryption_keys which returns information about the encryption state of each data warehouse node and its associated database encryption keys

Once TDE has been enabled there is also options to check whether it is enabled or not using T-SQL:

For further information refer to official MSFT documentation:

Transparent Data Encryption (TDE)

The SQL Server Security Blog on TDE with FAQ

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Exam Prep Resources for Microsoft Azure 70-473 Design and Implement Cloud Data Platform Solutions

I’m currently preparing for 70-473  Design and Implement Cloud Data Platform Solutions exam, so I’ve decided to compile a list of resources which may be useful to prepare for this exam. I’m going to append it with additional materials as I keep working on my preparation and I hope it may be useful to other test takers.

As with any MSFT exam your starting point has to be MSFT exam description page which contains run down of all exam topics as well as links to additional resources, so here it is – Exam 70-473 Designing and Implementing Cloud Data Platform Solutions. You should keep in mind that though this exam has been released in December 2015, it is being updated quarterly, so once in a while you need to check exam page to see if any new topics were added there. At the moment last update to this exam was made in June 2017 and changes are explained in exam 70-473 change document.

Paid resources:

70-473 Cloud Data Platform Solutions course by SoftwareArchitect.ca – this is an affordable (25$) online course which I bought and used during my preparation – good overview of all concepts at a fair price, and when I searched it was only 70-473 specific course from online training vendors which I was able to find. Author goes through all the “skills measured” topics as they stated in exam description. What I dislike about this course is amount of typos and some little issues like mismatch between numbering and naming  of videos in course navigation pane and inside of the videos themselves. One exactly the same video even inserted/listed twice there. So I would describe it as lack of QA/editing problem. My other complain would be lack of hands-on demos, there are some of them in the course but I wanted more. 🙂 Only after completion of the course I found that it is also available on Udemy and there it was priced 9,99$ with discount when I checked – so check both locations and compare prices if you want to try it.

Free resources and video recordings:

Certification Exam Overview: 70-473: Designing and Implementing Cloud Data Platform Solutions MVA course

Cert Exam Prep: Exam 70-473: Cloud Data Platform Solutions – exam overview video by MCT James Herring

Second link is YouTube video, looks like both of these links cover more or less the same material and delivered by the same person, yet YouTube session has newer slides, it seems, and they are not absolutely identical – so watch both of them.

Channel 9 – Keeping Sensitive Data Secure with Always Encrypted

YouTube – Secure your data in Azure SQL Database and SQL Data Warehouse

MSFT documentation:

Resolving Transact-SQL differences during migration to SQL Database

This article covers things which will work in SQL queries run on on-prem SQL Server while won’t work while run against Azure SQL DB. For example things you probably discovery very quickly is that USE statement is not supported.

Azure SQL Database – Controlling and granting database access

Article explains unrestricted administrative accounts, server-level administrative roles and non-administrator users + “access paths”.

Sizes for Windows virtual machines in Azure

General purpose virtual machine sizes

High performance compute VM sizes

You may expect questions around VM sizing based on given requirements so need to remember which series has premium storage and which not along with some other things which you can learn from the articles above.

Securing your SQL Database

Always Encrypted (Database Engine)

Always Encrypted Wizard

This article explains 2 very important things you should be aware of: key storage options and Always Encrypted Terms.

SQL Database dynamic data masking

Azure Blog – Microsoft Azure SQL Database provides unparalleled data security in the cloud with Always Encrypted

Azure SQL has loads of security features and you supposed to know them all 🙂 At least when to use, along with requirements and limitations.

Azure Cosmos DB: SQL API getting started tutorial

Get started with Azure Table storage and the Azure Cosmos DB Table API using .NET

ADO.NET Overview

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Simple walkthrough: Using K2 Database Consolidation Tool

Purpose of this blog post is to outline K2 databases consolidation process using K2 Database Consolidation Tool.

When you may need it? For older K2 deployments when initial installer used to create 14 separate databases instead of one “K2” database we expect to see with current K2 versions. Such environments even after upgrades to newer versions carry on to have these 14 databases and only starting from K2 4.7 databases consolidation is enforced and you cannot upgrade till you consolidate your databases into one. So you can still see non-consolidated K2 database en environments which run any version of K2 up to 4.6.11 including.

To perform consolidation of these 14 K2 databases into one you need to obtain appropriate version of K2 Database Consolidation Tool from K2 support. Below you can see basic steps you need to perform while performing K2 databases consolidation using this tool.

1) First we need to check collation of your existing K2 databases, this is necessary because consolidation tool won’t handle conversions from one locale to another and consolidation will fail. You can run this script to see collation of your non-consolidated K2 DBs:

As you can see on the screenshot below output of this script shows that my non-consolidated databases have Ukrainian_100_CI_AS collation:

2) Make sure that your target SQL Server service instance has the same collation as your databases either via GUI:

or script:

and copy your non-consolidated databases to the target server which will be hosting consolidated database (unless it is not the same server which was hosting them initially).

2) Obtain K2 Database Consolidation Tool from K2 support, extract it on your SQL server which hosts your K2 databases and launch SourceCode.Database.Consolidator.exe, once you start it you will be presented with the following UI:

3) It will detect your non-consolidated K2 DBs (<No Instance> in the Instance drop down means that you are connecting to default, not named SQL Server instance) and here you need to select your target DB – just select <New Database>, specify “Create Database Name” (I’m using default name used by K2 installer which is K2) and click Create:

4) Once you click Create, database K2 will be created in the same collation as your SQL Server instance (your target DB will contain all the required tables and structure but no data) and Start button become available to you so that you can start consolidation process:

5) Before clicking on Start make sure  your K2 service is stopped. Despite we just created our target “K2” database we still getting warning that all data in target DB will be truncated and we have to click Yes to start consolidation process:

Once you clicked on next you will have to wait for a while till consolidation completes (in the bottom of the tool window in its “status line” you will see current operations which are being performed during databases consolidation process. Time which is necessary to complete this process is heavily depends on your server performance and volume of data in your source databases.

In some scenarios (e.g. source and destination collations have different locale IDs or you moved source databases to another SQL server without re-creating their master key) consolidation process may fail leaving your non-consolidated databases databases in read-only state:

In such scenario you need to review consolidation log to identify and address errors and once done. Switch your source databases back to RW mode (as explained here), delete your target database and start again from step (2). When consolidation completes successfully source non-consolidated databases also stay in read-only mode.

If consolidation completes without errors you will get a message confirming this and also informing you that ReconfigureServer.ps1 script has been created:

You can also click on Log Directory link which will open consolidation log file location – as usual you can open it and make sure than neither ‘Logged Warning’ or ‘Logged Error’ can be found anywhere in this log beyond Legend section in the beginning.

6) In the directory which contains K2 Database Consolidation Tool you will need to take ReconfigureServer.ps1 script and copy it over to your K2 server. This script fires off K2 blackpearl Setup Manager while instructing it to connect to your new consolidated DB:

Here is this script code which you can copy/paste:

Once you run this script on K2 server it will start K2 Setup Manager where you need to go through all pages of “Configure K2 blackpearl” process:

You will see on the database configuration step of the wizard that thanks to PS script we already targeting our new consolidated DB:

Once reconfiguration process is completes (without errors and warnings) you can start testing how your K2 environments behaves after K2 consolidation process.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

SQL script to attach detached non-consolidated K2 DBs

I keep playing with SQL and non-consolidated K2 DBs and in previous post I covered bringing “these 14” back online, now I realized that other case where SSMS requires way too many click is attaching “these 14” back (let’s say after you rebuild your SQL instance system DBs to change instance collation).

Quick google allowed me to find relevant question on dba.stackexchange.com where I took script which generates CREATE DATABASE FOR ATTACH for all existing user databases. Next having my 14 non consolidated K2 DBs I generated the following script to attach them back in bulk:

You can either use this CREATE DATABASE FOR ATTACH for all existing user databases script while your K2 databases are still attached, of it they are not just replace paths in script listed above and execute modified script to attach them quickly.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

K2 Five collation requirement changes

In the past I’ve wrote some blog post promoting documented K2 collation requirement. With release K2 Five this requirement changed and I guess I have to mention it on my blog as essentially it makes my old blog posts about required K2 database collation incorrect.

So recently (with release of K2 Five) all K2 documentation was updated and states that our requited collation now is “SQL_Latin1_General_CP1_CI_AS“. Where to find this information?

In K2 Installation and Configuration Guide you can find “SQL and Reporting Services Operational Requirements” section which says that:

– Case-sensitive databases are NOT supported.

– The following collation setting is required for the K2 database: SQL_Latin1_General_CP1_CI_AS

And as usually you can find the same information in K2 Product Compatibility, Integration and Support matrix:

What is good about this change is that SQL_Latin1_General_CP1_CI_AS will be default collation if you installing SQL Server on top of Windows Server which has been installed with US location/language settings – so at least some people will meet this requirement by accident.

What is bad is that collation requirement was just silently changed in documentation with release of K2 Five without any explanations. According to my current knowledge collation which was mentioned in documentation before was a requirement only for pre-4.6.11 versions of K2. So in case you are doing new installation of K2 4.6.11 or newer make sure that your SQL Server instance provisioned with SQL_Latin1_General_CP1_CI_AS collation.

Somewhat mixed blessing is change which was made to K2 Five installer to enforce this collation: what it does at the moment is just enforces this collation on K2 DB level while ignoring SQL Server instance level collation. So in case you not provisioned SQL Server instance with the right collation you will get an errors post installation and will be forced to change SQL Server instance level collation to fix this. That’s something that I hope will be corrected in K2 installer in the future so that it warns you about wrong instance level collation issues before you start your installation.

Conclusion: read vendor documentation carefully before doing your installation even if you did it many times before 🙂

Update (Feb 2018): Current guidelines are:

– For upgrade installations be sure to make SQL server instance and K2 DB collation are the same (in case you moving K2 DB onto new server)

– For clean installs I would suggest stick to Latin1_General_CP1_CI_AS (especially if you do 4.7 install). I will probably write more detailed post to explain this position later

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Configuring SQL instance firewall rules via PowerShell

Doing distributed environments setups rather frequently these days I realized that I really don’t want wasting my time setting up SQL Server firewall rules via GUI (I described the process here) and luckily enough Ryan Mangan already created such script. All I had to do is try it (confirm that it works), save it on GitHub and share on my blog for the benefit of wider community:

Original blog post by Ryan/source of this script: PowerShell Script for SQL Firewall rules

Facebooktwittergoogle_plusredditpinterestlinkedinmail

How to: Drop multiple databases via SQL Script (no worries backup/restore is covered too :) )

Recently I did rather a lot of test requiring me to work with non-consolidated K2 DBs. Test included multiple DB restore/delete operations and I realized that I need some script to quickly drop all my K2 DBs and start from scratch. Here is this script:

Script selects every database prefixes with “K2” and you just need to copy its output into new query window and execute.

And in case you tend to backup things before you delete them, similar script for backup:

And for restore you can use the script below. Unfortunately it uses hard coded file paths but assuming your back up files have default DB names (and for example were created by the script above) you can get away with minimum find and replace adjustments (path to backup files and your SQL instance data directories may need to be adjusted). Here is the script for restore:

Facebooktwittergoogle_plusredditpinterestlinkedinmail

SQL Server: How to attach all databases from specific folder

I recently had to change SQL Server instance collation using the steps I described in my earlier post. After this operation dozen of databases which were hosted on that instance become detached and I was able to see only set of instance’s system DBs. So I had to look for an option to attach all DB from folder via script as doing this manually for 20 or so databases a bit too much of an effort to me 🙂

There is “Attach all the databases in one folder” script available in TechNet Script Center which was able to solve this problem for me. All you need it pass DatabaseDir parameter to the function specifying folder with your DBs:

Just in case I provide PS code of this function below in case you don’t want to grab the same from TechNet Script Center. All credit goes to Ivan Josipovic.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Changing SQL collation for deployed instance without reinstall (almost)

IMPORTANT UPDATE: If you use K2 4.6.11 or newer you have to have SQL_Latin1_General_CP1_CI_AS collation on a SQL Server instance which hosts K2 DB (more details)

Collation matters. Period. If you not sure just wait till you get an incident or issue which will teach you that it is. Some apps require you to have specific collation for their back end DB. For example for K2 you need to have  Latin1_General_CI_AS collation (see my earlier post on that _ ).

Nonetheless it can often be the case that your RTFM/Prerequisites_Check moment (I wonder if I can copyright/trademark this term? I heard that Michael Buffer trademarked his “let’s get ready to rumble” phrase _ ) happens after SQL server instance has been already provisioned. For newly created (clean install) K2 database you just need to delete database which was created on instance with wrong collation and allow K2 Setup Manager to create it again after you deal with your instance collation. There is an MSDN article detailing that _ but I will outline the procedure below.

Aforementioned MSDN article says that you have to have installation media to perform this operation, but in fact you can get away with Setup Bootstrap folder which should present on your server in location similar to one below (may vary from version to version, see details _ )

C:\Program Files\Microsoft SQL Server\120\Setup Bootstrap\SQLServer2014

So steps are the following:

0. In case you have any non system databases attached to this instance take a note of database(s) file location, frequently it is just default SQL instance data folder common to all databases. Save this path or paths to notepad. 

1. Locate setup bootstrap folder for your SQL server installation, open CMD or PS window (don’t forget “dot net sourcing” thing for the latter):

2. Execute the following command:

Once this operation completes  master, model, msdb, and tempdb system databases are rebuilt (dropped and re-created) with collation you specified in the command. Obviously all your modifications to these databases are lost (if you made any) and essentially it is sort of reinstall/repair type of operation just much quicker and without need to wade through SQL Setup GUI.

3. In case there were non system databases attached to this SQL instance (see step 0) they all are detached now. See my other blog post explaining how to attach all the databases from specific folder

Facebooktwittergoogle_plusredditpinterestlinkedinmail