T-SQL Tuesday #82 – To the cloud… And beyond!!!

T-SQL Tuesday is a recurring blog party, that is started by Adam Machanic (Blog | @AdamMachanic). Each month a blog will host the party, and everyone that want’s to can write a blog about a specific subject.

This month the subject is “To the cloud… And beyond!!!”. If you want to read the opening post, please click the image below to go to the invitation.

As I mentioned in the invitation, Azure allows us to easily host and scale our platform. But it also comes with a whole new set of challenges. Some of the observations I made in the last few months:

Agent is missing in action…
One of the biggest challenges for me when I started migrating stuff to Azure was the missing SQL Server Agent. I never thought about how often I actually used it until I missed it! There’s no more “let me quickly schedule this script for tomorrow morning” anymore.

And although there are multiple solutions to this problem available in Azure (Scheduler, Automation) I picked the easy way out. Because we’re running 2 platforms in parallel (an on-premise private cloud and Azure), I decided to create a VM in the private-cloud platform to run operational tasks like scheduled maintenance for example. This is not a solution I would like to keep for the upcoming years, but it allows us to easily run scheduled tasks, without losing time on figuring out how Azure Automation or Scheduler works.

For more information on this, please read this post.

Contained datastore
A thing that can make migrations to the cloud a bit more difficult, is that Azure SQL databases are basically a contained datastore (you would call it a “contained database” when you run it on-premise). This means that you (by default) can’t connect from one database to the other. This could mean that you need to rewrite your applications or stored procedures, or maybe even redesign your entire database/application/domain model.

This also means that running a stored procedure from the Ola Hallengren’s maintenance solution can only be done on the specific database, and not from the master database like the on-premise version does. These small challenges can be overcome, but it does mean code-duplication in your databases because the maintenance procedures need to be deployed to every single database.

Running database maintenance
Just like on-premise databases, Azure SQL databases also need to be serviced every now and then. To make sure I’m not reinventing the wheel I’ve re-used a solution that has proved its worth in practice: Ola Hallengren’s “SQL Server Maintenance Solution”.

But because every Azure SQL database is a contained datastore, I’ve made some minor changes to his solution. You can read more about that here.

Performance issues & alerting
One of the biggest differences for me between an on-premise database and an Azure SQL database is how I need to determine the cause of performance problems and solving them. When you’re working on a on-premise database it’s easy to just look at the CPU and RAM used by your instance, and watch the disk IO in the performance monitor in Windows. But you don’t have those in Azure of course. There you need to work with “DTU” percentages, and “Data IO” and “Log IO” counters. But DTU (Database Transaction Unit) makes monitoring performance a bit too abstract if you ask me. Just look at the explanation Microsoft gives us regarding DTU:


The Database Transaction Unit (DTU) is the unit of measure in SQL Database that represents the relative power of databases based on a real-world measure: the database transaction. We took a set of operations that are typical for an online transaction processing (OLTP) request, and then measured how many transactions could be completed per second under fully loaded conditions.

For example, a Premium P11 database with 1750 DTUs provides 350x more DTU compute power than a Basic database with 5 DTUs.

One thing I’m really happy with is that Adam Machanic published a new version of his sp_WhoIsActive for Azure a few months ago. This gives me the opportunity to quickly look at the results to find a cause of the performance issues.

When after a while you have a gut feeling about what the performance of your databases, you run into a new challenge with monitoring: the monitoring of DTU usage has a maximum retention of just 1 hour. This gives me another challenge, because I would like to see a longer retention. I definitely don’t want to over-scale my databases, so in some cases a long running process can trigger this alert. And when this happens during the day it’s annoying, but still okay. But it also woke me up in the middle of the night during an on-call rotation. This forced me to set some of the alerts on the maximum value of 45 minutes / 1 hour, and just disable others. In my opinion, this could use some work by Microsoft.

So in general I’m REALLY satisfied with the functionality that Azure SQL database provides us with, even though there’s always something to wish for of course. The move to Azure gives me as a DBA a new set of challenges and (in some cases) demands other/new skills, but it also allows me to do more than just manage a bunch of databases. I’m now also a part-time system administrator, network admin, (data)architect, developer, etc. This is something I personally really like, because it’s exactly that that allows me to broaden my horizon, and use a set of skills I didn’t for a while when working on on-premise databases.

So even though some people were afraid the DBA role would disappear because of the cloud-uprise, I’m convinced it will be here to stay except it will be a role with more skills needed than before, and I think that’s a good thing.

Invitation: T-SQL Tuesday #82 – To the cloud… And beyond!!!

T-SQL Tuesday is a recurring blog party, that is started by Adam Machanic (Blog | @AdamMachanic). Each month a blog will host the party, and everyone that want’s to can write a blog about a specific subject.

This month I’ll be the T-SQL Tuesday host, and I’m really honored! Thanks for inviting me to be the host Adam!

The topic
When Adam asked me if I wanted to host another T-SQL Tuesday, I immediately knew a topic I wanted to talk about: The cloud, and (if you want to) specifically about Azure SQL database.

Last time we blogged about the cloud was back in december of 2013, when Jorge Segarra hosted this monthly party. Since then, “the cloud” (to use that buzz-word again) has changed a lot, and I think the possibilities are endless nowadays.

The reason I would like to see you all blog about this topic, is that I’m working with Azure SQL databases a lot now since I switched jobs in December. Currently I’m working for a small start-up that has a cloud-first focus. This means the main (if possible) host for our (data)platform is Azure. And although hosting our platform in Azure makes it easier for us to scale parts of that platform, it also gives us new challenges to overcome. And I’m probably not the only one with that experience…

So with that said, I would like to give you the opportunity to blog about the cloud, in the broadest sense that you can imagine. I’m imagining really interesting blogs about migrating to the cloud, missing features in Azure SQL database, how you’re determining the cause of and solving performance problems, etc. Please surprise us all with your view on the cloud.

The rules
– Your post must be published between 00:00:00 UTC and 23:59:59 UTC on Tuesday 13th of September
– Include the T-SQL Tuesday logo in the top of your post, and your post must link back to this one (trackback and comments are moderated, so it might take some time before they’re visible) or tweet about it using the hashtag #TSQL2sDay
– If you like this, check Steve Jones’ (Blog | @way0utwest) blog post that contains the list of topics, and contact Adam Machanic (Blog | @AdamMachanic) if you’d like to host this party yourself

Running maintenance on Azure SQL databases

This article was recently published on dev.getroadmap.com:


To keep your data healthy and topfit, we all know you need to run something called database maintenance on your databases. This prevents things like performance problems or unrecoverable data from happening, and that is sort of the core business of DBA’s. And when we look at how this can be performed on a regular basis, an on-premise instance is a bit easier to maintain than an Azure SQL database. That’s why I want to share our experience with you on that, hoping that it can save you some time when you start implementing this.

Platform setup
In order to get a better understanding of why certain choices were made, I want to share a very basic version of the setup of our environment.

Our platform consists of 2 mayor parts: a private cloud part (that we see as “on-premise”), and an Azure part. Those 2 parts combined make our platform, and this platform hosts all of our services, applications, databases, etc. The development approach we use is everything should be designed with cloud-first in mind, but only if it’s the right tool for the job, and with the exclusion of some data.

The databases we use in Azure are all what Microsoft calls “Azure SQL databases”. This means that there are no virtual machines of any kind are running on our Azure-part of the platform, also known as DBaas (Database as a Service).

When I draw this platform, and only focus on the data part, it will look something like this:

One of the advantages of this setup is that we can leverage the power of both parts of the platform. As you’ll see later on in this blog, there are some things that we (need to) run from the on-premise instances and some things fully on Azure.

Big shoutout to Ola
Before I’m going into detail, I want to give full kudos to Ola Hallengren (Website | @olahallengren). He has spend a lot of his time to build a SQL Server Maintenance Solution that is completely free for everyone to use. And he did such an excellent job a lot companies (also huge companies) use his solution to run maintenance tasks on their databases.

None of the scripts below are written by me, but only small changes are made in order to make things more clear when the solution is deployed to an environment. The original scripts can be downloaded via the download page on Ola’s website.

Backups & Integrity check
Taking backups of your database and making sure there is no corruption in the datafiles is an essential part of the maintenance solution written by Ola. But because Azure SQL databases have a build-in maintenance solution (for example backups: full backups weekly, differentials hourly, and transaction log every 5 minutes, source), we don’t need to worry about that ourselves.

Index maintenance & Update Statistics
Indexes and statistics are the core of your performance-based maintenance processes. These make sure your queries run fast, and should provide you with a stable and predictable performance. This is especially needed on an Azure database, since you can’t monitor it like you would with an on-premise database.

Because Azure SQL databases are run on shared hardware that you can’t monitor yourself, Microsoft provides us with a number of different performance counters that we can use to check the status/health of our databases. The most important counters are CPU usage, Data IO, Log IO and DTU usage (a combination of the previously mentioned counters). The DTU counter is the most abstract (to me at least), because it’s explained by Microsoft as:


The Database Transaction Unit (DTU) is the unit of measure in SQL Database that represents the relative power of databases based on a real-world measure: the database transaction. We took a set of operations that are typical for an online transaction processing (OLTP) request, and then measured how many transactions could be completed per second under fully loaded conditions.

For example, a Premium P11 database with 1750 DTUs provides 350x more DTU compute power than a Basic database with 5 DTUs.

And for me, who is was used to monitoring physical hardware, that is a bit of a different approach when digging into performance-related issues. And it’s not that index and statistics maintenance isn’t important when you work on a on-premise database, but it’s a slightly bigger challenge to monitor the direct effects of a index rebuild or statistics update.

But because every Azure SQL database is a contained database, you need to deploy the stored procedures from Ola’s solution to every single database. So to keep it clear for everyone which table and stored procedures belong to the maintenance solution, I’ve changed Ola’s scripts slightly to create all objects in a specific schema named “dba”. So first of all, let’s create the schema:


And then create the used to log all of the maintenance commands and their outcome:

CREATE TABLE dba.CommandLog
	 DatabaseName sysname NULL,
	 SchemaName sysname NULL,
	 ObjectName sysname NULL,
	 ObjectType CHAR(2) NULL,
	 IndexName sysname NULL,
	 StatisticsName sysname NULL,
	 PartitionNumber INT NULL,
	 ExtendedInfo XML NULL,
	 CommandType NVARCHAR(60) NOT NULL,
	 ErrorNumber INT NULL,
	 ErrorMessage NVARCHAR(MAX) NULL)

Now that these are created, you can create the stored procedure that execute the actual index and statistics maintenance. They are too long to post here as code snippet, but you can download the CommandExecute script here, and the IndexOptimze script here.

But because we want to schedule these procedures later on, I decided to create an additional stored procedure in every database, that is deployed alongside the maintenance objects:

CREATE PROCEDURE dba.ExecuteMaintenance

EXECUTE dba.IndexOptimize
 @Databases = '<Insert database name>',
 @FragmentationLow = 'INDEX_REORGANIZE',
 @FragmentationLevel1 = 5,
 @FragmentationLevel2 = 30,
 @UpdateStatistics = 'ALL',
 @LogToTable = 'Y',
 @Execute = 'Y'


This allows us to run maintenance with specific options on the different databases.

After this was all set up, I needed to come up with a way to run these procedures on a schedule. And as you might know, Azure SQL databases don’t have a SQL Server Agent so that’s were our on-premise platform comes in handy. Just for this I created a new virtual machine in our private cloud, and installed SQL Server on that machine to utilize the SQL Server Agent. This server (operations server) runs all of the scheduled operational jobs (including maintenance and some monitoring jobs) on our Azure environment.

But to run a proces from this operations machine on one of our Azure databases I needed to create a linked server first:

EXEC sp_addlinkedserver 
 @srvproduct=N'Azure SQL Db',

EXEC sp_addlinkedsrvlogin 
 @rmtsrvname = 'ServerName__DatabaseName', 
 @useself = 'FALSE', 
 @rmtuser = '<USERNAME>',
 @rmtpassword = '<PASSWORD>'

And all that’s left now is to create a SQL Server Agent job, that executes the “ExecuteMaintenance” stored procedure on the Azure database:

EXEC [<LinkedServerName>].<DatabaseName>.dba.ExecuteMaintenance

The reason I choose this approach is that I didn’t want to lose any time on figuring out how the Azure Scheduler or Automation works, at the moment I was implementing the maintenance solution.

But if you want to have a good resource on that, please check out Pieter Vanhove (Blog | @Pieter_Vanhove) blog, because he wrote a great blog post about that.

When I look at how this solution is set up, I’m the first one who admits that this isn’t a perfect or ideal solution. But in the end, this gives the rest of the team a clear solution when they need to start or restart the maintenance process: Just log in to the operations server, start the SQL Server Management Studio (SSMS), open the SQL Server Agent jobs, find the job associated with the database they want to run the maintenance on, and that’s it. But for future scalability and maintainability, we might need to implement another solution to do this.

T-SQL Tuesday #48 – Cloud Atlas

T-SQL Tuesday is a recurring blog party, that is started by Adam Machanic (Blog | @AdamMachanic). Each month a blog will host the party, and everyone that want’s to can write a blog about a specific subject.

This month the subject is “Cloud Atlas”. If you want to read the opening post, please click the image below to go to the party-starter: Jorge Segarra (Blog | @SQLChicken).

In the last few years, “the cloud” has become more and more important in our lives. Not only in IT, or as a database- or data-professionals, but also in our personal lives. Take a look around you. How many people do you still see, carrying around a USB drive to store their data? Now do the same, and count the people that use a cloud solution for their data, like Dropbox, SkyDrive (if we are still allowed to call it that…), or Copy.com?

So everyone is storing their data in the cloud now. From personal information like a copy of a passport, to text files with peoples password lists. So without jumping to conclusions just yet, I guess we trust the companies that hold our data right…?

But now comes the hard (and controversial) part: we trust them with our personal data, but not our corporate data. It’s okay to store your passwords and private documents in the cloud, but it’s unthinkable that you store business data in the cloud!

So where is that distrust coming from? It probably has something to do with the whole NSA-thing. There, I said it! Without completely going off-topic, I would like to explain something about this statement.

My personal opinion is that people in the Netherlands are different from the rest of the world, when it comes to their privacy. They don’t care if the ISP is monitoring web traffic. They know it’s being monitored, but they accept that as a fact. When it comes to downloading games, music or movies, they think their entitled to that. But when it comes to government agencies monitoring the corporate data they put in the cloud, they draw the line.

Are you… the one…?
In the past few years, the discussion about on premise and off premise data intensified. People try to convince each other with arguments, and think the other is completely wrong.

A while ago, I encountered my first “cloud-company”. I’ve done some consulting for them, and they’ve set themselves the goal to move to the cloud within the next few years. The biggest advantages they see are costs, scalability and administration. And I fully agree with them.

Why use a cloud solution
Choosing a WASD (Windows Azure SQL Database) solution makes it easier to maintain your environment. You don’t have to monitor the hardware, and move to another server if your hardware fails or dies. This is all being taken care of by Microsoft.

Looking at the cost of a cloud solution is easy: it saves you money. When you run on premise servers, where you need a data center, electricity, maintenance team, etc. When you use a cloud solution, you only pay for the hardware you need. And if you’re done with it, you can just shut down the machine you were working on.

The same goes for scalability. For example, if you need to run a billing process, you could “spawn” twice as many cloud machines. This makes scalability a piece of cake. And again, when your done, just get rid of the machines you don’t use anymore. This makes it easier for companies to run big processes in a smaller amount of time.

Trying it out
The only time I’ve used WASD is on the machine that Jamie Thomson (Blog | @jamiet) made available to the SQL Family (read about it here). This was later taken over by Red-Gate, but I’m not sure this is still available.

But if you want to try it out, just create your own AdventureWorks on Azure. You can download the scripts here, and start your Azure trial here.

Building an empire, but how and where?

As you might have read in my previous post, I set myself some goals this year. One of these goals is to move my blog to a more permanent location. I started out on Geekswithblogs.net, but after a few months I’ve moved to this location. But now I want to move to a more permanent location, including my own domain name(s).

All I really need at this moment is a WordPress host. No need for extra storage (I use Dropbox and SkyDrive for file-storage), or any other stuff (as far as I know). I’m only looking for a WordPress host, with 2 or 3 domain names attached. But what are the options?

One of the options is Azure, where you can rent a virtual machine, or just host your website. One of the advantages of having a virtual machine is that you can manage your own machine. You can use your own machine as file storage, and maybe even to host more services in the future.

One of the disadvantages is the cost of a whole package. If you want to rent a small virtual machine and some storage, your looking at a monthly cost of about € 25,- (around $ 33,-). That’s a lot of money. But on the other hand, they’ll move your website and storage on hardware failure. So your website will have an almost guarantied up-time.

Web hosting package
Another option is buying a web hosting pack at for example GoDaddy. There you get a WordPress host, unlimited websites, unlimited space and bandwith (a pretty good deal if you ask me), and they also add a free domain name. This looks like the deal I want, except for all the bad stories I heard about GoDaddy…

Just domain name(s)
It’s also possible to buy a domain name, and point it to your WordPress.com blog. This can be purchased via WordPress.com. Then your blog is still hosted by WordPress.com, and you still can’t use the full potential of a WordPress.org install. On the other hand, WordPress.com is a big player in the market, and how often will it be offline?

That’s where YOU come in!
To be honest, I need your help. I’m not that into hosting plans, hosting providers, etc. So if you can help me out, I would greatly appreciate it! You can leave a comment below, or you can Tweet or mail me.