Snip/Shot by Toddy Mladenov: January 2010

Windows Azure Deployment Stuck in Initializing, Busy, Stopping – Why?

Recently I see a lot of posts on Windows Azure Forum related to deployments cycling between Initializing, Busy and Stopping states. In this post I would like to summarize the most common reasons why is this happening. Here they are:

Missing runtime dependencies (DLLs)
If you use libraries that are not includes in the standard .Net installation, you will need to include those in the package and deploy them together with your roles. In order to do that you need to set the Copy Local property of the reference to True.
Here is how you do this in Visual Studio 2010 (see screenshot below):
- Go to Solution Explorer and expand the Role tree
- Select the reference you want to add to your package
- In the Properties window for this reference change the value of Copy Local to True
Using incorrect platform version of a DLL
The OS running on Windows Azure VMs is 64 bit, and although you can start process that uses 32 bit assemblies you should NOT reference 32 bit assemblies from a Web or Worker role.
Your code or assembly you use requires admin access (elevated privileges)
Although your application may be running under Full Trust some calls or assemblies may require elevated privileges. Admin access is currently not supported in Windows Azure, and such calls will fail.
Misconfigured DiagnosticsConnectionString in Service Configuration file
There are a couple of issues you may hit here:
- By default the string points to Development Storage. When deploying to the cloud you need to make sure the string points to a storage account in the cloud.
- Starting diagnostics without specifying DiagnosticsConnectionString. The reason this will not work is obvious.
- Using brackets when specifying protocol. Apparently some blog post states that the protocol should be within square brackets, which is not true. The right format for specifying the string is the following:
  
  <Setting name="DiagnosticsConnectionString" value=”DefaultEndpointsProtocol=https; AccountName=myaccount; AccountKey=mykey" />
- Using HTTPS as endpoint protocol. You should use HTTPS as an endpoint protocol for Diagnostics.
Misconfigured DataConnectionString in Service Configuration file
You should check for the same things as described above for the DiagnosticsConnectionString.
web.config or app.config issues
Those can be:
- Invalid settings in web.config/app.config files
  One good example for such is using wrong providers.
- Malformed or invalid XML
Wrong CSDEF and/or CSCFG schemas
Although very unlikely your Service Definition or Service Configuration files may be invalid. Normally Visual Studio will show error if one of those files doesn’t comply with the schema, but if you use other tools or create the package manually those may not get validated.
Read from queue or table that doesn’t exist during initialization
Make sure all storage tables and queues are created prior starting the application, and any configuration data is populated in those.
Certificate without exportable private key
In order for your certificate to work in the cloud it needs to have exportable private key. Certificates without exportable private key are not supported. If you use Windows Certificates Manager you need to make sure to select “Yes, export the private key” option while exporting the certificate.
Returning from Run() method in a Worker Role
Windows Azure expects that you never return from Run() method. If you do so it considers it as an error condition and restarts the role.
Uncaught exception thrown during initialization
There are endless possibilities here, therefore make sure you catch every exception during initialization and log it properly.

Suggestions

Here are few suggestions that may help you if you hit this problem:

Make sure your code runs in Development Fabric and Development Storage
This is the first step you need to try but be aware that this test will not catch things like admin access (normally you are admin on your development machine) or wrong connection string.
Use the diagnostics feature to trace your application
If you don’t modify the generated code about diagnostics, errors will be traced automatically. Trace information is stored to the cloud storage on a regular basis. If your DiagnosticsConnectionString is configured properly to use cloud storage you should be able to find the traces in the WADInfrastructureLogsTable table.
If feasible delete and redeploy you application in the cloud
Sometimes your deployment may get stuck for no obvious reason. One way to solve it is to redeploy the service. If this is not possible you need to contact Windows Azure support team for assistance.

References

Here are few other blog posts that give more details about some of the issues described above:

Windows Azure role stuck in Initializing/Busy/Stopping [Anton Staykov’s Blog]

Another common reason for Windows Azure role stuck in Initializing/Busy/Stopping [Anton Staykov’s Blog]
Windows Azure: why is my service not starting? [Guy Shahine’s Blog]

Hopefully this information will make it easier for you to track down issues with your deployments.

Posted at 01:32 AM in Windows Azure | Permalink | Comments (7) | TrackBack (0)

Subscription and Service Administration in Windows Azure

On January 4th 2010 Windows Azure announced that you can start upgrading your CTP accounts to paid commercial subscription (see Sign up for a Windows Azure platform offer today and get visibility into your usage on Windows Azure Blog). In this post I would like to give you short introduction of the different roles available, as well as to point out certain limitations, and what to look for when migrating your CTP account.

Those of you who already migrated their CTP subscriptions know that the subscription administration is done in Microsoft Online Commerce Platform (MOCP), while service administration is done in the corresponding developer portal (in my case this will be Windows Azure Developer Portal). The steps you need to follow in order to upgrade your CTP subscription to paid commercial one are the following:

Go to Microsoft Online Commerce Platform (MOCP)
Sign-in with your Windows Live ID
Select an offer and provide your credit card details
Designate Service Administrator account

When you complete the process you can login with the Service Administrator Live ID to Windows Azure Developer Portal (if you have selected Windows Azure subscription of course) and you will see the new subscription.

Account Administrator and Service Administrator

First of all I would like to clarify two terms: Account Administrator and Service Administrator. What those two mean?

Account Administrator (AA) (also known as Account Owner) is the person who manages the subscription (or subscriptions) in MOCP. In general this is the person who is responsible for paying the bill, and normally has financial responsibilities in your company.
Service Administrator (SA) (or also Service Owner) is the person who manages the service that runs in the cloud (or Windows Azure, Microsoft SQL Azure or Windows Azure platform AppFabric). You can think of him or her as the IT person or developer who is responsible for the service operation.

With the introduction of those two roles you are able to separate the financial responsibilities from the operational responsibilities for your cloud account.

Limitations

There are certain limitations that some of you may find restrictive. In general they fall into two categories:

Granularity of access – Service Administrators are not able to delegate certain tasks in the developer portal to another person (i.e. LiveID), which makes the Service Administrator the only person who can manage the service.
By the way the same is true for the Account Administrator – there is no possibility to delegate the administration of particular subscription to another person.
Multiple access – for both the subscription administration and service administration you are not able to designate more than one person (i.e. Live ID).

CTP account migration

As part of the sign-up process in MOCP your CTP account may be migrated. What that means is, that your hosted services and storage accounts associated with your CTP account will be assigned to the newly created paid subscription. However, in order this to happen you need to do the following:

You need to use the same LiveID for Account Administrator as the one you used to sign up for the CTP account. I.e. when you sign up to MOCP use the same LiveID you used to sign up for CTP acocunt on the corresponding developer portal.
Once you sign-up for the paid commercial subscription, and verify in the developer portal that your account is migrated, you can return back to MOCP and designate another LiveID for Service Administrator.

Additional to that if you have more than one CTP subscription only the first one you created will be upgraded.

I hope this information will help you migrate your accounts smoothly.

Posted at 01:29 AM in Windows Azure | Permalink | Comments (0) | TrackBack (0)

Windows Azure Role Instance Limits Explained

In preparation for the Windows Azure Commercial Launch new limits for role instances were introduced. Those differ from the currently available Customer Technology Preview (CTP) limits, and may impact the way you deploy and configure your hosted services or storage accounts in Windows Azure.

This article explains what the relationship between the number of hosted services, number of roles, number of instances, VM (Virtual Machine) size and deployments is (for more details on the VM sizes visit Configuring Virtual Machine Size on MSDN). Additionally you will get an overview of what the limits for storage accounts are.

If you have already encountered issues with your deployments you should follow the guidelines below to calculate your needs and call Windows Azure Support team for assistance.

Here is a brief overview of the current and future limits imposed on your services in Windows Azure.

	Effective before Dec. 10th 2009	Effective after Dec. 10th 2009	Effective after Jan. 4th 2010
	Token (CTP)	Token (CTP)	Token (non-billing country)	Paying subscription
Deployment Slots	2	2	2	2
Hosted Services	1	1	20	20
Roles per deployment	5	5	5	5
Instances per Role	2	2	no limit	no limit
VM CPU Cores	no limit	8	8	20
Storage Accounts	2	2	5	5

CTP Program Limits

CTP program was free, but in order to participate you were required to request a token. One token entitled you to the following deployments:

2 deployment slots (1 production and 1 staging)
1 hosted service
5 roles per deployment
2 instances per role

In total this allowed you to have up to 20 instances deployed for free. Additional to that you were entitled to have up to 2 storage accounts, 50GB each.

If you were lucky to get additional CTP tokens you could “stack” those and easily double (or triple, quadruple etc.) the limits above.

The size of the VM was not considered into account when calculating the CTP limits. What this means is that you can configure your roles to be deployed on extra large VMs (8 CPU cores, 15 GB RAM, and 2TB local storage), and still deploy 20 instances for free, making it 160 CPU cores in total.

In summary the CTP program was a good way to consume large amount of resources for free.

Commercial Launch Limits

However, getting those free resource is not possible anymore. The new limits are already fact, and starting January 4th those will be different for customers who use tokens and customers who convert their subscription to billing. Those will be calculated using weighted formula that takes into account the size of the VM too. The weighted formula counts the number of CPU cores you use for your hosted service and limits your usage based on this number.

If you happen to be in a country that is not provisioned for billing for Windows Azure you will still be able to use tokens and deploy your application on Windows Azure. Of course those tokens will be controlled much more strictly than until now, and the corresponding limits will be consistently enforced.

What the numbers above mean? Don’t be mislead by the number 20 for Hosted Services. Theoretically you will be able to create so many, however you will hit the limit of VM CPU cores much earlier. Here is how this works.

Imagine you create Hosted Service with two roles in it, and each role has one instance with Small VM Size (1 CPU core). The math is as follows:

1 hosted service x 2 roles x 1 instance x 1 CPU core = 2 CPU cores

You have already used 2 out of your 8 allowed CPU cores with just one instance. If you decide to replace the VM size with Large one (4 CPU cores) you will reach the limit:

1 hosted service x 2 roles x 1 instance x 4 CPU core = 8 CPU cores

Thus you can easily reach or exceed the limit with only one Hosted Service.

The other number you should be careful with is the number of roles per deployment. You are entitled with up to 5 of those independent of the type (Web or Worker). Imagine you design your service to have 5 roles with 1 instance on a small VM each (not very redundant but this is for the purpose of illustration only). Imagine also that you want to use the staging environment for your pre-production tests, and you want to mimic exactly your production environment. If you have already your previous version deployed in your production slot, you will exceed the limit of VM CPU Cores when you try to deploy your new version on the staging slot (assuming you have limit of 8 CPU cores):

Prod Deployment = 5 roles x 1 instance x 1 CPU core = 5 CPU cores

Stage Deployment = 5 roles x 1 instance x 1 CPU core = 5 CPU cores

Total = 10 CPU cores

One thing to remember is that stacking tokens will not increase your limit as it will just increase the number of allowed Hosted Services but not the number of CPU cores you are entitled to. However stacking tokens will still give you more storage accounts. Of course this may change in the future.

Because the goal is to convert CTP users to paying users, billing subscription will have higher limits for compute, which is set to 20 by default. Also billing users will be able to call Windows Azure Support team, and request increase of their limit as long as they provide enough business justification and money guarantee.

Errors Thrown When Exceeding the Limit

If you happen to exceed the limit of VM CPU cores you will be presented with the following error in Windows Azure Portal:

The subscription policy count for resource was exceeded. Please visit the Accounts tab to learn how to increase your resource quota Dr. Watson Diagnostic ID: 89ac8ae2-c5a8-48ae-a7d6-751db6bdf22a

Similar error will be returned from the Service Management API.

Keep in mind that you will receive the above error in the following cases:

When you try to re-deploy your service (change the number of roles or the size of the VMs in csdef, or if your current usage is higher than the limit)
When you try to increase the number of instances for your service (in cscfg), which can also be part of the upgrade

If you happen to exceed the limit of roles per deployment you will receive the following error:

Unsupported number of roles Dr. Watson Diagnostic ID: 5cbd9c72-3f27-4f31-a54f-bfb100b92516

This error you will only receive if you try to re-deploy your hosted service.

Your service will not not impacted if you try to run or suspend it. If you already have running service that exceeds the limits above it will not be impacted either, but you need to make sure that you request increase of your limits before your next deployment.

One more thing is that stopped services count towards the limits. Stopping your application on staging will not help you if you want to increase the count on your production service; you will need to completely delete your staging deployment in order to free up some slots.

What is next?

All customers who reside in the billing countries will be required to migrate their CTP subscription to a billing one in January 2010. One thing you need to be aware of though is that during the migration process the limits will be reset to the above mentioned ones.

The best approach you can take is to wait until January and migrate your CTP subscription to a billing one first, and then request increase of your limit if you need to. Thus your service will continue to run uninterrupted. Of course you can do this only of you don’t plan any deployments that exceed the limits before January.

At some point of time after February 1st 2010 all CTP tokens in the billing countries will be evicted. Even when evicted you may be able to continue to run your service, but you will be prevented to do any deployments and upgrades.