Windows Azure Deployment Stuck in Initializing, Busy, Stopping – Why?

Tweet

Recently I see a lot of posts on Windows Azure Forum related to deployments cycling between Initializing, Busy and Stopping states. In this post I would like to summarize the most common reasons why is this happening. Here they are:

• Missing runtime dependencies (DLLs)
  If you use libraries that are not includes in the standard .Net installation, you will need to include those in the package and deploy them together with your roles. In order to do that you need to set the Copy Local property of the reference to True.
  Here is how you do this in Visual Studio 2010 (see screenshot below):
  • Go to Solution Explorer and expand the Role tree
  • Select the reference you want to add to your package
  • In the Properties window for this reference change the value of Copy Local to True
     
• Using incorrect platform version of a DLL
  The OS running on Windows Azure VMs is 64 bit, and although you can start process that uses 32 bit assemblies you should NOT reference 32 bit assemblies from a Web or Worker role.
• Your code or assembly you use requires admin access (elevated privileges)
  Although your application may be running under Full Trust some calls or assemblies may require elevated privileges. Admin access is currently not supported in Windows Azure, and such calls will fail.
• Misconfigured DiagnosticsConnectionString in Service Configuration file
  There are a couple of issues you may hit here:
  • By default the string points to Development Storage. When deploying to the cloud you need to make sure the string points to a storage account in the cloud.
  • Starting diagnostics without specifying DiagnosticsConnectionString. The reason this will not work is obvious.
  • Using brackets when specifying protocol. Apparently some blog post states that the protocol should be within square brackets, which is not true. The right format for specifying the string is the following:
    
    <Setting name="DiagnosticsConnectionString" value=”DefaultEndpointsProtocol=https; AccountName=myaccount; AccountKey=mykey" />
    
    
  • Using HTTPS as endpoint protocol. You should use HTTPS as an endpoint protocol for Diagnostics.
• Misconfigured DataConnectionString in Service Configuration file
  You should check for the same things as described above for the DiagnosticsConnectionString.
• web.config or app.config issues
  Those can be:
  • Invalid settings in web.config/app.config files
    One good example for such is using wrong providers.
  • Malformed or invalid XML
• Wrong CSDEF and/or CSCFG schemas
  Although very unlikely your Service Definition or Service Configuration files may be invalid. Normally Visual Studio will show error if one of those files doesn’t comply with the schema, but if you use other tools or create the package manually those may not get validated.
• Read from queue or table that doesn’t exist during initialization
  Make sure all storage tables and queues are created prior starting the application, and any configuration data is populated in those.
• Certificate without exportable private key 
  In order for your certificate to work in the cloud it needs to have exportable private key. Certificates without exportable private key are not supported. If you use Windows Certificates Manager you need to make sure to select “Yes, export the private key” option while exporting the certificate.
• Returning from Run() method in a Worker Role
  Windows Azure expects that you never return from Run() method. If you do so it considers it as an error condition and restarts the role.
• Uncaught exception thrown during initialization
  There are endless possibilities here, therefore make sure you catch every exception during initialization and log it properly.

 

Suggestions

Here are few suggestions that may help you if you hit this problem:

• Make sure your code runs in Development Fabric and Development Storage
  This is the first step you need to try but be aware that this test will not catch things like admin access (normally you are admin on your development machine) or wrong connection string.
• Use the diagnostics feature to trace your application
  If you don’t modify the generated code about diagnostics, errors will be traced automatically. Trace information is stored to the cloud storage on a regular basis. If your DiagnosticsConnectionString is configured properly to use cloud storage you should be able to find the traces in the WADInfrastructureLogsTable table.
• If feasible delete and redeploy you application in the cloud
  Sometimes your deployment may get stuck for no obvious reason. One way to solve it is to redeploy the service. If this is not possible you need to contact Windows Azure support team for assistance.

 

References

Here are few other blog posts that give more details about some of the issues described above:

Windows Azure role stuck in Initializing/Busy/Stopping [Anton Staykov’s Blog]

Another common reason for Windows Azure role stuck in Initializing/Busy/Stopping [Anton Staykov’s Blog]
Windows Azure: why is my service not starting? [Guy Shahine’s Blog] 

 

Hopefully this information will make it easier for you to track down issues with your deployments.

Advertisement

   

Comments

[Benjamin Guinebertière]

Having real time traces may also help
http://blogs.msdn.com/benjguin/archive/2010/02/22/traces-en-temps-r-el-sur-azure-2-2-real-time-traces-on-azure-2-2.aspx
For instance, I could detect things related DiagnosticsMonitor not working because the real time traces rely on a trace listener and Azure throws such traces.

[ToddySM]

Thank you Benjamin for your advice! For sure this will help people troubleshoot their issues.

[Greg]

The storage account name is case sensitive. If you have any caps at all in your config file, it won't work.

[satorg]

One more point: a deployment sticks if it has too long endpoint names within its csdef file. Details are here:
http://gdwin.tumblr.com/post/538256575/internal-endpoint-name-limitation

[niraja]

am using window 7 32 bit version , while am deploying print hellowword.cspkg on windows azure, deployement got stuck in busy and aborted status.

[ToddySM]

Hi Niraja,

As mentioned in the post above there may be many reasons for that. You will need to give more details in order to be able to help you.
What modifications have you done to the helloworld app if any? Can you point me to the package that you are trying to deploy?

[Hasitha Amaraseana]

Thanks for the post and its really helped me to figureout the problem I was having. Apparently, it was a Misconfigured DiagnosticsConnectionString in Service Configuration file. I recreated the file and was able to deploy the package without any issues.

Also, to share my experience, I mistakenly changed the http endpoint port to a different one from the default (80). I was able to deploy but the website and then giving me the page not found error. So, I recommend you to look twice on the endpoints before you move on with the deployement.

Thanks again for the post.