« August 2012 | Main | November 2012 »

September 2012

September 16, 2012

What is the Difference Between Apprenda and Windows Azure?

Since I started at Apprenda one of the most common questions I hear is: "What is the difference between Apprenda and Windows Azure?". Let me take a quick stab of what both platforms offer and how you can use their features in a complementary way.

First let's look from a platform-as-a-service (or PaaS) definition point of view. As you may already know both Apprenda and Windows Azure offer PaaS funtionality but because PaaS is a really broad term and is used widely in the industry, we need to make sure we use the same criteria when we compare two offerings. Hence we try to stick to Gartner's Reference Model for PaaS that allows us to make apples-to-apples comparison between the services. If you look at the definition, PaaS is a "category of cloud services that deliver functionality provided by platform, communicaiton and integration middleware". Gartner also lists typical services offered by the PaaS so let's see how Apprenda and Windows Azure compare at those:

  • Application Servers
    Both Apprenda and Windows Azure leverage the functionality of Microsoft's .NET Framework and IIS server.
    In the case of Apprenda IIS server is used to host the front-end tier of your applications while Apprenda's prorietary WCF container is used to host any services.
    In comparison when you develop applications for Windows Azure you use the Web Role to host your application's front-end and a Worker Role to host your services. If you use Windows Azure web sites then all your front-end and business logic is hosted in IIS in a multi-tenant manner.
  • Integration Middleware
    While Windows Azure offers Service Bus in the cloud at this point of time Apprenda does not have its own ServiceBus implementation. However Apprenda applications can easily integrate with any existing Service Bus implementation on premise or in the cloud.
  • Portals and other user experience enablers
    Both Apprenda and Windows Azure offer rich user experience.
    Apprenda has the System Operations Center portal that is targeted to the platform owners, the Developer Portal that is the main application management tool for development teams, and the User Portal where end-users (or tenants) can subscribe to applications provided by development teams. Apprenda also have rich documentation available at http://docs.apprenda.com/ as well as active support community at  http://support.apprenda.com. In addition when applications are deployed in a multi-tenant mode on Apprenda you are allowed to completely customize the login page, which allows for white-labeling support.
    Windows Azure on the other side offers the Management Portal available at http://windows.azure.com. Windows Azure management portal is targeted to the developers who use the platform to host their applications. Unlike Apprenda though and because Windows Azure is a public offering (I will come back to this later on) the management of the platform is done by Microsoft and platform management functionality is not exposed to the public. Windows Azure also offers Marketplace available at  http://datamarket.azure.com/ where developers can publish their and end-users can subscribe for applications and services. Extensive documentation for Windows Azure is available on their main web site at https://www.windowsazure.com/en-us/develop/overview/.
  • Database Management Services (DBMS)
    Both platforms offer rich database management functionality.
    Apprenda leverages SQL Server to offer relational data storage functionality for applications and enables lot of features on the data tier like resource throttling, data sharding and multi-tenancy. Apprenda is working to also deliver easy integration with popular no-SQL databases on a provider basis in its next version. This will allow your applications to leverage the functionality of MongoDB, Kasandra and others as well as imporved platform support like automatic data sharding.
    Windows Azure Database is the RDBMS analogue on Azure side. Unlike Apprenda though Windows Azure Database limits the databases to certin pre-defined sizes and requires you to handle the data sharding in your application. Windows Azure Storage offers proprietary no-SQL like functionality for applications that require large sets of semi-structured data.
  • Business Process Management Technologies
    At this point of time neither Apprenda nor Windows Azure offer build-in business process management technologies. However applications on both platforms can leverage Biztalk Server and Windows Workflow Foundation for business process management.
  • Application Lifecycle Management Tools
    Both Apprenda and Windows Azure offer distinct features that help you through your application lifecycle and allow multiple versions of your application to be hosted on the platform.
    Applications deployed on Apprenda go through the following phases:
    • Definition - this phase is used during the initial development phase of the application or a version of the application
    • Sandbox - this phase is used during functional, stress or performance testing of the application or application version
    • Production - this phase is used for live applications
    • Archived - this phase is used for older application versions

    In addition Apprenda stores the binaries for each application version in the repository so that developers can easily keep track of the evolution of the application.
    If you use Windows Azure cloud services the support for application lifecycle includes the two environments that you can choose from (Staging and Production) and the convinient way to easily switch between those (a.k.a VIP-Swap) as well as the hosted version of TFS that you can use to version, build and deploy your application. If you use Windows Azure web sites you also has the opportunity to use Git for pushing your application to the cloud. Keep in mind that at the time of this writing the TFS service is in Preview mode (and hance still free) and in the future it will be offered as paid service in the cloud. 

  • Application Governance Tools (including SOA, interface governance and registry/repository)
    At the moment neither of the platforms offers central repository of services but as mentioned above there are easily integrated with Biztalk.
    Using intelligent load-balancing both platforms ensure the DNS entries for the service endpoints are kept consistent so you don't need to reconfigure your applications if any of the servers fail.
  • Messaging and Event Processing Tools
    Apprenda and Windows Azure significantly differentiate in their messaging and event processing tools.
    Apprenda offers event processing capabilities in a publish-subscribe mode. Publisher applications can send events either at application or platform level and subscriber applications can consume those. Apprenda ensures that the event is visible only at the required level (application only or cross platform) and it doesn't require any additional configuration.
    Windows Azure offers several ways for messaging. ServiceBus Queues offer first-in-first-out queueing functionality and guarantees that the message will be delivered. ServiceBus Topics offer publish-subscribe messaging functionality. Windows Azure Queues is another Windows Azure service that offers similar capabilities where you can send a message to a queue and any application that has access to the queue can process it. Whether you use ServiceBus or Windows Azure Queues though you as developer are solely responsible for ensuring the proper access restrictions to your queues in order to avoid unauthorized access. Keep in mind that all Windows Azure services are publicly available and the burden of securing those lies on you.
  • Business Intelligence Tools and Business Activity Monitoring Tools
    At this point of time both platforms have no build-in business intelligence or activity monitoring functionality.
  • Integrated Applicaiton Development and Lifecycle Tools
    Because both platforms target .NET developers you can assume good integration with Visual Studio.
    Windows Azure has a rich integration with Visual Studio that allows you to choose from different project templates, build Windows Azure deployment archives, deploy and monitor the deployment progress from within Visual Studio.
    Apprenda as well offers Visual Studio project templates for applications using different Apprenda services as well as external tool that allows you to build deployment archive by pointing it to a Visual Studio solution file. Unlike Windows Azure package format though Apprenda's deployment package is open ZIP format and has very simple folder structure, which allows you to use any ZIP tool to build the package. In the next version of Apprenda SDK you will see even better Visual Studio integration that comes at parity of what Windows Azure has to offer.
  • Integrated self-service management tools
    As mentioned above both platforms offer self-service web portals for developers. Apprenda also offers similar portals for platform owners and users as well.
    On the command-line front Apprenda offers Apprenda Command Shell (ACS) that allows developers the ability to script their build, packaging and application deployment.
    Similarly Windows Azure SDK offers a set of Power Shell scripts that connect to Windows Azure management APIs and allow you to deploy, update, scale out/scale back etc. your application.

Now, that we have looked very thoroughly through the above bullet points from Gartner's Reference Model for PaaS you may think that there are a lot of simlarities between the two platform and wonder why should you use one versus the other. Hence it is time to look at the differences in more details.

  • Public vs. Private
    One of the biggest differences between Windows Azure and Apprenda is that they both are targeting complementary areas of the cloud computing space.
    As you may already know Windows Azure is public offering hosted by Microsoft and so far there is no offering from Microsoft that enables Azure like functionality in your own datacenter (DC).
    Apprenda on the other side is a software layer that you can install on any Windows infrastructure and turns this infrastructure into a Platform as a Service. Although Apprenda is mainly targeted to private datacenters it does not prevent you from installing it on any public infrastructure like Windows Azure IaaS, Amazon AWS, Rackspace etc. Thus you can use Apprenda to enable PaaS functionality similar to the Windows Azure one either in your datacenter or on a competitive public infrastructure.
  • Shared Hardware vs Shared Container
    One other big difference between Windows Azure and Apprenda is how the platform resources are managed.
    While Windows Azure spins up new Virtual Machine (VM) for each application you deploy (thus enabling you to share the hardware among different applications) Apprenda abstracts the underlying infrastructure even more and presents it as one unified pool of resources for all applications. Thus in the Apprenda case you are not limited to the one-to-one mapping between application and VM and you can deploy multiple applications on the same VM or even bare metal. The shared container approach that Apprenda uses allows for much better resource utilization, higher application density and true multi-tenancy then the app-to-VM one.
    One note that I need to add here is that with the introduction of Windows Azure web sites you can argue that Windows Azure also uses the shared container approach to increase the applicaiton density. Howeve Windows Azure web sites is strictly constraned to applications that run in IIS while Apprenda enables this functionality throughout all applicaiton tiers including services and data.
  • Legacy vs. New Applications
    One of the biggest complaints in the early days of Windows Azure was the support for legacy applications and the ability to migrate those to the cloud. Ever since Microsoft is trying to add functionality that will make the migration of such applications easier. Things significantly improved with the introduction of Windows Azure Infrastructure-as-a-Service (IaaS) but on the PaaS front Azure is till behind as you need to modify your application code in order to run it in Azure Web or Worker role.
    Migrating legacy application to Apprenda on the other side is much easier and in the majority of the cases the only thing you need to do is to repackage the binaries into an Apprenda archive and deploy them to the platform. As added bonus you get free support for authentication and authorization (AutH/AutZ) and multi-tenancy even if your application wasn't developed with those functionalities in mind.
  • Billing Support
    The last comparison point I want to touch on is the billing support on both platforms.
    As you may be aware ISVs are having hard time implementing different billing methods on Windows Azure because there are no good ways to tap into the billing infrastructure of the platform - there are no standard APIs exposed and the lag for processing billing data is significant (24h normally)
    Apprenda in contrast is implemented with the ISVs in mind and offers rich billing support that allows you to implement charge backs on functionality level (think API calls) as well as on resource level (either allocated or consumed). This allows developers to implement different monetization methods in their applications - like charging per feature, charging per user or per CPU usage for example (the latter is similar to Google AppEngine).

By now you should have very good understanding of the similarities and differences between Windows Azure and Apprenda. I bet that you already have good idea where can you use one versus the other. However I would like to throw at you few ideas where you can use both together to get the best of both in your advantage. Here are couple of use cases that you may find useful in your arsenal of solutions:

  • Burst Into the Cloud
    With the recent introduction of Windows Azure IaaS and Windows Azure Virtual Network (both still in Beta) you are not anymore limited to the capacity of your private datacenter. If you add Apprenda into the mix you can create unified PaaS layer on top of hybrid infrastructure and allow your applications to burst into the cloud when demand increases and scale back when it decreases. 
    There are several benefits you get from this.
    First, your development teams don't need to implement special code in their applications that runs conditional on where the applicaiton is deployed (in the private DC or in the cloud). They continue to develop the applicaitons as they are deployed on a stand-alone server, then they use Apprenda to abstract the applications from the underlying infrastructure.
    Second, the IT personel can dynamically manage the infrastructure and add capacity without the need to procurr new hardware. Thus they are not the bottleneck for applicaitons anymore and become enabler for faster time-to-market.
  • Data Sovereigncy
    For lot of organizations putting data in the public cloud is still out of questions. Hospitals, pharaceutical companies, banks and other financial institutions need to follow cetrain regulatory guidelines to ensure the sensitive data of their customers is well protected. However such organizations still want to benefit from the cloud. Thus using Apprenda as PaaS layer spnning your datacenter and Windows Azure IaaS you can ensure that the data tier is kept in your own datacenter while the services and front-end can scale into the cloud.
  • Easy and Smooth Migration of Legacy Apps to the Cloud
    With the build-in support for legacy applications Apprenda is a key stepping stone into the migration of those applications to Windows Azure. Using hybrid infrastructure (your own DC plus Windows Azure IaaS) with Apprenda PaaS layer on top you can leverage the benefits of the cloud for applications that will need substantial re-implementation in order to run on Azure.
  • Achieve True Vendor Independence
    The last but not least is that by abstracting your applications from your infrastructure with Apprenda's help you can achieve true independence from your public cloud provider. You can easily move applications between your own datacenter, Windows Azure, AWS, Rackspace and any other provider that offer Windows hosting. Even better, you are able to easily load ballance between instances on any of those cloud providers and ensure that if one has major failure your application continues to run uninterrupted.

I am pretty sure this post doesn't evaluate all possible features and capabilities of both platforms but I hope it gives you enough understanding of the basic differences of the platforms and how you can use them together. Having in mind that Apprenda is a close partner of Microsoft we are working to bring both platforms together. As always questions, feedback and your thoughts are highly appreciated.

September 03, 2012

Configuring Logging in Python - The Real Life Example

For some time I was playng with Python and few nights ago I finally reached the point that I had to implement configurable logging in order to capture information that would help me troubleshoot some issues. Although I am pretty familiar with Log4J, which is very similar to the Python logging module it took me some time to get my loggers configured properly. Unfortunately the documentation is not very rich on examples hence I thought it will be useful also for other people to publish a good Python logging example tailored to a real life scenario.

I will describe few gotchas that I discovered while playing with the logging module.

Python Logging Scenario

In order to demonstrate the logging functionality I will use a simple Python application that consist of the following packages:


Each package (except the rootpackage one) contains two modules with exactly the same code in it (I know, I know - it is against all rules of software development to repeat code:)). The code initializes the logger and also defines a function that logs a message with each log level. Here is it:

[1] import logging
[2] import logging.config
[3] from rootmodule.utils import get_logging_config

[4] # set up logging
[5] logging.config.fileConfig(get_logging_config())
[6] logger = logging.getLogger(__name__)

[7] def log_messages():
[8]     """Logs a message with each of the log levels
[9]     """
[10]   print logger.level
[11]     logger.critical("This is a critical message")
[12]     logger.error("This is an error message")
[13]     logger.warn("This is a warning message")
[14]     logger.info("This is an info message")
[15]     logger.debug("This is a debug message")

Line [6] above is important to note because this is the easiest way to get the full name of the module (including the package name). This will allow you to easily change the logging configuration using the package hierarchy structure.

The rootpackage package contains additional module called utils with just one function in it with the sole purpose to retrieve the absolute path to the logging configuration file. Here the code for it:

from os import path

def get_logging_config():
    """Returns the absolute path to the logging config file
    return path.join(path.split(rootpackage.__file__)[0], 'logging.conf')

The logging configuration file logging.conf is also stored in the folder where the rootpackage modules are stored.

Here is the structure of the files on the app.

+ rootpackage
    - __init__.py
    - logging.conf
    - root_logging.py
    - root_logging2.py
    - utils.py
    + levelonepackage
        - __init__.py
        - lovelone_logging.py
        - levelone_logging2.py
        + leveltwopackage
            - __init__.py
            - leveltwo_logging.py
            - leveltwo_logging2.py

Hence there are 3 packages(rootpackage, levelonepackage and leveltwopackage) in the sample app, 7 modules (root_logging,root_logging2, utils, levelone_logging, levelone_logging2
leveltwo_logging and leveltwo_logging2) and 1 logging configuration file (logging.conf). You can download the full source code for the Python logging example from my site.

Sample App Logging Goals

Now that you have looked at the app let's set the following four goals for logging from the app:

  • Limit the messages to a particular log level for a module in particular package
  • Limit the messages to a particular log level for a specific package
  • Log the messages from particular package in a file while the messages from other packages to the console
  • Log the messages from the same package in a file as well as on the console together with the messages from other packages

Those seems to be enough to give you a good idea what is possible as well as how to change the configuration to achieve the desired outcome. Let's start!

Configuring Python Logging

As you may already be aware from Python documentation there are three main objects in Python logging package that you need to configure. Those are the Logger, Handler and Fromatter. In this post I will concentrate on the Logger and the Handler objects and will use a simple Formatter as this one:

format=%(asctime)s - [%(name)s] - %(levelname)s - %(message)s

This formatter will print a message in the following format:

2012-08-23 23:46:03,463 - [rootpackage.root_logging] - DEBUG - This is a debug message

There are other object like Filter for example but I will not discuss those.

Here is the initial code for the logging.conf file that enables DEBUG level for all the the module in my sample app:

[1] [loggers]
[2] keys = root,rootlogging,rootlogging2,levelonelogging,

[3] [handlers]
[4] keys = console

[5] [formatters]
[6] keys = generic

[7] [logger_root]
[8] level = DEBUG
[9] handlers = console

[10] [logger_rootlogging]
[11] level = DEBUG
[12] handlers = console
[13] qualname = rootpackage.root_logging
[14] propagate = 0

[15] [logger_rootlogging2]
[16] level = DEBUG
[17] handlers = console
[18] qualname = rootpackage.root_logging2
[19] propagate = 0

[20] [logger_levelonelogging]
[21] level = DEBUG
[22] handlers = console
[23] qualname = rootpackage.levelonepackage.levelone_logging
[24] propagate = 0

[25] [logger_levelonelogging2]
[26] level = DEBUG
[27] handlers = console
[28] qualname = rootpackage.levelonepackage.levelone_logging2
[29] propagate = 0

[30] [logger_leveltwologging]
[31] level = DEBUG
[32] handlers = console
[33] qualname = rootpackage.levelonepackage.leveltwopackage.leveltwo_logging
[34] propagate = 0

[35] [logger_leveltwologging2]
[36] level = DEBUG
[37] handlers = console
[38] qualname = rootpackage.levelonepackage.leveltwopackage.leveltwo_logging2
[39] propagate = 0

[40] [handler_console]
[41] class = StreamHandler
[42] level = DEBUG
[43] formatter = generic
[44] args = (sys.stdout,)

[45] [formatter_generic]
[46] format=%(asctime)s - [%(name)s] - %(levelname)s - %(message)s

Except the Formatter that I explained earlier I have defined 7 Loggers (one for each of the modules in the app except for the utils one and one for the root Logger) and one Handler for handling console logs.

Lines 2, 4 and 6 from the logging.conf file define the dictionary keys for the configuration of the different logging objects (Loggers, Handlers and Formatters). If you define a configuration key for one of the objects you MUST have section in logging.conf that corresponds to this key. For example I have defined Logger configuration key levelonelogging. The corresponding section for this key is [logger_levelonelogging] and the information from that section will be used to configure the corresponding logger. Also, as Gevin Baker mentions in his blog post A real Python logging example, you SHOULD NOT add any spaces in the keys list.

If you try to call the log_messages() function from any of the modules you will see similar output on the console:

2012-08-26 20:32:40,302 - [rootpackage.root_logging] - CRITICAL - This is a critical message
2012-08-26 20:32:40,302 - [rootpackage.root_logging] - ERROR - This is an error message
2012-08-26 20:32:40,302 - [rootpackage.root_logging] - WARNING - This is a warning message
2012-08-26 20:32:40,302 - [rootpackage.root_logging] - INFO - This is an info message
2012-08-26 20:32:40,302 - [rootpackage.root_logging] - DEBUG - This is a debug message

The only difference you will see is that for different modules you will get the module names in the square brackets. The first row of the output is the numeric equivalent of the log level, and in the case of DEBUG this is 10.

Changing the Log Level for Python Module

Now, let's see what changes we need to make in order to limit the messages from one of the modules to only ERROR messages. Let's for example choose module rootpackage.levelonepackage.levelone_logging2 for our test. The configuration for that module is done in section [logger_levelonelogging2] starting at line 25. The only change you need to do is to change line 26 to the following:

[26] level = ERROR

and you will get the following output if you call log_messages() from the module rootpackage.levelonepackage.levelone_logging2:

2012-08-26 21:55:29,654 - [rootpackage.levelonepackage.levelone_logging2] - CRITICAL - This is a critical message
2012-08-26 21:55:29,654 - [rootpackage.levelonepackage.levelone_logging2] - ERROR - This is an error message

All other modules (including rootpackage.levelonepackage.levelone_logging) will continue to log with level DEBUG. As you can see this was pretty simple.

Changing the Log Level for Python Package

Next we wanted to have all the modules from particular package to use the same log level. Because we started with a module rootpackage.levelonepackage.levelone_logging2 in package rootpackage.levelonepackage let's configure the logging in a way that all modules in this package (both levelone_logging and levelone_logging2) log with level ERROR. We will need to do few more modifications to our configuration file. Here are the modifications that we need to make:

  • Line 2: Remove the levelonelogging2 key from the list. Here is how the line should look like:
    keys = root,rootlogging,rootlogging2,levelonelogging,
  • Lines 20-24: Change those as follows
    [20] [logger_levelonelogging]
    [21] level = ERROR
    [22] handlers = console
    [23] qualname = rootpackage.levelonepackage
    [24] propagate = 0

    Note that in line 23 we removed the module name and left just the name of the package
  • Lines 25-29: Remove those as we won't need them anymore

Now, if you call log_messages() from the modules rootpackage.levelonepackage.levelone_logging or rootpackage.levelonepackage.levelone_logging2 you will get the following result:

2012-08-26 21:55:29,654 - [rootpackage.levelonepackage.levelone_logging] - CRITICAL - This is a critical message
2012-08-26 21:55:29,654 - [rootpackage.levelonepackage.levelone_logging] - ERROR - This is an error message

2012-08-26 21:55:32,254 - [rootpackage.levelonepackage.levelone_logging2] - CRITICAL - This is a critical message
2012-08-26 21:55:32,254 - [rootpackage.levelonepackage.levelone_logging2] - ERROR - This is an error message

Note that the log levels for the loggers (in this particular case rootpackage.levelonepackage.levelone_logging and rootpackage.levelonepackage.levelone_logging2) are not set (value is 0) but it is set for the package logger and all the modules in the package will log with the same level if they don't have separate logger configured.

Changing the Log Handler for Python Package

Last two goals we had were to log the messages from particular package to a file or to a file and to the console at the same time. In order to achieve the former let's define new handler in our configuration file. Here are the steps for that:
  • Line 4: Add additional key file
  • Line 22: Change the handler from console to file for the rootpackage.levelonepackage package
    [22] handlers = file
  • At the bottom of the config file add the following section that defines the file handler
    class = FileHandler
    level = ERROR
    formatter = generic
    args = ('sample.log', 'a')

Now, if you call log_messages() from the modules rootpackage.levelonepackage.levelone_logging or rootpackage.levelonepackage.levelone_logging2 the only thing you will see on the console is the number 0 because the function prints the log level. If you exit Python and look in the folder where you started it you will find a file sample.log that contains the log messages. If you want to see the log messages on the cosole and also log them to a file the only change you will need to do is to add the handler to the list of handlers for the package as follows:

[22] handlers = console,file

And rememer - NO SPACES in the list!

Hope this will kick start your logging with Python and make your life easier.