Honing the craft.

Migrating to Python 3 on Google App Engine - Part 3 - Migrating from Webapp2 to Pyramid

Migrating to Pyramid from Webapp2 was the part of my journey to Python 3 on Google App Engine. This post details the changes I had to do, to accommodate my application for the Pyramid Web Framework.

Although migrating off Webapp2 is not a must if you want to move your application to the App Engine Python 3 Standard Environment, some changes to replace essential services no longer built into the App Engine runtime, like authorization, were due, so I ceased the opportunity to migrate to a more mainstream, pluggable and scalable framework.

If you don't want to move away from Webapp2 or your framework of choice, you can probably skip this post, in that case however, as services, like authentication and authorization no longer available in the App Engine runtime will be provided by Pyramid, you won't be able to use the information in the next blog post either.

In this post I go through all the changes that I had to make to have my application work identical to what I had with Webapp2.

Merging components into main.py

In the Python 2 Standard Environment, it is possible to define route patterns in app.yml and load different WSGI application depending on the matched route pattern. This feature is unavailable in the Python 3 Standard Environment, hence, before moving on with the following steps, I have merged the configuration, mainly routes, of my blog's public and admin interface, that have been running as completely separate applications up till this point.

The merged configuration and bootstrap code went into a module file called main.py, which is what the Python 3 Standard Environment looks for by default when a request comes in to your App Engine instance.

Changing the application's basic configuration

I have started the changes with refactoring the configuration and bootstrapping of my main application object in main.py to make it compatible with Pyramid.

The Webapp2 import statement has been removed:

# module: main
# removed this
import webapp2

and the Pyramid Configurator object was imported instead:

# module: main
from pyramid.config import Configurator

Then, the instantiation of the main application object had to be updated accordingly. In the case of Webapp2, you instantiate the WSGI application object and then use the instance to set up the basic configuration.

With Pyramid, you create an instance of the Configurator object, add your desired configuration, then use that instance to bootstrap the WSGI application object. The approach is a bit different, but the differences are mainly semantical, the statements used for the configuration are very similar as you will see very soon.

The main application object was instantiated like this in my case:

# module: main
app = NdbWsgiMiddleware(webapp2.WSGIApplication(debug=False))

Note, that the webapp2.WSGIApplication has been wrapped into the NdbWsgiMiddleware that provides the necessary client context for Cloud NDB. Hence, I will access the Webapp2 WSGI application object at app.wsgi_app as you will see later.

For Pyramid, what you can see above, has been replaced with:

# module: main
config = Configurator(settings=settings, root_factory='resources.Root')

Please ignore the parameters passed to the Configurator constructor for now, we will touch on those later on. In short, settings is a dictionary holding the configuration values for the core framework and included plugins, while root_factory is being used to configure RBAC resources for the authorization policy. I will expound on the latter in my next blog post.

Route configuration

Route configuration had to be updated to comply with the Pyramid API as well, for instance:

# module: main
app.wsgi_app.router.add(webapp2.Route('/blog/post/<slug>',
                                       handler=BlogPost,
                                       name='blog_post'))

was replaced with:

# module: main
config.add_route('blog_post', '/blog/post/{slug}')

This is a minor change:

• The route's name, the first parameter of add_route, remained the same.
• The route's URL pattern has only changed in that the syntax that captures the slug variable in the URL pattern into the request is wrapped in curly braces.
• There is no parameter registering the handler for the route as I will use a different method to link the route with the view handler.

Static routes, marked with the build_only boolean parameter in the webapp2.Route constructor, only used to facilitate URL generation in templates for example, had to be updated as well. In my case, I had the following route set up for building the correct references to CSS stylesheets:

# module: main
app.wsgi_app.router.add(webapp2.Route('/blog/styles/',
                                       build_only=True,
                                       name='style_path'))

This was replaced by:

# module: main
config.add_route('style_path', '/blog/styles/', static=True)

Configuring the template system - Jinja2

I have been using Jinja2 as my templating engine with Webapp2 and since Pyramid supports Jinja2 pretty much out of the box, it was a no brainer to stick with it.

For my Webapp2 handlers, I have set up the jinja2.Environment object and used it to render responses using my templates. This is how the environment object was set up:

# module: main
import jinja2
...
# Templating setup, we create the Jinja Environment
JINJA_ENVIRONMENT = jinja2.Environment(
        loader=jinja2.FileSystemLoader(os.path.dirname(__file__)),
        extensions=['jinja2.ext.autoescape'])
# We set up the webapp2.uri_for function for usage in the templates as URL helper
# Further parameters that need to be accessed from the templates universally should go here.
JINJA_ENVIRONMENT.globals = {'uri_for': webapp2.uri_for,
                             'markdown': markdown.markdown,
                             'md_ext': markdown_image_ext}
JINJA_ENVIRONMENT.filters['format_datetime'] = format_datetime

For Pyramid, you could also import Jinja2 separately as with Webapp2 and use it in the views in an explicit way, but it is more straightforward to add the pyramid_jinja2 extension instead. In order to use the extension in your application, you need to do three things:

• Make sure that the pyramid_jinja2 package is installed in your environment.
• Pass the right configuration into the Pyramid Configurator constructor to configure Jinja2.
• Include the extension using Pyramid's configuration API.

The configuration for Jinja2 can be set in the dictionary passed to the Configurator constructor as the settings keyword argument. I had four things configured for my pre-existing Jinja2 configuration:

• The webapp2.uri_for utility function generating URLs based on route names was made available as a global variable. This has been removed from the Jinja2 configuration for Pyramid, as Pyramid makes its own utility available through the request variable.
• The markdown.markdown function used to generate the HTML content of blog posts from my Markdown text.
• My extension to the markdown library to embed images in a special way.
• A template filter used to generate friendly dates.
• The jinja2.ext.autoescape extension.

This is how my settings dictionary looked like for the pyramid_jinja2 extension and was passed to the Configurator:

# module: main
from pyramid.config import Configurator
import blogutils
import markdown
import markdown_image_ext
...
settings = {'jinja2.filters': {'format_datetime': blogutils.format_datetime},
            'jinja2.globals': {'markdown': markdown.markdown,
                               'md_ext': markdown_image_ext}}
config = Configurator(settings=settings, root_factory='resources.Root')

This is pretty much the same thing as before just with a different API, so the transition is very straightforward. The autoescape feature is now (Jinja2 version 2.9+) part of the core functionality and it is enabled by default, hence it doesn't need to be set in the configuration.

Including the extension using the config object is easy enough as well:

# module: main
config.include('pyramid_jinja2')

Scanning views

With Webapp2 you have to reference the handler of the given route explicitly. In Pyramid you can take different approaches, but the most idiomatic way in my opinion is to use view decorators. Then Pyramid will look for these decorators in the modules it is configured to scan. I will cover how to set those decorators up, but before I do that, I configure Pyramid to scan the modules where my view handlers are located:

# module: main
config.scan('views')
config.scan('admin_views')

Note: You have to do this in your main application module, main.py in my case, where you configure your application object before the WSGI application object is instantiated.

Wrapping into the Cloud NDB middleware

Once all the configuration has been done, I have loaded the WSGI application object with config.make_wsgi_app() and wrapped it into my Cloud NDB middleware:

# module: main
app =  NdbWsgiMiddleware(config.make_wsgi_app())

Updating views

The next step was creating the views that Pyramid can invoke on the appropriate routes. I have repurposed my module containing the Webapp2 handlers as Pyramid views. First I have removed the current imports for:

• Webapp2
• Jinja2

And imported the Pyramid view decorator functions:

# module: views
from pyramid.view import view_config, view_defaults

Then I continued with making the necessary changes to the handlers, as follows.

Remove the Webapp2 dependency of handler classes

When you create request handlers in the Webapp2 framework, you create classes, that inherit from webapp2.RequestHandler and have methods matching the name of HTTP request methods, like get. These classes don't need to inherit from any particular class in Pyramid, so the reference to Webapp2 can be simply removed:

# module: views
class BlogHome(webapp2.RequestHandler):
    def get(self, **kwargs):
        ...

should be changed to:

# module: views
class BlogHome:
    def get(self, **kwargs):
        ...

Adding the view_config decorator to handler methods

The next step was to add the Pyramid view decorator.

Let's see how I have updated an existing Webapp2 handler method to add the decorator. The class looked like this on a high level after removing the inheritance reference to webapp2.RequestHandler:

# module: views
class BlogHome:
    def get(self, **kwargs):
        ...

After adding the decorator, the same handler looked like this:

# module: views
@view_defaults(renderer='blog/templates/bloglayout.jinja2')
class BlogHome:


    def __init__(self, request):
        self.request = request


    @view_config(request_method='GET', route_name='blog_home')
    @view_config(request_method='GET', route_name='blog_page')
    def show_posts(self):
        ...

The following changes were made:

• An initializer method has been added to store the request object in self.request. This is needed, as in case a method is being used as the view handler, the request object will be passed to the object constructor, and not the method directly.
• The @view_config decorator has been added to the handler method.
• The @view_defaults decorator has been added to the class.
• The handler method has been renamed to show_posts to make what the view does more explicit. For Webapp2, you need a method name that matches the HTTP request method, the request matching is now done in the view decorator, so this naming convention is no longer required.
• **kwargs has been removed from the parameter list.

There are two kinds of decorators used in the example above: @view_config and @view_defaults, they can take the same parameters, the only difference is their scope.

While you would use @view_config for concrete handler functions or methods, @view_defaults is intended for handler classes, and the options set with this decorator are automatically applied to all the methods of that class. This is useful, when you want to set a view configuration item, that is identical across multiple methods, so that you don't have to repeat the option for each method decorator.

I have used @view_defaults to set the renderer to a template, that will apply to all view handler methods of the class for instance.

From the decorator arguments used above, only route_name is mandatory, since that is the "link" between the view and the route it should be invoked on.

There are many additional arguments, that can control when the view is invoked and how it will provide its response. The ones I have used above are:

• request_method which should determine the HTTP request method the view should handle, which is GET in the example above. If it is not provided, the view will be invoked on any request method.
• renderer determines what templating system and template file should be used when sending the response.

Updating the handler methods

To make the view handler methods comply with the Pyramid API, there were two main areas in the code that I had to update:

• Statements working with the request object.
• The code generating the response.

Changes to request interrogation

I have used two methods of capturing request data with Webapp2:

• Replacement markers, that allow identifiers to be embedded into the request URL and then obtain the value of these identifiers in the corresponding handler.
• Data sent in the body of POST requests.

Accessing of request data captured with URL replacement markers:

• When using Webapp2, the captured data is passed to the handler through the **kwargs argument. The key matches the name of the replacement marker in the route pattern definition.
• For Pyramid, the data can be read from a request object attribute, namely self.request.matchdict, in case self.request is set up the same way as I have done in the object initializer.

Now let's get into my changes concretely.

For pagination on the list of blog posts, I have a route defined as /blog/page/{pagenum}. The page number parameter pagenum was obtained from the URL with Webapp2 like this:

# module: views
page = int(kwargs['pagenum'])

To work with Pyramid, I had to change this to:

# module: views
page = int(self.request.matchdict['pagenum']

Accessing of request data sent in the POST request body:

With Webapp2, you can obtain data items with:

# module: views
post.title = self.request.get('title')

To work with Pyramid, I have changed this to:

# module: views
post.title = self.request.POST['title']

or if it is expected, that a certain key might be missing (eg. if the corresponding checkbox is unchecked):

# module: views
post.published = self.request.POST.get('published', 'off')

Changing Webapp2 specific helpers

Up until now, I have used the webapp2_extras.json module to work with JSON data in some places:

# module: views
associatedImages = webapp2_extras.json.decode(self.request.get('post-images'))

To remove the dependency on Webapp2, I have changed those statements to use the standard library's json module instead:

# module: views
associatedImages = json.loads(self.request.POST['post-images'])

Sending a response without using a renderer

Sometimes you have special response scenarios, where none of the available view renderers fit your needs. In my case, this scenario is serving images from my Datastore image blobstore.

With Webapp2, I set the Content-Type header explicitly and wrote the raw image data on the response object:

# module: views
self.response.headers['Content-Type'] = str(img.mime)
self.response.write(img.image)

where img is my Datastore entity storing the image blob data and mime type.

With Pyramid I return the Response object with the image data and mime type set as constructor arguments:

# module: views
# from pyramid.response import Response
# In my view handler
return Response(body=img.image, content_type=str(img.mime))

Of course, this is just my example of using the Response object, the solution might be different depending on your needs.

Redirects from handlers

To redirect to a different route from a handler with Webapp2, you can use the redirect method inherited from webapp2.RequestHandler.

I have redirected to the category list page after adding a new category like this:

# module: views
self.redirect('/blog/admin/categories')

For the same effect, I have to raise a pyramid.httpexceptions.HTTPFound exception when using Pyramid, passing the URL it should redirect to, which can be generated by the request objects route_url method. You just have to pass the route name you need to redirect to:

# module: views
# from pyramid.httpexceptions import HTTPFound
# In my view handler
raise HTTPFound(self.request.route_url('categories'))

Getting route URLs

In Webapp2, I have obtained route URLs when needed with the uri_for helper:

# module: views
jsData['imgUploadBaseUrl'] = webapp2.uri_for('save_newimage')

I have to use the route_url helper with Pyramid now:

# module: views
jsData['imgUploadBaseUrl'] = self.request.route_url('save_newimage')

Response generation

When generating responses, I use two different approaches:

• Rendering HTML data to the user with the Jinja2 renderer.
• Returning JSON for some AJAX requests.

With Webapp2, both of these scenarios are invoked explicitly, while more work is done implicitly if you work with the Pyramid renderers.

Let me give you a few examples. Rendering an HTML page, and passing a template variable using Jinja2 looked like this in my code:

# module: views
template = JINJA_ENVIRONMENT.get_template('blog/templates/bloglayout.html')
self.response.write(template.render(template_values))

I have used JINJA_ENVIRONMENT, which is an instance of jinja2.Environment as mentioned earlier, to load the template and used the returned template object to render the output. The template_values argument can be used when rendering, this is a dictionary describing the template variables names made available in the template and their corresponding values (references).

The output rendered is passed to self.response.write that does the actual sending of the request.

The Pyramid version of the same handler, configured with the same Jinja2 template in its view decorator looks like this now:

# module: views
return template_values

You just return the dictionary describing the template variables you want to export into the the template and that's it.

The situation is very similar when responding with JSON data. With Webapp2, it is pretty explicit:

# module: views
self.response.headers['Content-Type'] = 'application/json'
# Do stuff to generate the JSON data
...
self.response.write(webapp2_extras.json.encode(data_dict))

where data_dict is a dictionary holding the data to be returned as JSON payload.

With Pyramid, I have used the built in JSON renderer by setting renderer='json' in @view_config, then, in the view handler, I was able to do this:

 # module: views
 # Do stuff to generate the JSON data
 ...
 return data_dict

There is no need to set the Content-Type response header, that is done by the renderer implicitly. Just return a dictionary holding the data to send back, and the renderer will encode it into proper JSON automatically.

Updating templates

You might have noticed already that I was using .html as the extension for my Jinja2 template files with Webapp2 and changed the extension to .jinja2 while transitioning to Pyramid. Although Pyramid is able to use .html files as Jinja2 templates, but it has to be configured to do so specifically, as it uses the extension to determine the renderer to use.

Since .html is a misleading extension to use for Jinja2 templates and Pyramid will use .jinja2 files as templates out of the box, I have rather just updated the extension of my template files to .jinja2.

extends statements in templates were also referring paths to template files, these also had to be updated. This:

{% extends "blog/templates/bloglayout.html" %}

had to be changed to this:

{% extends "blog/templates/bloglayout.jinja2" %}

Updating the URL helper

The helper function, that gives the URL for a given route name in Webapp2 is webapp2.uri_for, which was passed to the template, so that I can use it like this:

<link rel="stylesheet" href="{{ uri_for('style_path') }}screen.css"/>

Pyramid implicitly exports the request object into the template environment, which has the route_url method, that works the same way:

<link rel="stylesheet" href="{{ request.route_url('style_path') }}screen.css"/>

All I have done is replacing all instances of uri_for for request.route_url in similar fashion.

If you have a route with a replacement marker, like config.add_route('blog_page', '/blog/page/{pagenum}'), which is referred to in the template like this:

<a href="{{ uri_for('blog_page', pagenum=page+1) }}">

The API for request_url is the same, so I just had to replace the function name:

<a href="{{ request.route_url('blog_page', pagenum=page+1) }}">

Autoescape

Jinja2 version 2.9+ does automatic escaping of characters, that have a special meaning in an HTML context to prevent security problems like XSS attacks.

On earlier versions, like 2.6, that comes with the App Engine Python 2 Standard Environment, you had to enable the jinja2.ext.autoescape extension for this functionality. Strangely enough, although I have enabled this extension, the HTML content generated from Markdown and filled into the templates still worked somehow, however, once I have migrated to Pyramid and with that, to the latest version of Jinja2, the post contents were suddenly escaped and were not displayed correctly as a result. This makes sense and I am not sure why it was working before to begin with.

To overcome this problem with displaying the posts however, I had to disable autoescaping for these data items.

Previously, the place where the post intro text was populated looked like this for example:

<div class="blog-post-intro">
    {{ markdown(text=post.intro, extensions=[md_ext.ImageExtension(uri_for('serve_image_root'))]) }}
</div>

In the new environment, I had to use the safe template filter to make it work as expected:

<div class="blog-post-intro">
    {{ markdown(text=post.intro,extensions=[md_ext.ImageExtension(request.route_url('serve_image_root'))]) | safe }} 
</div>

where function markdown produces the HTML output from the Markdown text and the filter safe disables escaping. You can ignore the function parameters as they are not relevant in this scenario.

Handling errors

Custom error handlers

In order to display better looking error messages, I define custom error handlers, that are similar to normal view handlers. These handlers however are registered with the framework in a special way, so that they are invoked on the error they handle and return the correct HTTP response code.

To demonstrate the changes made, here is how my error handler for HTTP 404 Not Found errors was written with Webapp2:

# module: custom_error_handlers
def handle_404(request, response, exception):
    logging.exception(exception)
    template = JINJA_ENVIRONMENT.get_template('blog/templates/error.html')
    template_values = {'code': 404}
    response.write(template.render(template_values))
    response.set_status(404)

On a high level, this is what the handler does:

• Logs the exception.
• Renders my error template, exporting the error code.
• Sets the HTTP status code of the response.

To reach the same effect with Pyramid, I have changed the handler function like this:

# module: custom_error_handlers
# from pyramid.httpexceptions import HTTPNotFound
# from pyramid.view import view_config, view_defaults
@view_config(context=HTTPNotFound, renderer='blog/templates/error.jinja2')
def handle_404(exception, request)
    logging.exception(exception)
    return {'code': 404}

Short inventory of the changes:

• The function arguments have been updated, so that it takes two arguments, exception and request.
• The @view_config decorator has been added:
  • with HTTPNotFound as context. This tells the Pyramid view scanner, that the handler (view) applies to situations where the HTTPNotFound exception is raised.
  • with the custom error template set as the renderer.
• The handler just returns the dictionary containing the error code, that will determine how the error page will look like.
• All the other code rendering the template and setting the response code is removed. The HTTP response code is automatically set by Pyramid.

Registering custom error handlers

To have the handlers (views) properly invoked on errors, you need to tell the framework where to find them.

With Webapp2, this is how the custom error handlers are registered based on the HTTP response (error) code they handle:

# module: main
# import custom_error_handlers
app.wsgi_app.error_handlers[404] = custom_error_handlers.handle_404
app.wsgi_app.error_handlers[500] = custom_error_handlers.handle_500

Since I have used Pyramid's view decorator to mark my handlers, I can simply use the view scanner to register these, like normal views, when setting up the Configurator object:

# module: main
config.scan('custom_error_handlers')

Sending an error response from my code

There are some cases, when you want to trigger a particular HTTP response (error) code from your handler. This is done with the inherited abort method of the handler in Webapp2, while in Pyramid, you raise a specific exception for the same effect. See the examples below for the two error codes I am using for my application.

Returning an HTTP 404 Not Found response:

# module: main
# Webapp2
self.abort(404)
# Pyramid
# from pyramid.httpexceptions import HTTPNotFound
raise HTTPNotFound()

Returning an HTTP 500 Internal Server Error response:

# module: main
# Webapp2
self.abort(500)
# Pyramid
# from pyramid.httpexceptions import HTTPInternalServerError
raise HTTPInternalServerError()

Test drive

Once the changes outlined above were done, I have reached the point where all Webapp2 functionality I used was replaced with Pyramid.

I had two options for a next step:

• Test my application after these changes on the App Engine Python 2 Standard Environment.
• Move on with setting up authentication and authorization, then migrating to the Python 3 Standard Environment in one go.

Since I picked the latter option, I don't provide a detailed account of testing this with Python 2, however you probably know hot to do that if you have an existing application on App Engine. Just install the pyramid and pyramid_jinja2 packages in your 3^rd party library directory, make your code changes, don't forget to update your app.yaml file, so that all application routes trigger main.py and you should be good to go to test with dev_appserver.py.

For example, I would have run this to install the packages required in my library directory:

pip install -t pylibs/ pyramid pyramid_jinja2

Further opportunities for refactoring

Apart from the essential changes outlined above, I have done many refactoring in my code base to make the structure fit the Pyramid framework better. These were not elements impacting functionality, but code structure and modularity. These are specific to my code base, so no examples are given, but here are a few items that probably make sense for anybody:

• Since Webapp2 handlers are class methods, you need to have a class for each view. This is absolutely unnecessary for Pyramid, you can simply use view functions or merge existing related classes sharing common options.
• Some of my application specific helper functions were scattered in application modules, I have consolidated these into a common utility module.
• If you stick with using classes for views, it is probably a good idea to create a parent class the views inherit from to take care of setting self.request up.

Documentation references

• Pyramid
• Webapp2
• pyramid_jinja2 Docs
• Pyramid View Configuration
• Pyramid View Handlers

Next up

In the next blog post of the series, I will be sharing how I have set the authentication and authorization system up with Pyramid.