Tutorial: Asynchonous Python with Twisted; Part Three

Repository

• https://github.com/twisted/twisted

What Will I Learn?

In part one of this multipart tutorial, aimed at teaching you the use of asynchronous programming paradigms using Twisted, you will learned about the first two of the five pillars of the Twisted architecture: The Reactor and Deferreds.
In part two we moved on to the other three pillars: Transports, Protocols and Applications. You learned how to use logging, how to implement your own protocol, run it as an application using twistd and you learned about timed events and wrapping blocking APIs and long running code using a seperate thread or a Perspective Broker server running in a different process. In this third tutorial of this series we move on to the use of application level protocols with Twisted.

In this tutorial you will learn:

• How to use a relational database like PostgreSQL with Twisted for asynchronous database operations.
• How to use Redis asynchronically with Twisted.
• How to build asynchronous web applications with Twisted and Jinja2
• How to let your asynchronous web application run over SSL
• How to use Twisted and Qt together in a single application.

Requirements

To follow this tutorial, you should:

• Have a good working knowledge of Python
• Have read part one and part two of this series.

Difficulty

• Intermediate

Asynchronous Python with Twisted; Part 3

This is part three of a four part series of tutorials on the use of Twisted (and asyncio) for writing asynchronous applications in Python. This series of tutorials is based on a one-day training I gave to a group of eleven software engineers last month.

In the previous installment of this series of tutorials on the subject of asynchronous programming in Python, we looked, amongst other lower level things, at how you can use Twisted or, alternatively (or in conjunction), asyncio, to write protocols and make them run on a transport. We learned how to wrap synchronous API's by cheating a bit; that is, by wrapping them using a separate thread.

Now while in some cases, there is no way around it and you may need to opt for the wrapper option because there is no asynchronous library available for the specific protocol and writing one yourself may simply be too involved for the particular application level protocol involved. In a growing number of cases though, application level protocols are available for usage. Either, as is usually preferable, as an application level protocol that is part of the Twisted framework itself, or as an additional add on library. In the latter case, this could be a library implementing a Twisted protocol (and/or Transport), or a library built on top of Python3 async/await primitives.

Again we press, as discussed in part one of this tutorial series, that Twisted and asyncio really do play nice. There is a tiny bit of extra code needed to make things fit when mixing Deferreds and Futures in the same code base, but this is peanuts compared to the code needed to reinvent the wheel if that wheel was already invented for the alternate async core.

In this installment we are going to look at a few commonly used higher level protocols. We will look at databases and web applications and how to integrate HTML templating engines and database server-side code into our asynchronous web applications using Twisted.

Database usage

If the database we want to talk to is a relational database, the Twisted framework provides the core for connecting to that database in an asynchronous way. The twisted.enterprise.adbapi API provides a fully Deferred API for talking to relational databases. The adbapi uses dbapi database adapters such as pycopg2.

Before we look at how to use adbapi to talk to a database from our code, we first need to discuss a common way of talking to databases from code that should be considered bad practice.
Above we see some synchnronous code that talks to a database. An SQL statement is prepared in a safe way as to prevent silly injection attacks, and the statement is then executed. The above code might be acceptable if the SQL statements remain short, or if the database server is extremely limited in functionality.

As a general rule though, embedding SQL queries in your Python code is a rather bad idea. Most programmers aren't database administrators and have very little insights in profiling and optimizing database schema's and SQL statements for performance and/or resource usage. Database schema's and SQL queries tend to go through a number of iterations of profiling and refactoring, and a python program littered with long embedded SQL statements can quickly become a maintenance nightmare.

Today, most databases, that is, most relational databases will implement either database functions or procedures. These functions and procedures will run on the database server themselves. It is easy to move SQL statements from being embedded in your Python code to being part of a database stored procedure or function.

Keeping all SQL queries that query tables, inside of a stored procedure or function, has multiple advantages. It can keep your Python code free from containing SQL statement littered throughout the code base, but that isn't the most important benefit. The most important benefit is that the stored procedure of function provides an API to the data model. Programmers aren't (usually) database gurus. Look at a stored procedure or function as an interface between your code and the data model. The database administrator can profile the usage of all stored procedures by your code, completely refactor the database schema and keep the API exactly the same without ever having to even talk to a developer. Not only does this give benefit to both database administrator and developer in terms of separation of concerns, it also helps prevent version mismatch issues between database schema and code-base.

Now that we have established that the only SQL code we should see in our Python code base is code invoking stored procedures and functions, it is time to both look at the adbapi and to revisit the wildcard method from part one of the tutorial series. In the above code we use the psycopg2 database adapter for the PostgreSQL database server.
In our constructor we create an asynchronous connection pool for our database.

Then in our wildcard method, a single getattr method that returns a callable from a closure, where the returned callable implements the wildcard method itself. In the callable we construct a SELECT call that is a function invocation. In our case we take the convention of prefixing function names with function_. We asynchronously invoke the SQL call and return the deferred to the calling code.

Note that any database function we might define in our database will instantly get a working implementation in our proxy class that can be called without needing a single extra line of Python code.

Next to relational databases, there are also other types of databases, often for specific purposes. In recent years there has been a massive shift from the use of general purpose relational database servers base4d on SQL, to so called NoSQL servers more suitable for specific use cases. One notable example of such database is the distributed in-memory database Redis.

Like with many third party Twisted based asynchronous libraries, the name of the library for accessing Redis asynchonically has a name that starts with tx. In Redis, databases in the database server have a number and are basically key value stores.

The txredis API is a great example of an API that has you using Deferreds at multiple levels. In the above piece of sample code, we connect to database number 3 of the Redis server running on localhost. Creating a connection pool to the database returns a Deferred that we set callbacks on. On connect, we try to lookup a value by a key. The get method returns an other Deferred that again we add callbacks on. Now if the value was set previously we print it and stop the reactor to end our little program. If the value wasn't set yet, we use the set method to set the value for our key. Surprisingly, even setting a value, while a method that doesn't yield any value, again returns a Deferred. In this case we add a callback to both the success and error path in order to stop the reactor and end our little program once the set operation completes.

Web

Twisted also comes with a little Web sub-framework for implementing web based applications. In Twisted, a web application is build around a tree of Resource objects with one Resource representing the root of the tree.
The above code is a bare bones single page web server that in this case gets bound as a Transport on TCP port 4080.

As before , where we discussed that it is a very bad idea to embed any long or complex SQL statements in our Python code, the same is basically true for HTML code. Most programmers are rather poor web designers and most web designers are poor programmers. So, as before when we discussed stored procedures for databases, it is better to opt for an architecture that allows experts to stay in their lane.

The easiest way to keep ourselves from using Python embedded HTML with Twisted Web, is to combine Twisted Web with the use of the jinja2 library. Jinja2 is an easy to use templating library that augments twisted web perfectly. It allows you to completely decouple your HTML template design from your Python business logic.

The above example shows an example of a slightly more advanced template that iterates over a list of objects.

Let us have one last look at our embedded HTML example from before.

Here we bring things together and use jinja2 together with Twisted web. The two fit together quite well.

So far in our web example, we had the answer we wanted to send to our client straight away. But as we are discussing asynchronous programming here, chances are we will often need to do some more asynchronous work before we actually have something to return to our client. In the admittedly somewhat contrived example above, we show how something like that is done with Twisted web. What we do is we return the special value NOT_DONE_YET from our render_GET method. By doing so, we promise twisted that some Deferred callback will come along and provide the result. In our example above, a delayed respons* and response.finish combine to provide the client with its delayed response.

Now what if we wanted our little server to be a HTTPS server? Well, as we learned in part two of this series of tutorials, we can simply replace our transport with a different one. The most convenient and flexible way to do this is using endpoints.serverFromString. Note that in our special string we provide a number of attributes, including a certificate file and a private key file path.

To conclude our part on web related subjects, just for completeness, it is important to note that, naturally, we can also use Twisted to implement the client side of a web based infrastructure. You might use this in for example a Qt based asynchronous program, or if you wanted to implement something like a simple REST or JSON-RPC client, or maybe a web proxy of sorts.

Qt

As we discussed in both part one and two of this series, it is possible to use Twisted with Qt and build asynchronous networking application with a cross platform GUI. The above trivial example of using Twisted with Qt shows us how we can process Twisted timed events together with a simple GUI. We also see, that while initialization may be a bit fiddly; we basically replace the standard Twisted reactor with the Qt event loop, once initialized, things work quite seamless together.

Some subjects not covered.

While we couldn't cover everything in this tutorial, I hope we covered a few important high level protocols and we covered how we can connect to other notable libraries and frameworks that make everything just fit for creating real life asynchonous applications in Python. Some things not covered that you might want to look into yourself are subjects like AMQP, WSGI and LetsEncrypt that you can do using third party libraries such as hendrix, txacme and txsni. But also other parts of the Twisted framework itself, including SSH (with conch) , DNS, SMTP/IMAP and POP and subjects like working with credentials (ldap, pam, etc).

Finally two handy parts of Twisted not covered that you might want to look at are LineReceiver, a special Protocol base class that makes it easy to implement line oriented protocols, and FileSender, a very handy helper construct for sending large files asynchonously to a client.

---

In the next post in this series we will conclude this series with a tutorial covering testing and performance.

Curriculum

• Writing a simple STEEM statistics-script in Python with asyncsteem
• Writing an asynchronous STEEM web-app in Python with asyncsteem
• Building an asyncsteem python web app with micro-transaction-based steem user authentication, Redis and Jinja2
• Asynchonous Python with Twisted (and asyncio); Part One
• Asynchonous Python with Twisted (and asyncio); Part Two