dispy

dispy is a comprehensive framework for Python that allows you to execute parallel computations is a single cluster.

The tool allows you to combine the computing power of multiple processors in a single machine, in a cluster, a grid or a cloud.

With dispy, you can easily evaluate several Python functions or even stand alone programs, with various types and sizes of datasets. You can work independently from other computation tasks, without communication dependencies.

dispy integrates with asyncoro, a powerful Python framework, in order to create coroutines, generator functions and communication among tasks.

asyncoro is required for concurrent, asynchronous programming with coroutines.

The Python components and programs created with dispy, as well as their dependencies can be automatically distributed to local or network nodes.

SSL encryption can also be used for protecting information send via the nodes. Python functions can also transfer files to the client via nodes.

dispy provides a specific program, namely ‘dispynode.py’ that must run on each of the nodes for the jobs to be executed for the afferent clients.

Moreover, you can provide HTTP interface to any cluster, in order to visualize and monitor it through a Web browser.

dispy allows you to trace the results of computing Python functions or programs, as well as verify the output, track errors and exceptions.

It can also help you schedule tasks to be performed whenever a suitable node becomes available.

It can provide support for the automatic succession of executions so that it can start the scheduled function whenever the previous task is finished.

This process can come in handy in several practical situations, such as verifying the results when they become available.

dispy also supports both client-side and server-side fault recovery, for instance when a client is unexpectedly terminated and the scheduled tasks keep being executed on the nodes.