Friday, November 21, 2008

Notes for embedding python in your C/C++ app.

As I've written about before, I've embedded python into our sampler application to create a control-rate scripting language. The python code is run with (*gasp*) real-time priority in the audio thread, and performs quite well, as long s no one does a while True: pass. There were a ton of hurdles for embedding the app into the program, but there was also very little information available on the internets to help, so I thought I'd try to write some stuff down. Go Google.

Basic Concepts

There are two cases that I can think of for embedding python into your application. The first one is to provide a scripting interface to your application. This means that you allow your users to write scripts to control your app, and you provide a special API for access to your data engine, or something like that. This is what I did.

The second one is if you want to actually write some of your application in python. An example would be that you would write one of the gui widgets in python, and set the widget to be a child window of your application's main window - interesting stuff. What you would have to do is write some funky wrapper code to get a window handle or class pointer from your python module as an unsigned long long or uuint and pass that to your gui tool kit. I have code to do this with Qt.

Running scripts

The first example above is the most straight forward. You include some kind of script editor and a "run" button in your app, and call PyRun_SimpleString() or some other method from the pyhton library to execute your code. If you really want to get funky you can intercept sys.stdout and print console output. This makes your app way cooler.

I set my script source from utf8 like this:


/* normalize the source */
pySource = PyString_FromString((const char *) source);
fixed_source = normalize_line_endings(pySource);
Py_DECREF(pySource);
if (fixed_source == NULL)
goto set_source_error;

/* compile the source into code object */
code = Py_CompileString(PyString_AsString(fixed_source), modname, Py_file_input);
Py_DECREF(fixed_source);
if(code == NULL)
goto set_source_error;

setPythonCode(code);

set_source_error:
if(PyErr_Occurred())
postException();

If I'm setting the source from a "frozen" module, I do it like this:

PyObject *source = PyMarshal_ReadObjectFromString((char *) data, size);
if(source == NULL)
{
PyErr_Print();
goto set_frozen_source_exit;
}

const char *c_source = PyString_AsString(source);
m_source = c_source;

PyObject *code = compile_source((const char *) pathname, source);
if(PyErr_Occurred())
{
PyErr_Print();
goto set_frozen_source_exit;
}

setPythonCode(code);


setPythonCode(code) looks like this:


PyObject *d = NULL; // module dict
PyObject *v = NULL; // tmp var

/* initialize builtins */
d = PyModule_GetDict(m_pyScriptModule);
if (PyDict_SetItemString(d, "__builtins__", PyEval_GetBuiltins()) != 0)
{
postException();
return;
}

/* Remember the filename as the __file__ attribute */
v = ((PyCodeObject *)code)->co_filename;
Py_INCREF(v);
if (PyDict_SetItemString(d, "__file__", v) != 0)
PyErr_Clear(); /* Not important enough to report */
Py_DECREF(v);


The method that I used was to manually create an empty python module from scratch and compile the code from the editor each time the run button was clicked. This way you can intercept the compilation and module construction before it gets added to sys.modules. Scripts are grouped by instrument in our app, so I made the rule that __import__ would allow a script to import another script from the same instrument. I did this using an import hook. Maybe I'll write about that later.

Embedding Code

A really, really cool chunk of code included with the python library is the "frozen" app template. There is a python file to compile python source stored in a const char * buffer and store the byte-code in a giant char array, which you can use in your app later on. I used this to embed the standard python lib into our application, allowing it to be an entirely self-contained exe. Go check out Tools/freeze, and you can thank me later.

Part of our application code is also written in python. This means that I wrote some small wrappers and some basic functions as a python script, and added custom build steps to Visual Studio and XCode to "freeze" them into compilable C code, which is executed manually right after Py_Initialize().

Static Data

Python relies heavily on static variables. My experience is that it's a bad idea to try to call Py_Initialize() and Py_Finalize() more than once. Use a singleton class with an instance() method to do all your python initialization, and ensure it is only done once. Use the atexit() function to make sure that your lib is broken down when the app exits, even if you are writing some kind of dll or plugin. This method will work for both windows and mac, and save you tons of headache.

Statically linked extension modules

Once you link the python standard lib into your exe, you won't be able to stop. You'll write a couple of modules and want to link those in, too. You'll want to include PyQt so you can write gui code for your app as well. I wrote the API for our scripting engine using sip to generate the code for the python module, and that code is statically linked as well.

This all seems pretty easy, but you need to add --disable-shared to your python configure flags, and you need to do some special things to get the interpreter to initialize the modules correctly.

First, explicitly declare the init functions and a 'struct _inittab' for the modules you statically linked:
extern "C"
{

void initsip();
void initmymod();
}

static struct _inittab builtin_modules[] = {
{ "sip", initsip },
{ "mymod", initmymod },
{ NULL, NULL }
};

Then, call the import function explicitly followed by with _PyImport_FixupExtension() to register it properly. This isn't documented anywhere, and I got the tip from some nice guy in #python-dev.

Thread safety

At first I compiled python with threads, like any well-minded person would do. But then I got sick of having to manage all of the locks and thread states myself, and eventually compiled python without threads and started using my own static mutex. The performance is the same, but our thread code is gone and we don't have any more problems with excentric thread inits when host apps run our junk as a plugin.

But, If you use blocking calls like read(), write(), or open(), or will use C extension modules, then using your own lock is probably a bad idea. Those calls and extensions will probably release the GIL when possible allowing other threads to get a little CPU, so you'll take a performance hit if you disable that. We never make calls outside our app, so using our own lock is fine.

One bit of advice that I would give to coders with strict performance requirements is to separate your scripting engine from your main application code by way of some kind of abstract interface. We run our python code from multiple audio threads, so contention for the GIL causes CPU spikes at low latencies - as would any lock held in an audio thread, which is why they say NEVER use locks in an audio thread.

Anyway, the only feasible solution to this is to migrate all of our calls to the python lib into another process. Defining a good, clean abstract interface to your scripting engine will make this a no-brainer when it comes time to implement an RPC layer.

The unfeasible solution to this would be to patch the python interpreter to allow for separate interpreters to live in separate threads. But we don't want to get into THAT one now do we? *wink*. Well, if you really have a taste for the lions' den, then here you go:

http://mail.python.org/pipermail/python-list/2008-October/512902.html



Notes

If anyone wants more information, I'd be more than happy to provide code examples and further explanation. This problem is a fun one if you know what you are doing!

9 comments:

earobinson said...

I would love to see another post on this

David Boddie said...

Thanks for writing up your process. I've linked to this page from the PyQt Wiki, so hopefully people looking to embed Python into Qt applications will find their way here from there:

http://www.diotavelli.net/PyQtWiki/Overviews_and_Guides


It would be good to have a page about this on the PyQt Wiki, but I'm not putting any pressure on you to write it. :-)

On the other hand, perhaps it would be a good idea to merge your advice with the existing information on the PythonInfo Wiki:

http://wiki.python.org/moin/EmbedingPyQtTutorial

Patricio said...

I wrote a full wiki here:

http://trac2.assembla.com/pkaudio/wiki/pkaudio

Anonymous said...

Awesome article. I have been looking all over for this. I appreciate you describing how to embed external modules for C/C++.

I would love to see a follow-on post as well!

viagra online said...

Very interesting application, It would be a excelllent tool for help some problems with different languages.

Viagra Online said...

Hey.. Thanks for sharing this, it was pretty useful 'cause I'm studying something related so it help me a lot.

jackanderson said...

Hi ,
Thanks For posting this I am looking for this from so may days ...

kamagra online

dugi's guide said...

wow great i have read many articles about this topic and everytime i learn something new i dont think it will ever stop always new info , Thanks for all of your hard work!
dugi's guide review

discount viagra said...

That was an interesting piece of information on handwriting analysis. Please post more about graphology. Thank you!