[Pyublas] PyUblas with embedded python interpreter

[jsilva]

Hi,

I'm trying to use pyublas in a program that embeds the python
interpreter. I'm doing this because I couldn't find a simple and
effective way of calling C++ code from python that will allow the use of
OpenMP (using OpenMP from an extension wasn't stable for some reason).
I'm getting a segfault that I've reduced to the following:

/***************************************************/
#include <pyublas/numpy.hpp>

using namespace boost::python;

int main( int argc, char ** argv ) {
return 0;
}

/***************************************************/


I compile this with bjam to get an executable. Running it results in a
segfault:
#0 0x00000000 in ?? ()
#1 0xb7df9297 in PyEval_GetGlobals () at ../Python/ceval.c:3524
#2 0xb7e15af7 in PyImport_Import (module_name='numpy.core.multiarray')
at ../Python/import.c:2739
#3 0xb7e15d62 in PyImport_ImportModule (name=0x8048fe1
"numpy.core.multiarray")
at ../Python/import.c:2044
#4 0x08048974 in _import_array ()
at /usr/include/python2.6/numpy/__multiarray_api.h:1190
#5 0x08048b9b in _pyublas_array_importer (this=0x804a6e4)
at /usr/local/include/pyublas/numpy.hpp:47
#6 0x08048ce6 in __static_initialization_and_destruction_0
(__initialize_p=1,
__priority=65535) at /usr/local/include/pyublas/numpy.hpp:49
#7 0x08048d06 in global constructors keyed to python_ext.cpp ()
at python_ext.cpp:7
#8 0x08048fad in __do_global_ctors_aux ()
#9 0x080487d4 in _init ()
#10 0x08048f49 in __libc_csu_init ()
#11 0xb750dc14 in __libc_start_main () from /lib/i686/cmov/libc.so.6
#12 0x080488c1 in _start ()


Some searching sugests PyEval_GetGlobals only works when called from
within a python interpreter. Any tips on this? Does pyublas support
embedding (instead of extension modules)?

Regards,
Jo?o Silva

[jsilva]

I found your Oct/22 2009 reply about the same problem
(http://lists.tiker.net/pipermail/pyublas/2009-October/000055.html).
I've solved it by removing import_array from numpy.hpp and calling it
from Python. Wouldn't it be better to modify PyUblas so that it calls
import_array when the pyublas module is imported? As far as I know all
programs using PyUblas import that module, so this change would be
backward compatible.

Regards,
Jo?o Silva

[Andreas Kloeckner]

Hi Jo?o,

first off, sorry for the long wait for a reply--I've recently moved to a
new job and am just slowly working my way through the backlog.

Post by jsilva
I found your Oct/22 2009 reply about the same problem
(http://lists.tiker.net/pipermail/pyublas/2009-October/000055.html).
I've solved it by removing import_array from numpy.hpp and calling it
from Python. Wouldn't it be better to modify PyUblas so that it calls
import_array when the pyublas module is imported? As far as I know all
programs using PyUblas import that module, so this change would be
backward compatible.

I understand your argument, but moving import_array() to the module's
import sequence is actually not correct. Why? In a setting where all
modules are dynamically loaded (the usual case, really), every module
that makes numpy calls needs to call import_array(). This includes every
module that uses the facilities pyublas/numpy.hpp. As such, putting the
import only in pyublas's module init is not sufficient--all other
modules would need to do the same thing, manually. Perhaps that's
feasible, but it certainly wouldn't be backward compatible. The static
constructor is an easy way of ensuring that import_array() is called in
every separate dynamic module.

If you know of a C macro that is only defined when an embedded Python
module is built, we could use that as a switch between the current and
your suggested behavior. Or, if you have another idea, I'd also be open
to that.

Thanks for your input,
Andreas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://lists.tiker.net/pipermail/pyublas/attachments/20100904/f1e718c7/attachment.pgp>

[Andreas Kloeckner]

Post by Andreas Kloeckner
Hi Jo?o,
first off, sorry for the long wait for a reply--I've recently moved to a
new job and am just slowly working my way through the backlog.

[lines deleted]

Post by Andreas Kloeckner
If you know of a C macro that is only defined when an embedded Python
module is built, we could use that as a switch between the current and
your suggested behavior. Or, if you have another idea, I'd also be open
to that.
Thanks for your input,
Andreas

Thank you for your reply. Please disregard my suggestion to change where
import_array is called. For a few weeks I was trying to change my main
programming language to build computationally intensive simulations from
C++ (using OpenMP) to a mix of Python and C++. All the different ways
I've tried to do that failed, so I gave up went back to C++. IIRC I've
Python+Boost:Python+PyUblas+OpenMP (in a C++ extension) -> Never stable
enough, I think PyUblas isn't thread safe.

I have also tried this combination, with similar results. I actually do
not think that this is PyUblas's fault (there's nothing that could be
thread-dangerous), but rather some issue with using OpenMP in a C
extension. This hyopthesis would be easy to test--do you still have code
with which you could verify this?

Pyhton+MPI -> Spends too much time communicating, not fast enough.

This seems odd--MPI4Py (I assume that's what you were using) is just a
thin bit of glue around MPI (as long as you're handling numpy objects,
which should end up being the raw pointers handed to MPI).

Python+Boost:Python+ndarray (http://code.google.com/p/ndarray/)+OpenMP
-> ndarray doesn't seem to be thread safe.

Which would confirm the above.

Any insight you can provide would be much appreciated.

Andreas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://lists.tiker.net/pipermail/pyublas/attachments/20100905/5d823742/attachment.pgp>

[jsilva]

Post by Andreas Kloeckner

Post by Andreas Kloeckner
Hi Jo?o,
first off, sorry for the long wait for a reply--I've recently moved to a
new job and am just slowly working my way through the backlog.

[lines deleted]

Post by Andreas Kloeckner
If you know of a C macro that is only defined when an embedded Python
module is built, we could use that as a switch between the current and
your suggested behavior. Or, if you have another idea, I'd also be open
to that.
Thanks for your input,
Andreas

Thank you for your reply. Please disregard my suggestion to change where
import_array is called. For a few weeks I was trying to change my main
programming language to build computationally intensive simulations from
C++ (using OpenMP) to a mix of Python and C++. All the different ways
I've tried to do that failed, so I gave up went back to C++. IIRC I've
Python+Boost:Python+PyUblas+OpenMP (in a C++ extension) -> Never stable
enough, I think PyUblas isn't thread safe.

I have also tried this combination, with similar results. I actually do
not think that this is PyUblas's fault (there's nothing that could be
thread-dangerous), but rather some issue with using OpenMP in a C
extension. This hyopthesis would be easy to test--do you still have code
with which you could verify this?

All the tests I've did seem to indicate that it only gives trouble when
I'm trying to access data from python. In these cases it fails with
errors like these:

*** glibc detected *** ./python_ext: double free or corruption
(fasttop): 0x00000000031dddb0 ***
Segmentation fault

I've seen a thread on StackOverflow (
http://stackoverflow.com/questions/1934898/python-threads-the-gil-and-c
) that a call to PyEval_InitThreads() is required. I also added the
macros Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS at the beginning
and end of the parallel sections, but none of this fixed the problem.
The errors look like a mismatch of INCREFs and DECREFs leading to
erroneous reference counts. If both PyUblas and ndarray are thread safe
my next guess would be that Boost:Python isn't. Writing a C++ extension
without any library support such as Boost:Python doesn't seem to be
worth the trouble...

Post by Andreas Kloeckner

Pyhton+MPI -> Spends too much time communicating, not fast enough.

This seems odd--MPI4Py (I assume that's what you were using) is just a
thin bit of glue around MPI (as long as you're handling numpy objects,
which should end up being the raw pointers handed to MPI).

I don't think the python MPI wrapper is to blame here, it's just due to
different nature of shared memory and message passing. For problems that
have to exchange lots of data MPI will inevitably be slower than shared
memory parallelism. On the other hand while MPI can use many machines
I'm limited to just one with OpenMP.

Post by Andreas Kloeckner

Python+Boost:Python+ndarray (http://code.google.com/p/ndarray/)+OpenMP
-> ndarray doesn't seem to be thread safe.

Which would confirm the above.
Any insight you can provide would be much appreciated.
Andreas

I'm afraid I can't clarify this much more. I've searched the internet
without much luck, this seems to be a rather specialized topic, and the
few posts I've seen about this present problems and not solutions.

Regards,
Jo?o Silva

[Andreas Kloeckner]

Post by jsilva
All the tests I've did seem to indicate that it only gives trouble when
I'm trying to access data from python. In these cases it fails with
*** glibc detected *** ./python_ext: double free or corruption
(fasttop): 0x00000000031dddb0 ***
Segmentation fault

Hum. It's true that PyUblas doesn't try to get the GIL when it handles
Python objects, so that right there is cause for concern. I always lived
in the illusion that if you only use iterators and data access in the
threads, you should be fine, but I guess even that needs to be
questioned. Obtaining iterators should be guarded by GIL locks I guess,
but once you've obtained them, you should (theoretically) be fine, as
iterators are just pointers. Putting the locking into PyUblas would be a
bit of a mess, as Boost.Python accesses the Python in many places
(destructors for instance) implicitly.

Post by jsilva
I've seen a thread on StackOverflow (
http://stackoverflow.com/questions/1934898/python-threads-the-gil-and-c
) that a call to PyEval_InitThreads() is required. I also added the
macros Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS at the beginning
and end of the parallel sections, but none of this fixed the problem.
The errors look like a mismatch of INCREFs and DECREFs leading to
erroneous reference counts. If both PyUblas and ndarray are thread safe
my next guess would be that Boost:Python isn't. Writing a C++ extension
without any library support such as Boost:Python doesn't seem to be
worth the trouble...

Post by Andreas Kloeckner

Pyhton+MPI -> Spends too much time communicating, not fast enough.

This seems odd--MPI4Py (I assume that's what you were using) is just a
thin bit of glue around MPI (as long as you're handling numpy objects,
which should end up being the raw pointers handed to MPI).

I don't think the python MPI wrapper is to blame here, it's just due to
different nature of shared memory and message passing. For problems that
have to exchange lots of data MPI will inevitably be slower than shared
memory parallelism. On the other hand while MPI can use many machines
I'm limited to just one with OpenMP.

Ah, ok--I misunderstood that. Thanks for clarifying.

Post by jsilva
I'm afraid I can't clarify this much more. I've searched the internet
without much luck, this seems to be a rather specialized topic, and the
few posts I've seen about this present problems and not solutions.

Thanks for sharing your experience. I'll let you know if I come up with
a solution.

Andreas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://lists.tiker.net/pipermail/pyublas/attachments/20100915/2d0a6e6d/attachment.pgp>