Discussion:
[Python.NET] converting double array to numpy list
Mika S
2014-08-20 17:21:45 UTC
Permalink
How do I pass a double array as a numpy list from C# (calling python code
from c#) or passing something that can become a numpy list with minimal
overhead.

Right now, the naive way would be to create a new PyList. And to fill it
with elements from the double array.

But I read on a thread on this list (
https://mail.python.org/pipermail/pythondotnet/2014-May/001525.html) that
Marshal.copy was faster.
So, I tried Marshal.copy but I can't figure out how to instantiate a numpy
array in python. Numpy can instantiate a new array from native C array by
calling numpy.ctypeslib.as_array.
So, theoretically if i can do
Marshal.Copy(csharparray,0,addrofnative,csharparray.Length) and pass the
addrofnative (which is a IntPtr) I could be done. But I can only pass
PyObject when calling a module function.

Any ideas ?
Jeffrey Bush
2014-08-20 19:23:36 UTC
Permalink
Numpy arrays are almost always created in Python and there are dozens of
methods. That post you link to has the best solution for this case:

import numpy as np
dest = np.empty(len(src))

That creates a new numpy array of the same length of src with
undefined ("empty") values at every index. You could substitute len(src)
for whatever length you want (or a tuple if you want something not 1D). We
use empty since we are about to set every value to something so we don't
care that they have undefined values at that point.

Overall I don't understand your problem, can you be more clear? I don't
know if you are trying to do the copy in Python or C#, I am guessing you
are copying from C# array to a new Python NumPy array, which is exactly
what that post did (in Python).

Jeff
Post by Mika S
How do I pass a double array as a numpy list from C# (calling python code
from c#) or passing something that can become a numpy list with minimal
overhead.
Right now, the naive way would be to create a new PyList. And to fill it
with elements from the double array.
But I read on a thread on this list (
https://mail.python.org/pipermail/pythondotnet/2014-May/001525.html) that
Marshal.copy was faster.
So, I tried Marshal.copy but I can't figure out how to instantiate a numpy
array in python. Numpy can instantiate a new array from native C array by
calling numpy.ctypeslib.as_array.
So, theoretically if i can do
Marshal.Copy(csharparray,0,addrofnative,csharparray.Length) and pass the
addrofnative (which is a IntPtr) I could be done. But I can only pass
PyObject when calling a module function.
Any ideas ?
_________________________________________________
https://mail.python.org/mailman/listinfo/pythondotnet
Mika S
2014-08-20 21:45:25 UTC
Permalink
Thanks for replying. Here are some details that may be helpful in
understanding the problem.

I am using pythonnet to call python code from c#. This involves doing
something of this sort:

PythonEngine.Initialize();
PyObject testModule = PythonEngine.ImportModule("myModule");

var zero = new PyFloat(0.0);
var Lock = PythonEngine.AcquireLock();
var f = testModule.InvokeMethod("myModule", new PyObject[1] {
zero });
var g = (double)f[0].AsManagedObject(typeof(double));
PythonEngine.ReleaseLock(Lock);
PythonEngine.Shutdown();

This passed a python float with a value of 0 to the python module. Now if
my module, uses numpy arrays I need to pass at best a python list that can
be converted to a numpy array. The python list can be created using new
PyList and then assigning every double in the c# array into the pylist.
This should be inefficient I presume. So one way that I thought was to
marshal this into an unmanaged array and then pass the unmanaged array into
python. The python program can use numpy.ctypeslib.as_array on the
unmanaged array. Unfortunately, while calling python from c# I can only
pass a pyobject array while calling invokemethod - so that won't work.

In short: I want to call python code from a c# program. My python code uses
numpy arrays. I have a multidimensional array of doubles in the c#program
that I would like to pass to the python code.

Basically, the linked post already had the c# array in python using the clr
module. But in my case when I call python from c#, I do that using
InvokeMethod that would only allow me to pass an array of PyObjects. So
that's the challenge.
Post by Jeffrey Bush
Numpy arrays are almost always created in Python and there are dozens of
import numpy as np
dest = np.empty(len(src))
That creates a new numpy array of the same length of src with
undefined ("empty") values at every index. You could substitute len(src)
for whatever length you want (or a tuple if you want something not 1D). We
use empty since we are about to set every value to something so we don't
care that they have undefined values at that point.
Overall I don't understand your problem, can you be more clear? I don't
know if you are trying to do the copy in Python or C#, I am guessing you
are copying from C# array to a new Python NumPy array, which is exactly
what that post did (in Python).
Jeff
Post by Mika S
How do I pass a double array as a numpy list from C# (calling python code
from c#) or passing something that can become a numpy list with minimal
overhead.
Right now, the naive way would be to create a new PyList. And to fill it
with elements from the double array.
But I read on a thread on this list (
https://mail.python.org/pipermail/pythondotnet/2014-May/001525.html)
that Marshal.copy was faster.
So, I tried Marshal.copy but I can't figure out how to instantiate a
numpy array in python. Numpy can instantiate a new array from native C
array by calling numpy.ctypeslib.as_array.
So, theoretically if i can do
Marshal.Copy(csharparray,0,addrofnative,csharparray.Length) and pass the
addrofnative (which is a IntPtr) I could be done. But I can only pass
PyObject when calling a module function.
Any ideas ?
_________________________________________________
https://mail.python.org/mailman/listinfo/pythondotnet
_________________________________________________
https://mail.python.org/mailman/listinfo/pythondotnet
Jeffrey Bush
2014-08-21 23:04:19 UTC
Permalink
Hi,

So I have never used Python from C# (only the other way) however here are
my ideas:

1. numpy.ctypeslib.as_array is dangerous in this situation, you can only
use it safely on a pinned pointer, and as long as the returned array exists
you should not unpin that pointer, and pinned pointers should not be kept
around for very long. However, if you are using it quickly this is fine.
2. IntPtr is just a wrapper for a C# int or long, you can call the
ToInt64() function on it to get the value, and a long should easily be
sendable to Python.

The process would likely be the following (untested code):

C#:
var module = ...;
double[,] array = new double[,] { { ... } };
GCHandle pin = GCHandle.Alloc(array, GCHandleType.Pinned);
try:
long ptr = pin.AddrOfPinnedObject().ToInt64();
var output = module.InvokeMethod("func", new PyObject[3] { new
PyLong(ptr), new PyInt(array.GetLength(0)), new PyInt(array.GetLength(1))
});
finally:
if (pin.IsAllocated) { pin.Free(); }
// work with output here

Python:
def func(ptr, rows, cols):
array =
numpy.ctypeslib.as_array(ctypes.POINTER(ctypes.c_double).from_address(ptr),
(rows, cols))
# operate on array - can read and write, changing the C# array if
you write
output = numpy.sum(array) # for example
# after we return from this function, we can NEVER touch this
'array' again unless we copy the data in it
return output

Jeff
Post by Mika S
Thanks for replying. Here are some details that may be helpful in
understanding the problem.
I am using pythonnet to call python code from c#. This involves doing
PythonEngine.Initialize();
PyObject testModule = PythonEngine.ImportModule("myModule");
var zero = new PyFloat(0.0);
var Lock = PythonEngine.AcquireLock();
var f = testModule.InvokeMethod("myModule", new PyObject[1] {
zero });
var g = (double)f[0].AsManagedObject(typeof(double));
PythonEngine.ReleaseLock(Lock);
PythonEngine.Shutdown();
This passed a python float with a value of 0 to the python module. Now if
my module, uses numpy arrays I need to pass at best a python list that can
be converted to a numpy array. The python list can be created using new
PyList and then assigning every double in the c# array into the pylist.
This should be inefficient I presume. So one way that I thought was to
marshal this into an unmanaged array and then pass the unmanaged array into
python. The python program can use numpy.ctypeslib.as_array on the
unmanaged array. Unfortunately, while calling python from c# I can only
pass a pyobject array while calling invokemethod - so that won't work.
In short: I want to call python code from a c# program. My python code
uses numpy arrays. I have a multidimensional array of doubles in the
c#program that I would like to pass to the python code.
Basically, the linked post already had the c# array in python using the
clr module. But in my case when I call python from c#, I do that using
InvokeMethod that would only allow me to pass an array of PyObjects. So
that's the challenge.
Post by Jeffrey Bush
Numpy arrays are almost always created in Python and there are dozens of
import numpy as np
dest = np.empty(len(src))
That creates a new numpy array of the same length of src with
undefined ("empty") values at every index. You could substitute len(src)
for whatever length you want (or a tuple if you want something not 1D). We
use empty since we are about to set every value to something so we don't
care that they have undefined values at that point.
Overall I don't understand your problem, can you be more clear? I don't
know if you are trying to do the copy in Python or C#, I am guessing you
are copying from C# array to a new Python NumPy array, which is exactly
what that post did (in Python).
Jeff
Post by Mika S
How do I pass a double array as a numpy list from C# (calling python
code from c#) or passing something that can become a numpy list with
minimal overhead.
Right now, the naive way would be to create a new PyList. And to fill it
with elements from the double array.
But I read on a thread on this list (
https://mail.python.org/pipermail/pythondotnet/2014-May/001525.html)
that Marshal.copy was faster.
So, I tried Marshal.copy but I can't figure out how to instantiate a
numpy array in python. Numpy can instantiate a new array from native C
array by calling numpy.ctypeslib.as_array.
So, theoretically if i can do
Marshal.Copy(csharparray,0,addrofnative,csharparray.Length) and pass the
addrofnative (which is a IntPtr) I could be done. But I can only pass
PyObject when calling a module function.
Any ideas ?
_________________________________________________
https://mail.python.org/mailman/listinfo/pythondotnet
_________________________________________________
https://mail.python.org/mailman/listinfo/pythondotnet
_________________________________________________
https://mail.python.org/mailman/listinfo/pythondotnet
Dave Cook
2014-08-23 14:28:27 UTC
Permalink
On Wed, Aug 20, 2014 at 10:21 AM, Mika S <siddhupiddu-***@public.gmane.org> wrote:

How do I pass a double array as a numpy list from C# (calling python code
Post by Mika S
from c#) or passing something that can become a numpy list with minimal
overhead.
Based on input from an earlier thread here, I settled on using Marshal.Copy:

def ndarray_ptr(a): '''Return a .NET pointer to the underlying
ndarray data.''' return
IntPtr.__overloads__[int](a.__array_interface__['data'][0])def
marshal_to_ndarray(src): n = len(src) dest = np.empty(n,
dtype='f8')
Marshal.Copy(src, 0, ndarray_ptr(dest), n) return dest

Dave Cook

Loading...