我正在尝试从使用 ThreadPoolExecutor 并行化的 python 代码中调用 C++ 代码。似乎在进入一个 C++ 函数的那一刻,所有的 Python 线程都被冻结了。例子:
#include <boost/python.hpp>
#include <iostream>
#include <iomanip>
#include <thread>
#include <chrono>
using namespace std;
using namespace std::chrono;
using namespace std::this_thread;
using namespace boost::python;
void foo(int i)
{
cout<<"Enter " << " foo("<<i<<") (c++) thread: " << get_id() << endl;
sleep_for( seconds(2) );
cout<<"Exit " << " foo("<<i<<") (c++) thread: " << get_id() << endl;
}
BOOST_PYTHON_MODULE( test_py_async )
{
def("foo", &foo);
}
Python:
import test_py_async
import concurrent.futures
N=8
def foo(i):
print( f"Enter foo({i}) (py)")
test_py_async.foo(i)
print( f"Exit foo({i}) (py)")
return i
with concurrent.futures.ThreadPoolExecutor(max_workers=N) as executor:
print("submit tasks:")
futures=[ executor.submit( foo, i ) for i in range(N) ]
print("wait for tasks:")
results=[ future.result() for future in futures ]
print(f"results: {results}")
print(f"exit ThreadPoolExecutor")
assert( results == [ i for i in range(N) ] )
结论:
submit tasks:
Enter foo(0) (py)
Enter foo(0) (c++) thread: 11796
Exit foo(0) (c++) thread: 11796
Enter foo(1) (py)
Enter foo(1) (c++) thread: 16580
Exit foo(1) (c++) thread: 16580
Exit foo(0) (py)
Enter foo(2) (py)
Enter foo(2) (c++) thread: 20240
Exit foo(2) (c++) thread: 20240
Enter foo(3) (py)
Exit foo(1) (py)
Enter foo(3) (c++) thread: 11796
Exit foo(3) (c++) thread: 11796
Exit foo(2) (py)
Enter foo(4) (py)
Enter foo(4) (c++) thread: 17568
Exit foo(4) (c++) thread: 17568
Enter foo(6) (py)
Enter foo(6) (c++) thread: 19672
Exit foo(6) (c++) thread: 19672
Enter foo(5) (py)
Enter foo(7) (py)
Enter foo(7) (c++) thread: 16580
Exit foo(7) (c++) thread: 16580
wait for tasks:
Exit foo(3) (py)
Exit foo(6) (py)
Exit foo(7) (py)
Exit foo(4) (py)
Exit foo(5) (c++) thread: 16804
Exit foo(5) (py)
results: [0, 1, 2, 3, 4, 5, 6, 7]
exit ThreadPoolExecutor
从日志中可以看出,c++ 调用在执行时间上不会与其他 c++ 调用或 Python 代码重叠,尽管这些调用确实来自不同的线程。我究竟做错了什么?
从某种意义上说,Python 线程确实在扩展代码的持续时间内被冻结,其原因是GIL
运行函数的线程在函数完成之前会
foo
抓取GIL
并释放它。如果您的代码
sleep
从模块调用time
,或访问特定于 python 的 I/O 例程,或以其他方式与解释器交互,GIL
它可能在某个时候被释放,允许其他线程完全运行。您还可以通过使用特殊宏构建代码来手动释放它。
大致相同的事情发生在
time.sleep
此类代码不应与 python 对象和/或访问交互
API
。这在文档中有更详细的描述。