RError.com

RError.com Logo RError.com Logo

RError.com Navigation

  • 主页

Mobile menu

Close
  • 主页
  • 系统&网络
    • 热门问题
    • 最新问题
    • 标签
  • Ubuntu
    • 热门问题
    • 最新问题
    • 标签
  • 帮助
主页 / 问题 / 1006138
Accepted
Chorkov
Chorkov
Asked:2020-07-24 15:49:19 +0000 UTC2020-07-24 15:49:19 +0000 UTC 2020-07-24 15:49:19 +0000 UTC

并行调用 boost.python

  • 772

我正在尝试从使用 ThreadPoolExecutor 并行化的 python 代码中调用 C++ 代码。似乎在进入一个 C++ 函数的那一刻,所有的 Python 线程都被冻结了。例子:

#include <boost/python.hpp>
#include <iostream>
#include <iomanip>
#include <thread>
#include <chrono>

using namespace std;
using namespace std::chrono;
using namespace std::this_thread;
using namespace boost::python;

void foo(int i)
{
    cout<<"Enter " << " foo("<<i<<") (c++) thread: " << get_id() << endl;
    sleep_for(  seconds(2) );
    cout<<"Exit  " << " foo("<<i<<") (c++) thread: " << get_id() << endl;
}

BOOST_PYTHON_MODULE( test_py_async )
{
    def("foo", &foo);
}

Python:

import test_py_async
import concurrent.futures

N=8

def foo(i):
    print( f"Enter  foo({i}) (py)")
    test_py_async.foo(i)
    print( f"Exit   foo({i}) (py)")
    return i

with concurrent.futures.ThreadPoolExecutor(max_workers=N) as executor:

    print("submit tasks:")
    futures=[ executor.submit( foo, i ) for i in range(N) ]

    print("wait for tasks:")
    results=[ future.result() for future in futures ]

    print(f"results: {results}")

print(f"exit ThreadPoolExecutor")
assert( results == [ i for i in range(N) ] )

结论:

submit tasks:
Enter  foo(0) (py)
Enter  foo(0) (c++) thread: 11796
Exit   foo(0) (c++) thread: 11796
Enter  foo(1) (py)
Enter  foo(1) (c++) thread: 16580
Exit   foo(1) (c++) thread: 16580
Exit   foo(0) (py)
Enter  foo(2) (py)
Enter  foo(2) (c++) thread: 20240
Exit   foo(2) (c++) thread: 20240
Enter  foo(3) (py)
Exit   foo(1) (py)
Enter  foo(3) (c++) thread: 11796
Exit   foo(3) (c++) thread: 11796
Exit   foo(2) (py)
Enter  foo(4) (py)
Enter  foo(4) (c++) thread: 17568
Exit   foo(4) (c++) thread: 17568
Enter  foo(6) (py)
Enter  foo(6) (c++) thread: 19672
Exit   foo(6) (c++) thread: 19672
Enter  foo(5) (py)
Enter  foo(7) (py)
Enter  foo(7) (c++) thread: 16580
Exit   foo(7) (c++) thread: 16580
wait for tasks:
Exit   foo(3) (py)
Exit   foo(6) (py)
Exit   foo(7) (py)
Exit   foo(4) (py)
Exit   foo(5) (c++) thread: 16804
Exit   foo(5) (py)
results: [0, 1, 2, 3, 4, 5, 6, 7]
exit ThreadPoolExecutor

从日志中可以看出,c++ 调用在执行时间上不会与其他 c++ 调用或 Python 代码重叠,尽管这些调用确实来自不同的线程。我究竟做错了什么?

c++
  • 1 1 个回答
  • 10 Views

1 个回答

  • Voted
  1. Best Answer
    extrn
    2020-07-24T18:02:40Z2020-07-24T18:02:40Z

    从某种意义上说,Python 线程确实在扩展代码的持续时间内被冻结,其原因是GIL

    运行函数的线程在函数完成之前会foo抓取GIL并释放它。

    如果您的代码sleep从模块调用time,或访问特定于 python 的 I/O 例程,或以其他方式与解释器交互,GIL它可能在某个时候被释放,允许其他线程完全运行。

    您还可以通过使用特殊宏构建代码来手动释放它。

    Py_BEGIN_ALLOW_THREADS
    sleep_for(  seconds(2) );
    Py_END_ALLOW_THREADS
    

    大致相同的事情发生在time.sleep

    此类代码不应与 python 对象和/或访问交互API。
    这在文档中有更详细的描述。

    submit tasks:
    Enter  foo(0) (py)
    Enter  foo(0) (c++) thread: 140388371265280
    Enter  foo(1) (py)
    Enter  foo(1) (c++) thread: 140388362872576
    Enter  foo(2) (py)
    Enter  foo(2) (c++) thread: 140388354479872
    Enter  foo(3) (py)
    Enter  foo(3) (c++) thread: 140388346087168
    Enter  foo(4) (py)
    Enter  foo(4) (c++) thread: 140388337694464
    Enter  foo(5) (py)
    Enter  foo(5) (c++) thread: 140387985389312
    Enter  foo(6) (py)
    Enter  foo(6) (c++) thread: 140387976996608
    Enter  foo(7) (py)
    Enter  foo(7) (c++) thread: 140387968603904
    wait for tasks:
    Exit   foo(0) (c++) thread: 140388371265280
    Exit   foo(0) (py)
    Exit   foo(1) (c++) thread: 140388362872576
    Exit   foo(1) (py)
    Exit   foo(2) (c++) thread: 140388354479872
    Exit   foo(2) (py)
    Exit   foo(3) (c++) thread: 140388346087168
    Exit   foo(3) (py)
    Exit   foo(4) (c++) thread: 140388337694464
    Exit   foo(4) (py)
    Exit   foo(5) (c++) thread: 140387985389312
    Exit   foo(5) (py)
    Exit   foo(6) (c++) thread: 140387976996608
    Exit   foo(6) (py)
    Exit   foo(7) (c++) thread: 140387968603904
    Exit   foo(7) (py)
    results: [0, 1, 2, 3, 4, 5, 6, 7]
    exit ThreadPoolExecutor
    
    • 1

相关问题

Sidebar

Stats

  • 问题 10021
  • Answers 30001
  • 最佳答案 8000
  • 用户 6900
  • 常问
  • 回答
  • Marko Smith

    根据浏览器窗口的大小调整背景图案的大小

    • 2 个回答
  • Marko Smith

    理解for循环的执行逻辑

    • 1 个回答
  • Marko Smith

    复制动态数组时出错(C++)

    • 1 个回答
  • Marko Smith

    Or and If,elif,else 构造[重复]

    • 1 个回答
  • Marko Smith

    如何构建支持 x64 的 APK

    • 1 个回答
  • Marko Smith

    如何使按钮的输入宽度?

    • 2 个回答
  • Marko Smith

    如何显示对象变量的名称?

    • 3 个回答
  • Marko Smith

    如何循环一个函数?

    • 1 个回答
  • Marko Smith

    LOWORD 宏有什么作用?

    • 2 个回答
  • Marko Smith

    从字符串的开头删除直到并包括一个字符

    • 2 个回答
  • Martin Hope
    Alexandr_TT 2020年新年大赛! 2020-12-20 18:20:21 +0000 UTC
  • Martin Hope
    Alexandr_TT 圣诞树动画 2020-12-23 00:38:08 +0000 UTC
  • Martin Hope
    Air 究竟是什么标识了网站访问者? 2020-11-03 15:49:20 +0000 UTC
  • Martin Hope
    Qwertiy 号码显示 9223372036854775807 2020-07-11 18:16:49 +0000 UTC
  • Martin Hope
    user216109 如何为黑客设下陷阱,或充分击退攻击? 2020-05-10 02:22:52 +0000 UTC
  • Martin Hope
    Qwertiy 并变成3个无穷大 2020-11-06 07:15:57 +0000 UTC
  • Martin Hope
    koks_rs 什么是样板代码? 2020-10-27 15:43:19 +0000 UTC
  • Martin Hope
    Sirop4ik 向 git 提交发布的正确方法是什么? 2020-10-05 00:02:00 +0000 UTC
  • Martin Hope
    faoxis 为什么在这么多示例中函数都称为 foo? 2020-08-15 04:42:49 +0000 UTC
  • Martin Hope
    Pavel Mayorov 如何从事件或回调函数中返回值?或者至少等他们完成。 2020-08-11 16:49:28 +0000 UTC

热门标签

javascript python java php c# c++ html android jquery mysql

Explore

  • 主页
  • 问题
    • 热门问题
    • 最新问题
  • 标签
  • 帮助

Footer

RError.com

关于我们

  • 关于我们
  • 联系我们

Legal Stuff

  • Privacy Policy

帮助

© 2023 RError.com All Rights Reserve   沪ICP备12040472号-5