主页 / user-615386

TaPo4ek18's questions

Martin Hope
TaPo4ek18
Asked: 2024-08-15 22:09:02 +0000 UTC
  • 6

我正在将解析器优化为异步。服务器的响应是 500，但通过常规请求一切正常。有什么问题吗？当以通常的“同步”方法运行代码时，一切正常，但需要花费很多时间，因为......有很多数据。下面是我尝试异步的一段代码。据我了解，问题出在请求中，但我可能是错的。为了进行测试，我选择了 2 个产品类别，并显示带有 JSON 响应的 data_list 以查看其中的内容。

headers = {
    "Accept": "application/json, text/plain, */*",
    "Accept-Language": "en-U",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 YaBrowser/24.7.0.0 Safari/537.36"
}

data_list = []


async def get_page_data(session, id, page):        
    url=f"https://shop.deere.com/bff/v1/search?categoryId={id}&page={page}&size=24&productType=all&countryCode=US"

    async with session.post(url=url, headers=headers) as response:
        print(f"{id}/{page} data collected")
        data = await response.json()
        data_list.append(data)
        return data_list

async def gather_data():
    
    ids_for_query = ['ServiceMaintenanceKits', '5081']
    pages_for_query = [12, 27]
    
    async with aiohttp.ClientSession() as session:
        
        tasks = []

        for id, total_pages in zip(ids_for_query, pages_for_query):
            for page in range(1, total_pages + 1):
                    
                task = asyncio.create_task(get_page_data(session, id, page))
                tasks.append(task)

        await asyncio.gather(*tasks)
            

def main():
    asyncio.run(gather_data())
    print(data_list)

if __name__ == "__main__":
    main()

在输出中，我对每个请求都有一个答案：

{'errors': [{'type': 'Internal Server Error', 'description': None, 'details': [{'field': None, 'code': 'JD-COMMON-006', 'message': "Content-Type 'application/octet-stream;charset=UTF-8' is not supported"}

告诉我如何进行这项工作？

python