主页 / 问题 / 1594345
Accepted
Artem Gafarov
Asked:2024-09-20 17:45:19 +0000 UTC2024-09-20 17:45:19 +0000 UTC

HTTPS 请求 VK 服务器下载视频

  • 772

我正在编写代码从 VK 下载视频，而不使用 VK API。任务如下：获取视频链接，然后向该 URL 发送请求并接收可观看该视频的页面作为响应。即模拟来自真实浏览器的请求。

为此，我尝试从真实的浏览器请求中复制所有标头，如下所示：

    req, err := http.NewRequest("GET", URL, nil)

    req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 YaBrowser/24.7.0.0 Safari/537.36")
    req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7")
    req.Header.Set("Accept-Encoding", "gzip, deflate, br, zstd")
    req.Header.Set("Accept-Language", "ru,en;q=0.9")
    req.Header.Set("Cache-Control", "max-age=0")

    req.Header.Set("Sec-Ch-Ua", "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"YaBrowser\";v=\"24.7\", \"Yowser\";v=\"2.5\"")
    req.Header.Set("Sec-Ch-Ua-Mobile", "?0")
    req.Header.Set("Sec-Ch-Ua-Platform", "\"macOS\"")
    req.Header.Set("Sec-Fetch-Dest", "document")
    req.Header.Set("Sec-Fetch-Mode", "navigate")
    req.Header.Set("Sec-Fetch-Site", "same-origin")

    req.Header.Set("Upgrade-Insecure-Requests", "1")

    req.Header.Set("Priority", "u=0, I")

作为回应，我希望看到一个带有视频的页面以供进一步工作，但实际上我收到了一个页面，其中要求我更新/安装最新版本的浏览器。在 Pastebin 上收到的响应示例：URL。页面的视觉展示

问题是：如何在 Go 中模拟来自“现代”浏览器的请求？

完整的代码如下所示：

const URL = "https://m.vk.com/video-85466065_456241727"

req, err := http.NewRequest("GET", URL, nil)

req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 YaBrowser/24.7.0.0 Safari/537.36")
    req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7")
req.Header.Set("Accept-Encoding", "gzip, deflate, br, zstd")
req.Header.Set("Accept-Language", "ru,en;q=0.9")
req.Header.Set("Cache-Control", "max-age=0")

req.Header.Set("Sec-Ch-Ua", "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"YaBrowser\";v=\"24.7\", \"Yowser\";v=\"2.5\"")
req.Header.Set("Sec-Ch-Ua-Mobile", "?0")
req.Header.Set("Sec-Ch-Ua-Platform", "\"macOS\"")
req.Header.Set("Sec-Fetch-Dest", "document")
req.Header.Set("Sec-Fetch-Mode", "navigate")
req.Header.Set("Sec-Fetch-Site", "same-origin")

req.Header.Set("Upgrade-Insecure-Requests", "1")

req.Header.Set("Priority", "u=0, i")

client := &http.Client{}

res, err := client.Do(req)
golang

1 个回答

  1. Best Answer

    在我看来，您从服务器得到了正确的响应。浏览器也会接收它，但它们执行 JS 脚本并重新加载页面。

    证明：禁用 JS 的 chrome 截图 在此输入图像描述

    恕我直言，事情是这样的。服务器返回以下 cookie 集：

    remixlang=0; Path=/; Domain=vk.com; Expires=Sat, 20 Sep 2025 22:04:34 GMT; Secure; SameSite=None
remixstlid=9078596624758655055_ZqB9FcuPZghch6BGgDDoHd6B6dWFHp8zzggYkZA9Z4c; Path=/; Domain=vk.com; Expires=Sun, 21 Sep 2025 10:30:22 GMT; Secure; SameSite=None
remixstid=2019772071_3Tf1dSfwaoxqIPj5Cp2cBYNVL3TxXfC9sHqzoRmsRws; Path=/; Domain=vk.com; Expires=Wed, 24 Sep 2025 19:51:47 GMT; Secure; SameSite=None
remixua=15%7C612%7C171%7C3990382815; Path=/; Domain=vk.com; Expires=Wed, 17 Sep 2025 21:01:56 GMT; Secure; SameSite=None
remixff=0; Path=/; Domain=vk.com; Expires=Tue, 01 Oct 2024 10:30:22 GMT; Secure; SameSite=None
remixua=15%7C612%7C171%7C3990382815; Path=/; Domain=vk.com; Expires=Wed, 24 Sep 2025 21:52:38 GMT; Secure; SameSite=None

    但是浏览器发送这些 cookie（cookie 的值不匹配，因为有些是在程序中接收的，有些是从浏览器接收的 - 比较名称集）：

    remixlang=0; 
remixua=43%7C-1%7C214%7C2996390830;
remixstlid=9063256236099454727_RxskMlmfWzownbJe8AmH8YjmlSPWFAlQwzMC43sIbpD;
remixstid=359988006_2z3dEIQUE5D2Ea0lRSz5SqCIS4JgGf2rqQIhQKJLkIo;
remixff=0;
remixmdevice=2560/1440/1/!!-!!!!!!!!/1302

    remixmdevice浏览器不是从 header接收 cookie Set-Cookie，而是从页面中的脚本接收 cookie。如果没有这个 cookie，服务器会显示一个“Bad Browser”页面，有了它就会显示一个正常页面

    我尝试用手添加这个饼干，结果成功了。这是我的标题集：

        req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7")
    req.Header.Set("Accept-Encoding", "gzip, deflate, br, zstd")
    req.Header.Set("Accept-Language", "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,ko-KR;q=0.6")
    req.Header.Set("Cache-Control", "no-cache")

    // req.Header.Set("Cookie", `remixlang=0; remixua=43%7C-1%7C214%7C2996390830; remixstlid=9063256236099454727_RxskMlmfWzownbJe8AmH8YjmlSPWFAlQwzMC43sIbpD; remixstid=359988006_2z3dEIQUE5D2Ea0lRSz5SqCIS4JgGf2rqQIhQKJLkIo; remixff=0; remixmdevice=2560/1440/1/!!-!!!!!!!!/1302`)

    req.Header.Set("DNT", "1")
    req.Header.Set("Pragma", "no-cache")
    req.Header.Set("Priority", "u=0, i")
    req.Header.Set("Sec-Ch-Ua", `"Chromium";v="128", "Not;A=Brand";v="24", "Google Chrome";v="128"`)
    req.Header.Set("Sec-Ch-Ua-Mobile", "?0")
    req.Header.Set("Sec-Ch-Ua-Platform", `"Android"`)
    req.Header.Set("Sec-Fetch-Dest", "document")
    req.Header.Set("Sec-Fetch-Mode", "navigate")
    req.Header.Set("Sec-Fetch-Site", "same-origin")
    req.Header.Set("Upgrade-Insecure-Requests", "1")
    req.Header.Set("User-Agent", "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.6668.54 Mobile Safari/537.36")

    req.AddCookie(&http.Cookie{Name: "remixmdevice", Value: "2560/1440/1/!!-!!!!!!!!/1302"})

    我不知道如何动态形成这个cookie。但目前它正在发挥作用。

    附言。作为一个成年人，你当然应该使用 Selenium，但这真是一个魔鬼阿尔巴......

    • 1

相关问题