selenium 捕获网络请求

写在前面

之前分享过CDP的知识，现在有一个需求需要在selenium自动化时捕获网络请求，这里记录一下。

书接上回：浏览器自动化必须知道CDP协议

实现

开启日志记录

首先，我们要开启浏览器的日志记录，首先需要配置一个capabilities，它允许定义浏览器的一些特性。

import json
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service

chrome_option = Options()

chrome_option.set_capability("goog:loggingPrefs",{"performance":"ALL"})

service = Service("./chromedriver-win64/chromedriver.exe")

driver = webdriver.Chrome(service=service, options=chrome_option)

在最新的selenium的版本中，是通过Options()去配置浏览器的一些属性，在这里通过配置

'goog:loggingPrefs': {'performance': 'ALL'}

来打开浏览器的性能日志记录。

获取日志

接下来我们尝试获取一下访问百度的日志

driver.get("https://www.baidu.com")

performance_log = driver.get_log("performance")

在访问百度后，通过get_log(“performance”)来获取性能日志，这是一个由字典组成的列表。这里我打印一个看一下格式

{'level':'INFO','message':'{"message":{"method":"Network.requestWillBeSent","params":{"documentURL":"https://www.baidu.com/","frameId":"62715239374117F099DBA348C45736CD","hasUserGesture":false,"initiator":{"type":"other"},"loaderId":"C5F286A8A5744DEEB277D0718C4E34E8","redirectHasExtraInfo":false,"request":{"headers":{"Upgrade-Insecure-Requests":"1","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36","sec-ch-ua":"\"Not/A)Brand\";v=\"99\", \"Google Chrome\";v=\"115\", \"Chromium\";v=\"115\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\""},"initialPriority":"VeryHigh","isSameSite":true,"method":"GET","mixedContentType":"none","referrerPolicy":"strict-origin-when-cross-origin","url":"https://www.baidu.com/"},"requestId":"C5F286A8A5744DEEB277D0718C4E34E8","timestamp":1508.943033,"type":"Document","wallTime":1692620664.219256}},"webview":"62715239374117F099DBA348C45736CD"}','timestamp':1692620664216}

可以看到，关键的信息都在message中，注意message中是一个json字段

{"message":{"method":"Network.requestWillBeSent","params":{"documentURL":"https://www.baidu.com/","frameId":"62715239374117F099DBA348C45736CD","hasUserGesture":false,"initiator":{"type":"other"},"loaderId":"C5F286A8A5744DEEB277D0718C4E34E8","redirectHasExtraInfo":false,"request":{"headers":{"Upgrade-Insecure-Requests":"1","User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36","sec-ch-ua":"\"Not/A)Brand\";v=\"99\", \"Google Chrome\";v=\"115\", \"Chromium\";v=\"115\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\""},"initialPriority":"VeryHigh","isSameSite":true,"method":"GET","mixedContentType":"none","referrerPolicy":"strict-origin-when-cross-origin","url":"https://www.baidu.com/"},"requestId":"C5F286A8A5744DEEB277D0718C4E34E8","timestamp":1508.943033,"type":"Document","wallTime":1692620664.219256}},"webview":"62715239374117F099DBA348C45736CD"}

这是一个请求包的示例，大家看到method凡是以Network开头的都是网络请求。

获取网络返回包

我这个需求是想获取一个请求的返回包的数据，其实本身在performance日志中也有Network.response*相关的日志，但不是完整的请求包

所以，就需要requesetId这个字段了，通过CDP来获取

message = json.loads(packet.get("message")).get("message")
packet_method = message.get("method")if"Network"in packet_method:
    request_id = message.get("params").get("requestId")
    resp = driver.execute_cdp_cmd('Network.getResponseBody',{'requestId': request_id})
    body = resp.get("body")

这里就可以将返回包完整的获取到了

如果有定制化需求，比如想获取某个链接的返回包等等，都可以去通过分析数据通过条件判断来处理

完整代码

最后附上完整测试代码

import json
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service

chrome_option = Options()

chrome_option.set_capability("goog:loggingPrefs",{"performance":"ALL"})

service = Service("./chromedriver-win64/chromedriver.exe")

driver = webdriver.Chrome(service=service, options=chrome_option)

driver.get("https://www.baidu.com")

performance_log = driver.get_log("performance")for packet in performance_log:
    message = json.loads(packet.get("message")).get("message")
    packet_method = message.get("method")if"Network"in packet_method:
        request_id = message.get("params").get("requestId")try:
            resp = driver.execute_cdp_cmd('Network.getResponseBody',{'requestId': request_id})
            body = resp.get("body")print(body)except:pass