构建大规模账号池与本地部署：GitHub爬虫项目详解

账号池搭建

必要性

常见登录方式：

• 基于Session + Cookie的登录
• 基于JWT的登录：登录生成JWT字符串

账号池存储cookie或者JWT字符串 方便后续发请求爬取数据

本地部署

1. conda建立一个虚拟环境conda create -n new_env python=3.x # 替换 x 为你需要的 Python 版本
2. 激活新建环境conda activate new_env
3. 安装依赖项pip install -r requirments.txt
4. 修改setting.py配置文件- 配置redis数据库- # redis host REDIS_HOST = env.str('REDIS_HOST','127.0.0.1')# redis port REDIS_PORT = env.int('REDIS_PORT',6379)# redis password, if no password, set it to None REDIS_PASSWORD = env.str('REDIS_PASSWORD',None)# redis db, if no choice, set it to 0 REDIS_DB = env.int('REDIS_DB',0)- 配置检测网址- GENERATOR_MAP ={'antispider6':'Antispider6Generator','antispider7':'Antispider7Generator'}# integrated tester TESTER_MAP ={'antispider6':'Antispider6Tester','antispider7':'Antispider7Tester',} TEST_URL_MAP ={'antispider6':'https://antispider6.scrape.center/','antispider7':'https://antispider7.scrape.center/'}- 配置生成网址- defgenerate(self, username, password):""" generate main process """if self.credential_operator.get(username): logger.debug(f'credential of {username} exists, skip')return login_url ='https://antispider7.scrape.center/api/login' s = requests.Session() r = s.post(login_url, json={'username': username,'password': password })if r.status_code !=200: logger.error(f'error occurred while generating credential of {username}, error code {r.status_code}')return token = r.json().get('token') logger.debug(f'get credential {token}') self.credential_operator.set(username, token)
5. 配置账号密码的生成机制可以利用虚拟号接受验证码注册账号密码，需要花钱但不贵definit(self):""" do init """for i inrange(1, self.MAX_COUNT +1): self.account_operator.set(f'admin{i}',f'admin{i}')
6. 运行redis服务
7. 运行项目python run.py antispider7
8. 通过访问http://127.0.0.1:6379即可访问代理IP池的前台- //random：随机JWT字符串或者cookie- //count：数量

项目源码

GitHub - Python3WebSpider/AccountPool: Account Pool

更多精致内容:[CodeRealm]