python爬虫进阶篇：Scrapy中使用Selenium+Firefox浏览器爬取沪深A股股票行情

一、前言

上篇记录了Scrapy搭配selenium的使用方法，有了基本的了解后我们可以将这项技术落实到实际需求中。目前很多股票网站的行情信息都是动态数据，我们可以用Scrapy+selenium对股票进行实时采集并持久化，再进行数据分析、邮件通知等操作。

二、环境搭建

详情请看上篇笔记

三、代码实现

• items

classStockSpiderItem(scrapy.Item):# define the fields for your item here like:# name = scrapy.Field()# 股票代码
    stock_code = scrapy.Field()# 股票名称
    stock_name = scrapy.Field()# 最新价
    last_price = scrapy.Field()# 涨跌幅
    rise_fall_rate = scrapy.Field()# 涨跌额
    rise_fall_price = scrapy.Field()

• middlewares

def__init__(self):# ----------------firefox的设置------------------------------- #
        self.options = firefox_options()defspider_opened(self, spider):
        spider.logger.info('Spider opened: %s'% spider.name)
        spider.driver = webdriver.Firefox(options=self.options)# 指定使用的浏览器defprocess_request(self, request, spider):# Called for each request that goes through the downloader# middleware.# Must either:# - return None: continue processing this request# - or return a Response object# - or return a Request object# - or raise IgnoreRequest: process_exception() methods of#   installed downloader middleware will be called

        spider.driver.get("https://quote.eastmoney.com/center/gridlist.html#hs_a_board")returnNonedefprocess_response(self, request, response, spider):# Called with the response returned from the downloader.# Must either;# - return a Response object# - return a Request object# - or raise IgnoreRequest
        response_body = spider.driver.page_source

        return HtmlResponse(url=request.url, body=response_body, encoding='utf-8', request=request)

• settings设置

# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
SPIDER_MIDDLEWARES ={'stock_spider.middlewares.StockSpiderSpiderMiddleware':543,}# Enable or disable downloader middlewares# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES ={'stock_spider.middlewares.StockSpiderDownloaderMiddleware':543,}

• spider文件

defparse(self, response):# 股票代码
        stock_code = response.css("table.table_wrapper-table tbody tr td:nth-child(2) a::text").extract()# 股票名称
        stock_name = response.css("table.table_wrapper-table tbody tr td:nth-child(3) a::text").extract()# 最新价
        last_price = response.css("table.table_wrapper-table tbody tr td:nth-child(5) span::text").extract()# 涨跌幅
        rise_fall_rate = response.css("table.table_wrapper-table tbody tr td:nth-child(6) span::text").extract()# 涨跌额
        rise_fall_price = response.css("table.table_wrapper-table tbody tr td:nth-child(7) span::text").extract()for i inrange(len(stock_code)):
            item = StockSpiderItem()
            item["stock_code"]= stock_code[i]
            item["stock_name"]= stock_name[i]
            item["last_price"]= last_price[i]
            item["rise_fall_rate"]= rise_fall_rate[i]
            item["rise_fall_price"]= rise_fall_price[i]yield item

    defclose(self, spider):
        spider.driver.quit()

• pipelines持久化

defprocess_item(self, item, spider):"""
        接收到提交过来的对象后，写入csv文件
        """
        filename =f'stock_info.csv'withopen(filename,'a+', encoding='utf-8')as f:
            line = item["stock_code"]+","+ item["stock_name"]+","+ item["last_price"]+","+ \
                   item["rise_fall_rate"]+","+ item["rise_fall_price"]+"\n"
            f.write(line)return item

• readme文件

1.安装依赖包 
- python 3.0+- pip install -r requirements.txt

2.将最第二层stock_spider文件夹设置为根目录

3.将firefox驱动程序包放到python环境的Scripts文件夹里

4.必须要安装firefox浏览器才会调用到浏览器

5.执行spider_main.py文件启动爬虫