Scrapy的WebDriverWait问题

2017-03-28 02:17:16 308 阅读 AI 写同款 · GEO 优化AI 选题数据监测一键分发 ›

Scrapy 2.11 反爬虫实战：5种策略绕过验证码与IP封锁（附Selenium集成代码）本文详细介绍了Scrapy 2.11中五种实战验证的反爬策略，包括动态渲染与Selenium集成、智能代理IP池构建、验证码破解、请求指纹混淆技术以及分布式爬虫架构设计。通过具体代码示例和优化技巧，帮助开发者有效绕过验证码与IP封锁，提升数据采集效率。阅读详情



from datetime import datetime

from scrapy.spiders import CrawlSpider, Rule

from scrapy.shell import inspect_response

from tencent_app.items import TencentAppItem

import time

import scrapy

from selenium import webdriver

from selenium.webdriver.support.ui import WebDriverWait



class Yingyongbao(CrawlSpider):

    name = 'yingyongbao_app'

    def __init__(self):

        super(Yingyongbao,self).__init__()

        bid = ['qq']

        self.start_urls = ['http://sj.qq.com/myapp/search.htm?kw=%s' % i for i in bid]

        self.url = 'http://sj.qq.com/myapp/detail.htm?apkName='

        self.driver = webdriver.Firefox()



    def parse(self,response):

        url_set = set()

        self.driver.get(response.url)

        while True:

            try:

                wait = WebDriverWait(self.driver, 2)

                wait.until(lambda driver: driver.find_element_by_xpath('/html/body/div[3]/div[3]/div[2]/a'))

                #time.sleep(1)

                next_page = self.driver.find_element_by_xpath('/html/body/div[3]/div[3]/div[2]/a/i')

                next_page.click()

            except:

                break

我写的翻页程序，模拟浏览器，等待下一页标签出现，不断点击下一页的形式。但我不明白为什么wait = WebDriverWait(self.driver, 12,14)，wait.until(lambda driver: driver.find_element_by_xpath('/html/body/div[3]/div[3]/div[2]/a'))两行代码不管用，会报错

...全文