使用selenium获取不到response
在做scrapy+selenium+Phantomjs的爬虫时遇到一个问题,按照网上别人的爬虫自己做了一个,但是不能获取正确的
response,小白请大神帮忙看看哪出了问题?
输出的部分结果:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
<200 http://yuehui.163.com/searchusers.do>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
爬虫:
class YhSpider(scrapy.Spider):
name = 'yh'
allowed_domains = ['http://yuehui.163.com']
start_urls = ['http://yuehui.163.com/searchusers.do']
def parse(self, response):
print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
print(response)
print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
下载中间件:
class PageMiddleware(object):
def process_request(self, request, spider):
driver = webdriver.PhantomJS()
driver.get(request.url)
driver.find_element_by_xpath("//select[@name='province']/option[@value='2']").click()
driver.find_element_by_xpath("//div[@class='btnblock']/button[@id='commonsearchbtn']").click()
time.sleep(2)
content = driver.page_source.encode("utf-8")
driver.quit()
return HtmlResponse(request.url, encoding="utf-8",body=content, request=request)