python 爬取斗鱼直播视频信息，怎么翻页

[DoraU]

想爬取斗鱼的视频信息，http://www.douyutv.com/directory/game/LOL，需要翻页，利用chrome network得到地址为：http://www.douyutv.com/directory/game/LOL?page=1&isAjax=1，用python写程序就无法返回任何数据，是什么原因呢？

代码：
# -*- coding: utf-8 -*-
import scrapy
from scrapy.selector import Selector
from douyutv.items import VideoItem
import pymongo
from scrapy.selector import Selector

class VideoSpider(scrapy.Spider):
name = "video"
allowed_domains = ["http://www.douyutv.com"]
start_urls = []
def start_requests(self):
client = pymongo.MongoClient('mongodb://127.0.0.1')
db = client['douyutv']
items = db.CategoryItem_20160108102540.find({})
for item in items:
url = str('http://www.douyutv.com' + item['href'])
yield scrapy.http.Request(url=url, callback=self.parse, meta={'firstClass':item['firstClass'], 'secondClass':item['secondClass'], 'oriurl':url, 'page':1})

def parse(self, response):
item = VideoItem()
selector = Selector(response)
print response.meta['oriurl']
videos = selector.xpath('//div[@class="items items01 clearfix"]/ul/li')
allpage = int(selector.xpath('//input[@id="allPage"]/@value').extract()[0])
if response.meta['page'] <= allpage:
for video in videos:
title = video.xpath('a/@title').extract()[0]
href = video.xpath('a/@href').extract()[0]
author = video.xpath('a/div[@class="mes"]/div[2]/p[@class="moreMes"]/span[@class="nnt"]/text()').extract()
view = video.xpath('a/div[@class="mes"]/div[2]/p[@class="moreMes"]/span[@class="view"]/text()').extract()
item['firstClass'] = response.meta['firstClass']
item['secondClass'] = response.meta['secondClass']
item['title'] = title
if author:
item['author'] = author[0]
if view:
item['view'] = view[0]
item['href'] = href
yield item

page = response.meta['page'] + 1
url = response.meta['oriurl'] + '?page=' + str(page)

yield scrapy.http.Request(url=url, callback=self.parse, meta={'firstClass':response.meta['firstClass'], 'secondClass':response.meta['secondClass'], 'page':page, 'oriurl':response.meta['oriurl']})

[夜里的风]

你的程序太复杂了，我懒得看了 网址是这个没有错啦http://www.douyutv.com/directory/game/LOL?page=1&isAjax=1 page后面对应的参数是几就是第几页 你只要用urllib.open打开这个网页，得到response。read()的内容 再用正则式去匹配到你要的信息就好了。scarpy的框架有自己的格式，一般不是很大的爬虫的话，不建议用它，比较麻烦

[「已注销」]

phantomjs + scrapy