37,743
社区成员
发帖
与我相关
我的任务
分享python爬取淘宝评论
因为项目需要，要爬一个产品的五万多条评论，可是网上评论只能看99页，总共不到2000条，要怎么样才能把那五万多条数据都爬出来啊啊啊
...全文
请发表友善的回复…
发表回复
1学习者1 2020-02-07
- 打赏
- 举报
爬取淘宝的评论有分 淘宝集市的接口和天猫的接口，两者要传的商品字段也不一样，这是天猫店评论数据的基础代码
import requests
import json
url="https://rate.tmall.com/list_detail_rate.htm"
header={
"cookie":"cna=EYnEFeatJWUCAbfhIw4Sd0GO; x=__ll%3D-1%26_ato%3D0; hng=CN%7Czh-CN%7CCNY%7C156; uc1=cookie14=UoTaHYecARKhrA%3D%3D; uc3=vt3=F8dBy32hRyZzP%2FF7mzQ%3D&lg2=U%2BGCWk%2F75gdr5Q%3D%3D&nk2=1DsN4FjjwTp04g%3D%3D&id2=UondHPobpDVKHQ%3D%3D; t=ad1fbf51ece233cf3cf73d97af1b6a71; tracknick=%5Cu4F0F%5Cu6625%5Cu7EA22013; lid=%E4%BC%8F%E6%98%A5%E7%BA%A22013; uc4=nk4=0%401up5I07xsWKbOPxFt%2BwuLaZ8XIpO&id4=0%40UOE3EhLY%2FlTwLmADBuTfmfBbGpHG; lgc=%5Cu4F0F%5Cu6625%5Cu7EA22013; enc=ieSqdE6T%2Fa5hYS%2FmKINH0mnUFINK5Fm1ZKC0431E%2BTA9eVjdMzX9GriCY%2FI2HzyyntvFQt66JXyZslcaz0kXgg%3D%3D; _tb_token_=536fb5e55481b; cookie2=157aab0a58189205dd5030a17d89ad52; _m_h5_tk=150df19a222f0e9b600697737515f233_1565931936244; _m_h5_tk_enc=909fba72db21ef8ca51c389f65d5446c; otherx=e%3D1%26p%3D*%26s%3D0%26c%3D0%26f%3D0%26g%3D0%26t%3D0; l=cBa4gFrRqYHNUtVvBOfiquI8a17O4IJ51sPzw4_G2ICP9B5DeMDOWZezto8kCnGVL6mpR3RhSKO4BYTKIPaTlZXRFJXn9MpO.; isg=BI6ORhr9X6-NrOuY33d_XmZFy2SQp1Ju1qe4XLjXJRHsGyp1IJ9IG0kdUwfSA0oh",
"referer":"https://detail.tmall.com/item.htm",
"user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/6.2.4098.3 Safari/537.36",
}
params={ #必带信息
"itemId":"596285864342", #商品id
"sellerId":"2616970884",
"currentPage":"2", #页码
"callback":"jsonp2359",
}
req=requests.get(url,params,headers=header).content.decode('utf-8')[12:-1]; #解码，并且去除str中影响json转换的字符（\n\rjsonp(...)）;
result=json.loads(req);
print(result)
这是获取淘宝集市评论数据的教程和代码https://blog.csdn.net/u011280778/article/details/104197803
import requests
import json
url="https://rate.tmall.com/list_detail_rate.htm"
header={
"cookie":"cna=EYnEFeatJWUCAbfhIw4Sd0GO; x=__ll%3D-1%26_ato%3D0; hng=CN%7Czh-CN%7CCNY%7C156; uc1=cookie14=UoTaHYecARKhrA%3D%3D; uc3=vt3=F8dBy32hRyZzP%2FF7mzQ%3D&lg2=U%2BGCWk%2F75gdr5Q%3D%3D&nk2=1DsN4FjjwTp04g%3D%3D&id2=UondHPobpDVKHQ%3D%3D; t=ad1fbf51ece233cf3cf73d97af1b6a71; tracknick=%5Cu4F0F%5Cu6625%5Cu7EA22013; lid=%E4%BC%8F%E6%98%A5%E7%BA%A22013; uc4=nk4=0%401up5I07xsWKbOPxFt%2BwuLaZ8XIpO&id4=0%40UOE3EhLY%2FlTwLmADBuTfmfBbGpHG; lgc=%5Cu4F0F%5Cu6625%5Cu7EA22013; enc=ieSqdE6T%2Fa5hYS%2FmKINH0mnUFINK5Fm1ZKC0431E%2BTA9eVjdMzX9GriCY%2FI2HzyyntvFQt66JXyZslcaz0kXgg%3D%3D; _tb_token_=536fb5e55481b; cookie2=157aab0a58189205dd5030a17d89ad52; _m_h5_tk=150df19a222f0e9b600697737515f233_1565931936244; _m_h5_tk_enc=909fba72db21ef8ca51c389f65d5446c; otherx=e%3D1%26p%3D*%26s%3D0%26c%3D0%26f%3D0%26g%3D0%26t%3D0; l=cBa4gFrRqYHNUtVvBOfiquI8a17O4IJ51sPzw4_G2ICP9B5DeMDOWZezto8kCnGVL6mpR3RhSKO4BYTKIPaTlZXRFJXn9MpO.; isg=BI6ORhr9X6-NrOuY33d_XmZFy2SQp1Ju1qe4XLjXJRHsGyp1IJ9IG0kdUwfSA0oh",
"referer":"https://detail.tmall.com/item.htm",
"user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/6.2.4098.3 Safari/537.36",
}
params={ #必带信息
"itemId":"596285864342", #商品id
"sellerId":"2616970884",
"currentPage":"2", #页码
"callback":"jsonp2359",
}
req=requests.get(url,params,headers=header).content.decode('utf-8')[12:-1]; #解码，并且去除str中影响json转换的字符（\n\rjsonp(...)）;
result=json.loads(req);
print(result)
这是获取淘宝集市评论数据的教程和代码https://blog.csdn.net/u011280778/article/details/104197803
「已注销」 2019-06-10
- 打赏
- 举报
爬虫代码能分享先嘛？
生活，才能让你明白 2019-04-28
- 打赏
- 举报
楼主怎么爬到评论数据的？带cookie不能常爬吧？
一笑程序猴 2019-04-24
- 打赏
- 举报
可以根据url继续传入99页后面的页码试试能不能爬去到数据，如果不行的话估计就是剩下的评论数据平台没有放出来吧！
一笑程序猴 2019-04-24
- 打赏
- 举报
它就是没有放出来，手点都只能到99，那是不是就没有办法了[/quote]
你手动点到第99页，然后看一下url肯定会传个页码的参数的，假如传的是99，你传个100,101等试试看看能不能加载出数据，如何一样加载不出来，那就没办法了，那样的话估计就是后台直接把过早的评论过滤掉了。
要不要甜 2019-04-24
- 打赏
- 举报
它就是没有放出来，手点都只能到99，那是不是就没有办法了
要不要甜 2019-04-24
- 打赏
- 举报
它就是没有放出来，手点都只能到99，那是不是就没有办法了
