怎么爬取weibo的“微博实时热点”内容？

finer 2017-10-11 11:04:25

我要爬取weibo首页的“微博实时热点”内容，如“李小璐虹桥一姐求合照时，甜馨妈也是依旧这么少女，还记得《奋斗》中李小璐这小舞一跳，小歌一唱，真的太可爱了[心][心][心] #明星特别任务# http:\/\/t.cn\/RO6NYeh”，要怎么处理？

#coding:utf-8

import re
import urllib
from urllib import request
from bs4 import BeautifulSoup as bs
from pymongo import MongoClient

client = MongoClient('127.0.0.1',27017)
#client = MongoClient()
mydb= client.test
myC = mydb.UTWeiboHot

values={"username":"xxxx","password":"xxxxx"}
data=urllib.parse.urlencode(values).encode(encoding='UTF8')
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
referer='http://weibo.com'
url= "http://weibo.com/login"
headers={"User-Agent":user_agent,'Referer':referer}
request=request.Request(url,data,headers)
resp=urllib.request.urlopen(request)

html_data= resp.read().decode('utf-8')
soup=bs(html_data,"html.parser")

for strs in soup.find_all('div', class_='\"des_main S_txt2\">'):
print (strs)

我以上代码爬不到内容，哪里有问题呢？
----------------------------------------------------------------------------------------
<h3 class=\"list_title_s\"><a href=\"\/a\/hot\/8b8d0c582b02fbfb_0.html?type=grab\" class=\"S_txt1\" target=\"_blank\">李小璐虹桥一姐<\/a><\/h3>\n <div class=\"des_main S_txt2\">李小璐虹桥一姐求合照时，甜馨妈也是依旧这么少女，还记得《奋斗》中李小璐这小舞一跳，小歌一唱，真的太可爱了[心][心][心] #明星特别任务# http:\/\/t.cn\/RO6NYeh <\/div>\n <\/div>\n <\/div>\n <div class=\"UG_list_c\" action-type=\"feed_list_item\" href=\"\/a\/hot\/95e4a1cb77f6cd81_0.html?type=grab\" suda-uatrack=\"key=www_unlogin_home&value=hot02\">\n <div class=\"pic W_piccut_v\">\n <a href=\"\/a\/hot\/95e4a1cb77f6cd81_0.html?type=grab\" target=\"_blank\"><img src=\"http:\/\/ww4.sinaimg.cn\/thumb180\/ea170e04ly1fjza3y5evej20fb08et9g.jpg\" alt=\"\"><\/a>\n <\/div>\n <div class=\"list_des\">\n <h3 class=\"list_title_s\"><a href=\"\/a\/hot\/95e4a1cb77f6cd81_0.html?type=grab\" class=\"S_txt1\" target=\"_blank\">长期单身的姑娘<\/a><\/h3>\n <div class=\"des_main S_txt2\">【情感帖】长期单身的姑娘大概就是这样吧！ <\/div>\n <\/div>\n <\/div>\n <div class=\"UG_list_c\" action-type=\"feed_list_item\" href=\"\/a\/hot\/15779ded10c10bd7_0.html?type=grab\" suda-uatrack=\"key=www_unlogin_home&value=hot03\">\n <div class=\"pic W_piccut_v\">\n <a href=\"\/a\/hot\/15779ded10c10bd7_0.html?type=grab\" target=\"_blank\"><img src=\"http:\/\/ww4.sinaimg.cn\/thumb180\/61e89358ly1fke2l3wotxj20fa0c80tu.jpg\" alt=\"\"><\/a>\n <\/div>\n <div class=\"list_des\">\n <h3 class=\"list_title_s\"><a href=\"\/a\/hot\/15779ded10c10bd7_0.html?type=grab\" class=\"S_txt1\" target=\"_blank\">日本女高中生必须用iPhone<\/a><\/h3>\n <div class=\"des_main S_txt2\">【日本女高中生必须用iPhone：原因竟是这样】日本网络小说“野草莓”网站9月公布的一份调查问卷显示，iPhone稳居日本年轻人最受欢迎手机之首，其中绝大部分是女高中生。不过这个原因实在是没想到，具体的我们看文章#此处有码# ：http:\/\/t.cn\/ROJJM0B <\/div>\n <\/div>

...全文