python urllib2.urlopen.read()无法返回

[littleshoes]

大家好，我用python写了一个爬虫程序，获取网页页面，然后进行分析。但是，运行过程中出现了问题。

当获取页面很多的时候，urlopen后的结果read函数无法返回（说白了，就是调用之后，就停在那里了，不退出也不报错）。

我认为不是网站把我封了，因为如果封了，应该报错才对，但是我的没有报错，知识函数停在那里，然后就没有然后了。

我的代码如下，请大神帮我看看！！
出问题的代码在这里：

def get_soup(url, user_agent, funname):
    filename = "content.txt"
    headers = {"User-Agent": user_agent}
    req = urllib2.Request(url, headers=headers)
    try:
        write_log(filename, funname + ": Start down page" + "\n")
        response = urllib2.urlopen(req)
        write_log(filename, funname + ": End down page" + "\n")
    except urllib2.URLError, e:
        write_log(filename, e.reason + ": " + e.errno + "\n")

    write_log(filename, funname + ": Start read page" + "\n")
    html = response.read()
    write_log(filename, funname + ": End read page" + "\n")
    soup = BeautifulSoup(html, 'lxml')
    return soup

全部代码在这里：

import urllib
import urllib2
import re
from bs4 import BeautifulSoup
import time
import datetime

def write_log(filename, log_text):
    try:
        with open(filename, 'a') as f:
            f.write(log_text)
    except IOError, e:
        print e.errno + e.message


def get_soup(url, user_agent, funname):
    filename = "content.txt"
    headers = {"User-Agent": user_agent}
    req = urllib2.Request(url, headers=headers)
    try:
        write_log(filename, funname + ": Start down page" + "\n")
        response = urllib2.urlopen(req)
        write_log(filename, funname + ": End down page" + "\n")
    except urllib2.URLError, e:
        write_log(filename, e.reason + ": " + e.errno + "\n")

    write_log(filename, funname + ": Start read page" + "\n")
    html = response.read()
    write_log(filename, funname + ": End read page" + "\n")
    soup = BeautifulSoup(html, 'lxml')
    return soup


def get_post_next_url_list(url):
    soup = get_soup(url, user_agent, "get_post_next_url_list")
    next_page = soup.find_all(class_=re.compile("next.*pagination-item"))
    if len(next_page) > 0:
        next_page_url = next_page[0]["href"]
        return "http:" + next_page_url
    else:
        return ''

'''
def get_post_contents(urls):
    headers = {"User-Agent": user_agent}
    for url in urls:
        url = 'http://tieba.baidu.com' + url.a['href']
        filename = "content.txt"
        write_log(filename, url + "\n")
        req = urllib2.Request(url, headers=headers)
        response = urllib2.urlopen(req)
        html = response.read()
        soup = BeautifulSoup(html, 'lxml')
        divs = soup.find_all(class_=re.compile("l_post j_l_post l_post_bright"))

        for div in divs:
            name = div.find_all(class_="d_name")
            if len(name) > 0:
                name = name[0].find(class_=re.compile("p_author_name")).string

            content = div.find_all(class_=re.compile("d_post_content.*clearfix"))
            if len(content) > 0:
                content = content[0].get_text().strip()

            user_content = name + ":" + content + "\n"
            write_log(filename, user_content.encode("utf-8"))
        time.sleep(2)
        write_log(filename, "\n" + "\n")
'''

def get_post_url_list(url):
    soup = get_soup(url, user_agent, "get_post_url_list")
    urls = soup.find_all(class_="threadlist_title pull_left j_th_tit ")
    filename = "content.txt"
    for url in urls:
        write_log(filename, 'http://tieba.baidu.com' + url.a['href'] + "\n")
    return urls


def get_tieba_content(url):
    filename = "content.txt"

    while url.strip():
        write_log(filename, url + "\n")
        urls = get_post_url_list(url)
        #get_post_contents(urls)
        url = get_post_next_url_list(url)


if __name__ == '__main__':
    user_agent = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
    url = "http://tieba.baidu.com/f?ie=utf-8&kw=%E8%8B%B1%E5%9B%BD%E6%96%97%E7%89%9B%E7%8A%AC&fr=search"
    get_tieba_content(url)

[sanGuo_uu]

引用

当获取页面很多的时候，urlopen后的结果read函数无法返回

什么叫“获取页面很多”？是指多次循环吗。 你每次read，都只打开一个url，应该没问题啊 ----------------------------------------------------------------------- 我运行了下。 你Html根本就没print嘛，也没输入到content.txt，所以不会显示

[littleshoes]

@sanGuo_uu 是这样，html的确没有写入文件，我所说的read无法完成是指，程序需要执行一段时间，如果您执行我的代码，执行个十分钟左右，您就会发现，程序停在那里，不会继续运行了，

[sanGuo_uu]

好像不能上传文件 我39min运行完了。就是有点慢，其它没什么问题。（我看了任务管理器，也没有很占用内存,CPU） 速度大概是： #26min 22900 #39min 35900 它最后一页的pn就是pn=35900 -------------------------------------------------------------------- 调整思路： 1，文件IO一般是比较占用时间的，所以你这个write_log，老是打开关闭文件，可能是速度慢的原因之一 2，你说运行10+min就停了，你可以改下循环，让它只执行1/4，就停。然后下次设置初始值，从2/4开始执行。这样执行4次，就能总的都执行完了。

[qq_41095786]

请问楼主怎么解决的，我也碰到这个问题了