简单问题，获取网页的HTML

[liter156]

我先给一个

 HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
                webRequest.Timeout = 1000 * 20;//这里加了个时间控制
                HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
                Stream stream = webResponse.GetResponseStream();
                System.IO.StreamReader streamReader = new StreamReader(stream, System.Text.Encoding.GetEncoding(code));
                string content = streamReader.ReadToEnd();
                streamReader.Close();
                webResponse.Close();
                

大家谁还有其他的方法？？要求返回的是string不能直接打包下载到本地

上面的这个问题主要是网速比较慢时获取的HTML为空值，也不报错。。郁闷之极。。。加了Timeout也不行

解决马上结贴。。。。。。。。

[lyboyc]

input.Read(by, 0, (int)by.Length);
streamReader.ReadToEnd();
似乎是网速慢读不到数据就以为读完了。
你可以用ContentLength获得文件长度，然后一个个字符读，读不到继续，直到读这些长度的串，
不过你要设置不能断网了还一直循环执行下去。

[wiki14]

        /// <summary>
        /// 获取网页的HTML代码
         /// </summary>
        /// <param name="url">Url地址(http://www.baidu.com/)</param>
        /// <param name="Prog">进度条</param>
        /// <returns></returns>
        public static string GetUrlData(string url,ProgressBar Prog)
        {
            HttpWebResponse res = null;
            HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
                res = (HttpWebResponse)req.GetResponse();
            long  totalBytes = res.ContentLength;
            Stream input = null;
            input = res.GetResponseStream();
            int totalDownloadedByte = 0;
            byte[] by = new byte[10245];
            string Content=null; 
            System.Text.Encoding encoder = System.Text.Encoding.GetEncoding("GB2312");
            do
            {
                totalDownloadedByte = input.Read(by, 0, (int)by.Length); 
                Content+=encoder.GetString(by, 0, totalDownloadedByte);
                if (Prog.Value + totalDownloadedByte <= Prog.Maximum)
                {
                    Prog.Value += totalDownloadedByte;
                    Application.DoEvents();
                }
                else
                {
                    Prog.Value = Prog.Maximum;
                }               
                   
            }
            while (totalDownloadedByte != 0);
            
            res.Close();
            Prog.Value =0 ;
            return Content;
        }

[cwblaze]

UP

[段传涛]

我用mshtml 和dom 。 帮你顶

[zhulong1111]

mark

[ck11926375]

        private string GetHtml(string url, string charSet)
        {
            WebClient myWebClient = new WebClient(); 
            myWebClient.Credentials = CredentialCache.DefaultCredentials;            
            byte[] myDataBuffer = myWebClient.DownloadData(url);
            string strWebData = Encoding.Default.GetString(myDataBuffer);
            Match charSetMatch = Regex.Match(strWebData, "<meta([^<]*)charset=([^<]*)\"", RegexOptions.IgnoreCase | RegexOptions.Multiline);
            string webCharSet = charSetMatch.Groups[2].Value;
            if (charSet == null || charSet == "")
                charSet = webCharSet;

            if (charSet != null && charSet != "" && Encoding.GetEncoding(charSet) != Encoding.Default)
                strWebData = Encoding.GetEncoding(charSet).GetString(myDataBuffer);
            return strWebData;
        }

[liter156]

试了都不行。。。
郁闷

[liter156]

谁能给分析下原因。。。
按理说下载不成功会提示的，什么也不提示

[mngzilin]

[Quote=引用楼主 liter156 的回复:]
大家谁还有其他的方法？？要求返回的是string不能直接打包下载到本地
上面的这个问题主要是网速比较慢时获取的HTML为空值，也不报错。。郁闷之极。。。加了Timeout也不行
解决马上结贴。。。。。。。。
[/Quote]
网速比较慢，谁也救不了你。

webRequest.KeepAlive = true;去掉webRequest.Timeout = 1000 * 20;

[wuyq11]

使用webbrower看看
System.Net.WebClient wc = new System.Net.WebClient();
string content = wc.DownloadString("");

System.Net.HttpWebRequest request = (System.Net.HttpWebRequest)System.Net.WebRequest.Create("");
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)";
System.Net.WebResponse response = request.GetResponse();
System.IO.Stream resStream = response.GetResponseStream();
System.IO.StreamReader sr = new System.IO.StreamReader(resStream, encoding);
string html = sr.ReadToEnd();
resStream.Close();
sr.Close();

[以专业开发人员为伍]

在自己的网站里把网页缓存成html，我贴过一个demo，再贴一下：

<%@ Page Language="C#" %>

<%@ Import Namespace="System.IO" %>

<script runat="server">
    protected void Button1_Click(object sender, EventArgs e)
    {
        StringWriter wr = new StringWriter();
        Server.Execute(this.TextBox1.Text, wr);     //你可以使用第三个参数传递页面的更多初始数据
        this.Label1.Text = Server.HtmlEncode(wr.ToString());
        File.WriteAllText(Server.MapPath(this.TextBox2.Text), wr.ToString());
    }
</script>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title>演示取得当前网站中其它网页的源代码只需要两行代码就够了</title>
</head>
<body>
    <form id="form1" runat="server" defaultbutton="Button1">
    <div>
        请输入本网站的页面名称：<asp:TextBox ID="TextBox1" runat="server"></asp:TextBox>
        <br />
        请输入目标文件名称<asp:TextBox ID="TextBox2" runat="server"></asp:TextBox>
        <br />
        <asp:Button ID="Button1" runat="server" Text="Button" OnClick="Button1_Click" /><hr />
        <asp:Label ID="Label1" runat="server" Text="Label"></asp:Label>
    </div>
    </form>
</body>
</html>

[以专业开发人员为伍]

例如你可以输入

~/default.aspx?id=123456&name=abc

以及

~/app_data/default_123456_abc.html