.net中怎样抓取其他网页的内容

[追忆猛海]

就是我想获得别人网页上的数据，比如说一篇新闻报道。我在百度上查了好久，查出来的都不对，希望高手赐教。最好有事例

[孟子E章]

HttpWebRequest
得到源代码进行分析即可，

[追忆猛海]

[Quote=引用 1 楼 net_lover 的回复:]
HttpWebRequest
得到源代码进行分析即可，
[/Quote
能不能详细点啊 今天刚找到工作 这就是公司出的题啊 呜呜

[赳赳老陈]

//定义对象
            HttpWebRequest _request = null;
            HttpWebResponse _response = null;

            try
            {
                //创建web请求类		
                _request = (HttpWebRequest)WebRequest.Create(_destUrl);

                //取得响对象
                _response = (HttpWebResponse)_request.GetResponse();

                //获取数据
                _resultStr = ConvertResponseToString(_response);
            }
            catch (WebException ex)
            {
                //获取响应对象
                _response = ex.Response as HttpWebResponse;

                //处理错误
                if (_response.StatusCode == HttpStatusCode.InternalServerError)
                {
                    
                }
                else
                {
                    _resultStr = ex.Message;
                }
            }
            finally
            {
                if (_response != null) _response.Close();
            }

[qxyywy]

/// <summary>
        /// 获取指定页面的源代码
        /// </summary>
        /// <param name="PageURL"></param>
        /// <returns></returns>
        public String GetPageCode(string PageURL)
        {
            string Charset = "gb2312";
            try
            {
                //存放目标网页的html
                String strHtml = "";
                //连接到目标网页
                HttpWebRequest wreq = (HttpWebRequest)WebRequest.Create(PageURL);
                wreq.Headers.Add("X_FORWARDED_FOR", "101.0.0.11"); //发送X_FORWARDED_FOR头(若是用取源IP的方式，可以用这个来造假IP,对日志的记录无效)  

                wreq.Method = "Get";
                wreq.KeepAlive = true;
                wreq.ContentType = "application/x-www-form-urlencoded";
                wreq.AllowAutoRedirect = true;
                wreq.Accept = "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*";
                wreq.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)";

                CookieContainer cookieCon = new CookieContainer();
                wreq.CookieContainer = cookieCon;

                HttpWebResponse wresp = null;
                try
                {
                    wresp = (HttpWebResponse)wreq.GetResponse();
                }
                catch (WebException ex)
                {
                    wresp = (HttpWebResponse)ex.Response;
                }

                //采用流读取，并确定编码方式
                Stream s = wresp.GetResponseStream();
                StreamReader objReader = new StreamReader(s, System.Text.Encoding.GetEncoding(Charset));

                string strLine = "";
                //读取
                while (strLine != null)
                {
                    strLine = objReader.ReadLine();
                    if (strLine != null)
                    {
                        strHtml += strLine.Trim();
                    }
                }
                strHtml = strHtml.Replace("<br />", "\r\n");

                return strHtml;
            }
            catch (Exception n) //遇到错误，打印错误
            {
                return n.Message;
            }
        }

[追忆猛海]

ConvertResponseToString(_response);这个方法怎么写的啊？

[追忆猛海]

善良的小丑
[Quote=引用 5 楼 zhuiyimenghai 的回复:]
ConvertResponseToString(_response);这个方法怎么写的啊？
[/Quote]

[追忆猛海]

qxyywy
谢谢诶

[liji007]

莫非小弟面试的工作是做爬虫！

[qxyywy]

[Quote=引用 7 楼 zhuiyimenghai 的回复:]

qxyywy
谢谢诶
[/Quote]

[zouyong311]

现在很多不良网站，将其他如网站精华帖的内容copy到自己网页上当资源，然后挂上一堆广告。我很多次搜索技术文章baidu都把我引到这些网站上……

楼主不会是去这类公司了吧，嘿嘿。

[欢乐的小猪]

你百度下如何下载网页，下载完自己再进行解析

[知擎物联]

去下载这个吧，我以前写的Google搜索结果抓取的小工具。
http://download.csdn.net/detail/yeqi3000/4119987

[dongt2]

WebBrowser

[杨友山]

参考