用webBrowser翻页抓取

[hling]

页面有一个js翻页，想抓每页的内容，下面代码只能抓到第一页的数据。

 public Form1()
        {
            InitializeComponent();
            string url = "http://www.xxxx.cc/";
            webBrowser1.Navigate(url);
                }

        private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
        {
                      for (int i = 1; i < 8; i++)
            {
                string[] args = { "AspNetPager1", i.ToString() };
                webBrowser1.Document.InvokeScript("__doPostBack", args);
                if (webBrowser1.IsBusy == false && webBrowser1.StatusText == "完成")
                {
                    string body = webBrowser1.Document.Body.OuterHtml;

                    string exp = " 共([\\s\\S]*?)页，当前为第([\\s\\S]*?)页，每页10条";
                    Match match = Regex.Match(body, exp, RegexOptions.IgnoreCase);

                    string num = match.Groups[2].Value.ToString();
                             }
            
            }

        }

[viki117]

分页,也就是分开加载,你的页面只加载了第一页,当然只有一页的数据了,想抓去所有页,就要post和get每页的数据,然后分别获取数据,提取整合~~

[hling]

[以专业开发人员为伍]

我每一次都强调，对于王爷分析程序，不要扯到什么“正则”上。 给你写个

using System;
using System.Collections.Generic;
using System.Linq;
using System.Windows.Forms;

namespace WindowsFormsApplication1
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
            this.webBrowser1.DocumentCompleted += First_DocumentCompleted;
        }

        public class session
        {
            public int 页号;
            public Uri uri;
        }

        private List<session> Pages;

        private int TryInt(string s)
        {
            int x;
            if (!int.TryParse(s, out x))
                return -1;

            return x;
        }

        void First_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
        {
            if (this.webBrowser1.ReadyState == WebBrowserReadyState.Complete)
            {
                this.webBrowser1.DocumentCompleted -= First_DocumentCompleted;

                var nav_panel = (from x in this.webBrowser1.Document.All.OfType<HtmlElement>()
                                 where x.GetAttribute("className") == "page_nav"
                                 select x).First();
                Pages = (from li in nav_panel.GetElementsByTagName("LI").OfType<HtmlElement>()
                         from a in li.GetElementsByTagName("A").OfType<HtmlElement>()
                         let num = TryInt(a.InnerText)
                         where num >0
                         select new session
                         {
                             页号 = num,
                             uri = new Uri(a.GetAttribute("href"))
                         }).ToList();
                this.webBrowser1.DocumentCompleted += Flip_DocumentCompleted;
                Go();
            }
        }

        void Flip_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
        {
            if (this.webBrowser1.ReadyState == WebBrowserReadyState.Complete)
                Go();
        }

        private void Go()
        {
            var index = Pages.Count - 1;
            if (index >= 0)
            {
                var page = Pages[index];
                Pages.RemoveAt(index);
                this.webBrowser1.Navigate(page.uri);
                this.Text = string.Format("正在加载第 {0} 页：{1}", page.页号, page.uri.ToString());
            }
        }

    }
}

用正则，就等于是拿着鱼刺充鱼翅，根本不上档次。 只有非常具体的个别的地方才需要使用正则。有些人学点sql语句就入迷了，甚至取个机器时间、计算个四则运算，也要使用ado.net去调用一下sql数据库。csdn上有些人就好像对正则也持这种过份的姿态，你可以看到，对于你要分析网页这类简单任务来说，对比这个程序你会发现，正则显得成事不足败事有余，只给你添加了不少混乱代码而已。

[以专业开发人员为伍]

在你的Form1窗口中拖入一个webbrowser，然后设置其url地址为 http://bbs.csdn.net/forums/DotNET 运行一下，你就可以看到程序从最后一页开始，一页一页第完整加载每一页。那么你可以在里边加上你的处理代码。

[以专业开发人员为伍]

嗯，如果提高一丁点效率，我们可以改写那一句

        void First_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
        {
            if (this.webBrowser1.ReadyState == WebBrowserReadyState.Complete)
            {
                this.webBrowser1.DocumentCompleted -= First_DocumentCompleted;
                var nav_panel = (from x in this.webBrowser1.Document.GetElementsByTagName("div").OfType<HtmlElement>()
                                 where x.GetAttribute("className") == "page_nav"
                                 select x).First();
                Pages = (from li in nav_panel.GetElementsByTagName("LI").OfType<HtmlElement>()
                         from a in li.GetElementsByTagName("A").OfType<HtmlElement>()
                         let num = TryInt(a.InnerText)
                         where num > 0
                         select new session
                         {
                             页号 = num,
                             uri = new Uri(a.GetAttribute("href"))
                         }).ToList();
                this.webBrowser1.DocumentCompleted += Flip_DocumentCompleted;
                Go();
            }
        }

[hling]

它用了UpdatePanel不刷新翻页，还是翻不了

[xixihaha_2011_098]

获取分页的标签,模拟单击

[hling]

[hufanrong12345]

引用 3 楼 sp1234 的回复:

我每一次都强调，对于王爷分析程序，不要扯到什么“正则”上。 给你写个

using System;
using System.Collections.Generic;
using System.Linq;
using System.Windows.Forms;

namespace WindowsFormsApplication1
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
            this.webBrowser1.DocumentCompleted += First_DocumentCompleted;
        }

        public class session
        {
            public int 页号;
            public Uri uri;
        }

        private List<session> Pages;

        private int TryInt(string s)
        {
            int x;
            if (!int.TryParse(s, out x))
                return -1;

            return x;
        }

        void First_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
        {
            if (this.webBrowser1.ReadyState == WebBrowserReadyState.Complete)
            {
                this.webBrowser1.DocumentCompleted -= First_DocumentCompleted;

                var nav_panel = (from x in this.webBrowser1.Document.All.OfType<HtmlElement>()
                                 where x.GetAttribute("className") == "page_nav"
                                 select x).First();
                Pages = (from li in nav_panel.GetElementsByTagName("LI").OfType<HtmlElement>()
                         from a in li.GetElementsByTagName("A").OfType<HtmlElement>()
                         let num = TryInt(a.InnerText)
                         where num >0
                         select new session
                         {
                             页号 = num,
                             uri = new Uri(a.GetAttribute("href"))
                         }).ToList();
                this.webBrowser1.DocumentCompleted += Flip_DocumentCompleted;
                Go();
            }
        }

        void Flip_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
        {
            if (this.webBrowser1.ReadyState == WebBrowserReadyState.Complete)
                Go();
        }

        private void Go()
        {
            var index = Pages.Count - 1;
            if (index >= 0)
            {
                var page = Pages[index];
                Pages.RemoveAt(index);
                this.webBrowser1.Navigate(page.uri);
                this.Text = string.Format("正在加载第 {0} 页：{1}", page.页号, page.uri.ToString());
            }
        }

    }
}

用正则，就等于是拿着鱼刺充鱼翅，根本不上档次。 只有非常具体的个别的地方才需要使用正则。有些人学点sql语句就入迷了，甚至取个机器时间、计算个四则运算，也要使用ado.net去调用一下sql数据库。csdn上有些人就好像对正则也持这种过份的姿态，你可以看到，对于你要分析网页这类简单任务来说，对比这个程序你会发现，正则显得成事不足败事有余，只给你添加了不少混乱代码而已。

取精去糟！

[hufanrong12345]

引用 5 楼 sp1234 的回复:

嗯，如果提高一丁点效率，我们可以改写那一句

        void First_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
        {
            if (this.webBrowser1.ReadyState == WebBrowserReadyState.Complete)
            {
                this.webBrowser1.DocumentCompleted -= First_DocumentCompleted;
                var nav_panel = (from x in this.webBrowser1.Document.GetElementsByTagName("div").OfType<HtmlElement>()
                                 where x.GetAttribute("className") == "page_nav"
                                 select x).First();
                Pages = (from li in nav_panel.GetElementsByTagName("LI").OfType<HtmlElement>()
                         from a in li.GetElementsByTagName("A").OfType<HtmlElement>()
                         let num = TryInt(a.InnerText)
                         where num > 0
                         select new session
                         {
                             页号 = num,
                             uri = new Uri(a.GetAttribute("href"))
                         }).ToList();
                this.webBrowser1.DocumentCompleted += Flip_DocumentCompleted;
                Go();
            }
        }

取精去糟！