C# 下载文件远程服务器返回错误: (403) 已禁止,求各位朋友帮忙!!!

summer_chen_2008 2013-06-27 02:19:31
http://www.shutterstock.com 用浏览器下载图片没有问题,但通过我的C# AP程序不行

我已经通过获取头部Cookie以及下次Get页面set cookie保持了登陆状态
保持登陆状态的关键代码
 cookieHeader = response.Headers["Set-cookie"];  //get the cookie
webRequest.Headers.Add("Cookie", cookieHeader);

下载部分代码如下:

            //////////////////////////////////////////////////////////////////////////
//photo page method 1-3 down load file

string file_name = "";
url = GetDownloadUrl(responseData);
url_reference = GetDownloadUrlRe(url,responseData);

file_name = GetFileNameViaURL(url);
//DownloadFile(url, "f:\\" + file_name, "a");


webRequest = WebRequest.Create(url) as HttpWebRequest;
webRequest.MediaType = "GET";
webRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0.1) Gecko/20100101 Firefox/6.0.1";
webRequest.Host = "download.shutterstock.com";
webRequest.UseDefaultCredentials = true;
webRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
webRequest.KeepAlive = true;
webRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
webRequest.Headers.Add("Accept-Language", "zh-cn,en-us;q=0.5");

webRequest.Headers.Add("Cookie", cookieHeader);//session in cookie
webRequest.Headers.Add("Pragma", "no-cache");

response = (HttpWebResponse)webRequest.GetResponse();//will the 403 error



通过网络监控获取的两个不同的头部信息,我代码中已经尽量的模拟了浏览器的请求了。

浏览器的:
http://download.shutterstock.com/gatekeeper/W3siZSI6MTM3MjMyNjk3NywiYyI6Il9waG90b19zZXNzaW9uX2lkIiwiZGMiOiJpZGxfMTIwODA0MTgxIiwiayI6InBob3RvLzEyMDgwNDE4MS92ZWN0b3IuZXBzIiwibSI6IjEiLCJkIjoic2h1dHRlcnN0b2NrLW1lZGlhIn0sIiszUFppWVllN3dYYVRJSVc1VlVXMDc4RlJaSSJd/shutterstock_120804181.eps HTTP/1.1
Host: download.shutterstock.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Cookie: visitor_id=1259479000; visitor_id_version=1; psst=73:196,115:466,158:572,200:761,209:809,218:872,221:886,230:958,310:1081,322:1123,332:1160,371:1316,377:1334,389:1379,430:1486,433:1501,445:1534,448:1540; __utma=45497851.397993985.1368695244.1372234529.1372297959.37; __utmz=45497851.1370501396.26.2.utmcsr=bbs.csdn.net|utmccn=(referral)|utmcmd=referral|utmcct=/topics/390476920; exp=v2%2FMTAwMjk6MTAxMDUsMTAwMjM6MTAwNzMsMTAwMzM6MTAxMjcsMTAwNDU6MTAxNzM%3D; _photo_session_id=32db15b71c0d3eec3f90e0e32d9669cf; _video_session_id=; visit_exp=MjA5MTE0NDE2NC86LDEwMDQ1OjEwMTczLDEwMDIzOjEwMDcz; visit_id=2091144164; sst=212:815,380:1343; __utmb=45497851.6.10.1372297959; __utmc=45497851; idl_120804181=1
Connection: keep-alive

我AP的
GET http://download.shutterstock.com/gatekeeper/W3siZSI6MTM3MjM0MTk0NSwiYyI6Il9waG90b19zZXNzaW9uX2lkIiwiZGMiOiJpZGxfMTIwODA0MTgxIiwiayI6InBob3RvLzEyMDgwNDE4MS92ZWN0b3IuZXBzIiwibSI6IjEiLCJkIjoic2h1dHRlcnN0b2NrLW1lZGlhIn0sIm1CMW1zbW5OampFeVNRWTh0RXdFQVZFY092cyJd/shutterstock_120804181.eps HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0.1) Gecko/20100101 Firefox/6.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: zh-cn,en-us;q=0.5
Cookie: ssssidd=6e5751d5eaf3976a076b46e8d64c3980; path=/; expires=Sat, 27-Jul-2013 06:05:47 GMT,visitor_id=1367693476; domain=.shutterstock.com; path=/; expires=Tue, 27-Jun-2023 06:05:47 GMT,visitor_id_version=1; domain=.shutterstock.com; path=/; expires=Tue, 27-Jun-2023 06:05:47 GMT,visit_id=2091623910; domain=.shutterstock.com; path=/; expires=Thu, 27-Jun-2013 06:35:47 GMT,sst=212:815,380:1343; domain=.shutterstock.com; path=/; expires=Thu, 27-Jun-2013 06:35:47 GMT,psst=73:196,115:466,158:572,200:761,209:809,218:872,221:886,230:958,332:1160,377:1328,389:1382,430:1486,433:1501,445:1534,448:1540; domain=.shutterstock.com; path=/; expires=Wed, 25-Sep-2013 06:05:47 GMT
Pragma: no-cache
Host: download.shutterstock.com


我cookie部分没有红色字体的_photo_session_id,不知道是不是这个原因
...全文
509 5 打赏 收藏 转发到动态 举报
写回复
用AI写文章
5 条回复
切换为时间正序
请发表友善的回复…
发表回复
summer_chen_2008 2013-06-29
  • 打赏
  • 举报
回复
引用 1 楼 Carpathia 的回复:
那应该就是_photo_session_id的问题了
引用 2 楼 wonderfuly 的回复:
必须构造得一模一样才行。
引用 4 楼 athlon128 的回复:
看起来很高端,帮你置顶一下
多谢各位,找到问题了,确实需要构建 _photo_session_id=才行,核实发现页面时通过脚本写入的所有AP读不出来,只能手动的抓取页面的 <input type="hidden" name="session_id" value="da760bfbeca1ede1a22a4d7f495e5ed1" /> 再构建写入就好了
athlon128 2013-06-28
  • 打赏
  • 举报
回复
看起来很高端,帮你置顶一下
summer_chen_2008 2013-06-28
  • 打赏
  • 举报
回复
引用 2 楼 wonderfuly 的回复:
必须构造得一模一样才行。
_photo_session_id=32db15b71c0d3eec3f90e0e32d9669cf 这个值获取不到呀,我用
  
          CookieContainer cookies = new CookieContainer();

            //////////////////////////////////////////////////////////////////////////
            // first, request the login form to get the viewstate value
            HttpWebRequest webRequest = WebRequest.Create(url) as HttpWebRequest;
            webRequest.Method = "GET";
            webRequest.UserAgent = UserAgent;
           webRequest.CookieContainer =cookies;
            webRequest.Referer = "";
            webRequest.Host = "www.shutterstock.com";
            webRequest.Accept = "text/html, application/xhtml+xml, */*";
            webRequest.KeepAlive = true;
                        
            HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse();
            StreamReader responseReader = new StreamReader(response.GetResponseStream());
            
            cookieHeader = response.Headers["Set-cookie"];  //get the cookie

            OutputCookieData(url, response);
            

         private void OutputCookieData(String page_url,HttpWebResponse response)
        {
            StringBuilder sb = new StringBuilder();
            sb.Append("Url:"+page_url+Environment.NewLine);
            int i = 0;
            foreach (Cookie cook in response.Cookies)
            {
                i = i + 1;
                sb.Append(i+". Cookie:");

                sb.Append(String.Format("{0} = {1}", cook.Name, cook.Value));
                sb.Append(String.Format(" Domain: {0}", cook.Domain));
                sb.Append(String.Format(" Path: {0}", cook.Path));
                sb.Append(String.Format(" Port: {0}", cook.Port));
                sb.Append(String.Format(" Secure: {0}", cook.Secure));

                sb.Append(String.Format(" When issued: {0}", cook.TimeStamp));
                sb.Append(String.Format(" Expires: {0} (expired? {1})",
                    cook.Expires, cook.Expired));
                sb.Append(String.Format(" Don't save: {0}", cook.Discard));
                sb.Append(String.Format(" Comment: {0}", cook.Comment));
                sb.Append(String.Format(" Uri for comments: {0}", cook.CommentUri));
                sb.Append(String.Format(" Version: RFC {0}", cook.Version == 1 ? "2109" : "2965"));

                // Show the string representation of the cookie.
                sb.Append(String.Format("String: {0}", cook.ToString()));
                sb.Append(Environment.NewLine);
            }

            sb.AppendLine(Environment.NewLine);

            WriteCookieLog(sb.ToString());
        }
获取的值里面根本就没有,为啥浏览器里面有,我缺在哪里 rl:https://secure.shutterstock.com/login.mhtml Url:http://www.shutterstock.com/pic-120804181.html 1. Cookie:ssssidd = 7ea9c80b841b8f48fb78a43573e0b3c7 Domain: www.shutterstock.com Path: / Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 2013/7/19 15:39:16 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: ssssidd=7ea9c80b841b8f48fb78a43573e0b3c7 2. Cookie:visit_id = 2064191922 Domain: .shutterstock.com Path: / Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 2013/6/19 16:09:17 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: visit_id=2064191922 3. Cookie:sst = 212:815 Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: sst=212:815 4. Cookie:380:1343 = Domain: .shutterstock.com Path: / Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 2013/6/19 16:09:17 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 380:1343= 5. Cookie:psst = 73:196 Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: psst=73:196 6. Cookie:115:466 = Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 115:466= 7. Cookie:158:572 = Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 158:572= 8. Cookie:200:761 = Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 200:761= 9. Cookie:209:809 = Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 209:809= 10. Cookie:218:872 = Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 218:872= 11. Cookie:221:886 = Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 221:886= 12. Cookie:230:958 = Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 230:958= 13. Cookie:332:1160 = Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 332:1160= 14. Cookie:377:1328 = Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 377:1328= 15. Cookie:389:1379 = Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 389:1379= 16. Cookie:430:1486 = Domain: www.shutterstock.com Path: /pic-120804181.html Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 0001/1/1 0:00:00 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 430:1486= 17. Cookie:433:1498 = Domain: .shutterstock.com Path: / Port: Secure: False When issued: 2013/6/19 15:39:18 Expires: 2013/9/17 15:39:17 (expired? False) Don't save: False Comment: Uri for comments: Version: RFC 2965String: 433:1498=
游戏人间 2013-06-27
  • 打赏
  • 举报
回复
必须构造得一模一样才行。
Carpathia 2013-06-27
  • 打赏
  • 举报
回复
那应该就是_photo_session_id的问题了

111,125

社区成员

发帖
与我相关
我的任务
社区描述
.NET技术 C#
社区管理员
  • C#
  • Creator Browser
  • by_封爱
加入社区
  • 近7日
  • 近30日
  • 至今
社区公告

让您成为最强悍的C#开发者

试试用AI创作助手写篇文章吧