java如何下载网友内容？

[wonkju]

关键是需要登录， 有点难搞， 第二次请求的时候， 返回值说是200，但是什么内容也没有。

[finemi]

无非就是分析好请求所带的request头及response头信息

[oh_Maxy]

在打开目标页面URL之前，先通过HttpClient方式登录，这样再打开目标网址，就不会再跳转登录页了。

[JPF1024]

发起请求，模拟登录，然后在抓取分析HTML数据。

[wonkju]

引用 楼主 king769147 的回复:

关键是需要登录， 有点难搞， 第二次请求的时候， 返回值说是200，但是什么内容也没有。

我试过了很多次, 主要问题是, 获得是数据不是最新的. 代码为:

static final String LOGON_SITE = "szhsyh.ezhanyun.com" ;
	static final int     LOGON_PORT = 80; 
	@Override
	public String initCookie(User user) throws HttpException, IOException {
		 HttpClient client = new HttpClient(); 
	     client.getHostConfiguration().setHost(LOGON_SITE, LOGON_PORT); 

	      // 模拟登录页面 login.jsp->main.jsp 
//	      PostMethod post = new PostMethod( "/X_UserFileLogic/X_hesheng2.0/X_Login" ); 
//	      NameValuePair name = new NameValuePair( "AdminName" , "18688626226" ); 
//	      NameValuePair pass = new NameValuePair( "AdminPass" , "qaz123" ); 
//	      post.setRequestBody( new NameValuePair[]{name,pass});
	      GetMethod get=new GetMethod("/X_UserFileLogic/X_hesheng2.0/X_Login?X6AdminName=18688626226&X6AdminPass=qaz123"); 
	      int status = client.executeMethod(get);
System.out.println("status:" + status);	      
//	      System.out.println(post.getResponseBodyAsString()); 
	      //post.releaseConnection(); 

	      // 查看 cookie 信息 
	      CookieSpec cookiespec = CookiePolicy.getDefaultSpec(); 
	      Cookie[] cookies = cookiespec.match(LOGON_SITE, LOGON_PORT, "/" , false , client.getState().getCookies());
	      String cookie = null;
	      if (cookies.length == 0) { 
	         System.out.println( "None" ); 
	      } else { 
	         for ( int i = 0; i < cookies.length; i++) { 
	        	cookie = cookies[i].toString();
	            System.out.println(cookie); 
	         } 
	      } 

	      // 访问所需的页面 main2.jsp 
	      get=new GetMethod("/X_UserLogic/hesheng2015/320_JsonList/?CPPP=%E5%85%A8%E9%83%A8&CPLB=%E5%85%A8%E9%83%A8&keyword=");
	      get.setRequestHeader("Connection", "keep-alive");
	      get.setRequestHeader("Referer", "http://szhsyh.ezhanyun.com/x/?Action=List&Type=2&Module=320");
	      get.getParams().setCookiePolicy(CookiePolicy.RFC_2965);
	      get.setRequestHeader("Cookie", cookie);
	      client.executeMethod(get); 
  System.out.println("r:" + get.getResponseBodyAsString()); 
	      get.releaseConnection(); 
		return null;
	}

[wonkju]

上面的哥们, 帮看一下, 主要是获取不到最新的数据. 好像我没使用上cookie. 我通过浏览器分析头部信息为

Request URL:http://szhsyh.ezhanyun.com/X_UserLogic/hesheng2015/320_JsonList/?CPPP=%E5%85%A8%E9%83%A8&CPLB=%E5%85%A8%E9%83%A8&keyword=
Request Method:GET
Status Code:200 OK
Response Headers
 (6)
Request Headers
view source
Accept:application/json, text/javascript, */*; q=0.01
Accept-Encoding:gzip, deflate, sdch
Accept-Language:zh-CN,zh;q=0.8
Connection:keep-alive
Cookie:ASPSESSIONIDCSQCCCRA=JCEJEBHDKFKKMMMDEHKCFLJE; X6AdminLimit6=; X6AdminLimit5=; X6AdminLimit4=%2C4%2EB%2C4%2EC%2C4%2ED%2C4%2EE%2C3%2EB%2C3%2EC%2C3%2ED%2C3%2EE%2C10%2EA%2C5%2EB%2C5%2EC%2C5%2ED%2C5%2EE%2C56%2EB%2C56%2EC%2C56%2ED%2C56%2EE%2C6%2EB%2C6%2EC%2C6%2ED%2C6%2EE%2C400%2EA%2C401%2EB%2C401%2EC%2C401%2ED%2C; X6AdminLimit3=%2CModule4%2CModule3%2CModule5%2CModule56%2CModule6%2CModule2%2CModule1%2CModule319%2CModule403%2CModule404%2CModule408%2CModule388%2CModule357%2CModule358%2CModule359%2CModule360%2CModule362%2CModule361%2CModule327%2CModule320%2CModule321%2CModule322%2CModule323%2CModule324%2CModule325%2CModule341%2CModule396%2CModule295%2CModule296%2CModule297%2CModule298%2CModule299%2CModule300%2CModule301%2CModule302%2CModule303%2CModule304%2CModule305%2CModule316%2CModule317%2CModule318%2CModule409%2CModule447%2CModule448%2CModule449%2CModule450%2CModule451%2CModule452%2CModule453%2CModule454%2CModule7%2CModule90%2CModule91%2CMenu35%2CSee389%2CSee390%2CMenu33%2CMenu34%2CMenu36%2CSee120%2C4%2EB%2C4%2EC%2C4%2ED%2C4%2EE%2C3%2EB%2C3%2EC%2C3%2ED%2C3%2EE%2C10%2EA%2C5%2EB%2C5%2EC%2C5%2ED%2C5%2EE%2C56%2EB%2C56%2EC%2C56%2ED%2C56%2EE%2C6%2EB%2C6%2EC%2C6%2ED%2C6%2EE%2C400%2EA%2C401%2EB%2C401%2EC%2C401%2ED%2C; X6AdminLimit2=%2C; X6AdminLimit1=%2C; AdminLimitStr=%2C8%2C12%2C; X6AdminLimitId=1; X6AdminTime=2015%2D7%2D17+0%3A06%3A23; X6%5FAdmin%5FLimit=1; X6%5FAdmin%5FValueM=%E9%A1%BA%E4%B8%B0%2D23%3A20; X6%5FAdmin%5FValueJ=%E6%89%8B%E6%9C%BA; X6%5FAdmin%5FValueH=%E5%BA%97%E9%95%BF; X6%5FAdmin%5FValueG=755013669; X6%5FAdmin%5FValueF=%E4%BA%8C%E7%B1%BB%E4%BB%B7; X6%5FAdmin%5FValueE=%E9%A9%AC%E5%90%AF%E7%8F%8D; X6%5FAdmin%5FValueB=%E5%90%B4%E5%A4%A7%E5%BC%BA; X6%5FAdmin%5FValueA=%E7%8E%B0%E6%AC%BE%E5%AE%A2%E6%88%B7; X6%5FAdmin%5FCity=%E4%B8%9C%E8%8E%9E%E5%B8%82; X6%5FAdmin%5FProvince=%E5%B9%BF%E4%B8%9C%E7%9C%81; X6%5FAdmin%5FMobile=18688626226; X6%5FAdmin%5FAddress=%E9%95%BF%E5%AE%89%E9%95%87%E4%B9%8C%E7%BA%B1%E5%85%B4%E5%9B%9B%E8%B7%AF%EF%BC%91%EF%BC%90%EF%BC%90%E5%8F%B7; X6%5FAdmin%5FName=%E5%B9%BF%E4%B8%9C%E4%B8%9C%E8%8E%9E%E9%95%BF%E5%AE%89%E4%B8%AD%E5%9B%BD%E7%94%B5%E4%BF%A1%E9%A9%AC%E5%90%AF%E7%8F%8DX%2FC; X6AdminPass=8855c4efa678474c; X6AdminName=18688626226; CloseSysHidden=0; X6AdminIp=183%2E16%2E88%2E83; X6AdminId=12896
Host:szhsyh.ezhanyun.com
Referer:http://szhsyh.ezhanyun.com/x/?Action=List&Type=2&Module=320
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36
X-Requested-With:XMLHttpRequest
Query String Parameters
view source
view URL encoded
CPPP:全部
CPLB:全部
keyword: