关于网页内容抓取

[keephope]

public class FirstSocket
{
public static void main(String args[])
{
String strServer = "s5.warlord.cn";
String strPage = "/index.jsp";

try
{
String hostname = strServer;
int port = 80;
InetAddress addr = InetAddress.getByName(hostname);
Socket socket = new Socket(addr, port); // 建立一个Socket

// 发送命令
BufferedWriter wr = new BufferedWriter(new OutputStreamWriter(
socket.getOutputStream(), "UTF-8"));

wr.write("GET /module/map.jsp?lid=1&aid=276&tcid=undefined&session=ec4e42f7e7af HTTP/1.1");
wr.write("Accept: */*");
wr.write("Accept-Language: zh-cn");
wr.write("Referer: http://s5.warlord.cn/main/index.jsp?session=ec4e42f7e7af#");
wr.write("Accept-Encoding: gzip, deflate");
wr.write("User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)");
wr.write("Host: s5.warlord.cn");
wr.write("Connection: Keep-Alive");
wr.write("Cookie: newbieFlag=1; warlord2v3.identify=f13bd0e357e2d324; JSESSIONID=6030B5B0E7918C25D37ED5C08A733176");

wr.flush();

// 接收返回的结果
BufferedReader rd = new BufferedReader(new InputStreamReader(socket
.getInputStream()));
String line;
while ((line = rd.readLine()) != null)
{
System.out.println(line);
}
wr.close();
rd.close();
} catch (Exception e)
{
System.out.println(e.toString());
}
}
}


我想捕捉的是一个网页游戏里面的数据

在IE上输入该地址是可以正确获得内容的

但是这么写代码的话是无法捕捉到内容的

中间内容:

wr.write("GET /module/map.jsp?lid=1&aid=276&tcid=undefined&session=ec4e42f7e7af HTTP/1.1");
wr.write("Accept: */*");
wr.write("Accept-Language: zh-cn");
wr.write("Referer: http://s5.warlord.cn/main/index.jsp?session=ec4e42f7e7af#");
wr.write("Accept-Encoding: gzip, deflate");
wr.write("User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)");
wr.write("Host: s5.warlord.cn");
wr.write("Connection: Keep-Alive");
wr.write("Cookie: newbieFlag=1; warlord2v3.identify=f13bd0e357e2d324; JSESSIONID=6030B5B0E7918C25D37ED5C08A733176");


是用抓包工具从IE里抓到的内容

[kokobox]

lz可以用URLConnection 发送url然后用输入流读取页面内容。

[wopopo]

可参照：

import java.net.URL;
import java.net.MalformedURLException;
import java.io.*;

class ReadFromURL
{
public static void main(String args[ ])
{

URL root = null;
URL url = null;
String readstring;
try
{
root = new URL("file:///e:/lxt008/javacode/");
url = new URL(root,"ReadFromURL.java");
BufferedReader d = new BufferedReader(new InputStreamReader(url.openStream()));

while((readstring = d.readLine())!=null)
{
System.out.println(readstring);
}
System.out.println("***** end of the file *****");
d.close();
}
catch(MalformedURLException e)
{
System.out.println("MalformedURLException: "+e);
}
catch(IOException e)
{
System.out.println("IOException: "+e);
}
}
}

[joejoe1991]

我感觉楼主这样应该可以的吧？谁解释一下为什么不行呢？

[zealVampire]

原因是HTTP 头应该是额外需要发送几个\r\n的。 Try following codes.
Cookies如果有需要的话，可以获取之后继续添加倒headers发送。 还有一些redirect或者是refernce的头有时是需要反馈过去的。

package test;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.Reader;
import java.net.Socket;
import java.net.UnknownHostException;

/**
* @author zealvampire
* @version 1.0 2008-4-23
*/
public class HttpClient {

private int port = 80;
private String host = null;
private String getPath = null;
private StringBuffer content = null;







public HttpClient(String pHost, String pGetPath) {
this(pHost, 80, pGetPath);
}
public HttpClient(String pHost, int pPort, String pGetPath) {
this.host = pHost;
this.port = pPort;
if (pGetPath == null || pGetPath.length()==1)
pGetPath = "/";
this.getPath = pGetPath;
}

public int getURLContent() {
Socket sock = null;
OutputStream out = null;
BufferedReader br = null;
try {
sock = new Socket(this.host, this.port);
out = sock.getOutputStream();
out.write(this.generateHeaders().toString().getBytes());
out.flush();
System.err.println("Request's sent.");
br = new BufferedReader(new InputStreamReader(sock.getInputStream()));
String line = null;
this.content = new StringBuffer();
while ((line = br.readLine()) != null) {
System.err.println(line);
this.content.append(line).append("\r\n");
}

} catch (UnknownHostException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return 1;
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return 2;
}
finally {
close(out);
close(br);
close(sock);
}
//System.err.println("--Response below---");
//System.err.println(this.content.toString());
return 0;

}

private void close(Socket sock) {
if (sock != null)
try {
sock.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private void close(OutputStream out) {
if (out != null)
try {
out.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
private void close(Reader reader) {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

private String generateHeaders() {
StringBuffer buf = new StringBuffer();
buf.append("GET ").append(this.getPath).append(" HTTP/1.1\r\n");
buf.append("Host: ").append(this.host).append("\r\n");
buf.append("User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14\r\n");
buf.append("Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\n");
buf.append("Accept-Language: zh-cn,zh;q=0.5\r\n");
//Never add the header as some servers will gzip the outputstream. Then You need to unzip the output
//buf.append("Accept-Encoding: gzip,deflate\r\n");
buf.append("Accept-Charset: gb2312,utf-8;q=0.7,*;q=0.7\r\n");
buf.append("Keep-Alive: 300\r\n");
buf.append("Connection: keep-alive\r\n\r\n\r\n");
return buf.toString();

}




/**
* @param args
*/
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
HttpClient client = new HttpClient("www.sina.com.cn", "/");
int retCode = client.getURLContent();

}

}

[zealVampire]

HTTP是基于TCP的， 直接使用SOCKET的话第一步应该是三次握手。 SYN - SYN-ACK -...

可以使用java.net.URL开连接方式，建议使用JDK1.6 因为它优化了似乎支持cookies,
或者使用Apache http client包

[zealVampire]

Hold on... Maybe I am wrong dude, while the socket is created and returns no error, the three way hand shakes should be ready..
Ooop, I should do some tests to confirm it.

[shadowlin]

直接用java的URL和HttpConnection试试?
httpconnection可以设置http的header的.比你这种设置可靠很多啊.
你那个sessionId我怀疑是有问题的..