关于HttpClient和爬虫的问题

just_a_name_ 2016-07-19 12:25:24

1.关于java设置代理服务器的问题
HttpClient httpclient=new HttpClient();
httpClient.getHostConfiguration().setProxy("192.168.0.1", 9527);
httpClient.getParams().setAuthenticationPreemptive(true);
httpClient.getParams().setParameter(CredentialsProvider.PROVIDER, new MyProxyCredentialsProvider());
httpClient.getState().setProxyCredentials( new AuthScope("192.168.0.1",AuthScope.ANY_PORT,AuthScope.ANY_REALM),
new UsernamePasswordCredentials("username", "password"));
其中new MyProxyCredentialsProvider()对象new 不出来是为什么。。
我导的包有
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.apache.commons.httpclient.Header;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.auth.CredentialsProvider;
import org.apache.commons.httpclient.methods.PostMethod;

第二个问题是我对着书写了一段代码，但是运行的时候有问题
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.apache.commons.collections4.map.StaticBucketMap;
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.methods.PostMethod;
import org.omg.CORBA.PRIVATE_MEMBER;
import org.omg.CORBA.PUBLIC_MEMBER;
public class RetrivePage {
private static HttpClient httpClient = new HttpClient();
static {
httpClient.getHostConfiguration().setProxy("172.17.8.18.84", 8080);
}

public static boolean downLoadPage (String path) throws HttpException, IOException{
InputStream input = null;
OutputStream output = null;
//得到post方法
PostMethod postMethod = new PostMethod(path);
//设置post方法的参数

NameValuePair[] postData = new NameValuePair[2];
postData[0] = new NameValuePair("name", "lietu");
postData[1] = new NameValuePair("password", "*****");
postMethod.addParameters(postData);
//执行，返回状态码
int statusCode= httpClient.executeMethod(postMethod);
//针对状态码进行处理，先是只处理返回值为200的状态码
if(statusCode == HttpStatus.SC_OK) {
input = postMethod.getResponseBodyAsStream();
//得到文件名
String filename = path.substring(path.lastIndexOf('/') + 1);
//获得文件输出流
output = new FileOutputStream(filename);

//输出文件
int tempByte = -1;

while((tempByte = input.read()) > 0) {
output.write(tempByte);
}
//关闭输出流
if(input != null) {
input.close();
}
if (output != null){
output.close();
}
return true;
}
return false;
}
/*
* 测试代码
*/
public static void main(String[] args) throws IOException {
//抓取首页，输出
try {
RetrivePage.downLoadPage("http://www.lietu.com/");
}catch(HttpException e) {
//TODO Auto-generated catch block
e.printStackTrace();
}
}
}
我出现的问题是
七月 19, 2016 12:08:00 上午 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
信息: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect
七月 19, 2016 12:08:00 上午 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry

说链接超时，我感觉是httpClient.getHostConfiguration().setProxy("172.17.8.18.84", 8080);这段有问题，那个ip地址我ping不上，但是又不知道改成什么好。。求大神指教。还有我想想问问学习爬虫有什么好的资料推荐么？

...全文