自己动手写网络爬虫

[Andyduflun]

java小白一名，看了自己动手写爬虫这本书，第一个例子完全按书中的代码敲下来的，编译无错误，但运行总是出现异常，下边是代码和错误内容，求大神帮忙
package zhuawang;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.NameValuePair;
import org.apache.commons.httpclient.methods.*;

public class zhuawang {
private static HttpClient httpClient=new HttpClient();
//设置代理服务器
static{
//设置代理服务器的IP地址和端口
httpClient.getHostConfiguration().setProxy("172.27.35.1", 8080);
}
public static boolean downloadPage(String path) throws HttpException,IOException{
InputStream input=null;
OutputStream output=null;
//得到POST方法
PostMethod postMethod=new PostMethod(path);
//测试post方法的参数
NameValuePair[] postData = new NameValuePair[2];
postData[0] = new NameValuePair("name","baidu");
postData[1] = new NameValuePair("pasword","123456");
postMethod.addParameters(postData);
//执行，返回状态码
int statusCode = httpClient.executeMethod(postMethod);
//针对状态码进行处理（简单起见，只处理返回值为200的状态码）
if(statusCode == HttpStatus.SC_OK)
{
input = postMethod.getResponseBodyAsStream();
//得到文件名
String filename = path.substring(path.lastIndexOf('/')+1);
//获得文件输出流
output = new FileOutputStream(filename);

//输出到文件
int tempByte = -1;
while((tempByte=input.read())>0)
{
output.write(tempByte);
}
//关闭输出流
if(input!=null)
input.close();
if(output!=null)
output.close();
return true;
}
return false;
}
//测试代码
public static void main(String[] args) {
//抓取百度首页
try{
zhuawang.downloadPage("http://www.baidu.com");
}catch (HttpException e){
e.printStackTrace();
}catch (IOException e){
e.printStackTrace();
}
}
}

异常提示
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
at org.apache.commons.httpclient.HttpClient.<clinit>(HttpClient.java:66)
at zhuawang.zhuawang.<clinit>(zhuawang.java:17)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 2 more

[Andyduflun]

没人吗。。。

[小伙真帅]

少引用了包 commons-logging.jar java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory 这一行都写得很明白了。

[Andyduflun]

谢谢，我按您的导入了下确实不出现这个问题了，但总是连接超时： 四月 22, 2016 9:14:46 下午 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry 信息: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect 四月 22, 2016 9:14:46 下午 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry 信息: Retrying request 四月 22, 2016 9:15:07 下午 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry 信息: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect 四月 22, 2016 9:15:07 下午 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry 信息: Retrying request 四月 22, 2016 9:15:28 下午 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry 信息: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connect 四月 22, 2016 9:15:28 下午 org.apache.commons.httpclient.HttpMethodDirector executeWithRetry 信息: Retrying request java.net.ConnectException: Connection timed out: connect at java.net.DualStackPlainSocketImpl.connect0(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.<init>(Socket.java:425) at java.net.Socket.<init>(Socket.java:280) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at zhuawang.zhuawang.downloadPage(zhuawang.java:35) at zhuawang.zhuawang.main(zhuawang.java:64)

[wxdawxq]

172.27.35.1 这个地址也是书里面的吗？

[Andyduflun]

引用 4 楼 wxdawxq的回复:

172.27.35.1 这个地址也是书里面的吗？

已解决，就是这个问题跪谢

[数据工厂]

推荐一个大牛的爬虫开发教程，绝对干货：http://blog.csdn.net/youmumzcs/article/details/51373830

[ZJ729286823]

引用 5 楼 blueskiesss 的回复:

[quote=引用 4 楼 wxdawxq的回复:]172.27.35.1 这个地址也是书里面的吗？

已解决，就是这个问题跪谢[/quote] 楼主咋么解决的啊 我也遇到了这个问题 刚学这个

[忘川闻境]

你们是怎么学的，用的书还是10年第一次出版的那本吗？

[qq_38060440]

请问该怎么解决？

[m0_37329573]

同楼上两位问的问题

[辰宇Yyyy]

想问一下jar包在哪里获取呢？