scrapy运行过程中停住且无任何报错,请问该如何定位

汇讯猎头 汇讯猎头 墨衍会员 · AI 创作全网分发 墨衍智能分发 通过墨衍一键同步至各平台 可同步平台 CSDN微信公众号微博知乎掘金百家号博客园抖音小红书 你的文章也可以这样分发 墨衍会员 ¥399起/年,写一次发全网 我也要 + 加入社区
操作系统环境:mac
python版本:2.7

现象:
运行scrapy的过程中,突然之间停止,且没有任何报错,日志显示如下,请问哪位高人帮忙看看为何停止了,遇到这类问题该如何定位:谢谢!


……前面略

2017-01-23 13:42:39 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4720106
2017-01-23 13:42:39 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4720094
2017-01-23 13:42:39 [scrapy] DEBUG: Crawled (200) <GET http://nocutnews.co.kr/news/list?c1=254&c2=259> (referer: None)
2017-01-23 13:42:40 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:60033/wd/hub/session {"requiredCapabilities": {}, "desiredCapabilities": {"platform": "ANY", "browserName": "phantomjs", "version": "", "javascriptEnabled": true}}
2017-01-23 13:42:40 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2017-01-23 13:42:40 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:60033/wd/hub/session/c13fd4f0-e12e-11e6-a80e-87e393dbd863/url {"url": "http://nocutnews.co.kr/news/list?c1=254&c2=259", "sessionId": "c13fd4f0-e12e-11e6-a80e-87e393dbd863"}
2017-01-23 13:42:46 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2017-01-23 13:42:46 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:60033/wd/hub/session/c13fd4f0-e12e-11e6-a80e-87e393dbd863/source {"sessionId": "c13fd4f0-e12e-11e6-a80e-87e393dbd863"}
2017-01-23 13:42:46 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2017-01-23 13:42:46 [selenium.webdriver.remote.remote_connection] DEBUG: DELETE http://127.0.0.1:60033/wd/hub/session/c13fd4f0-e12e-11e6-a80e-87e393dbd863 {"sessionId": "c13fd4f0-e12e-11e6-a80e-87e393dbd863"}
2017-01-23 13:42:46 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2017-01-23 13:42:46 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4708177
2017-01-23 13:42:47 [scrapy] DEBUG: Crawled (200) <GET http://nocutnews.co.kr/news/list?c1=254&c2=260> (referer: None)
2017-01-23 13:42:48 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:60119/wd/hub/session {"requiredCapabilities": {}, "desiredCapabilities": {"platform": "ANY", "browserName": "phantomjs", "version": "", "javascriptEnabled": true}}
2017-01-23 13:42:48 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2017-01-23 13:42:48 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:60119/wd/hub/session/c5b49430-e12e-11e6-a4d5-e100c4c26ce4/url {"url": "http://nocutnews.co.kr/news/list?c1=254&c2=260", "sessionId": "c5b49430-e12e-11e6-a4d5-e100c4c26ce4"}
2017-01-23 13:43:51 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2017-01-23 13:43:51 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:60119/wd/hub/session/c5b49430-e12e-11e6-a4d5-e100c4c26ce4/source {"sessionId": "c5b49430-e12e-11e6-a4d5-e100c4c26ce4"}
2017-01-23 13:43:51 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2017-01-23 13:43:51 [selenium.webdriver.remote.remote_connection] DEBUG: DELETE http://127.0.0.1:60119/wd/hub/session/c5b49430-e12e-11e6-a4d5-e100c4c26ce4 {"sessionId": "c5b49430-e12e-11e6-a4d5-e100c4c26ce4"}
2017-01-23 13:43:51 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4722146
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4721697
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4721446
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4721364
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4718899
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4716210
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4715884
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4715666
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4715796
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4715192
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4712395
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4711118
2017-01-23 13:43:51 [root] DEBUG: bloomfilter follow link = http://nocutnews.co.kr/news/4710328
2017-01-23 13:43:51 [scrapy] INFO: Crawled 434 pages (at 3 pages/min), scraped 390 items (at 0 items/min)
2017-01-23 13:43:52 [scrapy] DEBUG: Crawled (200) <GET http://nocutnews.co.kr/news/list?c1=282> (referer: None)
2017-01-23 13:43:54 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:60210/wd/hub/session {"requiredCapabilities": {}, "desiredCapabilities": {"platform": "ANY", "browserName": "phantomjs", "version": "", "javascriptEnabled": true}}
2017-01-23 13:43:54 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2017-01-23 13:43:54 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:60210/wd/hub/session/ed09b380-e12e-11e6-ba50-b128e0b318bc/url {"url": "http://nocutnews.co.kr/news/list?c1=282", "sessionId": "ed09b380-e12e-11e6-ba50-b128e0b318bc"}

...全文

相关推荐

python3 安装scrapy报错的解决方法

方法一:在文档里有连接,电机进去下载对应的文件,再pip安装。这样的话稍微有点复杂。 方法二:直接安装依赖包,然后直接pip install scrapy就行。

松松的私房菜

裁判文书网Selenium爬虫工程化实践指南

裁判文书网作为典型动态反爬网站,其前端JS校验、浏览器指纹识别、行为序列建模与IP信誉联动构成多层防御体系。理解其反爬原理是构建稳定采集系统的基础,而Selenium凭借对真实浏览器环境的高保真模拟,成为处理动态token、Shadow DOM、iframe嵌套及人机行为模拟的主流技术选型。该方案强调工程化落地:从chromedriver版本精准匹配、登录流程模块化封装,到验证码OCR优化、Vue列表页翻页稳定性保障,再到字段结构化解析与法律合规脱敏,最终实现可复用、可审计、可交接的工业级数据采集能力。

weixin_33824363

scrapy安装教程

为安装scrapy烦恼,为安装scrapy搜集各种资源和教程,可以来这看一看

work_you_will_see

python3 scrapy安装教程(详细)

详细的python3中scrapy模块的安装教程,希望大家都能顺利安装和编程,

赢得青楼薄幸名

重装系统后python环境问题后续——scrapy生成spider报错:Fatal error in launcher: Unable to create process using

重装系统后python环境问题后续:scrapy报错:Fatal error in launcher: Unable to create process using'”c:\users\administrator\appdata\local\programs\python\python37\python.exe” “D:\Users\Administrator\AppData\Local\Programs\Python\Python37\Scripts\scrapy.exe” startproject spider’: ???????????解决路径 报错:Fatal error in lau

weixin_38599412
评论 4
收藏 已收藏
分享
返回顶部
汇讯猎头
脚本语言 37,738 成员 34,208 内容