selenium webdriver 不能得到网页全部内容
I am trying to parse a web page, such as this one. Selenium could return some of the content of this page, but not all of them. For example "Professionally Managed by: B & A Associates" is in the web page, but its not returned by the variable 'content' in the script. Any idea why is that, how to solve this problem?
我的脚本程序试着在分析一个页面,如 https://www.forrent.com/apartment-community-profile/1012635
但是selenium webdrive 不能返回页面中的完整内容。比如说页面中的‘Professionally Managed by: B & A Associates’。请问这是什么原因?有什么解决办法吗?
from time import sleep
from geocoder_helpers import normalized
import os
import urllib2
from bs4 import BeautifulSoup
import json
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
url = 'https://www.forrent.com/apartment-community-profile/1012635'
driver = webdriver.Firefox(executable_path='/home/yliu/repos/funnel_objects/listing_sites/geckodriver')
try:
driver.set_page_load_timeout(20)
driver.get(url)
#WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.ID, "contactHeading")))
WebDriverWait(driver, 40)
html = driver.page_source
content = BeautifulSoup(html,"lxml")
driver.quit()
return content
except TimeoutException:
print('time out from contact')
return None
相关推荐
selenium webdriver爬取动态网页
selenium webdriver 爬虫爬取动态网页,里面有chromedriver,windows平台可用,linux需要更换linux的chromedriver。修改代码可爬取其他网站的
selenium-WebDriver的API文档
selenium官方API文档,有RC的,有WebDriver的,哈哈,都是好东西,适合用来做自动化测试。。
webdriver不能启动浏览器,报module ‘selenium.webdriver’ has no attribute”问题解决办法
webdriver不能启动chrome浏览器,报module ‘selenium.webdriver’ has no attribute”问题解决办法 1、先配置下环境变量(谷歌浏览器为例) 步骤:下载好浏览器驱动chromedriver.exe(下载地址:http://chromedriver.storage.googleapis.com/index.html) 拷贝chromedriver.exe放在谷歌浏览器安装目录下和python安装目录下,环境变量path配置浏览器安装路径(如C:\Program Files (x86)\Google\Chrome\Application)和pyth
selenium webdriver+chrome插件.zip
selenium ide的chrome的插件,chrome和firfox的webdriver插件,具体使用和问题查看我博客文章
Selenium webdriver添加cookie实现过程详解
主要介绍了Selenium webdriver添加cookie实现过程详解,文中通过示例代码介绍的非常详细,对大家的学习或者工作具有一定的参考学习价值,需要的朋友可以参考下



