双11狂欢大促

盲盒赢 定制羽绒服

开启惊喜盲盒,抽取你的专属好礼

— 惊喜奖品抢先看 —

selenium webdriver 不能得到网页全部内容

iwsyang iwsyang 墨衍会员 · AI 创作全网分发 墨衍智能分发 通过墨衍一键同步至各平台 可同步平台 CSDN微信公众号微博知乎掘金百家号博客园抖音小红书 你的文章也可以这样分发 墨衍会员 ¥399起/年,写一次发全网 我也要 + 加入社区
https://www.forrent.com/apartment-community-profile/1012635

I am trying to parse a web page, such as this one. Selenium could return some of the content of this page, but not all of them. For example "Professionally Managed by: B & A Associates" is in the web page, but its not returned by the variable 'content' in the script. Any idea why is that, how to solve this problem?
我的脚本程序试着在分析一个页面,如 https://www.forrent.com/apartment-community-profile/1012635

但是selenium webdrive 不能返回页面中的完整内容。比如说页面中的‘Professionally Managed by: B & A Associates’。请问这是什么原因?有什么解决办法吗?

from time import sleep
from geocoder_helpers import normalized

import os
import urllib2
from bs4 import BeautifulSoup
import json

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from pyvirtualdisplay import Display

display = Display(visible=0, size=(800, 600))
display.start()

url = 'https://www.forrent.com/apartment-community-profile/1012635'
driver = webdriver.Firefox(executable_path='/home/yliu/repos/funnel_objects/listing_sites/geckodriver')
try:
driver.set_page_load_timeout(20)
driver.get(url)

#WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.ID, "contactHeading")))
WebDriverWait(driver, 40)
html = driver.page_source
content = BeautifulSoup(html,"lxml")
driver.quit()
return content
except TimeoutException:
print('time out from contact')
return None
...全文

相关推荐

评论 7
收藏 已收藏
分享
返回顶部
iwsyang
脚本语言 37,738 成员 34,208 内容