如何在表格输入Selenium后抓取第二页？

piok6c0g 于 8个月前发布在其他

关注(0)|答案(1)|浏览(70)

我有这个代码，在第一页上输入一个邮政编码，但我越来越卡在如何刮第二页。我只想收集表单中的所有字段并将其打印到CSV。任何帮助将不胜感激。我尝试使用xpath和element. id。第二个页面有一个动态网址，所以我不能直接引用。

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import requests

chrome_options = Options()
chrome_options.add_argument("start-maximized")
chrome_options.add_argument("disable-infobars")
chrome_options.add_argument("--disable-extensions")
option = webdriver.ChromeOptions()
option.add_experimental_option("detach", True)
driver = webdriver.Chrome(options = option)
driver.get("https://www.thegeneral.com")
element = WebDriverWait(driver, 1).until(EC.element_to_be_clickable((By.ID, "free-quote-zip")))
element.click()
element.clear()
element.send_keys("63013")
element.submit()
def click_through_to_new_page(link_text):
  link = browser.find_element_by_link_text('my link')
  link.click()
  def link_has_gone_stale():
    try:
      # poll the link with an arbitrary call
      link.find_elements_by_id('doesnt-matter')
      return False
    except StaleElementReferenceException:
      return True
  wait_for(link_has_gone_stale)
results = driver.find_elements(By.XPATH, "//[@id="autoInsurance"]")
print(results)

selenium

来源：https://stackoverflow.com/questions/77317474/how-to-scrape-second-page-after-form-input-selenium

1条答案

按热度按时间

disho6za1#

您可以填写以下表格。

import time
from asyncio import wait_for

from selenium import webdriver
from selenium.common import StaleElementReferenceException
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import requests

chrome_options = Options()
chrome_options.add_argument("start-maximized")
chrome_options.add_argument("disable-infobars")
chrome_options.add_argument("--disable-extensions")
option = webdriver.ChromeOptions()
option.add_experimental_option("detach", True)
driver = webdriver.Chrome(options = option)
driver.get("https://www.thegeneral.com")
element = WebDriverWait(driver, 1).until(EC.element_to_be_clickable((By.ID, "free-quote-zip")))
element.click()
element.clear()
element.send_keys("63013")
element.submit()


 time.sleep(5)
file = open('csvfile.csv','w')
autoinsurance_options = driver.find_element(By.ID, "autoInsurance").find_elements(By.TAG_NAME, "option")
for option in autoinsurance_options:
    file.write(option.text + "\n")

time.sleep(5)
own_home_options = driver.find_element(By.NAME, "ownsHomeFlag").find_elements(By.TAG_NAME, "option")
for option in own_home_options:
    file.write(option.text + "\n")

//time.sleep(5)
//driver.find_element(By.ID, "autoInsurance").send_keys("No")
//driver.find_element(By.NAME, "ownsHomeFlag").send_keys("Rent")
//driver.find_element(By.ID, "emailAddress").send_keys("[email protected]")

赞(0）回复(0）举报 8个月前

我来回答

如何在表格输入Selenium后抓取第二页？

1条答案

相关问题

热门标签

最新问答