python 我怎么能一次从一个只在滚动后显示响应的网站上抓取所有数据？

x7yiwoj4 于 5个月前发布在 Python

关注(0)|答案(1)|浏览(79)

我试图从一个网站刮大学的名称和地址：https://www.collegenp.com/2-science-colleges/，但问题是，我只得到了列表中的前11所大学的数据，而没有得到其他大学的数据.我已经尝试了我所知道的一切.但没有方法工作.
我的代码是：

from selenium import webdriver
import bs4
from bs4 import BeautifulSoup
import requests
import pandas as pd
from time import sleep

driver=webdriver.Chrome('C:/Users/acer/Downloads/chromedriver.exe')
driver.get('https://www.collegenp.com/2-science-colleges/')

driver.refresh()
sleep(20)

page=requests.get("https://www.collegenp.com/2-science-colleges/")

college = []
location=[]

soup= BeautifulSoup(page.content,'html.parser')

for a in soup.find_all('div',attrs={'class':'media'}):
  name=a.find('h3',attrs={'class':'college-name'})
  college.append(name.text)
  loc=a.find('span',attrs={'class':'college-address'})
  location.append(loc.text)

df=pd.DataFrame({'College name':college,'Locations':location})
df.to_csv('hell.csv',index=False,encoding='utf-8')

字符串
有没有什么方法可以让我把所有的数据都 * 出来？

python

来源：https://stackoverflow.com/questions/67358466/how-can-i-scrape-all-data-at-once-from-a-website-which-shows-the-response-only-a

1条答案

按热度按时间

q35jwt9p1#

您可以使用此代码从下一页获取信息：

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = "https://www.collegenp.com/2-science-colleges/"

headers = {"X-Requested-With": "XMLHttpRequest"}
data = {"state": "on", "action": "filter", "count": "0"}

all_data = []
for page in range(0, 5):  # <-- increase number of pages here
    print("Getting page {}..".format(page))

    data["count"] = page * 10
    soup = BeautifulSoup(
        requests.post(url, data=data, headers=headers).content,
        "html.parser",
    )

    for c in soup.select(".college-name"):
        all_data.append(
            {
                "College name": c.get_text(strip=True),
                "Location": c.find_next(class_="college-address").get_text(
                    strip=True
                ),
            }
        )

df = pd.DataFrame(all_data)
print(df)
df.to_csv("data.csv", index=False)

字符串
印刷品：

College name                  Location
0                     Caspian Valley College,Lalitpur      Kumaripati, Lalitpur
1      Advance Academy and Republica College,Lalitpur      Kumaripati, Lalitpur
2              Araniko International Academy,Lalitpur       Satdobato, Lalitpur
3   Bagiswori Secondary School, Taulachhen, Bhakta...    Chyamhasing, Bhaktapur
4              Bajra Barahi Secondary School,Lalitpur       Chapagaon, Lalitpur
5              Bhanubhakta Memorial College,Kathmandu       Lazimpat, Kathmandu
6                  Damak Model Secondary School,Jhapa              Damak, Jhapa
7                         Damak Multiple Campus,Jhapa              Damak, Jhapa
8                           Einstein Academy,Lalitpur       Thasikhel, Lalitpur
9                   Hari Khetan Multiple Campus,Parsa            Birganj, Parsa
10                       Kankai Adarsha Campus,Morang         Birtamode, Morang
11          Lumbini Adarsh Degree College,Nawalparasi     Kawasoti, Nawalparasi
12            Madhyabindu Multiple Campus,Nawalparasi     Kawasoti, Nawalparasi
13                Marshyangdi Multiple Campus,Lamjung       Besishahar, Lamjung

...

型
并保存data.csv（来自LibreOffice的屏幕截图）：

的数据