Apartment Scraping with Selenium/Python - Can't Scrape a New Tab?
I've Put Together a Small Program Data Harvester with the Help of My Humble Assistant, Chatgpt, to Scrape Apartments. Com and Look for All the Names, Price...
I've put together a small program data harvester with the help of my humble assistant, ChatGPT, to scrape Apartments.com and look for all the names, price ranges, numbers, etc of the apartment complexes here in my city.
It functions partially, but cant seem to find the .css code for ".phoneNumber" on a new tab. I've tried to make it look for different obvious CSS, HTML, hrefs, in a few different ways now. It just can't seem to find any of it as soon as it looks away from the main tab.
Now I admit I'm pretty inexperienced at coding and have never put together anything complex, but it looks like it ought to work to me. If I could get some help I'd be super appreciative! The output and code is below:
Console Log:
beginning pagination
Park Wilshire
2424 Wilshire Blvd, Los Angeles, CA 90057
$1,495 - 2,870
Studio - 1 Bed
Traceback (most recent call last):
File "C:\Users\...\aptScraper\main.py", line 1108, in <module>
phone_link = driver.find_element(By.XPATH, "//a[contains(@class,'.phoneNumber')]")
And now the real code:
from selenium import webdriver
import csv
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
import time
import re
# Open a browser and navigate to apartments.com
driver = webdriver.Chrome()
driver.get("")
# Find the search box and input "Los Angeles Ca"
search_box = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "searchBarLookup")))
search_box.send_keys("Los Angeles, CA")
# Click the search button
search_box.send_keys(Keys.RETURN)
# Wait for the first page of listings to load
apartments = WebDriverWait(driver, 15).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".placard")))
print("beginning pagination")
# Store the information in a CSV file
with open('apartments.csv', mode='w', encoding='utf-8', newline='') as file:
writer = csv.writer(file)
writer.writerow(['Name', 'Address', 'Rent', 'Bedrooms', 'Phone'])
while True:
for apartment in apartments:
try:
name = apartment.find_element(By.CSS_SELECTOR, ".property-title, .js-placardTitle")
print(name.text)
except:
continue
address = apartment.find_element(By.CSS_SELECTOR, ".property-address")
print(address.text)
try:
rent = apartment.find_element(By.CSS_SELECTOR, ".property-pricing, .property-rents")
print(rent.text)
except:
continue
bedrooms = apartment.find_element(By.CSS_SELECTOR, ".property-beds")
print(bedrooms.text)
phone_number = None
apartment_link = apartment.find_element(By.CSS_SELECTOR, ".property-link").get_attribute("href")
driver.execute_script(f"window.open('{apartment_link}');")
driver.switch_to.window(driver.window_handles[-1])
time.sleep(1)
#problem code is here
phone_link = driver.find_element(By.XPATH, "//a[contains(@class,'.phoneNumber')]")
if phone_link:
phone_number = re.search(r'\d{10}', phone_link.get_attribute('href')).group()
print("phone number found!", phone_number, " for: ", name.text)
writer.writerow([name.text, address.text, rent.text, bedrooms.text, phone_number])
else:
print(f"No phone number found for {name.text} at {address.text}")
driver.close()
driver.switch_to.window(driver.window_handles[0])
# Check if there is a next page button
time.sleep(1)
next_button = driver.find_element(By.CSS_SELECTOR, ".next")
if "disabled" in next_button.get_attribute("class"):
break
next_button.click()
apartments = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".placard")))
And - needless to say - it crashes before completion as it cannot find a valid ".phoneNumber" entry. It is definitely there when I inspect the page elements though. What gives?
The desired apartment's tab opens, but I cannot figure out how to make the selenium locate the ".phoneNumber" element within the new tab, or really any element at all. Please advise
1 Answer
Solved!
The issue was 2 sided. One, when I would update the find_element() to point to a valid element on the page, it would then not run the writer without crashing, as the original element references from the last page were lost.
Solution in my case was to use copy.deepcopy(rent, etc.) on whatever was lost during the transition.