In this post we will be creating a Twitter bot that will scrape some content online and post this as tweets to a Twitter account. Please not that this is for educational purposes only and in this post we will be using a news site to get hold of the details of their latest story.
Twitter offer an API with variations on access levels, it is important to note that for this project an ‘Elevated’ access level will be needed. Meaning you will need to create a Twitter account and register for the ‘Essential’ access level API and then register again for the additional ‘Elevated’ access level. This is fairly easy to do and Twitter is able to approve these very quickly.
Just a small point on the automated tweets, the same tweet can not be posted more than once, this in a way works to our advantage as it means we’d never post the same story more than once.
We begin by specifying the location of our web driver, the driver.get() method will then navigate to the given URL Address. In this example are going to the BBC news page for UK stories. We then apply a delay to allow for all elements of the web page to load.
driver =webdriver.Chrome(executable_path="C:\chromedriver.exe")
driver.get("https://www.bbc.co.uk/news/uk")
driver.implicitly_wait(1)On the web page itself I am interested only in the very latest story, my aim is to scrape the headline and details of the story as marked in green via the image below. To take this I inspect the webpage and take the full XPath of both elements.
Headline = driver.find_element_by_xpath('/html/body/div[8]/div/div/div[2]/div/div[1]/div/div[1]/div/div[1]/div/div[2]/div[1]/a/h3').text
Detail = driver.find_element_by_xpath('/html/body/div[8]/div/div/div[2]/div/div[1]/div/div[1]/div/div[1]/div/div[2]/div[1]/p').text
Once done I move on to Authenticating the Twitter API and then prep the tweet which will contain the scraped content. After this I post the tweet and close the web driver.
auth = tweepy.OAuthHandler(Key1, Key2)
auth.set_access_token(Key3, Key4)
api = tweepy.API(auth)
Tweet = "Latest News: " + Headline + " - " + Detail + " #LatestNews #News #Scriptopia"
api.update_status(Tweet)
driver.close()
driver.quit()The full source code for this project can be found below:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from bs4 import BeautifulSoup
import re
import tweepy
driver = webdriver.Chrome(executable_path="C:\chromedriver.exe")
driver.get("https://www.bbc.co.uk/news/uk")
driver.implicitly_wait(1)
Headline = driver.find_element_by_xpath('/html/body/div[8]/div/div/div[2]/div/div[1]/div/div[1]/div/div[1]/div/div[2]/div[1]/a/h3').text
Detail = driver.find_element_by_xpath('/html/body/div[8]/div/div/div[2]/div/div[1]/div/div[1]/div/div[1]/div/div[2]/div[1]/p').text
auth = tweepy.OAuthHandler(Key1, Key2)
auth.set_access_token(Key3, Key4)
api = tweepy.API(auth)
Tweet = "Latest News: " + Headline + " - " + Detail + " #LatestNews #News #Scriptopia"
api.update_status(Tweet)
driver.close()
driver.quit()