As part of the quiz/puzzle site one of its key features will be the quiz questions and answers. To begin with they will be in plain text, but I will later move on to multiple choice quizzes to help increase user engagement. I was wanting to create a function that would automatically populate the site with questions and answers daily and to do this I have decided to make use of a web scraper.
Looking online I noticed there were many online sites that were providing this data for free, to make the scripting easier I was particularly interested in those that were making use of incremental URL’s i.e.
Using a while loop in Python I am then able to iterate over each URL and extract data out of each page using Python’s Beautiful Soup library. Most pages would be sectioned by keywords for example the word ‘Questions’ and the word ‘Answers’. In my script I lookout especially for the word “Answers” and place ad code right before it, this way a wonderful banner ad is presented to the user once all questions have been read.
Once prepped with ad code I move on to inserting the quiz in to a MYSQL database using Python’s mysql.connector driver which feeds in to the website online. Creating a scrape tool in Python has required very little effort and I have successfully set them up for multiple sources, scraping quiz content from a different source each day of the week.
If you are wanting a copy of the code for reference please email me via the contact page.