{"id":1549,"date":"2021-11-27T14:55:53","date_gmt":"2021-11-27T14:55:53","guid":{"rendered":"https:\/\/laserphotonics.uk\/?p=1549"},"modified":"2021-11-27T14:55:57","modified_gmt":"2021-11-27T14:55:57","slug":"controlling-the-web-with-python","status":"publish","type":"post","link":"https:\/\/laserphotonics.uk\/?p=1549","title":{"rendered":"Controlling the Web with Python"},"content":{"rendered":"\n

<\/h1>\n\n\n\n
\"Will<\/a><\/figure>\n\n\n\n

Will Koehrsen<\/a>Mar 10, 2018\u00b79 min read<\/a><\/a><\/p>\n\n\n\n

https:\/\/towardsdatascience.com\/controlling-the-web-with-python-6fceb22c5f08<\/a><\/p>\n\n\n\n

An adventure in simple web automation<\/strong><\/p>\n\n\n\n

Problem:<\/strong> Submitting class assignments requires navigating a maze of web pages so complex that several times I\u2019ve turned an assignment in to the wrong place. Also, while this process only takes 1\u20132 minutes, it sometimes seems like an insurmountable barrier (like when I\u2019ve finished an assignment way too late at night and I can barely remember my password).<\/p>\n\n\n\n

Solution: <\/strong>Use Python to automatically submit completed assignments! Ideally, I would be able to save an assignment, type a few keys, and have my work uploaded in a matter of seconds. At first this sounded too good to be true, but then I discovered selenium<\/a>, a tool which can be used with Python to navigate the web for you.<\/p>\n\n\n\n

\"\"\/
Obligatory XKCD<\/a><\/figcaption><\/figure>\n\n\n\n

Anytime we find ourselves repeating tedious actions on the web with the same sequence of steps, this is a great chance to write a program to automate the process for us. With selenium and Python, we just need to write a script once, and which then we can run it as many times and save ourselves from repeating monotonous tasks (and in my case, eliminate the chance of submitting an assignment in the wrong place)!<\/p>\n\n\n\n

Here, I\u2019ll walk through the solution I developed to automatically (and correctly) submit my assignments. Along the way, we\u2019ll cover the basics of using Python and selenium to programmatically control the web. While this program does work (I\u2019m using it every day!) it\u2019s pretty custom so you won\u2019t be able to copy and paste the code for your application. Nonetheless, the general techniques here can be applied to a limitless number of situations. (If you want to see the complete code, it\u2019s available on GitHub<\/a>).<\/p>\n\n\n\n

Approach<\/h1>\n\n\n\n

Before we can get to the fun part of automating the web, we need to figure out the general structure of our solution. Jumping right into programming without a plan is a great way to waste many hours in frustration. I want to write a program to submit completed course assignments to the correct location on Canvas (my university\u2019s \u201clearning management system\u201d<\/a>). Starting with the basics, I need a way to tell the program the name of the assignment to submit and the class. I went with a simple approach and created a folder to hold completed assignments with child folders for each class. In the child folders, I place the completed document named for the particular assignment. The program can figure out the name of the class from the folder, and the name of the assignment by the document title.<\/p>\n\n\n\n

Here\u2019s an example where the name of the class is EECS491 and the assignment is \u201cAssignment 3 \u2014 Inference in Larger Graphical Models\u201d.<\/p>\n\n\n\n

\"\"\/
File structure (left) and Complete Assignment (right)<\/figcaption><\/figure>\n\n\n\n

The first part of the program is a loop to go through the folders to find the assignment and class, which we store in a Python tuple:<\/p>\n\n\n\n

# os for file management
import os# Build tuple of (class, file) to turn in
submission_dir = 'completed_assignments'dir_list = list(os.listdir(submission_dir))for directory in dir_list:
    file_list = list(os.listdir(os.path.join(submission_dir, 
directory)))
    if len(file_list) != 0:
        file_tup = (directory, file_list[0])
    
print(file_tup)('EECS491', 'Assignment 3 - Inference in Larger Graphical Models.txt')<\/strong><\/pre>\n\n\n\n
This takes care of file management and the program now knows the program and the assignment to turn in. The next step is to use selenium to navigate to the correct webpage and upload the assignment.<\/p>\n\n\n\n
Web Control with Selenium<\/h2>\n\n\n\n
To get started with selenium, we import the library and create a web driver, which is a browser that is controlled by our program. In this case, I\u2019ll use Chrome as my browser and send the driver to the Canvas website where I submit assignments.<\/p>\n\n\n\n
import selenium# Using Chrome to access web
driver = webdriver.Chrome()# Open the website
driver.get('https:\/\/canvas.case.edu'<\/a>)<\/pre>\n\n\n\n
When we open the Canvas webpage, we are greeted with our first obstacle, a login box! To get past this, we will need to fill in an id and a password and click the login button.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Imagine the web driver as a person who has never seen a web page before: we need to tell it exactly where to click, what to type, and which buttons to press. There are a number of ways to tell our web driver what elements to find, all of which use selectors. A selector<\/a> is a unique identifier for an element on a webpage. To find the selector for a specific element, say the CWRU ID box above, we need to inspect the webpage. In Chrome, this is done by pressing \u201cctrl + shift + i\u201d or right clicking on any element and selecting \u201cInspect\u201d. This brings up the Chrome developer tools<\/a>, an extremely useful application which shows the HTML underlying any webpage<\/a>.<\/p>\n\n\n\n
To find a selector for the \u201cCWRU ID\u201d box, I right clicked in the box, hit \u201cInspect\u201d and saw the following in developer tools. The highlighted line corresponds to the id box element (this line is called an HTML tag).<\/p>\n\n\n\n
\"\"\/
HTML in Chrome developer tools for the webpage<\/figcaption><\/figure>\n\n\n\n
This HTML might look overwhelming, but we can ignore the majority of the information and focus on the id = \"username\"<\/code> and name=\"username\"<\/code> parts. (these are known as attributes of the HTML tag).<\/p>\n\n\n\n
To select the id box with our web driver, we can use either the id<\/code> or name<\/code> attribute we found in the developer tools. Web drivers in selenium have many different methods for selecting elements on a webpage and there are often multiple ways to select the exact same item:<\/p>\n\n\n\n
# Select the id box
id_box = driver.find_element_by_name('username')# Equivalent Outcome! 
id_box = driver.find_element_by_id('username')<\/pre>\n\n\n\n
Our program now has access to the id_box<\/code> and we can interact with it in various ways, such as typing in keys, or clicking (if we have selected a button).<\/p>\n\n\n\n
# Send id information
id_box.send_keys('my_username')<\/pre>\n\n\n\n
We carry out the same process for the password box and login button, selecting each based on what we see in the Chrome developer tools. Then, we send information to the elements or click on them as needed.<\/p>\n\n\n\n
# Find password box
pass_box = driver.find_element_by_name('password')# Send password
pass_box.send_keys('my_password')# Find login button
login_button = driver.find_element_by_name('submit')# Click login
login_button.click()<\/pre>\n\n\n\n
Once we are logged in, we are greeted by this slightly intimidating dashboard:<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
We again need to guide the program through the webpage by specifying exactly the elements to click on and the information to enter. In this case, I tell the program to select courses from the menu on the left, and then the class corresponding to the assignment I need to turn in:<\/p>\n\n\n\n
# Find and click on list of courses
courses_button = driver.find_element_by_id('global_nav_courses_link')courses_button.click()# Get the name of the folder
folder = file_tup[0]
    
# Class to select depends on folder
if folder == 'EECS491':
    class_select = driver.find_element_by_link_text('Artificial Intelligence: Probabilistic Graphical Models (100\/10039)')elif folder == 'EECS531':
    class_select = driver.find_element_by_link_text('Computer Vision (100\/10040)')# Click on the specific class
class_select.click()<\/pre>\n\n\n\n
The program finds the correct class using the name of the folder we stored in the first step. In this case, I use the selection method find_element_by_link_text<\/code> to find the specific class. The \u201clink text\u201d for an element is just another selector we can find by inspecting the page. :<\/p>\n\n\n\n
\"\"\/
Inspecting the page to find the selector for a specific class<\/figcaption><\/figure>\n\n\n\n
This workflow may seem a little tedious, but remember, we only have to do it once when we write our program! After that, we can hit run as many times as we want and the program will navigate through all these pages for us.<\/p>\n\n\n\n
We use the same \u2018inspect page \u2014 select element \u2014 interact with element\u2019 process to get through a couple more screens. Finally, we reach the assignment submission page:<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
At this point, I could see the finish line, but initially this screen perplexed me. I could click on the \u201cChoose File\u201d box pretty easily, but how was I supposed to select the actual file I need to upload? The answer turns out to be incredibly simple! We locate the Choose File<\/code> box using a selector, and use the send_keys<\/code> method to pass the exact path of the file (called file_location<\/code> in the code below) to the box:<\/p>\n\n\n\n
# Choose File button
choose_file = driver.find_element_by_name('attachments[0][uploaded_data]')# Complete path of the file
file_location = os.path.join(submission_dir, folder, file_name)# Send the file location to the button
choose_file.send_keys(file_location)<\/pre>\n\n\n\n
That\u2019s it! By sending the exact path of the file to the button, we can skip the whole process of navigating through folders to find the right file. After sending the location, we are rewarded with the following screen showing that our file is uploaded and ready for submission.<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Now, we select the \u201cSubmit Assignment\u201d button, click, and our assignment is turned in!<\/p>\n\n\n\n
# Locate submit button and click
submit_assignment = driver.find_element_by_id('submit_file_button')
submit_assignent.click()<\/pre>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Cleaning Up<\/h2>\n\n\n\n
File management is always a critical step and I want to make sure I don\u2019t re-submit or lose old assignments. I decided the best solution was to store a single file to be submitted in the completed_assignments<\/code> folder at any one time and move files to asubmitted_assignments<\/code> folder once they had been turned in. The final bit of code uses the os module to move the completed assignment by renaming it with the desired location:<\/p>\n\n\n\n
# Location of files after submission
submitted_file_location = os.path.join(submitted_dir, submitted_file_name)# Rename essentially copies and pastes files
os.rename(file_location, submitted_file_location)<\/pre>\n\n\n\n
All of the proceeding code gets wrapped up in a single script, which I can run from the command line. To limit opportunities for mistakes, I only submit one assignment at a time, which isn\u2019t a big deal given that it only takes about 5 seconds to run the program!<\/p>\n\n\n\n
Here\u2019s what it looks like when I start the program:<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
The program provides me with a chance to make sure this is the correct assignment before uploading. After the program has completed, I get the following output:<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
While the program is running, I can watch Python go to work for me:<\/p>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n
Conclusions<\/h1>\n\n\n\n
The technique of automating the web with Python works great for many tasks, both general and in my field of data science. For example, we could use selenium to automatically download new data files every day (assuming the website doesn\u2019t have an API<\/a>). While it might seem like a lot of work to write the script initially, the benefit comes from the fact that we can have the computer repeat this sequence as many times as want in exactly the same manner. The program will never lose focus and wander off to Twitter.<\/mark> It will faithfully carry out the same exact series of steps with perfect consistency (which works great until the website changes).<\/p>\n\n\n\n
I should mention you do want to be careful before you automate critical tasks. This example is relatively low-risk as I can always go back and re-submit assignments and I usually double-check the program\u2019s handiwork. Websites change, and if you don\u2019t change the program in response you might end up with a script that does something completely different than what you originally intended!<\/p>\n\n\n\n
In terms of paying off, this program saves me about 30 seconds for every assignment and took 2 hours to write. So, if I use it to turn in 240 assignments, then I come out ahead on time! However, the payoff of this program is in designing a cool solution to a problem and learning a lot in the process. While my time might have been more effectively spent working on assignments rather than figuring out how to automatically turn them in, I thoroughly enjoyed this challenge. There are few things as satisfying as solving problems, and Python turns out to be a pretty good tool for doing exactly that.<\/p>\n\n\n\n
As always, I welcome feedback and constructive criticism. I can be reached on Twitter @koehrsen_will<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
Will KoehrsenMar 10, 2018\u00b79 min read https:\/\/towardsdatascience.com\/controlling-the-web-with-python-6fceb22c5f08 An adventure in simple web automation Problem: Submitting class assignments requires navigating a maze of web pages so complex that several times I\u2019ve turned an assignment in to the wrong place. Also, while this process only takes 1\u20132 minutes, it sometimes seems like an insurmountable barrier (like when I\u2019ve […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[2],"tags":[],"_links":{"self":[{"href":"https:\/\/laserphotonics.uk\/index.php?rest_route=\/wp\/v2\/posts\/1549"}],"collection":[{"href":"https:\/\/laserphotonics.uk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/laserphotonics.uk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/laserphotonics.uk\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/laserphotonics.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1549"}],"version-history":[{"count":1,"href":"https:\/\/laserphotonics.uk\/index.php?rest_route=\/wp\/v2\/posts\/1549\/revisions"}],"predecessor-version":[{"id":1550,"href":"https:\/\/laserphotonics.uk\/index.php?rest_route=\/wp\/v2\/posts\/1549\/revisions\/1550"}],"wp:attachment":[{"href":"https:\/\/laserphotonics.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1549"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/laserphotonics.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1549"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/laserphotonics.uk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1549"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}