Forming a nasty POST request using Requests Python library

Question

I am working on a Python script that automatically helps me add classes in my school's add-a-class web interface. What I've done so far is, I am able to be authenticated into the website and access the HTML of this add-a-class page. The interface works like so:

The items under Status indicate classes I am currently taking this term.

Under "Add Classes Worksheet", you have a few input 5 digit number called a CRN i.e. 34688. You can type in 10 CRNs at a time.

I was thinking I could simply query the URL with the CRNs that I am interested in adding. Thinking it would be simple, I opened up devtools, and it turns out the POST is a total mess.

Here is the raw form data that I get from the URL it queries under Chrome Dev Tools when I try to add a class in the "Add Classes Worksheet"

term_in=201535&RSTS_IN=DUMMY&assoc_term_in=DUMMY&CRN_IN=DUMMY&start_date_in=DUMMY&end_date_in=DUMMY&SUBJ=DUMMY&CRSE=DUMMY&SEC=DUMMY&LEVL=DUMMY&CRED=DUMMY&GMOD=DUMMY&TITLE=DUMMY&MESG=DUMMY&REG_BTN=DUMMY&MESG=DUMMY&RSTS_IN=&assoc_term_in=201535&CRN_IN=34688&start_date_in=03%2F28%2F2016&end_date_in=06%2F11%2F2016&SUBJ=ECEC&CRSE=451&SEC=001&LEVL=Undergraduate+Quarter&CRED=++++3.000&GMOD=Standard+Letter&TITLE=Computer+Arithmetic&MESG=DUMMY&RSTS_IN=&assoc_term_in=201535&CRN_IN=31109&start_date_in=03%2F28%2F2016&end_date_in=06%2F11%2F2016&SUBJ=BIO&CRSE=141&SEC=073&LEVL=Undergraduate+Quarter&CRED=++++0.000&GMOD=Non+Gradeable+Unit&TITLE=Essential+Biology&MESG=DUMMY&RSTS_IN=&assoc_term_in=201535&CRN_IN=31099&start_date_in=03%2F28%2F2016&end_date_in=06%2F11%2F2016&SUBJ=BIO&CRSE=141&SEC=B&LEVL=Undergraduate+Quarter&CRED=++++4.500&GMOD=Standard+Letter&TITLE=Essential+Biology&MESG=DUMMY&RSTS_IN=&assoc_term_in=201535&CRN_IN=30091&start_date_in=03%2F28%2F2016&end_date_in=06%2F11%2F2016&SUBJ=ECE&CRSE=493&SEC=001&LEVL=Undergraduate+Quarter&CRED=++++4.000&GMOD=Standard+Letter&TITLE=Senior+Design+Project+III&MESG=DUMMY&RSTS_IN=&assoc_term_in=201535&CRN_IN=31117&start_date_in=03%2F28%2F2016&end_date_in=06%2F11%2F2016&SUBJ=BIO&CRSE=141&SEC=001&LEVL=Undergraduate+Quarter&CRED=++++0.000&GMOD=Non+Gradeable+Unit&TITLE=Essential+Biology&MESG=DUMMY&RSTS_IN=&assoc_term_in=201535&CRN_IN=31525&start_date_in=03%2F28%2F2016&end_date_in=06%2F11%2F2016&SUBJ=BIO&CRSE=141&SEC=072&LEVL=Undergraduate+Quarter&CRED=++++0.000&GMOD=Non+Gradeable+Unit&TITLE=Essential+Biology&MESG=DUMMY&RSTS_IN=&assoc_term_in=201535&CRN_IN=30013&start_date_in=03%2F28%2F2016&end_date_in=06%2F11%2F2016&SUBJ=EXAM&CRSE=080&SEC=001&LEVL=Undergraduate+Quarter&CRED=++++0.000&GMOD=Non+Gradeable+Unit&TITLE=Common+Exam+Period+-+I&MESG=DUMMY&RSTS_IN=&assoc_term_in=201535&CRN_IN=34039&start_date_in=03%2F28%2F2016&end_date_in=06%2F11%2F2016&SUBJ=ENTP&CRSE=205&SEC=001&LEVL=Undergraduate+Quarter&CRED=++++3.000&GMOD=Standard+Letter&TITLE=Ready%2C+Set%2C+Fail&RSTS_IN=WR&CRN_IN=34889&assoc_term_in=&start_date_in=&end_date_in=&RSTS_IN=WR&CRN_IN=&assoc_term_in=&start_date_in=&end_date_in=&RSTS_IN=WR&CRN_IN=&assoc_term_in=&start_date_in=&end_date_in=&RSTS_IN=WR&CRN_IN=&assoc_term_in=&start_date_in=&end_date_in=&RSTS_IN=WR&CRN_IN=&assoc_term_in=&start_date_in=&end_date_in=&RSTS_IN=WR&CRN_IN=&assoc_term_in=&start_date_in=&end_date_in=&RSTS_IN=WR&CRN_IN=&assoc_term_in=&start_date_in=&end_date_in=&RSTS_IN=WR&CRN_IN=&assoc_term_in=&start_date_in=&end_date_in=&RSTS_IN=WR&CRN_IN=&assoc_term_in=&start_date_in=&end_date_in=&RSTS_IN=WR&CRN_IN=&assoc_term_in=&start_date_in=&end_date_in=&regs_row=8&wait_row=0&add_row=10&REG_BTN=Submit+Changes

From what it appears, it seems in the request, not only do you add the new CRNs that you want to add, the classes you have added so far also becomes part of the new request.

Looking deeper into the request, it seems each class, whether it's already a class you currently have in your schedule or it's a class you are adding via the textbox, it follows the following format:

assoc_term_in=201535
CRN_IN=34688
start_date_in=03%2F28%2F2016
end_date_in=06%2F11%2F2016
SUBJ=ECEC
CRSE=451
SEC=001
LEVL=Undergraduate+Quarter
CRED=++++3.000
GMOD=Standard+Letter
TITLE=Computer+Arithmetic
MESG=DUMMY
RSTS_IN=

Each class has 13 associated query keys and values. I am actually not sure if it's required to re-submit classes that you've already added, and I am willing to perform some tests if necessary.

The request itself is a POST:

Request URL:https://school.edu/pls/duprod/bwckcoms.P_Regs 
Request Method:POST 
Status Code:200 OK 
Remote Address:[some IP address]

My question is:

1. Can I submit CRNs directly to the textbox avoiding having the re-form the nasty POST request? (i.e. some Python library that does so). Seems like Mechanize is an option, but is it possible to use Requests with Mechanize? I prefer not to re-write all the authentication it took to get into that form page.

2. If the above option is not possible, any ideas how I can re-form the POST request efficiently?

Answer 1

One thing you could do is to use a dict in order to hold all of the information for the post request per Requests documentation. For example your code may look like:

import json
import requests

payload = {
  "assoc_term_in":"201535",
  "CRN_IN":"34688",
  "start_date_in":"03%2F28%2F2016",
  "end_date_in":"06%2F11%2F2016",
  "SUBJ":"ECEC",
  "CRSE":"451",
  "SEC":"001",
  "LEVL":"Undergraduate+Quarter",
  "CRED":"++++3.000",
  "GMOD":"Standard+Letter",
  "TITLE":"Computer+Arithmetic",
  "MESG":"DUMMY",
  "RSTS_IN":""
}

r = requests.post("https://school.edu/pls/duprod/bwckcoms.P_Regs", data=json.dumps(payload))
print(r.text)

Then you can change any of the values that you need to as you would with a normal dictionary. If you wanted to go with #1, one of your options would be to look into something like Selenium, however, I personally feel if you already know what the POST request looks like doing something above may be easier.