Data Pipeline for processing large loads of course data from the Web Scraper
Make a proof-of-concept pipeline using PySpark to focus on finding viable data transformations to the different Postgres tables
Potentially migrate to Scala (mostly because Spark code is natively written in Scala)
There's a lot of course data to process, not sure if Pandas would be enough. Also, converting from Dataframe to SQL table should be easy once data transformations are figured out