Optimize the processing to make it more reliable and easier to test

[samsmithnz]

• Split the nightly processing into two steps
  • Add the processing into a queue on a schedule
  • For each item in the queue, spawn a function and process the data