Process breaks when percentage completion goes backwards

[smithara]

I've had one instance where a request has crashed because the percentage completion has decreased (which presumably means a bug in the server percentage reporting), causing the progress bar update to raise an error.

[19/37] Processing:   21%|███████████████████████▎                                                                                       |  [ Elapsed: 02:55, Remaining: 14:59 ]
Traceback (most recent call last):
  File "make_AEJ_estimates.py", line 139, in <module>
    main(sat_ID)
  File "make_AEJ_estimates.py", line 131, in main
    df = fetch_data(sat_ID)
  File "make_AEJ_estimates.py", line 48, in fetch_data
    data_S = request.get_between(start_time, end_time)
  File "/jupyter/s1556306/miniconda3_envs/py37/lib/python3.7/site-packages/viresclient/_client.py", line 521, in get_between
    show_progress=show_progress
  File "/jupyter/s1556306/miniconda3_envs/py37/lib/python3.7/site-packages/viresclient/_client.py", line 395, in _get
    status_handler=progressbar.update
  File "/jupyter/s1556306/miniconda3_envs/py37/lib/python3.7/site-packages/viresclient/_wps/wps.py", line 151, in retrieve_async
    status_handler(wpsstatus)
  File "/jupyter/s1556306/miniconda3_envs/py37/lib/python3.7/site-packages/viresclient/_client.py", line 178, in update
    self.refresh_tqdm()
  File "/jupyter/s1556306/miniconda3_envs/py37/lib/python3.7/site-packages/viresclient/_client.py", line 144, in refresh_tqdm
    self.tqdm_pbar.update(self.percentCompleted-self.lastpercent)
  File "/jupyter/s1556306/miniconda3_envs/py37/lib/python3.7/site-packages/tqdm/_tqdm.py", line 1025, in update
    raise ValueError("n ({0}) cannot be negative".format(n))
ValueError: n (-21) cannot be negative

[pacesm]

@smithara
FYI, This can happen when processing daemon gets restarted. In that case, all running jobs are terminated and started again when the server gets up and running again.

[smithara]

Ok I guess it will be worthwhile if I spend some time in the future to address these robustness issues more systematically.