Adjust download blocker task retries to a sensible number

[nrejac]

idb-backend/idigbio_workers/tasks/download.py

Lines 80 to 83 in 4cf5efe

	@app.task(bind=True, ignore_result=True, max_retries=None)
	def blocker(self, rid, pollbase=1.25):
	"""Wait for an AsyncResult to be ready, then return its result.

The download blocker task currently never stops retrying. There is a reference in the commit message that says these were terminating early. However, retrying endlessly can result in stuck tasks that never time out.

Adjust to a sensible number, preferably one that can be tweaked at run time.

[danstoner]

The "download all of idigbio" task seems like it would need a very large number of retries. Or maybe I don't understand how this actually works.

I don't know what this comment really means, either:

This can be used to append tasks to a running one, block on the
    running and chain after this.

[danstoner]

Reading some more of the context in our Slack...

If an exception is being "swallowed" and we keep retrying the same task, it seems like that is a bug that maybe ought to be fixed.

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/idb-backend/idigbio_workers/tasks/download.py", line 94, in blocker
    countdown=pollbase ** self.request.retries)
  File "/usr/local/lib/python2.7/site-packages/celery/app/task.py", line 687, in retry
    raise ret
Retry: Retry in 108.420217249s

That's tricky, some exceptions we would want to retry (such as a service temporarily unavailable).