Skip to content

Client-side batching: wholesale chunk failures are reported under chunk-local indices, so errors collide and describe the wrong objects #2165

Description

@feiiiiii5

What happens

With client-side batching, when a whole chunk fails to be sent, BatchObjectReturn.errors
reports fewer failures than actually happened, and the surviving keys describe the wrong
objects.

BatchObjectReturn documents the opposite (weaviate/collections/classes/batch.py, class
docstring):

The keys of the errors and uuids dictionaries will always be equivalent to the
original_index of the objects as you added them to the batching loop

Root cause

The wholesale-failure handler in _BatchBase.__send_batch keys its errors by the position
inside the chunk that was sent:

# weaviate/collections/batch/base.py:648
errors_obj = {
    idx: ErrorObject(message=repr(e), object_=obj) for idx, obj in enumerate(objs)
}

Every other place that fills this dictionary uses the global index — obj.index in
grpc_batch.py:157, sync.py:367/:410, async_.py:409/:452 — because
BatchObjectReturn.__add__ merges chunk results with a plain self.errors.update(other.errors)
(classes/batch.py:231) and does no re-keying. So with a chunk size smaller than the batch,
chunk N re-uses keys 0..len(chunk)-1 and overwrites chunk N-1's entries.

Two observable symptoms:

  1. len(result.errors) under-reports the number of failed objects.
  2. If one chunk succeeds and a later chunk fails wholesale, the successful chunk's uuids
    keys and the failed chunk's errors keys collide and describe different objects —
    result.uuids[2] and result.errors[2] refer to two different inputs, so a caller that
    retries "the failed indices" re-inserts rows that already went in and skips rows that did not.

Reproduction

Offline, mock gRPC only, no server and no API keys. Four objects with
fixed_size(batch_size=2), and the mock rejects every object of both chunks (which makes
_BatchGRPC.objects raise WeaviateInsertManyAllFailedError, landing in that except Exception):

result.errors keys : [0, 1]        # 4 objects failed
len(result.errors) : 2
len(failed_objects): 4
errors[0] -> object_.index=2 (input position 2)
errors[1] -> object_.index=3 (input position 3)

And for "first chunk succeeds, second fails wholesale": uuids keys [0, 1] while errors
keys are also [0, 1], describing objects 2 and 3.

Expected: errors keys [0, 1, 2, 3] in the first case and [2, 3] in the second, each
describing the object it names.

Tested on main @ 142d798, Python 3.11.15. I have a patch and mock_tests/ regression tests
for both symptoms, in #2166

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions