Repository navigation
Actor state save best practices #854
Copy link
Copy link
Closed
Labels
Description
Expected Behavior
Saving actor state does not fail.
Actual Behavior
Saving an actor state fails with the following:
java.util.ConcurrentModificationException: null
at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1597)
at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1630)
at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1628)
at io.dapr.actors.runtime.ActorStateManager.flush(ActorStateManager.java:286)
at io.dapr.actors.runtime.ActorStateManager.lambda$save$15(ActorStateManager.java:272)
at reactor.core.publisher.MonoRunnable.call(MonoRunnable.java:73)
at reactor.core.publisher.MonoRunnable.call(MonoRunnable.java:32)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:228)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onComplete(FluxHide.java:147)
at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:89)
at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:89)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1840)
at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:62)
at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
at reactor.core.publisher.MonoIgnorePublisher.subscribe(MonoIgnorePublisher.java:57)
at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:203)
at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
at reactor.core.publisher.Mono.subscribe(Mono.java:4475)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
at reactor.core.publisher.Mono.block(Mono.java:1741) Steps to Reproduce the Problem
Implement an actor with two methods like the following:
@Override
public void enqueue(String taskId) {
var mockPayload = "{}";
this.getActorStateManager().set("payload_%s".formatted(taskId), mockPayload).block();
System.out.println("Saved payload %s".formatted(taskId));
this.saveState().block();
}
@Override
public void ack(String taskId) {
var mockPayload = this.getActorStateManager().get("payload_%s".formatted(taskId), String.class).block();
System.out.println("Retrieved payload %s".formatted(taskId));
System.out.println("Payload was %s".formatted(mockPayload));
this.getActorStateManager().remove("payload_%s".formatted(taskId)).block();
this.saveState().block();
}And play around, try to call:
enqueue(1)enqueue(2)enqueue(3)ack(1)
Should fail.
Controller for the sake of completeness:
@RestController
@RequiredArgsConstructor
public class SerializerActorController {
private final ActorProxyBuilder<SerializerActor> actorProxyBuilder;
@GetMapping(value = "/enqueue/{taskId}")
public ResponseEntity<Void> enqueue(@PathVariable("taskId") String taskId) {
try {
var actorId = new ActorId("actorId");
var actor = actorProxyBuilder.build(actorId);
actor.enqueue(taskId);
return ResponseEntity.ok().build();
} catch (Exception ex) {
ex.printStackTrace();
return ResponseEntity.internalServerError().build();
}
}
@GetMapping(value = "/ack/{taskId}")
public ResponseEntity<Void> ack(@PathVariable("taskId") String taskId) {
try {
var actorId = new ActorId("actorId");
var actor = actorProxyBuilder.build(actorId);
actor.ack(taskId);
return ResponseEntity.ok().build();
} catch (Exception ex) {
ex.printStackTrace();
return ResponseEntity.internalServerError().build();
}
}
}Release Note
RELEASE NOTE:
FIX Solved concurrency on actor state saving stage.
Reactions are currently unavailable
Activity
mukundansundar commentedon May 12, 2023
@nitroin What is the version of Dapr that you are using? Also what version of SDK are you using?
manuelserradev commentedon May 12, 2023
@mukundansundar
Latest SDK version (1.8.0).
mukundansundar commentedon May 18, 2023
@nitroin Can you check if this happens with 1.10.7 runtime?
And what actor state store is being used?
artursouza commentedon May 18, 2023
This error might be inside the ActorStateManager code in SDK and not in runtime.
manuelserradev commentedon May 26, 2023
I, also, do think that is not runtime related.
What I can see is that in the actor-sdk the incorrect type of collection is used (for thread safety and such), but this contradicts with the expectation of the "actor" threading model - but again looks like to me the problem arise from underlying layer (sdk and state interaction).
That being said I'll do try on the latest runtime.
manuelserradev commentedon May 26, 2023
I can confirm the same issue with:
And sdk version 1.8.0.
I will copy-paste the full actor:
Controller:
App config:
And actor factory, needed to inject app configs into actors:
fnurglewitz commentedon May 26, 2023
I'm having the same issue with dapr 1.10
manuelserradev commentedon Feb 20, 2024
@artursouza care to explain a little further? This error still happens and I'd like to understand any actionable on our side.
javier-aliaga commentedon Oct 15, 2026
closed vy #1017