After a rolling update of the cluster, I found that one bucket of a PK table could not initialize RocksDB from the snapshot and WAL. The relevant failure logs are shown below:
Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist: /rbf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/snap-408/420d54b8-9af9-4047-85df-747876f4a750
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:86)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:156)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2063)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:796)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:458)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:539)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1110)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1050)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:973)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1918)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2982)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1569) ~[hadoop-common-3.2.2.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1515) ~[hadoop-common-3.2.2.jar:?]
at org.apache.hadoop.ipc.Client.call(Client.java:1412) ~[hadoop-common-3.2.2.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:244) ~[hadoop-common-3.2.2.jar:?]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:129) ~[hadoop-common-3.2.2.jar:?]
at jdk.proxy2.$Proxy27.getBlockLocations(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:333) ~[hadoop-hdfs-client-3.2.2.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:426) ~[hadoop-common-3.2.2.jar:?]
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:169) ~[hadoop-common-3.2.2.jar:?]
But the _metadata file exists. the content as follows:
{
"version": 1,
"table_id": 149,
"bucket_id": 100,
"snapshot_id": 408,
"snapshot_location": "rbf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/snap-408",
"kv_snapshot_handle": {
"shared_file_handles": [
{
"kv_file_handle": {
"path": "rbf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/shared/aaf668b2-847d-4ea4-b798-62c5c1bcfeab",
"size": 19301626
},
"local_path": "000755.sst"
},
{
"kv_file_handle": {
"path": "bf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/shared/11051cc5-5cb8-4962-a642-45cda6cafef6",
"size": 27621774
},
"local_path": "000733.sst"
},
{
"kv_file_handle": {
"path": "rbf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/shared/5bfa680a-652e-4621-9088-f22edc6b0cac",
"size": 67651196
},
"local_path": "000704.sst"
},
{
"kv_file_handle": {
"path": "bf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/shared/04871351-8c5b-415f-b84c-3a4ae89e60c6",
"size": 67647338
},
"local_path": "000705.sst"
},
{
"kv_file_handle": {
"path": "rbf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/shared/986074a4-4511-4994-97cf-265234902006",
"size": 59108533
},
"local_path": "000706.sst"
},
{
"kv_file_handle": {
"path": "rbf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/shared/fa74dc40-9010-4952-afc4-9f7332355906",
"size": 52334
},
"local_path": "000775.sst"
},
{
"kv_file_handle": {
"path": "rbf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/shared/b971948c-7878-4b38-bfe0-daf1d82c70fa",
"size": 10772241
},
"local_path": "000774.sst"
}
],
"private_file_handles": [
{
"kv_file_handle": {
"path": "rbf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/snap-408/1929184d-aa34-46b0-b914-8f70080d4c87",
"size": 2377
},
"local_path": "MANIFEST-000766"
},
{
"kv_file_handle": {
"path": "rbf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/snap-408/dde56efa-6b62-43e4-8c85-294df4739868",
"size": 6411
},
"local_path": "OPTIONS-000769"
},
{
"kv_file_handle": {
"path": "rbf/data/fluss/kv/fluss_lake/opal_state_v1_fg_2456-149/100/snap-408/420d54b8-9af9-4047-85df-747876f4a750",
"size": 16
},
"local_path": "CURRENT"
}
],
"snapshot_incremental_size": 10833379
},
"log_offset": 6782001
}
Search before asking
Fluss version
0.9.0 (latest release)
Please describe the bug 🐞
After a rolling update of the cluster, I found that one bucket of a PK table could not initialize RocksDB from the snapshot and WAL. The relevant failure logs are shown below:
But the _metadata file exists. the content as follows:
This bug looks similar to #1304. However, because the “file not found” error message is different, the KV initialization process cannot fall back to the previous snapshot, even though that snapshot appears to be valid.
Solution
No response
Are you willing to submit a PR?