-
Notifications
You must be signed in to change notification settings - Fork 245
feat(tracing): Add Store, P2P and Config tracing #2972
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
chatton
wants to merge
43
commits into
main
Choose a base branch
from
tracing-part-6
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+798
−12
Open
Changes from all commits
Commits
Show all changes
43 commits
Select commit
Hold shift + click to select a range
78b54f2
wip: adding tracing
chatton ae87a59
chore: only having the first tracing decorator
chatton fdd943b
chore: remove comment
chatton 74d4a5f
deps: adding pin to genproto version
chatton d796589
chore: ensuring errors reported, adding unit tests
chatton d52f187
chore: add check to validate basic
chatton b2b3218
chore: modified default
chatton 41bce54
chore: adding logging of possible error
chatton fd34073
chore: updated flag test
chatton fd7425a
chore: bump endpoint to correct port
chatton f30a577
wip: adding propagating client to engine and eth client
chatton 570509b
chore: simplify construction of rpc opts
chatton caa0684
chore: address PR feedback
chatton c154f23
chore: ensure consistent propagation settings
chatton 607f4a3
chore: adding interface for engine client and tracing implementation
chatton c5d7c41
chore: mrege main
chatton 80e2b17
chore: refactored wiring to use bool
chatton 07a45b6
chore: tidy all fix
chatton 423bb15
chore: fix go mod conflicts
chatton 3e373ce
chore: addressing PR feedback
chatton ed217d7
chore: adding eth client tracing
chatton 17eb5aa
chore: merge main
chatton 931d2ac
chore: add payload id as attribute
chatton ee2d158
chore: handle merge conflicts
chatton 8d7fc84
Merge branch 'cian/add-tracing-part-3' into cian/add-tracing-part-4
chatton 32db6c8
chore: merge main
chatton a3fa329
chore: adding tracing for DA client
chatton 776a2ea
chore: add *.test to gitignore
chatton d3bdb1e
chore: updated test
chatton 4d1d3ff
chore: adding hex encoded namespace
chatton ed675a8
feat: add Phase 1 RPC server tracing instrumentation
chatton b1a827e
chore: make tidy all
chatton 30edc0c
chore: tidy all and add wrappers
chatton 1ff13f3
chore: merge main
chatton 9d726af
chore: merge part 5
chatton db319d2
chore: removed unused tracer
chatton 2edad3e
chore: remove unused attribute
chatton b1aaa84
Merge branch 'main' into tracing-part-6
chatton 3ab312c
chore: tidy all
chatton fecea22
Merge branch 'main' into tracing-part-6
chatton d63f852
chore: addressing PR feedback
chatton b1246eb
chore: removing accidental file
chatton cb10d69
Merge branch 'main' into tracing-part-6
chatton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,270 @@ | ||
| package server | ||
|
|
||
| import ( | ||
| "context" | ||
| "encoding/hex" | ||
|
|
||
| "connectrpc.com/connect" | ||
| "go.opentelemetry.io/otel" | ||
| "go.opentelemetry.io/otel/attribute" | ||
| "go.opentelemetry.io/otel/codes" | ||
| "go.opentelemetry.io/otel/trace" | ||
| "google.golang.org/protobuf/types/known/emptypb" | ||
|
|
||
| pb "github.com/evstack/ev-node/types/pb/evnode/v1" | ||
| "github.com/evstack/ev-node/types/pb/evnode/v1/v1connect" | ||
| ) | ||
|
|
||
| // tracedStoreServer decorates a StoreServiceHandler with OpenTelemetry spans. | ||
| type tracedStoreServer struct { | ||
| inner v1connect.StoreServiceHandler | ||
| tracer trace.Tracer | ||
| } | ||
|
|
||
| // WithTracingStoreServer decorates the provided store service handler with tracing spans. | ||
| func WithTracingStoreServer(inner v1connect.StoreServiceHandler) v1connect.StoreServiceHandler { | ||
| return &tracedStoreServer{ | ||
| inner: inner, | ||
| tracer: otel.Tracer("ev-node/store-service"), | ||
| } | ||
| } | ||
|
|
||
| func (t *tracedStoreServer) GetBlock( | ||
| ctx context.Context, | ||
| req *connect.Request[pb.GetBlockRequest], | ||
| ) (*connect.Response[pb.GetBlockResponse], error) { | ||
| var attrs []attribute.KeyValue | ||
| switch identifier := req.Msg.Identifier.(type) { | ||
| case *pb.GetBlockRequest_Height: | ||
| attrs = append(attrs, attribute.Int64("height", int64(identifier.Height))) | ||
| case *pb.GetBlockRequest_Hash: | ||
| if identifier.Hash != nil { | ||
| attrs = append(attrs, attribute.String("hash", hex.EncodeToString(identifier.Hash))) | ||
| } | ||
| } | ||
|
|
||
| ctx, span := t.tracer.Start(ctx, "StoreService.GetBlock", trace.WithAttributes(attrs...)) | ||
| defer span.End() | ||
|
|
||
| res, err := t.inner.GetBlock(ctx, req) | ||
| if err != nil { | ||
| span.RecordError(err) | ||
| span.SetStatus(codes.Error, err.Error()) | ||
| return nil, err | ||
| } | ||
|
|
||
| span.SetAttributes( | ||
| attribute.Bool("found", res.Msg.Block != nil), | ||
| ) | ||
| if res.Msg.Block != nil && res.Msg.Block.Data != nil { | ||
| totalSize := 0 | ||
| for _, tx := range res.Msg.Block.Data.Txs { | ||
| totalSize += len(tx) | ||
| } | ||
| span.SetAttributes( | ||
| attribute.Int("block_size_bytes", totalSize), | ||
| attribute.Int("tx_count", len(res.Msg.Block.Data.Txs)), | ||
| ) | ||
| } | ||
| return res, nil | ||
| } | ||
|
|
||
| func (t *tracedStoreServer) GetState( | ||
| ctx context.Context, | ||
| req *connect.Request[emptypb.Empty], | ||
| ) (*connect.Response[pb.GetStateResponse], error) { | ||
| ctx, span := t.tracer.Start(ctx, "StoreService.GetState") | ||
| defer span.End() | ||
|
|
||
| res, err := t.inner.GetState(ctx, req) | ||
| if err != nil { | ||
| span.RecordError(err) | ||
| span.SetStatus(codes.Error, err.Error()) | ||
| return nil, err | ||
| } | ||
|
|
||
| if res.Msg.State != nil { | ||
| span.SetAttributes( | ||
| attribute.Int64("height", int64(res.Msg.State.LastBlockHeight)), | ||
| attribute.String("app_hash", hex.EncodeToString(res.Msg.State.AppHash)), | ||
| attribute.Int64("da_height", int64(res.Msg.State.DaHeight)), | ||
| ) | ||
| } | ||
| return res, nil | ||
| } | ||
|
|
||
| func (t *tracedStoreServer) GetMetadata( | ||
| ctx context.Context, | ||
| req *connect.Request[pb.GetMetadataRequest], | ||
| ) (*connect.Response[pb.GetMetadataResponse], error) { | ||
| ctx, span := t.tracer.Start(ctx, "StoreService.GetMetadata", | ||
| trace.WithAttributes( | ||
| attribute.String("key", req.Msg.Key), | ||
| ), | ||
| ) | ||
| defer span.End() | ||
|
|
||
| res, err := t.inner.GetMetadata(ctx, req) | ||
| if err != nil { | ||
| span.RecordError(err) | ||
| span.SetStatus(codes.Error, err.Error()) | ||
| return nil, err | ||
| } | ||
|
|
||
| span.SetAttributes( | ||
| attribute.Int("value_size_bytes", len(res.Msg.Value)), | ||
| ) | ||
| return res, nil | ||
| } | ||
|
|
||
| func (t *tracedStoreServer) GetGenesisDaHeight( | ||
| ctx context.Context, | ||
| req *connect.Request[emptypb.Empty], | ||
| ) (*connect.Response[pb.GetGenesisDaHeightResponse], error) { | ||
| ctx, span := t.tracer.Start(ctx, "StoreService.GetGenesisDaHeight") | ||
| defer span.End() | ||
|
|
||
| res, err := t.inner.GetGenesisDaHeight(ctx, req) | ||
| if err != nil { | ||
| span.RecordError(err) | ||
| span.SetStatus(codes.Error, err.Error()) | ||
| return nil, err | ||
| } | ||
|
|
||
| span.SetAttributes( | ||
| attribute.Int64("genesis_da_height", int64(res.Msg.Height)), | ||
| ) | ||
| return res, nil | ||
| } | ||
|
|
||
| func (t *tracedStoreServer) GetP2PStoreInfo( | ||
| ctx context.Context, | ||
| req *connect.Request[emptypb.Empty], | ||
| ) (*connect.Response[pb.GetP2PStoreInfoResponse], error) { | ||
| ctx, span := t.tracer.Start(ctx, "StoreService.GetP2PStoreInfo") | ||
| defer span.End() | ||
|
|
||
| res, err := t.inner.GetP2PStoreInfo(ctx, req) | ||
| if err != nil { | ||
| span.RecordError(err) | ||
| span.SetStatus(codes.Error, err.Error()) | ||
| return nil, err | ||
| } | ||
|
|
||
| span.SetAttributes( | ||
| attribute.Int("store_count", len(res.Msg.Stores)), | ||
| ) | ||
| return res, nil | ||
| } | ||
|
|
||
| // tracedP2PServer decorates a P2PServiceHandler with OpenTelemetry spans. | ||
| type tracedP2PServer struct { | ||
| inner v1connect.P2PServiceHandler | ||
| tracer trace.Tracer | ||
| } | ||
|
|
||
| // WithTracingP2PServer decorates the provided P2P service handler with tracing spans. | ||
| func WithTracingP2PServer(inner v1connect.P2PServiceHandler) v1connect.P2PServiceHandler { | ||
| return &tracedP2PServer{ | ||
| inner: inner, | ||
| tracer: otel.Tracer("ev-node/p2p-service"), | ||
| } | ||
| } | ||
|
|
||
| func (t *tracedP2PServer) GetPeerInfo( | ||
| ctx context.Context, | ||
| req *connect.Request[emptypb.Empty], | ||
| ) (*connect.Response[pb.GetPeerInfoResponse], error) { | ||
| ctx, span := t.tracer.Start(ctx, "P2PService.GetPeerInfo") | ||
| defer span.End() | ||
|
|
||
| res, err := t.inner.GetPeerInfo(ctx, req) | ||
| if err != nil { | ||
| span.RecordError(err) | ||
| span.SetStatus(codes.Error, err.Error()) | ||
| return nil, err | ||
| } | ||
|
|
||
| span.SetAttributes( | ||
| attribute.Int("peer_count", len(res.Msg.Peers)), | ||
| ) | ||
| return res, nil | ||
| } | ||
|
|
||
| func (t *tracedP2PServer) GetNetInfo( | ||
| ctx context.Context, | ||
| req *connect.Request[emptypb.Empty], | ||
| ) (*connect.Response[pb.GetNetInfoResponse], error) { | ||
| ctx, span := t.tracer.Start(ctx, "P2PService.GetNetInfo") | ||
| defer span.End() | ||
|
|
||
| res, err := t.inner.GetNetInfo(ctx, req) | ||
| if err != nil { | ||
| span.RecordError(err) | ||
| span.SetStatus(codes.Error, err.Error()) | ||
| return nil, err | ||
| } | ||
|
|
||
| if res.Msg.NetInfo != nil { | ||
| span.SetAttributes( | ||
| attribute.String("node_id", res.Msg.NetInfo.Id), | ||
| attribute.Int("listen_address_count", len(res.Msg.NetInfo.ListenAddresses)), | ||
| ) | ||
| } | ||
| return res, nil | ||
| } | ||
|
|
||
| // tracedConfigServer decorates a ConfigServiceHandler with OpenTelemetry spans. | ||
| type tracedConfigServer struct { | ||
| inner v1connect.ConfigServiceHandler | ||
| tracer trace.Tracer | ||
| } | ||
|
|
||
| // WithTracingConfigServer decorates the provided config service handler with tracing spans. | ||
| func WithTracingConfigServer(inner v1connect.ConfigServiceHandler) v1connect.ConfigServiceHandler { | ||
| return &tracedConfigServer{ | ||
| inner: inner, | ||
| tracer: otel.Tracer("ev-node/config-service"), | ||
| } | ||
| } | ||
|
|
||
| func (t *tracedConfigServer) GetNamespace( | ||
| ctx context.Context, | ||
| req *connect.Request[emptypb.Empty], | ||
| ) (*connect.Response[pb.GetNamespaceResponse], error) { | ||
| ctx, span := t.tracer.Start(ctx, "ConfigService.GetNamespace") | ||
| defer span.End() | ||
|
|
||
| res, err := t.inner.GetNamespace(ctx, req) | ||
| if err != nil { | ||
| span.RecordError(err) | ||
| span.SetStatus(codes.Error, err.Error()) | ||
| return nil, err | ||
| } | ||
|
|
||
| span.SetAttributes( | ||
| attribute.String("header_namespace", res.Msg.HeaderNamespace), | ||
| attribute.String("data_namespace", res.Msg.DataNamespace), | ||
| ) | ||
| return res, nil | ||
| } | ||
|
|
||
| func (t *tracedConfigServer) GetSignerInfo( | ||
| ctx context.Context, | ||
| req *connect.Request[emptypb.Empty], | ||
| ) (*connect.Response[pb.GetSignerInfoResponse], error) { | ||
| ctx, span := t.tracer.Start(ctx, "ConfigService.GetSignerInfo") | ||
| defer span.End() | ||
|
|
||
| res, err := t.inner.GetSignerInfo(ctx, req) | ||
| if err != nil { | ||
| span.RecordError(err) | ||
| span.SetStatus(codes.Error, err.Error()) | ||
| return nil, err | ||
| } | ||
|
|
||
| span.SetAttributes( | ||
| attribute.String("signer_address", hex.EncodeToString(res.Msg.Address)), | ||
| ) | ||
| return res, nil | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This block of code for handling errors from the inner service call and updating the span is repeated in almost every traced method in this file. To improve maintainability and reduce boilerplate, consider extracting this logic into a generic helper function.
For example, you could create a function that takes the context, request, the inner function call, and returns the response and error, handling the span creation and error recording internally. Here's a conceptual example:
Applying this pattern would make the tracing decorators much more concise.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think having a bit of duplication isn't too bad to avoid a generics function that takes a function as an arg, makes things harder to reason about, but if anyone else wants this to be implemented I can do it.