Repository navigation
feat!: prepare v5 release - #1991
Conversation
* chore!: require Python 3.11 or newer Python 3.10 reaches end of life in October 2026. Raise requires-python to >=3.11, target py311 in ruff, drop 3.10 from the CI unit matrix, and remove the pre-3.11 fallbacks for asyncio.create_task(context=...) and typing.NotRequired. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(observe): keep detecting legacy-marked coroutine functions inspect.iscoroutinefunction ignores asyncio's _is_coroutine marker, which asgiref's markcoroutinefunction sets on Python < 3.12. Accept both so such functions keep the async wrapper and their observation ends after the coroutine runs. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
* chore!: require Python 3.11 or newer Python 3.10 reaches end of life in October 2026. Raise requires-python to >=3.11, target py311 in ruff, drop 3.10 from the CI unit matrix, and remove the pre-3.11 fallbacks for asyncio.create_task(context=...) and typing.NotRequired. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(openai)!: drop support for OpenAI SDK versions below 1.0 The OpenAI integration now always uses the v1 client shapes. Remove the pre-1.0 method definitions and every _is_openai_v1() branch, and raise the dev dependency floor to openai>=1.0.0. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(observe): keep detecting legacy-marked coroutine functions inspect.iscoroutinefunction ignores asyncio's _is_coroutine marker, which asgiref's markcoroutinefunction sets on Python < 3.12. Accept both so such functions keep the async wrapper and their observation ends after the coroutine runs. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
* chore!: require Python 3.11 or newer Python 3.10 reaches end of life in October 2026. Raise requires-python to >=3.11, target py311 in ruff, drop 3.10 from the CI unit matrix, and remove the pre-3.11 fallbacks for asyncio.create_task(context=...) and typing.NotRequired. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(openai)!: drop support for OpenAI SDK versions below 1.0 The OpenAI integration now always uses the v1 client shapes. Remove the pre-1.0 method definitions and every _is_openai_v1() branch, and raise the dev dependency floor to openai>=1.0.0. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(otel)!: send x-langfuse-ingestion-version: 4 by default The default OTLP exporter now sends x-langfuse-ingestion-version: 4 so servers in dual-write or preview mode process exported spans directly. It is a no-op on events_only deployments, and additional_headers can still override it. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(observe): keep detecting legacy-marked coroutine functions inspect.iscoroutinefunction ignores asyncio's _is_coroutine marker, which asgiref's markcoroutinefunction sets on Python < 3.12. Accept both so such functions keep the async wrapper and their observation ends after the coroutine runs. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(otel): pin that a mixed-case ingestion-version override wins Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * chore: re-run CI Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
* chore!: require Python 3.11 or newer Python 3.10 reaches end of life in October 2026. Raise requires-python to >=3.11, target py311 in ruff, drop 3.10 from the CI unit matrix, and remove the pre-3.11 fallbacks for asyncio.create_task(context=...) and typing.NotRequired. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(openai)!: drop support for OpenAI SDK versions below 1.0 The OpenAI integration now always uses the v1 client shapes. Remove the pre-1.0 method definitions and every _is_openai_v1() branch, and raise the dev dependency floor to openai>=1.0.0. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(otel)!: send x-langfuse-ingestion-version: 4 by default The default OTLP exporter now sends x-langfuse-ingestion-version: 4 so servers in dual-write or preview mode process exported spans directly. It is a no-op on events_only deployments, and additional_headers can still override it. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client)!: remove blocked_instrumentation_scopes blocked_instrumentation_scopes was deprecated in favor of should_export_span. Remove it from Langfuse, the resource manager and the span processor, and drop the leftover is_active entry from the create_prompt docstring. LANGFUSE_HOST and host keep working, but now log a deprecation warning when they decide the base URL. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(observe): keep detecting legacy-marked coroutine functions inspect.iscoroutinefunction ignores asyncio's _is_coroutine marker, which asgiref's markcoroutinefunction sets on Python < 3.12. Accept both so such functions keep the async wrapper and their observation ends after the coroutine runs. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(otel): pin that a mixed-case ingestion-version override wins Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client): keep host and LANGFUSE_HOST working without a warning Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * chore: re-run CI Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
* chore!: require Python 3.11 or newer Python 3.10 reaches end of life in October 2026. Raise requires-python to >=3.11, target py311 in ruff, drop 3.10 from the CI unit matrix, and remove the pre-3.11 fallbacks for asyncio.create_task(context=...) and typing.NotRequired. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(openai)!: drop support for OpenAI SDK versions below 1.0 The OpenAI integration now always uses the v1 client shapes. Remove the pre-1.0 method definitions and every _is_openai_v1() branch, and raise the dev dependency floor to openai>=1.0.0. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(otel)!: send x-langfuse-ingestion-version: 4 by default The default OTLP exporter now sends x-langfuse-ingestion-version: 4 so servers in dual-write or preview mode process exported spans directly. It is a no-op on events_only deployments, and additional_headers can still override it. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client)!: remove blocked_instrumentation_scopes blocked_instrumentation_scopes was deprecated in favor of should_export_span. Remove it from Langfuse, the resource manager and the span processor, and drop the leftover is_active entry from the create_prompt docstring. LANGFUSE_HOST and host keep working, but now log a deprecation warning when they decide the base URL. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(tracing)!: JSON-serialize propagated metadata values propagate_attributes(metadata=...) coerced non-string values with str(), so a boolean became "True" and a list became its Python repr. Serialize them as JSON instead, like observation metadata, and drop None values instead of sending "None". Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): serialize propagated metadata like JSON.stringify Use compact separators, keep non-ASCII characters, and send None as "null", so the Python and JS SDKs emit byte-identical propagated metadata values. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(observe): keep detecting legacy-marked coroutine functions inspect.iscoroutinefunction ignores asyncio's _is_coroutine marker, which asgiref's markcoroutinefunction sets on Python < 3.12. Accept both so such functions keep the async wrapper and their observation ends after the coroutine runs. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(otel): pin that a mixed-case ingestion-version override wins Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): keep the exact digits of large integer metadata values Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): match JS for nested big ints and non-finite metadata Integers keep their exact digits as JSON numbers at any depth, and propagated metadata values containing NaN or Infinity are dropped with a warning instead of being stored, matching the JS SDK. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client): keep host and LANGFUSE_HOST working without a warning Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
…taset_run (#1933) * chore!: require Python 3.11 or newer Python 3.10 reaches end of life in October 2026. Raise requires-python to >=3.11, target py311 in ruff, drop 3.10 from the CI unit matrix, and remove the pre-3.11 fallbacks for asyncio.create_task(context=...) and typing.NotRequired. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(openai)!: drop support for OpenAI SDK versions below 1.0 The OpenAI integration now always uses the v1 client shapes. Remove the pre-1.0 method definitions and every _is_openai_v1() branch, and raise the dev dependency floor to openai>=1.0.0. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(otel)!: send x-langfuse-ingestion-version: 4 by default The default OTLP exporter now sends x-langfuse-ingestion-version: 4 so servers in dual-write or preview mode process exported spans directly. It is a no-op on events_only deployments, and additional_headers can still override it. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client)!: remove blocked_instrumentation_scopes blocked_instrumentation_scopes was deprecated in favor of should_export_span. Remove it from Langfuse, the resource manager and the span processor, and drop the leftover is_active entry from the create_prompt docstring. LANGFUSE_HOST and host keep working, but now log a deprecation warning when they decide the base URL. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(tracing)!: JSON-serialize propagated metadata values propagate_attributes(metadata=...) coerced non-string values with str(), so a boolean became "True" and a list became its Python repr. Serialize them as JSON instead, like observation metadata, and drop None values instead of sending "None". Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client)!: remove get_dataset_run, get_dataset_runs and delete_dataset_run These helpers call the dataset-run endpoints that Langfuse v4 no longer serves, so they always fail with a 404. Read experiment runs through langfuse.api.experiments.list() and list_items() instead. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): serialize propagated metadata like JSON.stringify Use compact separators, keep non-ASCII characters, and send None as "null", so the Python and JS SDKs emit byte-identical propagated metadata values. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(observe): keep detecting legacy-marked coroutine functions inspect.iscoroutinefunction ignores asyncio's _is_coroutine marker, which asgiref's markcoroutinefunction sets on Python < 3.12. Accept both so such functions keep the async wrapper and their observation ends after the coroutine runs. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(otel): pin that a mixed-case ingestion-version override wins Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): keep the exact digits of large integer metadata values Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * chore: re-run CI Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): match JS for nested big ints and non-finite metadata Integers keep their exact digits as JSON numbers at any depth, and propagated metadata values containing NaN or Infinity are dropped with a warning instead of being stored, matching the JS SDK. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client): keep host and LANGFUSE_HOST working without a warning Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
* chore!: require Python 3.11 or newer Python 3.10 reaches end of life in October 2026. Raise requires-python to >=3.11, target py311 in ruff, drop 3.10 from the CI unit matrix, and remove the pre-3.11 fallbacks for asyncio.create_task(context=...) and typing.NotRequired. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(openai)!: drop support for OpenAI SDK versions below 1.0 The OpenAI integration now always uses the v1 client shapes. Remove the pre-1.0 method definitions and every _is_openai_v1() branch, and raise the dev dependency floor to openai>=1.0.0. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(otel)!: send x-langfuse-ingestion-version: 4 by default The default OTLP exporter now sends x-langfuse-ingestion-version: 4 so servers in dual-write or preview mode process exported spans directly. It is a no-op on events_only deployments, and additional_headers can still override it. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client)!: remove blocked_instrumentation_scopes blocked_instrumentation_scopes was deprecated in favor of should_export_span. Remove it from Langfuse, the resource manager and the span processor, and drop the leftover is_active entry from the create_prompt docstring. LANGFUSE_HOST and host keep working, but now log a deprecation warning when they decide the base URL. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(tracing)!: JSON-serialize propagated metadata values propagate_attributes(metadata=...) coerced non-string values with str(), so a boolean became "True" and a list became its Python repr. Serialize them as JSON instead, like observation metadata, and drop None values instead of sending "None". Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client)!: remove get_dataset_run, get_dataset_runs and delete_dataset_run These helpers call the dataset-run endpoints that Langfuse v4 no longer serves, so they always fail with a 404. Read experiment runs through langfuse.api.experiments.list() and list_items() instead. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(experiments)!: record experiment runs via OTEL only Generate the experiment id client-side instead of creating dataset run items via POST /dataset-run-items. Dataset-backed runs derive a stable id from project, dataset and run name (matching the server's derivation), so re-runs with the same run name keep grouping into one experiment. Local runs keep a random id. - set langfuse.experiment.item.version from the pinned dataset version - always persist run evaluations as scores on the experiment, incl. local data - return experiment_id / experiment_url (v4 results page) on results; dataset_run_id / dataset_run_url remain as deprecated aliases BREAKING CHANGE: run_experiment no longer creates Postgres dataset runs; dataset_run_id / dataset_run_url are deprecated aliases that are now also set for local-data experiments, and the URL points to the experiments page. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(experiments): read experiment e2e results via v4 APIs Replace dataset run, dataset run item and legacy trace reads with the experiments, observations and v3 scores APIs, and cover stable experiment ids, run-level scores for local data and the recorded item version. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(experiments): pin the cross-SDK experiment id vectors Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): serialize propagated metadata like JSON.stringify Use compact separators, keep non-ASCII characters, and send None as "null", so the Python and JS SDKs emit byte-identical propagated metadata values. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(observe): keep detecting legacy-marked coroutine functions inspect.iscoroutinefunction ignores asyncio's _is_coroutine marker, which asgiref's markcoroutinefunction sets on Python < 3.12. Accept both so such functions keep the async wrapper and their observation ends after the coroutine runs. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(otel): pin that a mixed-case ingestion-version override wins Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): keep the exact digits of large integer metadata values Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(experiments): skip orphaned run scores and offline project lookups Run-level scores are only sent when at least one item trace is sampled, because a fully sampled-out experiment never reaches the server. Clients with tracing disabled no longer resolve the project id. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * chore: re-run CI Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): match JS for nested big ints and non-finite metadata Integers keep their exact digits as JSON numbers at any depth, and propagated metadata values containing NaN or Infinity are dropped with a warning instead of being stored, matching the JS SDK. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client): keep host and LANGFUSE_HOST working without a warning Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
* feat(langchain)!: drop LangChain v0 support The CallbackHandler now imports only from langchain_core and no longer requires the langchain package to be installed. Remove the langchain.__version__ branch and the langchain.callbacks / langchain.schema fallbacks for LangChain v0. The serializer fallback tests now hide langchain_core, the module the serializer actually imports, and reload the serializer module. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * build(deps)!: require langchain-core>=1 and langchain-openai>=1 Add langchain-core>=1,<2 to the dev dependencies and raise langchain-openai from >=0.0.5,<0.4 to >=1,<2. The lock moves langchain-openai 0.3.34 -> 1.1.12, which keeps langchain-core 1.2.22 and openai 2.29.0 unchanged. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
…s API (#1936) * chore!: require Python 3.11 or newer Python 3.10 reaches end of life in October 2026. Raise requires-python to >=3.11, target py311 in ruff, drop 3.10 from the CI unit matrix, and remove the pre-3.11 fallbacks for asyncio.create_task(context=...) and typing.NotRequired. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(openai)!: drop support for OpenAI SDK versions below 1.0 The OpenAI integration now always uses the v1 client shapes. Remove the pre-1.0 method definitions and every _is_openai_v1() branch, and raise the dev dependency floor to openai>=1.0.0. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(otel)!: send x-langfuse-ingestion-version: 4 by default The default OTLP exporter now sends x-langfuse-ingestion-version: 4 so servers in dual-write or preview mode process exported spans directly. It is a no-op on events_only deployments, and additional_headers can still override it. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client)!: remove blocked_instrumentation_scopes blocked_instrumentation_scopes was deprecated in favor of should_export_span. Remove it from Langfuse, the resource manager and the span processor, and drop the leftover is_active entry from the create_prompt docstring. LANGFUSE_HOST and host keep working, but now log a deprecation warning when they decide the base URL. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(tracing)!: JSON-serialize propagated metadata values propagate_attributes(metadata=...) coerced non-string values with str(), so a boolean became "True" and a list became its Python repr. Serialize them as JSON instead, like observation metadata, and drop None values instead of sending "None". Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client)!: remove get_dataset_run, get_dataset_runs and delete_dataset_run These helpers call the dataset-run endpoints that Langfuse v4 no longer serves, so they always fail with a 404. Read experiment runs through langfuse.api.experiments.list() and list_items() instead. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(experiments)!: record experiment runs via OTEL only Generate the experiment id client-side instead of creating dataset run items via POST /dataset-run-items. Dataset-backed runs derive a stable id from project, dataset and run name (matching the server's derivation), so re-runs with the same run name keep grouping into one experiment. Local runs keep a random id. - set langfuse.experiment.item.version from the pinned dataset version - always persist run evaluations as scores on the experiment, incl. local data - return experiment_id / experiment_url (v4 results page) on results; dataset_run_id / dataset_run_url remain as deprecated aliases BREAKING CHANGE: run_experiment no longer creates Postgres dataset runs; dataset_run_id / dataset_run_url are deprecated aliases that are now also set for local-data experiments, and the URL points to the experiments page. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(experiments): read experiment e2e results via v4 APIs Replace dataset run, dataset run item and legacy trace reads with the experiments, observations and v3 scores APIs, and cover stable experiment ids, run-level scores for local data and the recorded item version. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(experiments): pin the cross-SDK experiment id vectors Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(batch-evaluation)!: evaluate observations via the v2 observations API Batch evaluation now reads exclusively from GET /api/public/v2/observations with cursor pagination, so it works on Langfuse platform v4. BREAKING CHANGES: - scope="traces" is replaced by scope="root_observations", which evaluates the logical root observation of each trace and attaches scores to the trace. scope="observations" is unchanged in intent. - Mappers receive an ObservationV2. input/output are raw strings and only the requested field groups are populated. - fetch_trace_fields is replaced by fields (v2 field groups), defaulting to "core,basic,io,metadata" in both the client and the runner. - filter must be a JSON array in the v2 observations filter format. - Resume tokens carry the pagination cursor and are also returned when max_items is reached while more items exist. - The private _additional_trace_tags option is removed. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(batch-evaluation): rewrite e2e suite for v2 observations Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): serialize propagated metadata like JSON.stringify Use compact separators, keep non-ASCII characters, and send None as "null", so the Python and JS SDKs emit byte-identical propagated metadata values. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(observe): keep detecting legacy-marked coroutine functions inspect.iscoroutinefunction ignores asyncio's _is_coroutine marker, which asgiref's markcoroutinefunction sets on Python < 3.12. Accept both so such functions keep the async wrapper and their observation ends after the coroutine runs. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(otel): pin that a mixed-case ingestion-version override wins Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): keep the exact digits of large integer metadata values Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(experiments): skip orphaned run scores and offline project lookups Run-level scores are only sent when at least one item trace is sampled, because a fully sampled-out experiment never reaches the server. Clients with tracing disabled no longer resolve the project id. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(batch-evaluation): one root per trace and safe resume - root_observations scope evaluates each trace once, preferring the physical root over SDK-marked app roots - resuming reuses the token's filter and rejects a different one, since the cursor only continues the query that produced it - the cursorless start-time fallback is inclusive so tied observations are not skipped Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * chore: re-run CI Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * chore: re-run CI Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): match JS for nested big ints and non-finite metadata Integers keep their exact digits as JSON numbers at any depth, and propagated metadata values containing NaN or Infinity are dropped with a warning instead of being stored, matching the JS SDK. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(client): keep host and LANGFUSE_HOST working without a warning Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(tracing)!: remove trace-level input/output setters (#1935) Remove the deprecated Langfuse.set_current_trace_io() and span.set_trace_io(), the langfuse.trace.input/output attributes, and their media handling. On Langfuse v4 a trace's input and output come from its root observation: set them with update() on the root span or update_current_span(). Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test: run the e2e suites on Langfuse v4 events_only (#1937) * feat(tracing)!: remove trace-level input/output setters Remove the deprecated Langfuse.set_current_trace_io() and span.set_trace_io(), the langfuse.trace.input/output attributes, and their media handling. On Langfuse v4 a trace's input and output come from its root observation: set them with update() on the root span or update_current_span(). Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test: read e2e assertions through the v4 observations API Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * ci: run e2e against an events_only server Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test: pin that create_score logs invalid input without enqueueing Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test: remove the v3 wait_for_trace helper Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test: keep scalar text IO as strings and read full response_format The v2 observations API returns IO as raw strings, so only JSON objects and arrays are parsed back; text like "2" stays a string. The response_format assertion requests the untruncated metadata value. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test: dedupe observation rows and assert streamed text as a string The events table can briefly return several rows for one span, so the observation helper keeps the latest row per id. A streamed completion of "2" is text and is now asserted as a string. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * feat(batch-evaluation)!: evaluate observations only Remove the scope parameter and root-observation mode, so batch evaluation always runs on the observations matching the filter and scores each observation. Resume tokens no longer carry a scope. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(batch-evaluation): evaluate duplicate observation rows once The events table can briefly return several rows for one observation, within a page or across a page boundary. Keep the most recently updated row per id so each observation is evaluated and scored once. Also build exception messages before raising. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
…1929) * fix(tracing)!: require the gen_ai. prefix in the default span filter is_genai_span matched any attribute key starting with "gen_ai", so lookalikes such as gen_ai_custom or gen_airflow.model exported unrelated third-party spans. Match only the gen_ai.* namespace. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * chore: re-run CI Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * chore: re-run CI Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * chore: re-run CI Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * chore: re-run CI Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
The default OTLP exporter now gzips span batches unless configured otherwise. Resolution order: the otel_compression argument, LANGFUSE_OTEL_COMPRESSION, OTEL_EXPORTER_OTLP_TRACES_COMPRESSION, OTEL_EXPORTER_OTLP_COMPRESSION, then gzip. Values are case-insensitive and an invalid value falls through to the next setting, matching the JS SDK. Set any of them to "none" to send uncompressed. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
…1955) * chore(api): drop references to API types removed from the v4 spec langfuse.model no longer re-exports DatasetRun, Observation and TraceWithFullDetails, which the regenerated API client removes. The api docstrings point to the v2 observations, scores_v3 and experiments reads instead of trace.get/list and the legacy v1 resources. Two unit tests no longer mock the removed dataset_run_items resource. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * chore: re-run CI Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
* feat(api): update API spec from langfuse/langfuse 21b9b33 * fix(api): adapt the legacy score-create alias to the CreateScoresRequest union scores.create now takes a request body that is one score or a list. The legacy score_v1.create alias keeps its keyword signature, builds a CreateScoreRequest from the passed arguments, and calls scores.create(request=...). The postprocessor upgrades an alias applied by an earlier template instead of leaving it in place. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: langfuse-bot <langfuse-bot@langfuse.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
Co-authored-by: langfuse-bot <langfuse-bot@langfuse.com>
* fix(tracing): keep metadata within the span attribute limit Each metadata key is written as its own OTel attribute. Once a span holds more than SpanLimits.max_span_attributes (default 128) attributes, the OTel SDK evicts the oldest ones, which are the Langfuse core attributes such as input, model and version. Before writing observation attributes, count the span's existing attribute keys plus the new keys and drop the excess new metadata keys from the tail. Non-metadata attributes and overwrites of existing keys are never dropped. Log one warning per write that dropped keys instead of raising. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(tracing): reserve attribute slots for later observation updates Capping metadata at exactly the span attribute limit left no room for attributes written in later updates, so e.g. update(output=...) made the OTel SDK evict the oldest attribute (langfuse.internal.is_app_root, then input, model, ...). Metadata now also leaves one slot for each reserved observation attribute that is not on the span or in the current write yet, matching langfuse-js. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(tracing): pin span attribute limit in metadata limit tests The tests assert the OTel default of 128 attributes, so a custom OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT or OTEL_ATTRIBUTE_COUNT_LIMIT in the environment made them fail. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(tracing): drop newest attributes instead of evicting on span attribute overflow The Python OTel SDK evicts the oldest attribute once a span is full, while OTel JS drops the newest. After capping metadata, the limit helper now also drops excess new non-metadata keys from the tail when the span has no physical capacity left, so Langfuse writes never evict existing attributes. Overwrites of existing keys are always kept, and at most one warning is logged per write. Langfuse write paths on observation spans now go through the guard: the observation constructor, update(), media-processed updates, set_trace_as_public, the experiment item attribute writes, as_root marking, propagated attributes in the span processor and propagate_attributes, and the app-root marker. The warning now names SpanLimits.max_span_attributes alongside OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT, matching langfuse-js. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(experiments): keep experiment run keys when metadata hits the span limit The task span metadata put experiment_name, experiment_run_name and the dataset ids after item and experiment metadata, so the span attribute limit dropped them first. Insert the run keys first so they survive truncation, and again last so they still win over user keys of the same name. Mirrors langfuse-js. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(tracing): reserve attribute slots by observation type Span-like observations and events never write model, usage, cost, completion start time, model parameters or prompt attributes, so reserving slots for them wasted 7 metadata keys. Reserve the 7 shared keys (type, level, status message, version, environment, input, output) for span-like types and events, and all 14 for generation-like types and for writes whose observation type is unknown (as_root, propagated attributes, app-root marker). Mirrors langfuse-js. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(experiments): propagate experiment metadata as one JSON attribute Experiment metadata and experiment item metadata were propagated as one attribute per flattened key (langfuse.experiment.item.metadata.<path>), so large item metadata filled the task span and its children and left no room for later attributes such as the output. Send each as a single serialized JSON attribute (langfuse.experiment.metadata and langfuse.experiment.item.metadata), like langfuse-js. The server parses the blob at the bare prefix and flattens it to the same paths. Experiment attributes are set by the SDK and skip the 200-character validation of user-propagated values, matching langfuse-js. Remove the now unused _flatten_and_serialize_metadata_values. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(experiments): write task span metadata after the experiment attributes The experiment task span got its observation metadata at start, before the experiment attributes were copied onto it. The metadata cap could then fill the span up to the reserved slots, and the copied attributes used those slots, so the later output was dropped. Start the task span without metadata and set it inside the propagation block, before the task runs, so the cap counts the copied attributes and trims the observation metadata instead. The experiment run keys still go first. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…#1956) * fix(tracing): mask metadata per key when the mask function fails When a custom mask function raised on dict metadata, the fallback string was written to the bare langfuse.observation.metadata attribute. The server parses that attribute as a JSON object, so the fallback was lost, and nothing showed which keys had been masked. For dict metadata, write the fallback under each key of the failed call instead, so it merges with keys from earlier updates like any other update. Non-dict metadata, input and output keep the plain-string fallback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(tracing): apply mask to propagated trace metadata propagate_attributes now applies the client's mask to each propagated trace metadata value (keys are kept), on the raw value before serialization and the 200 character check. If the mask raises, only that value becomes the fallback string. The masked values are stored in the context and baggage, so the current span, child spans, and outgoing baggage all carry them. Matches langfuse-js #1003. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(tracing): document which client's mask applies to propagated metadata The mask for propagated trace metadata comes from the client for the public key in the execution context, otherwise the only initialized client. With several clients and no public key in context, or with no client yet, propagated metadata is not masked. Document this and the workarounds, and pin the behavior with a test. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * revert(tracing): do not mask propagated trace metadata Propagated metadata is passed explicitly by the developer. Masking it adds client-resolution ambiguity and baggage/length-limit ordering complexity, and diverges from the JS SDK. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): keep string fallback when the mask fails on empty dict metadata Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
* test(api): pin how POST /scores responses parse into the union Cover the 200, 202 and 207 response shapes of scores.create, sync and async, against a mocked transport. CI also runs them on pydantic 2.7, whose union matching depends on the member order. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * test(api): assert the pydantic 2.7 overlay and describe its coverage Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
) * fix(tracing): JSON-encode metadata attribute values Per-key `langfuse.observation.metadata.<key>` attributes now carry `json.dumps(value, cls=EventSerializer)` for every value, strings included. Previously str and int were passed raw/natively, so ints beyond int64 made the OTLP encoder drop the key, ints beyond 2^53 lost precision on the server, and a server that JSON-decodes values could not tell the string "123" from the number 123. None values are still skipped so updates keep earlier keys. The bare non-dict metadata attribute is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(tracing)!: JSON-encode non-dict metadata too The bare `langfuse.observation.metadata` attribute (non-dict metadata) is now sent as `json.dumps(metadata, cls=EventSerializer)`, strings included, matching the per-key values and langfuse-js #1005. Previously strings were passed through raw, so `metadata="foo"` sent `foo` in Python but `"foo"` in JS. None is still not written. Input/output and other `_serialize` callers are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(e2e): match metadata expectations to server decoding The server JSON-decodes per-key metadata values only for Python SDK major >= 5 (langfuse/langfuse#18436). On 4.x the experiment items API returns the raw JSON string, so compare against the encoded value until the version bump. test_concurrency now writes an int so it round-trips the same way with and without server-side decoding. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(tracing): write metadata JSON with compact separators and raw non-ASCII Encode observation and trace metadata values with separators=(",", ":") and ensure_ascii=False so the output is byte-identical to JS JSON.stringify, matching propagated metadata (#1932). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(tracing): debug-log metadata keys skipped because their value is None Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(tracing)!: JSON-encode propagated trace metadata like observation metadata propagate_attributes(metadata=...) now uses the same encoder as observation metadata, so a string such as "123" stays a string once the server decodes langfuse.trace.metadata.<key>. Ints outside the JS-safe range and NaN/Infinity become JSON strings instead of exact digits or being dropped. The 200 character limit applies to the encoded value. Plain str values skip EventSerializer and go straight to json.dumps; the output is byte-identical and pinned against EventSerializer and JSON.stringify. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(media): extract top-level data URIs from JSON-encoded metadata values Metadata values are JSON-encoded, so a top-level data URI arrives quoted and the exporter no longer recognized it, shipping the base64 inline. Decode the JSON string for Langfuse metadata attributes, process the media, and re-encode the reference string. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * style(tests): format span attribute limit test Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * fix(tracing): drop None propagated metadata values with a warning Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> * perf(media): extract JSON-encoded metadata data URIs without json.loads Strip the quotes of a JSON-encoded top-level data URI in place instead of decoding and re-encoding it. Values with backslashes, inner quotes, or no ;base64 header in the first 256 characters are left unchanged. Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
* fix(tracing): do not mask attributes the caller never set
`_process_media_and_apply_mask` runs for input, output and metadata on
every span creation and update, regardless of whether the caller provided
them. An unset attribute is still handed to the user's mask function as
`None`. A mask that assumes a dict throws, `_mask_attribute` then fails
closed, and the fallback string is not `None` -- so it survives the
`v is not None` filters in `create_span_attributes` /
`create_generation_attributes`, and lands on the span as an attribute the
caller never set. Each unset attribute also logs one ERROR per span,
which makes it look like the user's mask function is broken.
Repro with a mask that assumes a dict and only `output` set:
mask receives [None, None, None, None, {'answer': '42'}, None]
span carries langfuse.observation.input / .metadata =
'<fully masked due to failed mask function>'
plus 5 ERROR logs ("'NoneType' object is not a mapping")
Return early when `data is None`: there is nothing to process or mask.
Attributes that do have a value keep the existing fail-closed behaviour
and the per-key metadata fallback.
Adds two regression tests asserting that only attributes the caller set
reach the mask, and that unset attributes stay unset.
* push
---------
Co-authored-by: Hassieb Pakzad <68423100+hassiebp@users.noreply.github.com>
The test asserted raw metadata values, but metadata values are JSON-encoded on prepare-v5-release, so it failed there. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>
* docs(api): document metadata is set / is not set filter operators (#1946) Regenerated langfuse/api from the upstream Fern definition: baseline langfuse/langfuse 619dbe3 (last sync, #1866) plus langfuse/langfuse#17357 and #18404. Split out of the bundled API-spec bot PR #1943. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(api): type Dataset JSON-schema response fields as objects (#1949) Regenerated langfuse/api from the upstream Fern definition: baseline langfuse/langfuse 619dbe3 (last sync, #1866) plus langfuse/langfuse#17892 and the datasets.yml/opentelemetry.yml part of #18406. Split out of the bundled API-spec bot PR #1943. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * refactor(api): share cursor pagination meta via CursorMetaResponse (#1948) Regenerated langfuse/api from the upstream Fern definition: baseline langfuse/langfuse 619dbe3 (last sync, #1866) plus langfuse/langfuse#17888. Split out of the bundled API-spec bot PR #1943. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(api): document fromStartTime as required on experiments endpoints (#1947) * docs(api): document fromStartTime as required on experiments endpoints Regenerated langfuse/api from the upstream Fern definition: baseline langfuse/langfuse 619dbe3 (last sync, #1866) plus langfuse/langfuse#17603. Split out of the bundled API-spec bot PR #1943. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(api): use language-neutral wording for experiments time window Regenerated from langfuse/langfuse branch docs/experiments-from-start-time-neutral-wording. The fromStartTime docstrings named `toStartTime`, but the Python parameter is `to_start_time`; they now say "an upper bound" instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(api): type read-side metadata as optional objects (#1950) * fix(api): type read-side metadata as optional objects Regenerated langfuse/api from the upstream Fern definition: baseline langfuse/langfuse 619dbe3 (last sync, #1866) plus langfuse/langfuse#18122 and the commons.yml part of #18406. Split out of the bundled API-spec bot PR #1943. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(api): pin read-side metadata parsing on generated models Cover omitted, null, empty and nested-JSON metadata plus rejection of non-object metadata for Trace, Observation, ObservationV2, BaseScore, BaseScoreV1 and every Score / ScoreV1 / GetScoresResponseData variant. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Niklas Semmler <1880943+niklassemmler@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* test: run unit tests without langchain and langgraph installed Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * build: lock dependencies with Python 3.15 wheels Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * ci: run unit tests on Python 3.15 and declare supported versions Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(scores)!: send scores via POST /api/public/scores Scores are now batched to the public scores endpoint instead of being sent as score-create events to /api/public/ingestion. The consumer keeps flush_at, flush_interval and the batch byte limit, retries transport errors, 429 and 5xx with backoff, does not retry other 4xx, and logs rejected items from a 207 partial result without retrying them. Scores for traces that the tracer sampler drops are now dropped as well; the trace id was previously never read, so sampling was skipped. The trace-create ingestion path used for trace tags is removed. BREAKING CHANGE: batch score delivery needs Langfuse platform v4.55.0 or later. Custom score timestamps need the platform release that accepts timestamp on POST /scores; older servers record the receive time. Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * chore(api)!: drop the legacy score create compatibility layer Removes legacy.score_v1.create, the CreateScore* aliases it exposed, and the script that patched them into the generated client. Score deletion through legacy.score_v1.delete is unchanged. BREAKING CHANGE: use api.scores.create to create scores. Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * chore(integrations)!: stop passing the deprecated usage argument BREAKING CHANGE: OpenAI and LangChain generations only report usage_details. Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * style: format test_media Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * test(e2e): expect custom score timestamps to fail until the platform supports them Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * fix(scores): send naive score timestamps as UTC The scores API rejects timestamps without a UTC offset. Naive datetimes were serialized with the local offset; they are now treated as UTC. Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * feat(scores): hint at the minimum server version when a score batch gets a 400 Langfuse servers before v4.55.0 reject list bodies on POST /scores. The first 400 in a process now logs that batched score creation requires v4.55.0 or later and that the scores were dropped. Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * fix(scores): match the JS minimum server version hint and log repeats at debug Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * refactor(scores): drop the minimum server version hint Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * fix(scores): keep scores unless the trace-id sampler drops their trace Re-running a parent-based, random or attribute-based sampler with only the trace id can disagree with the decision the trace actually got, so scores are only dropped for Langfuse's trace-id ratio sampler. Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * fix(scores): cap LANGFUSE_MAX_BATCH_SIZE_BYTES at the scores body limit Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: langfuse-bot <langfuse-bot@langfuse.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 43412fbce3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def _is_observation_metadata_key(key: str) -> bool: | ||
| prefix = LangfuseOtelSpanAttributes.OBSERVATION_METADATA | ||
| return key == prefix or key.startswith(prefix + ".") |
There was a problem hiding this comment.
Treat langfuse.trace.metadata.* attributes as metadata here as well. When propagate_attributes(metadata=...) supplies enough keys to approach OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT, on_start() writes those keys before the observation wrapper writes its core attributes; because this predicate recognizes only observation metadata, the metadata budget retains every trace-metadata key and later writes such as observation type, input, output, and model are dropped. With the default limit, 128 propagated metadata keys can therefore silently strip all Langfuse observation data instead of trimming metadata.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in #1993. Propagated langfuse.trace.metadata.* keys now share the metadata budget and the reserved slots, and a regression test checks that type, input, model, and output survive.
| try: | ||
| decision = sampler.should_sample( | ||
| parent_context=None, trace_id=int(trace_id, 16), name="experiment" | ||
| ).decision |
There was a problem hiding this comment.
Do not recompute sampling with parent_context=None. For a parent-based or attribute-based sampler, this can differ from the decision that was made when the experiment span started—for example, a sampled remote parent with an always-off root policy records and exports the item span, but this call returns DROP and causes every run-level evaluation score to be skipped; the inverse can enqueue orphan scores. Capture the actual span's sampled trace flag while processing the item instead of invoking the sampler again.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in #1993. Run scores are now gated on the sampling decision recorded on the real experiment-item-run spans, not on a synthetic sampler call. A regression test uses a name-based sampler.
* docs(api): document metadata is set / is not set filter operators (#1946) Regenerated langfuse/api from the upstream Fern definition: baseline langfuse/langfuse 619dbe3 (last sync, #1866) plus langfuse/langfuse#17357 and #18404. Split out of the bundled API-spec bot PR #1943. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(api): type Dataset JSON-schema response fields as objects (#1949) Regenerated langfuse/api from the upstream Fern definition: baseline langfuse/langfuse 619dbe3 (last sync, #1866) plus langfuse/langfuse#17892 and the datasets.yml/opentelemetry.yml part of #18406. Split out of the bundled API-spec bot PR #1943. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * refactor(api): share cursor pagination meta via CursorMetaResponse (#1948) Regenerated langfuse/api from the upstream Fern definition: baseline langfuse/langfuse 619dbe3 (last sync, #1866) plus langfuse/langfuse#17888. Split out of the bundled API-spec bot PR #1943. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(api): document fromStartTime as required on experiments endpoints (#1947) * docs(api): document fromStartTime as required on experiments endpoints Regenerated langfuse/api from the upstream Fern definition: baseline langfuse/langfuse 619dbe3 (last sync, #1866) plus langfuse/langfuse#17603. Split out of the bundled API-spec bot PR #1943. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(api): use language-neutral wording for experiments time window Regenerated from langfuse/langfuse branch docs/experiments-from-start-time-neutral-wording. The fromStartTime docstrings named `toStartTime`, but the Python parameter is `to_start_time`; they now say "an upper bound" instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(api): type read-side metadata as optional objects (#1950) * fix(api): type read-side metadata as optional objects Regenerated langfuse/api from the upstream Fern definition: baseline langfuse/langfuse 619dbe3 (last sync, #1866) plus langfuse/langfuse#18122 and the commons.yml part of #18406. Split out of the bundled API-spec bot PR #1943. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(api): pin read-side metadata parsing on generated models Cover omitted, null, empty and nested-JSON metadata plus rejection of non-object metadata for Trace, Observation, ObservationV2, BaseScore, BaseScoreV1 and every Score / ScoreV1 / GetScoresResponseData variant. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Niklas Semmler <1880943+niklassemmler@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| decision = sampler.should_sample( | ||
| parent_context=None, trace_id=int(trace_id, 16), name="experiment" | ||
| ).decision |
There was a problem hiding this comment.
_is_trace_sampled asks the sampler again with name="experiment", rather than using the decision made for the item span. If a custom sampler keeps experiment-item-run and experiment-item-task but drops other names, the experiment is exported, yet this check suppresses all its run-level scores. The returned result still contains those scores.
Keep the actual sampling decision, or restrict this check to samplers whose decision depends only on the trace ID.
Knowledge Base Used: Experiments and batch evaluation
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/_client/client.py
Line: 2806-2808
Comment:
**Experiment scores disappear**
`_is_trace_sampled` asks the sampler again with `name="experiment"`, rather than using the decision made for the item span. If a custom sampler keeps `experiment-item-run` and `experiment-item-task` but drops other names, the experiment is exported, yet this check suppresses all its run-level scores. The returned result still contains those scores.
Keep the actual sampling decision, or restrict this check to samplers whose decision depends only on the trace ID.
**Knowledge Base Used:** [Experiments and batch evaluation](https://app.greptile.com/personal-org-4986/-/custom-context/knowledge-base/langfuse/langfuse-python/-/docs/experiments-and-batch-evaluation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Confirmed and fixed in #1993. The gate now uses each item span's recorded sampling decision.
| def _is_observation_metadata_key(key: str) -> bool: | ||
| prefix = LangfuseOtelSpanAttributes.OBSERVATION_METADATA | ||
| return key == prefix or key.startswith(prefix + ".") |
There was a problem hiding this comment.
_is_observation_metadata_key excludes langfuse.trace.metadata.*, where propagate_attributes(metadata=...) writes its keys. When propagated metadata exceeds the span's attribute limit, the new helper fills the physical capacity without leaving the reserved slots free. During on_start, this can fill the span before the wrapper writes its type and input. Later output updates are dropped too.
Apply the metadata budget to propagated metadata as well.
Knowledge Base Used: OpenTelemetry propagation and export
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/_client/span.py
Line: 67-69
Comment:
**Metadata crowds out observation data**
`_is_observation_metadata_key` excludes `langfuse.trace.metadata.*`, where `propagate_attributes(metadata=...)` writes its keys. When propagated metadata exceeds the span's attribute limit, the new helper fills the physical capacity without leaving the reserved slots free. During `on_start`, this can fill the span before the wrapper writes its type and input. Later output updates are dropped too.
Apply the metadata budget to propagated metadata as well.
**Knowledge Base Used:** [OpenTelemetry propagation and export](https://app.greptile.com/personal-org-4986/-/custom-context/knowledge-base/langfuse/langfuse-python/-/docs/opentelemetry-propagation-and-export.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Confirmed and fixed in #1993. Trace metadata keys are now budgeted with observation metadata.
| resumed_item_id = ( | ||
| resume_from.last_processed_id | ||
| if resume_from is not None | ||
| and resume_from.cursor is None | ||
| and resume_from.last_processed_timestamp | ||
| else None | ||
| ) | ||
| previous_page_ids: Set[Optional[str]] = set() |
There was a problem hiding this comment.
Resuming with a cursor resets previous_page_ids and does not exclude last_processed_id. The events table can return several rows for one observation across a page boundary. If max_items stops after the first row, resuming processes the next copy again and creates another score for the same observation.
Carry the boundary page's IDs in the resume token and restore them before fetching the next page.
Knowledge Base Used: Experiments and batch evaluation
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/batch_evaluation.py
Line: 892-899
Comment:
**Resuming creates duplicate scores**
Resuming with a cursor resets `previous_page_ids` and does not exclude `last_processed_id`. The events table can return several rows for one observation across a page boundary. If `max_items` stops after the first row, resuming processes the next copy again and creates another score for the same observation.
Carry the boundary page's IDs in the resume token and restore them before fetching the next page.
**Knowledge Base Used:** [Experiments and batch evaluation](https://app.greptile.com/personal-org-4986/-/custom-context/knowledge-base/langfuse/langfuse-python/-/docs/experiments-and-batch-evaluation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Confirmed and fixed in #1993. The resume token now carries last_page_ids, and a resumed run skips them. A regression test splits duplicate rows across the page boundary.
| Examples | ||
| -------- | ||
| from langfuse import LangfuseAPI | ||
| from langfuse.scores import CreateScoreRequest |
There was a problem hiding this comment.
The new examples import from langfuse.scores and langfuse.unstable.skills, but these packages live under langfuse.api. Copying either example raises ModuleNotFoundError before making a request. Use langfuse.api for CreateScoreRequest and langfuse.api.unstable.skills for the skill types. The new batch-score example has the same wrong paths.
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/api/scores/client.py
Line: 53
Comment:
**New examples fail on import**
The new examples import from `langfuse.scores` and `langfuse.unstable.skills`, but these packages live under `langfuse.api`. Copying either example raises `ModuleNotFoundError` before making a request. Use `langfuse.api` for `CreateScoreRequest` and `langfuse.api.unstable.skills` for the skill types. The new batch-score example has the same wrong paths.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Not fixing this here. langfuse/api/ is generated by Fern and must not be hand-edited. These docstring examples come from the upstream Fern definitions in langfuse/langfuse, so the fix has to happen there and be regenerated. It doesn't affect runtime behavior.
| ) | ||
| return self._create_observation_id() | ||
|
|
||
| import json |
There was a problem hiding this comment.
The new import json inside _create_experiment_id violates the repository instruction to move imports to the top of the module instead of placing them inside functions or methods. Move the new imports to module scope before merging to satisfy this requirement. _is_trace_sampled and _format_experiment_item_version do this too, and the pattern appears elsewhere in the new code.
Rule Used: Move imports to the top of the module instead of placing them within functions or methods. (source)
Learned From
langfuse/langfuse-python#1387
Prompt To Fix With AI
This is a comment left during a code review.
Path: langfuse/_client/client.py
Line: 2785
Comment:
**Imports break the team rule**
The new `import json` inside `_create_experiment_id` violates the repository instruction to move imports to the top of the module instead of placing them inside functions or methods. Move the new imports to module scope before merging to satisfy this requirement. `_is_trace_sampled` and `_format_experiment_item_version` do this too, and the pattern appears elsewhere in the new code.
**Rule Used:** Move imports to the top of the module instead of placing them within functions or methods. ([source](https://app.greptile.com/personal-org-4986/-/custom-context?memory=c960fc07-9928-409f-a18b-a780cbdded12))
**Learned From**
[langfuse/langfuse-python#1387](https://github.com/langfuse/langfuse-python/pull/1387)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Fixed in #1993: json and serialize_datetime are now imported at module level.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🔴
langfuse/_client/client.py— With a custom sampler that is not a pure trace-id ratio sampler, run-level experiment scores can be silently skipped even though the item traces were exported._is_trace_sampledre-runs any sampler withparent_context=None,name="experiment"and no attributes, then treats a DROP as "sampled out". Fix: use the item span's real sampled decision (for example, recordspan._otel_span.get_span_context().trace_flags.sampledoris_recording()inExperimentItemResult). Or matchadd_score_task, which only trustsTraceIdRatioBasedand otherwise keeps the score.Why this was flagged
Trigger:
Langfuse(tracer_provider=...), or a globally registered provider, whose sampler decides by span name or attributes (rule-based or attribute-based samplers). client.py:614-621 keepsrun_evaluationsonly if_is_trace_sampled(result.trace_id)holds for some item._is_trace_sampled(client.py:696-711) callssampler.should_sample(parent_context=None, trace_id=..., name="experiment"), with a span name and attributes the realexperiment-item-runspans never had. If that synthetic call returns DROP, the code logs 'Skipping run-level scores' at debug level and never sends the run scores, even though the items were recorded. On the base branch, run evaluations were stored whenever a dataset run id existed, with no sampler re-check.add_score_task(resource_manager.py:500-503) is in this same diff and explicitly avoids this: 'Parent-based, random or attribute-based samplers may have recorded the trace, so keep its scores.'Verification:
_is_trace_sampled(2797-2812) takes whatevertracer_provider.sampleris and callssampler.should_sample(parent_context=None, trace_id=int(trace_id, 16), name="experiment"). The SDK's own score path in resource_manager.py (lines 496-506) avoids this deliberately. At the base commit, run evaluations were stored whenever they existed, with no sampling check.
| def _resolve_compression(otel_compression: Optional[str]) -> Compression: | ||
| """Resolve compression for the default exporter; gzip unless configured otherwise. | ||
| Order: the ``otel_compression`` argument, ``LANGFUSE_OTEL_COMPRESSION``, | ||
| ``OTEL_EXPORTER_OTLP_TRACES_COMPRESSION``, ``OTEL_EXPORTER_OTLP_COMPRESSION``, | ||
| then gzip. Values are case-insensitive, and an invalid value falls through to | ||
| the next setting, matching the JS SDK. | ||
| """ | ||
| candidates = ( | ||
| ("otel_compression", otel_compression, True), | ||
| (LANGFUSE_OTEL_COMPRESSION, os.environ.get(LANGFUSE_OTEL_COMPRESSION), True), | ||
| ( | ||
| OTEL_EXPORTER_OTLP_TRACES_COMPRESSION, | ||
| os.environ.get(OTEL_EXPORTER_OTLP_TRACES_COMPRESSION), | ||
| False, | ||
| ), | ||
| ( | ||
| OTEL_EXPORTER_OTLP_COMPRESSION, | ||
| os.environ.get(OTEL_EXPORTER_OTLP_COMPRESSION), | ||
| False, | ||
| ), | ||
| ) | ||
| for setting, raw_value, warn in candidates: | ||
| compression = _parse_compression(raw_value, setting=setting, warn=warn) | ||
| if compression is not None: | ||
| return compression | ||
|
|
||
| return Compression.Gzip |
There was a problem hiding this comment.
🔴 Self-hosted users on Langfuse servers older than v3.30.0 would have every span export rejected after upgrading, with no warning. The base SDK sent uncompressed spans by default. _resolve_compression now falls back to Compression.Gzip when nothing is configured. The diff also deletes the note in environment_variables.py and the otel_compression docstring that said "gzip requires Langfuse server v3.30.0 or later". Fix: either document a minimum server version for v5 where users will see it and warn when spans are rejected, or keep uncompressed as the default.
Why this was flagged
Trigger: a user leaves otel_compression, LANGFUSE_OTEL_COMPRESSION, OTEL_EXPORTER_OTLP_TRACES_COMPRESSION and OTEL_EXPORTER_OTLP_COMPRESSION all unset, and the server is older than v3.30.0. This happens to every self-hosted instance that has not been upgraded. In that case span_processor.py:125 returns Compression.Gzip, and line 204 passes it to OTLPSpanExporter. On the base branch _resolve_compression returned None, so the exporter used OTel's default, which is no compression. The old docstring said gzip needs server v3.30.0 or later, and this diff removes that note without stating a minimum server version anywhere in the repo (no README or AGENTS.md mentions one). A server that cannot decode the gzipped protobuf body rejects each OTLP request. The BatchSpanProcessor only logs the failure, so all traces from an upgraded SDK are lost. Users only recover if they know to set otel_compression="none".
Verification: Trigger: a self-hosted Langfuse server older than v3.30.0 with no compression setting configured. The new _resolve_compression in langfuse/_client/span_processor.py reaches return Compression.Gzip (line 125) when nothing is set. On the base, it returned None, so OTel's default of no compression applied. Nothing is logged at setup to warn users.
There was a problem hiding this comment.
Intended, so not a bug. v5's minimum server version is v4.59.0, and every supported server accepts gzip OTLP. Proxies that can't handle gzip can opt out with otel_compression="none".
| config_id=config_id, | ||
| environment=environment or self._environment, | ||
| metadata=metadata, | ||
| timestamp=timestamp or _get_timestamp(), |
There was a problem hiding this comment.
🔴 Users who pass timestamp= to create_score will have it silently ignored, so backdated scores are stored at the time the request arrives. The base branch kept the timestamp. Scores now go to POST /api/public/scores, and the change's own e2e test says the server does not read timestamp there yet. Fix: ship this only with a minimum-server-version requirement that is documented and checked, or keep scores that carry an explicit timestamp on a path that honours it until that release is out. Older self-hosted servers will never honour it.
Why this was flagged
A caller backfills evaluations with langfuse.create_score(name=..., value=..., trace_id=..., timestamp=datetime.now(timezone.utc) - timedelta(hours=1)). client.py:1959 puts the value into the CreateScoreRequest body, and ScoreConsumer sends it to POST /api/public/scores (score_consumer.py:226). This PR marks tests/e2e/test_core_sdk.py:363-370 (test_create_score_with_custom_timestamp) as xfail with the reason "Needs the Langfuse platform release that adds timestamp to POST /scores". So the server CI runs against (langfuse HEAD) drops the field, and the score gets the time the request was received. On the base branch the same test ran without xfail: the timestamp went as the ingestion event timestamp to /api/public/ingestion and was stored as the score time. Nothing warns: the request returns 2xx/202, and create_score's docstring still promises the custom timestamp. Self-hosted servers older than that unreleased version will drop it permanently.
Verification: client.py:1959 sets timestamp=timestamp or _get_timestamp() inside the CreateScoreRequest body, sent to POST /api/public/scores. The field doc (langfuse/api/scores/types/create_score_request.py:80-82) says it "Defaults to the time the request is received." The base put it on the score-create ingestion event, and its test passed without xfail. The marker at tests/e2e/test_core_sdk.py:363-369 notes the gap, and the SDK has no version check.
There was a problem hiding this comment.
This is real, but it depends on the platform, not the SDK. POST /scores on server main (v4.58.0) doesn't accept timestamp yet. That support is in langfuse/langfuse#18516, which has to ship as the v4.59.0 minimum before this SDK release. The e2e test stays xfail until then. No SDK change is needed.
| # The API rejects timestamps without a UTC offset; naive values are UTC. | ||
| if timestamp is not None and timestamp.tzinfo is None: | ||
| timestamp = timestamp.replace(tzinfo=timezone.utc) |
There was a problem hiding this comment.
🔴 Score timestamps passed as naive datetimes, such as datetime.now(), are shifted by the host's UTC offset. On the base branch they were read as local time. create_score now stamps naive values with timezone.utc (client.py:1942-1944). The score dict still goes through EventSerializer in the consumer, and serialize_datetime already attached the local offset to naive values. So the comment's reason for the change (the API rejects timestamps with no offset) never applied. Fix: drop the replace(tzinfo=timezone.utc) and keep reading naive values as local time like before, or document the change to UTC as a breaking change.
Why this was flagged
A caller on a host set to UTC+2 calls langfuse.create_score(..., timestamp=datetime.now()) with a naive local datetime. The change at langfuse/_client/client.py:1943-1944 replaces tzinfo with timezone.utc, so 14:00 local is sent as 14:00Z. The score lands two hours in the future (or in the past for negative offsets), and time-based filters, charts and ordering are off by that much. On the base branch the score event went through json.dumps(..., cls=EventSerializer), and serialize_datetime (langfuse/_utils/serializer.py:231-234) treated naive datetimes as local time, giving 14:00+02:00. The new path is the same: request.dict(by_alias=True) keeps the datetime object, and ScoreConsumer._next (langfuse/_task_manager/score_consumer.py) serializes it with EventSerializer. So the offset was never missing, and the replace only changes what naive values mean.
Verification: normal. Conflicts with stated purpose: the test test_naive_score_timestamp_is_sent_as_utc (tests/unit/test_score_consumer.py:361-374) pins naive-as-UTC on purpose. Trigger: a caller passes a naive local datetime (for example datetime.now()) as timestamp= to create_score / span.score() on a host whose timezone is not UTC.
Mechanism, verified:
- client.py:1943-1944 runs
if timestamp is not None and timestamp.tzinfo is None: timestamp = timestamp.replace(tzinfo=timezone.utc). This relabels the wall-clock time as UTC without converting it.
There was a problem hiding this comment.
This change is intentional. The v4 to v5 migration guide (langfuse-docs#4008) says "Naive score timestamps are sent as UTC", and test_naive_score_timestamp_is_sent_as_utc runs under a non-UTC TZ to pin that behavior. The SDK no longer guesses the host's local timezone. Users who want local time can pass an aware datetime.
| if current is None or (item.updated_at or item.start_time) > ( | ||
| current.updated_at or current.start_time | ||
| ): |
There was a problem hiding this comment.
🔴 With the default fields, batch evaluation can score a stale duplicate row of an observation instead of the newest one. _deduplicate_page claims to prefer the most recently updated row, but it compares item.updated_at. That field belongs to the time field group, and DEFAULT_BATCH_EVALUATION_FIELDS ("core,basic,io,metadata") never requests it. So updated_at is always None, the tie falls back to the equal start_time, and whichever row comes first wins. Fix: always request time (for example, append ",time" when it is missing) so the comparison has data, or document that dedup is arbitrary without it.
Why this was flagged
run_batched_evaluation with default fields="core,basic,io,metadata" (client.py:3140, batch_evaluation.py:809) calls observations.get_many with those groups. Per the generated docs (langfuse/api/observations/client.py:65), updatedAt is only returned in the time group. So ObservationV2.updated_at is None on every item. When the events table returns several unmerged rows for one id (the case the docstring at batch_evaluation.py:1409-1410 describes), (item.updated_at or item.start_time) > (current.updated_at or current.start_time) compares two equal start times. It is False, so the first row seen is kept, whatever version it is. The mapper and evaluators then run on that row's input/output. If it is an older partial row, they score incomplete data, and the observation is never re-evaluated. The comment's "preferring the most recently updated" holds only if the caller happens to include time in fields. This code is new in this PR; the base branch read v1 observations, which have no such dedup.
Verification: Trigger: the server returns more than one unmerged row for the same observation id while run_batched_evaluation runs with the default fields. DEFAULT_BATCH_EVALUATION_FIELDS = "core,basic,io,metadata" (batch_evaluation.py:33) leaves out the time group, so updated_at is None. _deduplicate_page (batch_evaluation.py:1417-1419) then compares equal start_time, so the first row seen is kept. This dedup logic is new in this diff, so the base does not have this defect.
There was a problem hiding this comment.
Confirmed and fixed in #1993. Batch-evaluation fetches now always add the time field group, so updatedAt is available when the dedup picks the newest row.
Records origin/main (bf11ec1) as a parent so prepare-v5-release shares main's history. #1986 and #1992 carried this merge, but both were squash-merged (the repo allows only squash merges), which dropped the ancestry. All conflicts resolve to prepare-v5-release; the tree is identical to the previous branch tip. Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com>
* fix: budget propagated trace metadata against the span attribute limit Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * fix: gate experiment run scores on the item span's sampling decision Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * fix(batch-evaluation): skip boundary duplicates on resume and fetch updatedAt for dedup Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * style: format span metadata key helper Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * fix: count experiment items as sampled only after experiment attributes are set Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * fix(batch-evaluation): request updatedAt when no field groups are given Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: langfuse-bot <langfuse-bot@langfuse.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ae1e56467d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| items_to_process = self._deduplicate_page( | ||
| items, skip_ids=previous_page_ids | {resumed_item_id} | ||
| ) |
There was a problem hiding this comment.
When unmerged versions of an observation straddle two pages and the stale version appears first, adding every previous-page ID to skip_ids discards the newer row before _deduplicate_page() can compare updated_at. The first page is therefore evaluated and scored using stale input/output, while the fresh version on the next page is silently ignored; retain enough boundary state to compare the versions and process the newest row instead.
Useful? React with 👍 / 👎.
…#1996) Co-authored-by: Cursor Agent <cursoragent@cursor.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 63fcde76fd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| `api.observations.get_many(trace_id=...)` (`/api/public/v2/observations`, | ||
| cursor-paginated; request field groups with `fields`). A trace's name, | ||
| user, session, tags and input/output are on its root observation. Read | ||
| scores with `api.scores_v3` and experiments with `api.experiments`. |
There was a problem hiding this comment.
Replace api.scores_v3 with api.scores and get_many_v3() with get_many(). The regenerated LangfuseAPI in this commit removes the scores_v3 property, so following this public API documentation raises AttributeError; the newly added calls in tests/e2e/test_batch_evaluation.py, tests/e2e/test_experiments.py, and tests/support/utils.py use the same removed accessor and will fail the e2e shards before polling any scores.
AGENTS.md reference: AGENTS.md:L132-L133
Useful? React with 👍 / 👎.
#1998) * test: read scores via api.scores.get_many in e2e helpers Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> * docs: point Langfuse.api docstring at api.scores.get_many Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e68a2bfb97
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| logger.info("No more items to fetch") | ||
| break | ||
|
|
||
| batch_number += 1 | ||
| total_items_fetched += len(items) |
There was a problem hiding this comment.
When the v2 API returns multiple unmerged rows for one observation, this increments the limit counter for every raw row even though _deduplicate_page() evaluates only one. For example, with max_items=2, a first page containing stale and fresh versions of the same ID reaches the limit after processing just one logical observation and returns early despite other matching observations being available. Track unique items processed/attempted for the limit, fetching further rows as needed.
Useful? React with 👍 / 👎.
* feat(api): update API spec from langfuse/langfuse f0ea0c3 * test: drop the single-score case from the scores create response test Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com> --------- Co-authored-by: langfuse-bot <langfuse-bot@langfuse.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6599a97429
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-authored-by: langfuse-bot <langfuse-bot@langfuse.com>
# Conflicts: # uv.lock Co-authored-by: hassieb.pakzad <hassieb.pakzad@clickhouse.com>
5 participants
What does this PR do?
Releases Python SDK v5. For most users this is a small upgrade. If you trace with
@observe,start_as_current_observation(), or an integration, and you create scores and fetch prompts, your code keeps working as is.Requirements: Python 3.11+, and Langfuse Cloud or self-hosted Langfuse v4.59.0 or later. Self-hosted Langfuse v3 users should stay on SDK v4.
Migration guide: Python v4 → v5 (draft in langfuse/langfuse-docs#4008).
Breaking changes (new in v5)
Ordered by impact.
pip install langfusestill resolves to the latest v4.openai >= 1.0: the pre-1.0 code paths inlangfuse.openaiare removed.langchain-core >= 1: the LangChainCallbackHandlerimports only fromlangchain_core, so thelangchainmeta-package is no longer needed. The import path and usage are unchanged.api.scorestakes request objects (feat(api): update API spec from langfuse/langfuse 21b9b33 #1961, feat(scores)!: send scores via POST /api/public/scores #1987).langfuse.api.scores.create(...)now takesrequest=CreateScoreRequest(...), or a list of them, and returnsCreateScoresResponse.api.legacy.score_v1.deletemoves toapi.scores.delete, andapi.legacy.score_v1.createis gone.create_score(),span.score(), andspan.score_trace()are unchanged.run_batched_evaluation()reads observations only (feat(batch-evaluation)!: evaluate observations via the v2 observations API #1936). It now reads from the v2 observations API:scopeis removed.fetch_trace_fieldsis renamed tofields. The default is"core,basic,io,metadata".ObservationV2.filtermust be a v2 filter JSON array._additional_trace_tagsis removed.ValueError.api.scores.get_many(feat(api): update API spec from langfuse/langfuse 738b05d #1995, test: read scores via api.scores.get_many in e2e helpers and docstring #1998).api.scores_v3is removed:api.scores_v3.get_many_v3(...)becomesapi.scores.get_many(...), with the same arguments andGetScoresV3Response. The reused method name filters byid=, not the v4 v2-readscore_ids=.Previously deprecated, now removed
v4 already deprecated each of these and named the replacement. If you moved to the replacement, nothing changes.
api.legacynamespace anymore.api.trace.get/list,api.sessions.*,api.legacy.observations_v1.*,api.legacy.metrics_v1.*,api.scores.get_many/get_by_id,api.dataset_run_items.*, andapi.datasets.get_run/get_runs/delete_runare removed, together with their types (Trace*,Observation,ObservationsView*,Session*,DatasetRun*,ScoreV1,GetScoresResponse*, and the v1MetricsResponse). The replacements areapi.observations,api.scores.get_many,api.metrics, andapi.experiments.api.trace.delete/delete_multipleandapi.datasets.list/get/createstay.langfuse.modelno longer re-exportsDatasetRun,Observation, orTraceWithFullDetails.set_current_trace_io()andspan.set_trace_io()(feat(batch-evaluation)!: evaluate observations via the v2 observations API #1936). Trace input and output now come from the root observation, so set them withspan.update(input=..., output=...).get_dataset_runs(),get_dataset_run(), anddelete_dataset_run()(feat(client)!: remove get_dataset_run, get_dataset_runs and delete_dataset_run #1933). Their endpoints are not served on Langfuse v4. Useapi.experiments.list()andlist_items()instead.blocked_instrumentation_scopes(feat(client)!: remove blocked_instrumentation_scopes #1931). Composeshould_export_spanwithis_default_export_spaninstead.New features
run_experiment()results haveexperiment_idandexperiment_url, which link to the experiment view.dataset_run_idanddataset_run_urlremain as deprecated aliases. Dataset-backed runs get a stable id derived from the project, dataset, and run name, so re-runs with the same run name group into one experiment.langfuse.experiment.item.version.api.scores.create()accepts a list ofCreateScoreRequestobjects.Under the hood
None of these need code changes.
POST /api/public/scoresinstead of being sent as ingestion events (feat(scores)!: send scores via POST /api/public/scores #1987).flush_at,flush_interval, and the batch byte limit still apply. Network errors, 429, and 5xx are retried with backoff. Other 4xx responses are logged and not retried. Rejected items in a 207 partial response are logged. Naive score timestamps are sent as UTC. Scores are dropped when Langfuse's trace-id ratio sampler drops their trace.otel_compression="none"orLANGFUSE_OTEL_COMPRESSION=none.x-langfuse-ingestion-version: 4(feat(otel)!: send x-langfuse-ingestion-version: 4 by default #1930), so Langfuse v4 processes them in real time. You can override it withadditional_headers.run_experiment()no longer callsPOST /api/public/dataset-run-items. Experiment and item metadata are propagated as single JSON attributes.Nonevalues are skipped. Media data URIs inside metadata are still extracted. The 200-character limit on propagated metadata applies to the encoded value.gen_ai.prefix (fix(tracing)!: require the gen_ai. prefix in the default span filter #1929), so lookalike attributes such asgen_ai_customno longer export unrelated spans.updatedAtis always fetched (fix: address v5 release review findings #1993).usage_details(feat(scores)!: send scores via POST /api/public/scores #1987).738b05d(feat(api): update API spec from langfuse/langfuse 21b9b33 #1961, feat(api): update API spec from langfuse/langfuse 5beee5d #1962, feat(api): update API spec from langfuse/langfuse 99da25d #1990, feat(api): update API spec from langfuse/langfuse 738b05d #1995).Type of change
Verification
CI on this branch runs ruff, mypy, unit tests on Python 3.11–3.15, both e2e shards, and the live-provider suite. Every listed change landed through its own reviewed PR into
prepare-v5-release.Checklist
code_review.md..env.templateif needed (docs: upgrade guides and changelog for Python SDK v5 and JS/TS SDK v6 langfuse-docs#4008).