docs: notes from a long-running client

docs/advanced/long-running.md

@@ -38,6 +38,95 @@ connect-time `Self` call finds a lost authorization, `OnAuthLost` is called firs
 `OnSelfError` if it returned `nil`. Use `telegram.IsAuthLost(err)` to check errors
 yourself.
 
+### In practice
+
+- **What it looks like.** Terminating a session from the official app (Settings →
+  Devices) shows up as `AUTH_KEY_UNREGISTERED`, not `SESSION_REVOKED`. That is the
+  `reason` to expect on `gotd.client.auth.lost`.
+- **When it's noticed.** Detection rides on requests: the loss is found by the next
+  request that needs authorization. A client that is idle, or only receives updates,
+  notices it on its next call.
+- **Several accounts in one process.** "Stop one account, keep the others" holds only if
+  each `Run` is supervised on its own. If all accounts' `Run` calls share one `errgroup`,
+  the `*AuthLostError` of one cancels the group and stops every account. Either run each
+  account under its own supervisor, or return `nil` from `OnAuthLost` (log, count and
+  alert) and stop that account yourself.
+- **Testing it.** Use a spare account, never one in use: a terminated session can't be
+  restored, and whatever happens until a new login is lost. Give the test client a
+  distinctive `DeviceConfig.DeviceModel` so its session is easy to find in the app's
+  device list, then terminate it there while the client sends a request every few
+  seconds.
+
+### Banned or deleted accounts
+
+`telegram.AuthLostKindOf(err)`, or `Kind()` of `*AuthLostError`, tells why authorization
+was lost. Use it to decide whether signing in again can help:
+
+| Kind | Error | Meaning |
+| --- | --- | --- |
+| `AuthLostBanned` | `USER_DEACTIVATED_BAN` | The account is banned by Telegram. |
+| `AuthLostDeleted` | `USER_DEACTIVATED` | The account is deleted. |
+| `AuthLostRevoked` | `SESSION_REVOKED`, `AUTH_KEY_DROP_*` | The session was terminated, or the server asked to drop the key. |
+| `AuthLostUnregistered` | `AUTH_KEY_UNREGISTERED` | The key isn't authorized: usually a terminated session. |
+| `AuthLostExpired` | `SESSION_EXPIRED` | The session expired. |
+| `AuthLostKeyInvalid` | `AUTH_KEY_INVALID`, `AUTH_KEY_DUPLICATED` | The key is invalid or used by another connection. |
+
+```go
+OnAuthLost: func(ctx context.Context, err error) error {
+	switch telegram.AuthLostKindOf(err) {
+	case telegram.AuthLostBanned, telegram.AuthLostDeleted:
+		markDead(account) // signing in again won't help
+	default:
+		askToSignIn(account)
+	}
+	return err
+},
+```
+
+- **What clients do.** TDLib and the official apps treat any unauthorized (401) error on
+  the home DC as a logout, and only TDLib adds a ban message. gotd does the same: the kind
+  is logged and added to `gotd.client.auth.lost`, and bans and deletions are logged at
+  error level.
+- **`AUTH_KEY_DROP_*` notifications.** The server can ask a client to drop its key with
+  an `updateServiceNotification` of this type, e.g. after a key leak. Telegram Desktop
+  logs out on it, so gotd reports it via `OnAuthLost` as a `*tgerr.Error` with code 401
+  and the notification type and text (`telegram.AuthKeyDropError`). Return `nil` to keep
+  running: if the session still works, a later loss is reported again.
+- **The kind isn't always known.** A ban or deletion can drop the auth key on the server.
+  The server then doesn't know the key at all (transport error `-404`), gotd creates a
+  new one, and the next request fails with `AUTH_KEY_UNREGISTERED`. The kind is then
+  `AuthLostUnregistered`, not `AuthLostBanned`.
+- **Signing in with a banned number** fails with `PHONE_NUMBER_BANNED`. The client logs it
+  at error level and counts it in `gotd.client.account.restricted`.
+  `telegram.IsAccountBanned(err)` matches it and `USER_DEACTIVATED_BAN`.
+
+## Frozen accounts
+
+Telegram can freeze an account instead of banning it. A frozen account stays signed in
+and can read, but most methods fail with `FROZEN_METHOD_INVALID` (code 420, not a flood
+wait) or `FROZEN_PARTICIPANT_MISSING`, until it is deleted at the end of the freeze or
+an appeal succeeds. Authorization isn't lost, so `OnAuthLost` isn't called.
+
+The freeze state comes from app config (`freeze_since_date`, `freeze_until_date`,
+`freeze_appeal_url`). After a frozen error, the client fetches app config, at most once
+an hour like Telegram Desktop, and calls `OnFrozen` if the account is frozen:
+
+```go
+client := telegram.NewClient(appID, appHash, telegram.Options{
+	OnFrozen: func(ctx context.Context, state telegram.FreezeState) {
+		alert(account, state.Since, state.Until, state.AppealURL)
+	},
+})
+```
+
+`OnFrozen` runs on its own goroutine with `Run`'s context. It is only called after a
+request fails, so check `client.FreezeState(ctx)` at start too. Use `telegram.IsFrozen(err)`
+to skip retries of refused requests. Frozen errors are counted in
+`gotd.client.account.restricted`.
+
+See [examples/account-banned](https://github.com/gotd/td/tree/main/examples/account-banned)
+for a complete client.
+
 ## Connection events
 
 `OnConnectionState` reports the primary connection state (connecting, ready,

docs/advanced/observability.md

@@ -61,6 +61,39 @@ abnormal ones (see [Logs](#logs)). Raise the level when you need to see more.
 Metric attributes never contain user, chat or channel IDs, file locations or message
 content, so the number of series stays bounded. IDs only appear in logs.
 
+### Several clients in one process
+
+gotd doesn't label metrics by client. Two clients (e.g. two accounts) sharing one
+`MeterProvider` add up into the same counters, and their gauges such as
+`gotd.client.pools` overwrite each other. Give each client its own provider that adds an
+instrumentation-scope attribute to every meter gotd creates:
+
+```go
+import (
+	"go.opentelemetry.io/otel/attribute"
+	"go.opentelemetry.io/otel/metric"
+)
+
+// scoped adds attrs to the instrumentation scope of every meter.
+type scoped struct {
+	metric.MeterProvider
+	attrs []attribute.KeyValue
+}
+
+func (p scoped) Meter(name string, opts ...metric.MeterOption) metric.Meter {
+	return p.MeterProvider.Meter(name, append(opts, metric.WithInstrumentationAttributes(p.attrs...))...)
+}
+
+mp := scoped{MeterProvider: shared, attrs: []attribute.KeyValue{attribute.String("account", name)}}
+// Use mp in telegram.Options and updates.Config of that account.
+```
+
+`client.Downloader()` uses the client's provider, so its metrics are separated too. A
+`Downloader` you share between clients is not.
+
+With Prometheus' OTLP receiver, set `otlp.promote_scope_metadata: true` to turn the
+scope attribute into a label (`otel_scope_account` above).
+
 ## Metrics
 
 ### Client (`telegram.Options.MeterProvider`)
@@ -72,7 +105,8 @@ content, so the number of series stays bounded. IDs only appear in logs.
 | `gotd.client.connection.reconnects` | counter | `dc` | Reconnects of the primary connection. |
 | `gotd.client.connection.restarts` | counter | `reason` (`timeout`, `migration`) | Restarts requested by the client itself; each is also counted in `connection.reconnects`. |
 | `gotd.client.migrations` | counter | `dc` (target) | Home DC changes (`*_MIGRATE`). |
-| `gotd.client.auth.lost` | counter | `reason` | Authorized → lost transitions (see `OnAuthLost`). |
+| `gotd.client.auth.lost` | counter | `reason`, `kind` | Authorized → lost transitions (see `OnAuthLost`). `reason` is the error type, `kind` is `banned`, `deleted`, `revoked`, `unregistered`, `expired` or `key_invalid`. |
+| `gotd.client.account.restricted` | counter | `reason` | Errors telling that the account is restricted: `FROZEN_METHOD_INVALID`, `FROZEN_PARTICIPANT_MISSING`, `PHONE_NUMBER_BANNED`. |
 | `gotd.client.request.timeouts` | counter | `result` (`resent`, `failed`), `method` | Requests that missed their `RequestTimeout` deadline. |
 | `gotd.client.pool.created` | counter | `dc`, `kind` (`file`, `migrate`), `result` (`ok`, `error`, `canceled`) | Connection pools created for other DCs. |
 | `gotd.client.pool.create.duration` | histogram, s | `kind`, `result` | Time to create a pool (connect and authorization transfer). |
@@ -135,6 +169,11 @@ defer reg.Unregister()
 
 ### What to watch
 
+- **Counters versus state.** `connection.state` counts transitions. The first `ready`
+  after start is a counter's first value, which PromQL's `increase()` doesn't show, and
+  an idle client looks the same as a dead one. To show disconnects, chart increases of
+  `state="disconnected"` and `connection.dead`. "Connected now" can't be read from these
+  counters.
 - **The client is healthy:**
   - `connection.state{state="ready"}` grows only on reconnects;
   - `connection.dead` and `request.timeouts` stay near zero;
@@ -143,7 +182,9 @@ defer reg.Unregister()
   - `difference.requests{result="too_long"}` and `channel.too_long` are rare;
   - `gaps{action="detected"}` is followed by a resolution;
   - `before_run{action="dropped"}` is zero, unless you chose `BeforeRunDrop`.
-- **Authorization:** any increase of `auth.lost` needs attention.
+- **Authorization:** any increase of `auth.lost` needs attention; `kind="banned"` or
+  `"deleted"` means the account is gone. Any `account.restricted` means a frozen
+  account or a banned number.
 - **Downloads keep up:**
   - `downloads{result="error"}`, `hash.mismatches` and `retries.exhausted` stay near zero;
   - `scheduler.window` recovers after `scheduler.backoffs`.
@@ -156,8 +197,9 @@ IDs (channel, DC, pts) appear here, never in metrics.
 | Level | Component | Events |
 | --- | --- | --- |
 | Debug | client | Connection state, dead data and CDN connections, pool created, `FILE_MIGRATE` hop, restart requested |
-| Info | client | Home DC migration (from, to) |
-| Warn | client | Dead primary connection, pool creation failed, pool evicted after `AUTH_KEY_UNREGISTERED`, lost authorization, request timeout |
+| Info | client | Home DC migration (from, to), popup service notifications (type, text) |
+| Warn | client | Dead primary connection, pool creation failed, pool evicted after `AUTH_KEY_UNREGISTERED`, lost authorization, request timeout, `AUTH_KEY_DROP_*` notification, frozen account (since, until, appeal URL) |
+| Error | client | Lost authorization of a banned or deleted account, `PHONE_NUMBER_BANNED` |
 | Debug | updates | Each difference request with result and duration, channel tracked, update received before `Run` |
 | Info | updates | Channel gap (channel, local and remote pts, top message, number of messages), replay of updates buffered before `Run` |
 | Warn | updates | `updates.differenceTooLong`, channel pts that did not advance, failed `TrackChannel` |
@@ -175,6 +217,7 @@ Use hooks to act on events. Use metrics and logs to watch them.
 | `OnConnectionStateEvent func(ConnectionStateEvent)` | The primary connection state changes; `DC`, `State`. Prefer it to `OnConnectionState`. |
 | `OnConnDead func(ConnDeadEvent)` | Any connection is dead; `DC`, `Kind`, `Err`. Prefer it to `OnDead`. |
 | `OnAuthLost func(ctx, err) error` | An authorized session is lost. Runs on its own goroutine with `Run`'s context. Return an error to stop `Run` with `*AuthLostError`. See [Long-running clients](long-running). |
+| `OnFrozen func(ctx, FreezeState)` | A request failed because the account is frozen, and app config confirms it. Checked at most once an hour. See [Frozen accounts](long-running#frozen-accounts). |
 | `OnSelfError` / `OnSelfSuccess` | The `Self` call after connecting fails or succeeds. |
 | `OnTransfer` | Authorization is transferred to another DC. |
 | `RequestTimeout func(bin.Encoder) time.Duration` | Per-request response deadline; one resend after a reconnect. Return 0 for requests that must not be sent twice. |

docs/helpers/updates-recovery.md

@@ -66,6 +66,60 @@ sequence and everything else to the common one. It is opt-in, mirrors TDLib's
 behavior, and is a no-op before `gaps.Run` — so install it alongside `UpdateHook`
 as shown above. See [gotd/td#1782](https://github.com/gotd/td/pull/1782).
 
+## Your own sends
+
+Telegram pushes the result of `messages.sendMessage`, `messages.sendMedia`,
+`messages.editMessage` and similar calls to the account's **other** sessions, not to the
+one that made the call: the session only gets it as the call's result. Without
+`updhook.UpdateHook` (see [Wiring it up](#wiring-it-up)) your handler sees your own
+message only after a later update reveals the pts gap and `getDifference` fetches it,
+which can take minutes.
+
+With `UpdateHook` the pts stays in sync, but one result still loses data. A text sent
+with `messages.sendMessage` comes back as `updateShortSentMessage`, which has the new
+message ID, date and pts, but no text and no peer. The engine can't rebuild it, so the
+handler gets a `tg.MessageEmpty` with that ID. If you record your own messages, build the
+message from the request in a middleware that sees both, as Telegram Web A does for its
+local messages:
+
+```go
+func sentHook(gaps *updates.Manager, self int64) telegram.Middleware {
+	return telegram.MiddlewareFunc(func(next tg.Invoker) telegram.InvokeFunc {
+		return func(ctx context.Context, input bin.Encoder, output bin.Decoder) error {
+			if err := next.Invoke(ctx, input, output); err != nil {
+				return err
+			}
+			box, ok := output.(*tg.UpdatesBox)
+			if !ok {
+				return nil
+			}
+			u := box.Updates
+			if s, ok := u.(*tg.UpdateShortSentMessage); ok {
+				if req, ok := input.(*tg.MessagesSendMessageRequest); ok {
+					u = &tg.UpdateShort{Date: s.Date, Update: &tg.UpdateNewMessage{
+						Message: &tg.Message{
+							ID: s.ID, Date: s.Date, Out: true,
+							FromID:   &tg.PeerUser{UserID: self},
+							PeerID:   peerOf(req.Peer), // your InputPeer → Peer mapping
+							Message:  req.Message,
+							Entities: s.Entities, // the server's entities, e.g. links
+						},
+						Pts: s.Pts, PtsCount: s.PtsCount,
+					}}
+				}
+			}
+			// The call succeeded: don't fail it because handling did; recovery fills
+			// the gap later.
+			_ = gaps.Handle(ctx, u)
+			return nil
+		}
+	})
+}
+```
+
+Use it instead of `updhook.UpdateHook(gaps.Handle)`, not next to it, so results aren't
+handled twice. Media sends return full `updates`, so only plain text needs this.
+
 ## Running the engine
 
 After authenticating, start the engine with your user ID. It blocks, keeping the client
@@ -130,6 +184,20 @@ gaps.Run(ctx, api, self.ID, updates.AuthOptions{
 })
 ```
 
+### "Failed to load channel state"
+
+On start the engine loads each channel with a stored pts, and needs its access hash
+from `ChannelAccessHasher` to call `updates.getChannelDifference`. If the pts is stored
+but the hash isn't, it logs "Failed to load channel state, skipping channel" (or calls
+`OnLoadChannelStateFailed`) and skips the channel until its next update arrives, so
+updates missed while offline are not fetched.
+
+Usually the hash store is behind the state store: another database, a cache that was
+cleared, or a channel the account left or that was deleted. Fix it by giving the hasher
+the hash (any store you have works, e.g. your own archive of chats), or by calling
+`TrackChannel` with the dialog's pts and hash as above. The stored state of a channel
+that is gone stays, so the same warning repeats on every start.
+
 ## Updates before `Run`
 
 Updates can arrive before the manager's `Run` has loaded the state, e.g. while the
@@ -160,6 +228,24 @@ updates.Config{
 }
 ```
 
+## Channels that aren't pushed
+
+Telegram doesn't push updates of every channel to every session. For such channels the
+engine only learns of new posts from its idle `getChannelDifference`, which runs
+15 minutes after the channel's last update and then every 15 minutes while none arrive
+(difference trigger `channel-idle-timeout` in the [metrics](../advanced/observability)). Their messages therefore arrive 0–15
+minutes late, about 7 minutes on average.
+
+The official clients do the same for channels that aren't open, and poll often only the
+open ones, at the `timeout` returned by the last `channelDifference`
+(Telegram Desktop `Updates::addActiveChat`, TDLib `channel_get_difference_timeout_` for
+open chats). With gotd, poll a channel you need sooner yourself, e.g. with
+`messages.getHistory` from its last known message.
+
+To see whether you're affected, compare update lag per channel with
+`gotd.updates.difference.triggers{reason="channel-idle-timeout"}`: a median lag of
+several minutes for a few channels matches the idle poll.
+
 The [`userbot`](https://github.com/gotd/td/tree/main/examples/userbot) example uses
 persistent state; the [`updates`](https://github.com/gotd/td/tree/main/examples/updates)
 example is a minimal in-memory version.