Skip to content

Make Appwrite source report resilient to per-resource failures #199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
premtsd-code wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Make Appwrite source report resilient to per-resource failures #199

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
318a867
Make Appwrite source report resilient to per-resource failures
premtsd-code Jun 23, 2026
af39ae5
Skip only unsupported (404) resources, fail fast on other errors
premtsd-code Jun 23, 2026
5f8deb7
Fail fast on report errors except aggregated missing scopes
premtsd-code Jun 23, 2026
a712360
Trim report() comments
premtsd-code Jun 23, 2026
ec09e6d
Fail fast on invalid credentials from optional reporters
premtsd-code Jun 23, 2026
bddc529
Fail fast with a clear message on invalid credentials
premtsd-code Jun 23, 2026
c2febe0
Pass $report by reference so reporter writes are kept
premtsd-code Jun 23, 2026
6335a88
Stop suppressing Appwrite report failures
premtsd-code Jun 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 61 additions & 46 deletions src/Migration/Sources/Appwrite.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,18 +280,8 @@ public function report(array $resources = [], array $resourceIds = []): array
$resources = $this->getSupportedResources();
}

// Reachability gate: unauthenticated, so a failure is a wrong/unreachable endpoint.
try {
$this->reportAuth($resources, $report, $resourceIds);
$this->reportDatabases($resources, $report, $resourceIds);
$this->reportStorage($resources, $report, $resourceIds);
$this->reportFunctions($resources, $report, $resourceIds);
$this->reportMessaging($resources, $report, $resourceIds);
$this->reportSites($resources, $report, $resourceIds);
$this->reportIntegrations($resources, $report, $resourceIds);
$this->reportBackups($resources, $report, $resourceIds);
$this->reportProjects($resources, $report, $resourceIds);
$this->reportDomains($resources, $report, $resourceIds);

$report['version'] = $this->call(
'GET',
'/health/version',
Expand All @@ -301,18 +291,67 @@ public function report(array $resources = [], array $resourceIds = []): array
]
)['version'];
} catch (\Throwable $e) {
if ($e->getCode() === 403) {
throw new \Exception('Missing required scopes.', $e->getCode(), $e);
} else {
throw new \Exception($e->getMessage(), $e->getCode(), $e);
throw new \Exception('Unable to reach the migration source endpoint.', $e->getCode(), $e);
}

// First-class callables (not fn()) so $report is passed by reference at call time;
// arrow functions would capture a copy and discard each reporter's writes.
$reporters = [
Transfer::GROUP_AUTH => $this->reportAuth(...),
Transfer::GROUP_DATABASES => $this->reportDatabases(...),
Transfer::GROUP_STORAGE => $this->reportStorage(...),
Transfer::GROUP_FUNCTIONS => $this->reportFunctions(...),
Transfer::GROUP_MESSAGING => $this->reportMessaging(...),
Transfer::GROUP_SITES => $this->reportSites(...),
Transfer::GROUP_INTEGRATIONS => $this->reportIntegrations(...),
Transfer::GROUP_BACKUPS => $this->reportBackups(...),
Transfer::GROUP_PROJECTS => $this->reportProjects(...),
Transfer::GROUP_DOMAINS => $this->reportDomains(...),
];

$missingScopes = [];

foreach ($reporters as $group => $reporter) {
try {
$reporter($resources, $report, $resourceIds);
} catch (\Throwable $e) {
$code = $e->getCode();

if ($this->isMissingScopeError($e)) {
$missingScopes[] = $group;
continue;
}

if ($code === Exception::CODE_UNAUTHORIZED) {
throw new \Exception('Invalid credentials for the migration source.', $code, $e);
}

throw $e;
Comment thread
greptile-apps[bot] marked this conversation as resolved.
}
}

if (!empty($missingScopes)) {
throw new \Exception('Missing required scopes for: ' . \implode(', ', $missingScopes) . '.', Exception::CODE_FORBIDDEN);
}

$this->previousReport = $report;

return $report;
}

private function isMissingScopeError(\Throwable $e): bool
{
if (!\in_array($e->getCode(), [Exception::CODE_UNAUTHORIZED, Exception::CODE_FORBIDDEN], true)) {
return false;
}

$message = $e->getMessage();

return \str_contains($message, 'general_unauthorized_scope')
|| \str_contains($message, 'missing scopes')
|| \str_contains($message, 'required scopes');
}

/**
* @param array $resources
* @param array $report
Expand Down Expand Up @@ -1649,11 +1688,7 @@ protected function reportBackups(array $resources, array &$report, array $resour
private function reportDomains(array $resources, array &$report, array $resourceIds = []): void
{
if (\in_array(Resource::TYPE_RULE, $resources)) {
try {
$report[Resource::TYPE_RULE] = $this->proxy->listRules([Query::limit(1)])->total;
} catch (\Throwable) {
$report[Resource::TYPE_RULE] = 0;
}
$report[Resource::TYPE_RULE] = $this->proxy->listRules([Query::limit(1)])->total;

@greptile-apps greptile-apps Bot Jun 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Optional endpoints now abort

This optional reporter no longer catches non-auth endpoint failures. On an older Appwrite source, or when domain rules are disabled, listRules() can return 404 and the outer report loop rethrows it, aborting discovery for every later group. The intended behavior for optional reporters is to keep returning 0 for unsupported resources while still surfacing credential and scope errors.

}
}

Expand All @@ -1665,11 +1700,7 @@ private function reportProjects(array $resources, array &$report, array $resourc
resourceIds: $resourceIds,
limit: 1
);
try {
$report[Resource::TYPE_PROJECT_VARIABLE] = $this->project->listVariables($variableQueries)->total;
} catch (\Throwable) {
$report[Resource::TYPE_PROJECT_VARIABLE] = 0;
}
$report[Resource::TYPE_PROJECT_VARIABLE] = $this->project->listVariables($variableQueries)->total;

@greptile-apps greptile-apps Bot Jun 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 One project failure hides siblings

listVariables() now runs without the optional-resource catch and it executes before the singleton project counts are written. If variables are unavailable on the source and return 404, reportProjects() exits before setting project-protocols, project-labels, project-services, or email templates, even though those counts do not depend on variables. Isolate the optional endpoint failure so unrelated project resources can still be reported.

}

if (\in_array(Resource::TYPE_PROJECT_PROTOCOLS, $resources)) {
Expand All @@ -1688,12 +1719,8 @@ private function reportProjects(array $resources, array &$report, array $resourc
}

if (\in_array(Resource::TYPE_PROJECT_EMAIL_TEMPLATE, $resources)) {
try {
// total:true returns the real count without fetching every row.
$report[Resource::TYPE_PROJECT_EMAIL_TEMPLATE] = $this->project->listEmailTemplates([Query::limit(1)], total: true)->total;
} catch (\Throwable) {
$report[Resource::TYPE_PROJECT_EMAIL_TEMPLATE] = 0;
}
// total:true returns the real count without fetching every row.
$report[Resource::TYPE_PROJECT_EMAIL_TEMPLATE] = $this->project->listEmailTemplates([Query::limit(1)], total: true)->total;
}
}

Expand Down Expand Up @@ -2845,11 +2872,7 @@ private function reportIntegrations(array $resources, array &$report, array $res
resourceIds: $resourceIds,
limit: 1
);
try {
$report[Resource::TYPE_PLATFORM] = $this->project->listPlatforms($platformQueries)->total;
} catch (\Throwable) {
$report[Resource::TYPE_PLATFORM] = 0;
}
$report[Resource::TYPE_PLATFORM] = $this->project->listPlatforms($platformQueries)->total;

@greptile-apps greptile-apps Bot Jun 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Integration counts lose isolation

The integration reporter now lets listPlatforms() throw directly. If platforms are unsupported or disabled on the source, this aborts the whole integrations group before API keys, webhooks, and the SMTP singleton are counted. Keep each optional integration endpoint isolated so one unsupported resource returns 0 without hiding the rest of the group.

}

if (\in_array(Resource::TYPE_API_KEY, $resources)) {
Expand All @@ -2858,11 +2881,7 @@ private function reportIntegrations(array $resources, array &$report, array $res
resourceIds: $resourceIds,
limit: 1
);
try {
$report[Resource::TYPE_API_KEY] = $this->project->listKeys($keyQueries)->total;
} catch (\Throwable) {
$report[Resource::TYPE_API_KEY] = 0;
}
$report[Resource::TYPE_API_KEY] = $this->project->listKeys($keyQueries)->total;
}

if (\in_array(Resource::TYPE_WEBHOOK, $resources)) {
Expand All @@ -2871,11 +2890,7 @@ private function reportIntegrations(array $resources, array &$report, array $res
resourceIds: $resourceIds,
limit: 1
);
try {
$report[Resource::TYPE_WEBHOOK] = $this->webhooks->list($webhookQueries)->total;
} catch (\Throwable) {
$report[Resource::TYPE_WEBHOOK] = 0;
}
$report[Resource::TYPE_WEBHOOK] = $this->webhooks->list($webhookQueries)->total;
}

if (\in_array(Resource::TYPE_SMTP, $resources)) {
Expand Down
Loading