You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
docs: add self-hosted upgrade and rollback guide - #8890
Adds a new page, "Upgrading and rolling back", to the self hosting docs. It covers:
what to do before an upgrade: release notes, the current Helm revision and a database backup
the upgrade command, and how migrations run (the migrate-db init container by default, and where to read its logs)
verifying the upgrade
why helm rollback does not revert the database, and a restore procedure that worked in testing
a troubleshooting table
the removal of the bundled InfluxDB in chart 0.85.0
How did you test this code?
I ran every command on a local kind cluster, upgrading chart 0.80.0 (Flagsmith 2.222.0) to chart 0.85.0 (Flagsmith
2.286.0) with default values and the PostgreSQL instance deployed by the chart. I took a pg_dump backup, upgraded (91
migrations applied), rolled back with helm rollback (Flagsmith 2.222.0 then failed with a missing table error), and
restored the backup, after which 2.222.0 worked again with the original data.
Docs checks: I ran npx prettier --check on the page and npm run build. [Edit this line to match what you ran, and
mention any unrelated build warnings.]
Full write up with logs and screenshots:
Not tested: custom values, an external or managed database, the jobs.migrateDb Job, the analytics migration job,
other version pairs, and downtime during the upgrade. Happy to rerun any of these.
Questions for reviewers: I placed the page next to the Edge Proxy page with sidebar_position: 4, and put the diagram
in docs/diagrams. Happy to move either.
The change adds a self-hosted Helm upgrade and rollback guide. It describes database backup, upgrade settings, migration checks, and deployment verification. It explains that chart rollback does not reverse database migrations and provides a tested database restore procedure. The guide also lists troubleshooting steps, chart 0.85.0 analytics-storage options, and the configurations and scenarios that were not tested.
Revise the guide before merging: following its recovery steps can delete a database before a usable restore is established, while upgrade instructions can lose configuration or interrupt requests. Analytics migration also needs an explicit reachable InfluxDB source.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Tip
🎯 Customize your risk assessments
Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.
✅ Automatic approvals, with your rules
Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 12
Review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: ASSERTIVE
Plan: Advanced
Run ID: 21fcfeb9-050b-457d-a11f-3b57776d70c2
Commits
Reviewing files that changed from the base of the PR and between 5051ac2 and a96256c.
Files ignored due to path filters: 1
docs/docs/diagrams/upgrade-rollback-restore.svg is excluded by !**/*.svg
Files selected for processing: 1
docs/docs/deployment-self-hosting/upgrading.md
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.
python3 - <<'PY'import urllib.request, tarfile, iofor ver in ['0.80.0','0.85.0']: url=f'https://github.com/Flagsmith/flagsmith-charts/releases/download/flagsmith-{ver}/flagsmith-{ver}.tgz' print('ARCHIVE',url) try: data=urllib.request.urlopen(url,timeout=20).read() chart=tarfile.open(fileobj=io.BytesIO(data),mode='r:gz') names=chart.getnames() print('POSTGRES FILES',*[x for x in names if 'postgres' in x.lower()][:80],sep='\n') for m in chart.getmembers(): if m.name in ['flagsmith/Chart.yaml','flagsmith/Chart.lock','flagsmith/values.yaml'] or ('postgresql' in m.name and (m.name.endswith('statefulset.yaml') or m.name.endswith('values.yaml'))): if m.isfile(): print('\nFILE',m.name) for i,l in enumerate(chart.extractfile(m).read().decode().splitlines(),1): if m.name.endswith('values.yaml'): if any(x in l.lower() for x in ['postgresql','/tmp','resources','persistence','volume','ephemeral']): print(f'{i}: {l}') else: print(f'{i}: {l}') except Exception as e: print(type(e).__name__,str(e))PY
Repository: Flagsmith/flagsmith
Length of output: 42236
Stream the PostgreSQL dump outside the pod.
The command first writes /tmp/backup.dump inside the PostgreSQL container. A dump larger than the container's writable layer can fail with insufficient space, and node local-storage pressure can evict the pod. Stream pg_dump to the local machine, then update the validation and restore commands to use that local file.
pg_restore -l lists archive entries but does not verify table data. A corrupt backup can pass this check, then fail after dropdb deletes the current database. Restore the backup into a separate database and verify its contents before deleting the current database.
Helm upgrade --values missing file error open no such file official documentation source
Result:
Sources consulted:
- helm upgrade | Helm — https://helm.sh/docs/helm/helm_upgrade/
- os package - os - Go Packages — https://pkg.go.dev/os?GOOS=linux
- helm upgrade | Helm — https://docs.helm.sh/docs/helm/helm_upgrade/
- Relative filesystem schema references in `values.schema.json` broken · Issue #10481 · helm/helm · GitHub — GitHub issue 10481 in helm/helm (link omitted to avoid creating a cross-reference)
- Chart Development Tips and Tricks | Helm — https://helm.sh/docs/v2/developing_charts/charts_tips_and_tricks/
- …and 8 more
Remove the mandatory values file from the default upgrade command.
The basic installation uses chart defaults and creates no values.yaml. The documented command therefore fails for that installation unless the operator creates a file. Use --reuse-values, and document -f values.yaml only for installations that have a values file.
Suggested fix
--- "a/docs/docs/deployment-self-hosting/upgrading.md"+++ "b/docs/docs/deployment-self-hosting/upgrading.md"@@ -56,7 +56,7 @@
## Upgrade with Helm
```bash
-helm upgrade -n flagsmith flagsmith flagsmith/flagsmith --version <chart-version> -f values.yaml --wait --timeout 20m+helm upgrade -n flagsmith flagsmith flagsmith/flagsmith --version <chart-version> --reuse-values --wait --timeout 20m
```
Pass the same values file you installed with. Without `-f` or `--reuse-values`, Helm applies the chart defaults and
Committable suggestion
IMPORTANT
Review the replacement and indentation, then test before committing.
Quiesce the old API before applying incompatible migrations.
The default chart uses a rolling update, so an old ready API pod can serve requests while the new pod runs migrate. In Flagsmith 2.222.0, the features list queries FeatureFlagCodeReferencesScan, which migration 0003 deletes. Document a Recreate strategy or another quiescing procedure before this upgrade.
Preserve all Helm value overrides during upgrades.
If the installation used additional -f files or --set, the documented command supplies only values.yaml. Helm then omits the other release overrides. Use every values file and command-line override again, or add --reuse-values, so settings such as databaseExternal and taskProcessor.enabled are not replaced by chart defaults.
Helm --wait does not guarantee that every replica is ready. It accepts a Deployment when the minimum number of ready replicas is available, based on Desired - maxUnavailable. State this limitation so operators do not treat a partially ready rollout as fully healthy.
Suggested documentation fix
--- "a/docs/docs/deployment-self-hosting/upgrading.md"+++ "b/docs/docs/deployment-self-hosting/upgrading.md"@@ -62,7 +62,9 @@
Pass the same values file you installed with. Without `-f` or `--reuse-values`, Helm applies the chart defaults and
discards values you set earlier. The test used default values only, so this behaviour was not tested here.
-`--wait` prints nothing until every pod is ready. Downloading images can take several minutes on a slow connection. In+`--wait` prints nothing until Helm's readiness checks succeed. For Deployments, Helm waits for the minimum number of+ready replicas (`Desired - maxUnavailable`), not necessarily every replica. Check all replicas separately after the+upgrade. Downloading images can take several minutes on a slow connection. In
the test, a first install with a 10 minute timeout failed because the PostgreSQL image took more than five minutes to
download, even though the pods were healthy shortly afterwards.
Committable suggestion
IMPORTANT
Review the replacement and indentation, then test before committing.
Suggested change
`--wait` prints nothing until every pod is ready. Downloading images can take several minutes on a slow connection. In
`--wait` prints nothing until Helm's readiness checks succeed. For Deployments, Helm waits for the minimum number of
ready replicas (`Desired - maxUnavailable`), not necessarily every replica. Check all replicas separately after the
upgrade. Downloading images can take several minutes on a slow connection. In
Sources consulted:
- PostgreSQL: Documentation: 18: DROP DATABASE — https://www.postgresql.org/docs/current/sql-dropdatabase.html
- PostgreSQL: Documentation: 18: pg_restore — https://www.postgresql.org/docs/18/app-pgrestore.html
- PostgreSQL: Documentation: 18: 22.5. Destroying a Database — https://www.postgresql.org/docs/current/manage-ag-dropdb.html
- PostgreSQL: Documentation: 17: pg_restore — https://www.postgresql.org/docs/17/app-pgrestore.html
- pg_restore ## Name pg_restore — restore a Postgre — https://www.postgresql.org/files/documentation/pdf/7.4/postgresql-7.4.2-A4.pdf
- …and 12 more
Copy the backup back into the pod before deleting the database.
If the PostgreSQL pod has been replaced, /tmp/backup.dump may no longer exist. The documented commands then delete the database before pg_restore fails. Copy ./backup.dump into the pod and verify it before running dropdb.
Sources consulted:
- GitHub - Flagsmith/flagsmith-charts: Helm Charts for Flagsmith. Feature flagging and remote config service. Host yourself or use our hosted version. · GitHub — https://github.com/Flagsmith/flagsmith-charts
- flagsmith 0.2.9 · flagsmith/flagsmith — https://artifacthub.io/packages/helm/flagsmith/flagsmith/0.2.9
- flagsmith/docker-compose.yml at main · Flagsmith/flagsmith · GitHub — https://github.com/Flagsmith/flagsmith/blob/main/docker-compose.yml
- flagsmith/docker-compose.pgpool.yml at main · Flagsmith/flagsmith · GitHub — https://github.com/Flagsmith/flagsmith/blob/main/docker-compose.pgpool.yml
- …and 25 more
Script executed:
nl -ba docs/docs/deployment-self-hosting/hosting-guides/kubernetes-openshift.md | sed -n '200,245p'
git status --short
Repository: Flagsmith/flagsmith
Length of output: 1978
Preserve the application role during restore.
If the API uses a non-postgres role, --no-owner makes postgres own restored tables. PostgreSQL grants no table privileges to PUBLIC by default, so an application role that relied on ownership can no longer query them. Restore original ownership or grant the application role the required privileges before restarting the API.
github Flagsmith flagsmith-charts 0.80.0 charts flagsmith api deployment readinessProbe httpGet health PostgreSQL image tag
Result:
Sources consulted:
- GitHub - Flagsmith/flagsmith-charts: Helm Charts for Flagsmith. Feature flagging and remote config service. Host yourself or use our hosted version. · GitHub — https://github.com/Flagsmith/flagsmith-charts
- flagsmith 0.2.4 · flagsmith/flagsmith — https://artifacthub.io/packages/helm/flagsmith/flagsmith/0.2.4
- flagsmith 0.1.8 · flagsmith/flagsmith — https://artifacthub.io/packages/helm/flagsmith/flagsmith/0.1.8
- …and 41 more
Stop before restarting the API when the restore fails.
pg_restore continues after errors by default. It can leave an incomplete database before returning a failure. Add --single-transaction and instruct operators to inspect the exit status before scaling flagsmith-api up.
When api.replicacount is greater than one, this command leaves the deployment at one replica after the restore. The chart does not enable the API HPA by default. Record the current count before scaling down, then use it when starting the API.
git diff 5051ac24bff70bfed8a428531be4816acef822ff a96256c2837e9069c4ba868965909c4d3515c74f -- docs/docs/deployment-self-hosting/upgrading.md
printf'\n--- guide ---\n'
nl -ba docs/docs/deployment-self-hosting/upgrading.md | sed -n '1,210p'printf'\n--- roots and chart refs ---\n'
ls -a
rg -n -i -F -e 'migrateAnalyticsData' -e 'UsePostgresForAnalytics' -e 'flagsmith-charts' -e 'influxdbExternal' --glob '!docs/docs/deployment-self-hosting/upgrading.md'.||test"$?" -eq 1
Repository: Flagsmith/flagsmith
Length of output: 26419
Script executed:
printf'%s\n''--- analytics references ---'
rg -n -i -e 'migrate.*analytics|analytics.*migrat|use_postgres_for_analytics|usepostgresforanalytics' api docs --glob '*.py' --glob '*.md'||test"$?" -eq 1
printf'%s\n''--- chart releases and tags ---'
curl -fsSL --max-time 20 'https://api.github.com/repos/Flagsmith/flagsmith-charts/git/matching-refs/tags/'| python3 -c 'import json,sys; a=json.load(sys.stdin); print("\n".join(str(x) for x in a if "0.85.0" in x["ref"] or "0.80.0" in x["ref"]))'
python3 - <<'PY'import json, urllib.requestcommit='a99c231d61dd89e1bc1924a4429619447b5e764f'base='https://api.github.com/repos/Flagsmith/flagsmith-charts/git/trees/'+commit+'?recursive=1'with urllib.request.urlopen(base, timeout=25) as f: tree=json.load(f)['tree']for x in tree: p=x['path'] if p.startswith('charts/flagsmith/') and ('migrat' in p.lower() or p.endswith(('Chart.yaml','values.yaml','NOTES.txt','_helpers.tpl'))): print(p)PYprintf'\n--- migration implementation ---\n'
nl -ba api/app_analytics/management/commands/migrate_analytics.py
nl -ba api/app_analytics/migrate_to_pg.py
printf'\n--- analytics read branches ---\n'
nl -ba api/app_analytics/analytics_db_service.py | sed -n '1,115p;185,245p;265,325p'
Repository: Flagsmith/flagsmith
Length of output: 12253
Script executed:
python3 - <<'PY'import json, urllib.requestfor label, commit in [('0.85.0','a99c231d61dd89e1bc1924a4429619447b5e764f'), ('0.80.0','fe3cdf0237a96a02a81f26b82742ffa05fb20b66')]: print(f'--- {label} ---') for path in [ 'charts/flagsmith/values.yaml', 'charts/flagsmith/templates/jobs-migrate-analytics-data.yaml', 'charts/flagsmith/templates/jobs-migrate-db.yaml', 'charts/flagsmith/templates/NOTES.txt', 'charts/flagsmith/Chart.yaml', ]: url=f'https://raw.githubusercontent.com/Flagsmith/flagsmith-charts/{commit}/{path}' try: text=urllib.request.urlopen(url, timeout=20).read().decode() except Exception as e: print(f'### {path}: {e}') continue print(f'### {path}') if path.endswith('values.yaml'): lines=text.splitlines() for i,line in enumerate(lines): if any(k.lower() in line.lower() for k in ('influx','migrateanalytics','taskprocessor','postgresforanalytics')): for x in lines[max(0,i-3):min(len(lines),i+8)]: print(x) else: print(text)PY
Repository: Flagsmith/flagsmith
Length of output: 21303
Script executed:
python3 - <<'PY'import urllib.requestfor label, commit in [('0.85.0','a99c231d61dd89e1bc1924a4429619447b5e764f'), ('0.80.0','fe3cdf0237a96a02a81f26b82742ffa05fb20b66')]: for path in ['charts/flagsmith/templates/_api_environment.yaml','charts/flagsmith/templates/jobs-migrate-analytics-data.yaml','charts/flagsmith/Chart.yaml','charts/flagsmith/values.yaml']: url=f'https://raw.githubusercontent.com/Flagsmith/flagsmith-charts/{commit}/{path}' print(f'\n### {label} {url}') try: lines=urllib.request.urlopen(url, timeout=20).read().decode().splitlines() except Exception as e: print(e); continue if path.endswith('values.yaml'): indices=set() for i,l in enumerate(lines): if any(k.lower() in l.lower() for k in ('influxdb2:', 'influxdbExternal:', 'UsePostgresForAnalytics:', 'migrateAnalyticsData:')): indices.update(range(max(0,i-1),min(len(lines),i+14))) elif path.endswith('_api_environment.yaml'): indices=set() for i,l in enumerate(lines): if any(k.lower() in l.lower() for k in ('influx', 'postgresforanalytics')): indices.update(range(max(0,i-4),min(len(lines),i+8))) else: indices=set(range(len(lines))) for i in sorted(indices): print(f'{i+1:4} {lines[i]}')PYprintf'\n--- application version tags ---\n'
git tag -l '*2.286.0*'printf'\n--- InfluxDB client and settings ---\n'
rg -n -F -e 'get_client' -e 'INFLUXDB_URL' -e 'INFLUXDB_TOKEN' -e 'INFLUXDB_ORG' -e 'INFLUXDB_BUCKET' -- api/app_analytics/influxdb_wrapper.py api/app/settings/common.py
nl -ba api/app_analytics/influxdb_wrapper.py | sed -n '1,125p'
Repository: Flagsmith/flagsmith
Length of output: 19391
Script executed:
python3 - <<'PY'import urllib.requestroot='https://raw.githubusercontent.com/Flagsmith/flagsmith/'for ref in ('v2.286.0','2.286.0'): print('\n###',ref) for path in ('api/app_analytics/migrate_to_pg.py','api/app_analytics/management/commands/migrate_analytics.py','api/app_analytics/analytics_db_service.py','api/app/settings/common.py','api/app_analytics/influxdb_wrapper.py'): try: text=urllib.request.urlopen(root+ref+'/'+path,timeout=15).read().decode().splitlines() except Exception as e: print(path, e); continue print('---',path) if path.endswith('migrate_to_pg.py') or path.endswith('migrate_analytics.py'): indices=range(len(text)) else: wanted=('USE_POSTGRES_FOR_ANALYTICS','get_feature_evaluation_data(', 'get_feature_evaluation_data_from_local_db(', 'def get_client(', 'INFLUXDB_TOKEN =','INFLUXDB_BUCKET =','INFLUXDB_URL =','INFLUXDB_ORG =') indices=sorted({j for i,l in enumerate(text) if any(w in l for w in wanted) for j in range(max(0,i-2),min(len(text),i+12))}) for i in indices: print(f'{i+1:4} {text[i]}')url='https://raw.githubusercontent.com/Flagsmith/flagsmith-charts/a99c231d61dd89e1bc1924a4429619447b5e764f/charts/flagsmith/templates/_api_environment.yaml'print('\n### full chart 0.85.0 API env')for i,l in enumerate(urllib.request.urlopen(url,timeout=15).read().decode().splitlines()): print(f'{i+1:4} {l}')PY
Repository: Flagsmith/flagsmith
Length of output: 12076
Web query:
Helm 3 upgrade removes resources no longer present in new chart release Kubernetes official documentation
Result:
Sources consulted:
- Changes Since Helm 2 | Helm — https://docs.helm.sh/docs/v3/faq/changes_since_helm2/
- helm upgrade | Helm — https://docs.helm.sh/docs/v3/helm/helm_upgrade/
- Chart Development Tips and Tricks | Helm — https://helm.sh/docs/v3/howto/charts_tips_and_tricks/
- Using Helm | Helm — https://helm.sh/docs/intro/using_helm/
- helm/pkg/action/upgrade.go at main · helm/helm · GitHub — https://github.com/helm/helm/blob/main/pkg/action/upgrade.go
- …and 7 more
Script executed:
python3 - <<'PY'import io, tarfile, urllib.requesturl='https://helm.influxdata.com/influxdb2-2.1.1.tgz'print('###',url)try: data=urllib.request.urlopen(url,timeout=30).read() with tarfile.open(fileobj=io.BytesIO(data),mode='r:gz') as tar: for member in tar.getmembers(): if member.name.endswith(('/templates/service.yaml','/templates/deployment.yaml','/templates/statefulset.yaml')): print('\n---',member.name) text=tar.extractfile(member).read().decode().splitlines() for i,l in enumerate(text): print(f'{i+1:4} {l}')except Exception as e: print('FETCH ERROR',repr(e))for path, needles in [ ('pkg/kube/client.go', ('func (c *Client) Update(', 'original.Difference(target)', 'ResourcePolicyAnno', 'DeleteWithOptions')), ('pkg/action/upgrade.go', ('KubeClient.Update(', 'execHook(', 'WaitWithJobs(', 'Wait(')),]: url='https://raw.githubusercontent.com/helm/helm/v3.18.0/'+path print('\n###',url) try: lines=urllib.request.urlopen(url,timeout=20).read().decode().splitlines() except Exception as e: print('FETCH ERROR',repr(e)); continue indices=set() for i,l in enumerate(lines): if any(n in l for n in needles): indices.update(range(max(0,i-5),min(len(lines),i+14))) for i in sorted(indices): print(f'{i+1:4} {lines[i]}')PY
Repository: Flagsmith/flagsmith
Length of output: 6450
Document the required InfluxDB source before enabling analytics migration.
When UsePostgresForAnalytics.enabled is true, chart 0.85.0 gives the migration Job PostgreSQL settings but no INFLUXDB_* settings. Chart 0.85.0 also removes the bundled InfluxDB. Configure the old data in influxdbExternal, run the migration, then switch analytics reads to PostgreSQL.
Thanks a lot, @Cydo73 . Could you take a look at CodeRabbit's comments before I jump in, please?
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Thanks for submitting a PR! Please check the boxes below:
docs/if required so people know about the feature.Contributes to #7429
Changes
Adds a new page, "Upgrading and rolling back", to the self hosting docs. It covers:
migrate-dbinit container by default, and where to read its logs)helm rollbackdoes not revert the database, and a restore procedure that worked in testingHow did you test this code?
I ran every command on a local kind cluster, upgrading chart 0.80.0 (Flagsmith 2.222.0) to chart 0.85.0 (Flagsmith
2.286.0) with default values and the PostgreSQL instance deployed by the chart. I took a pg_dump backup, upgraded (91
migrations applied), rolled back with
helm rollback(Flagsmith 2.222.0 then failed with a missing table error), andrestored the backup, after which 2.222.0 worked again with the original data.
Docs checks: I ran
npx prettier --checkon the page andnpm run build. [Edit this line to match what you ran, andmention any unrelated build warnings.]
Full write up with logs and screenshots:
Not tested: custom values, an external or managed database, the
jobs.migrateDbJob, the analytics migration job,other version pairs, and downtime during the upgrade. Happy to rerun any of these.
Questions for reviewers: I placed the page next to the Edge Proxy page with
sidebar_position: 4, and put the diagramin
docs/diagrams. Happy to move either.