From f1befa437964fb9616da96251db06e017a016826 Mon Sep 17 00:00:00 2001 From: kik-kik Date: Wed, 14 Jan 2026 19:48:12 +0100 Subject: [PATCH 1/3] feat: update apple_app_store reference --- .../non_desktop/apple_app_store/reference.md | 109 +++--------------- 1 file changed, 15 insertions(+), 94 deletions(-) diff --git a/src/datasets/non_desktop/apple_app_store/reference.md b/src/datasets/non_desktop/apple_app_store/reference.md index f47c212ec..105e7830e 100644 --- a/src/datasets/non_desktop/apple_app_store/reference.md +++ b/src/datasets/non_desktop/apple_app_store/reference.md @@ -4,114 +4,35 @@ # Introduction -The [`apple_app_store`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&folder=&organizationId=&p=moz-fx-data-marketing-prod&d=apple_app_store&page=dataset) dataset is used to understand the acquisition performance for non-desktop products on the Apple App Store along the key metrics and dimensions. Apple’s documentation for all metrics and dimensions can be found [in the app store connect reference](https://help.apple.com/app-store-connect/#/itc21781223f). +The [`app_store_v2_syndicate`](https://console.cloud.google.com/bigquery?project=moz-fx-data-shared-prod&ws=!1m8!1m3!3m2!1smoz-fx-data-shared-prod!2sapp_store!1m3!3m2!1smoz-fx-data-shared-prod!2sapp_store_v2_syndicate) dataset is used to understand the acquisition performance for non-desktop products on the Apple App Store along the key metrics and dimensions. Apple’s documentation for all metrics and dimensions can be found [in the app store connect reference](https://help.apple.com/app-store-connect/#/itc21781223f). # Contents -The [`apple_app_store`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&folder=&organizationId=&p=moz-fx-data-marketing-prod&d=apple_app_store&page=dataset) dataset contains a collection of aggregated tables by a singular dimension that explains the performance of acquisition activity through the Apple App Store. +The `app_store_v2_syndicate` dataset contains a collection of pre-aggregated tables by a singular dimension that explains the performance of acquisition activity through the Apple App Store. The tables available are as follows: -The dimensions (saved as individual derived tables) include: +- `moz-fx-data-shared-prod.app_store_v2_syndicate.app_crash_daily` - Daily metrics to understand crashes for your App Store apps by app version and device type. -- [`moz-fx-data-marketing-prod.apple_app_store.metrics_by_platform`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store&t=metrics_by_platform&page=table) - The device type on which the app was downloaded or used. E.g. iPad, iPod, or iPhone. -- [`moz-fx-data-marketing-prod.apple_app_store.platform version`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store&t=metrics_by_platform_version&page=table) - The OS version on which the app was downloaded or used. App Units, In-App Purchases, and Sales are based on the version on which the app is downloaded. Active in Last 30 Days, Product Page Views, Retention, and Sessions are based on the iOS version on which the app is used. -- [`moz-fx-data-marketing-prod.apple_app_store.app version`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store&t=metrics_by_app_version&page=table) - The version of the app displayed on the app store at the time of activity. -- [`moz-fx-data-marketing-prod.apple_app_store.region`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store&t=metrics_by_region&page=table) - The App Store region in which purchases were made, based on the customer’s billing address. Regions include (USA and Canada, Europe, Latin America and The Caribbean, Asia Pacific, Africa, The Middle East, and India) -- [`moz-fx-data-marketing-prod.apple_app_store.storefront`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store&t=metrics_by_storefront&page=table) - The app store country in which purchases were made, based on the customer’s billing address. -- [`moz-fx-data-marketing-prod.apple_app_store.source`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store&t=metrics_by_source&page=table) - The source from which a customer tapped a link to your App Store product page to view your app or download it for the first time. You can view metrics based on the source from which users are finding your app. Source types include (App Store Browse, App Store Search, App Referrers, Web Referrers) -- [`moz-fx-data-marketing-prod.apple_app_store.app referrer`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store&t=metrics_by_app_referrer&page=table) - People landing on the app store via links from within other apps. This also includes apps using the store kit API and excludes the native Safari. -- [`moz-fx-data-marketing-prod.apple_app_store.web Referrer`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store&t=metrics_by_web_referrer&page=table) - Previously top websites. Shows the referring website for the app download. Web referrals must be from Safari on devices with iOS 8 or tvOS 9 or later. Taps from websites using web browsers like chrome are attributed to that app. -- [`moz-fx-data-marketing-prod.apple_app_store.campaign`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store&t=metrics_by_campaign&page=table) - Previously top campaigns. Tracks app and website referrals to measure the impact of an advertising campaign. Tracked by adding two tokens to any app store link to see results in app analytics. -- [`moz-fx-data-marketing-prod.apple_app_store.total`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store&t=metrics_total&page=table): Total metric activity without any dimension breakdown. +- `moz-fx-data-shared-prod.app_store_v2_syndicate.app_session_detailed_daily` - Daily metrics to understand how often people open your app, and how long they spend in your app. +- `moz-fx-data-shared-prod.app_store_v2_syndicate.app_session_standard_daily` - Daily metrics to understand how often people open your app, and how long they spend in your app. -The metrics included in the aggregated tables are: +- `moz-fx-data-shared-prod.app_store_v2_syndicate.app_store_discovery_and_engagement_detailed_daily` - Daily metrics on how users discover and engage with your app on the App Store. +- `moz-fx-data-shared-prod.app_store_v2_syndicate.app_store_discovery_and_engagement_standard_daily` - Daily metrics on how users discover and engage with your app on the App Store. -- **`app_units`** - The number of first-time app purchases made on the App Store using iOS 8 and tvOS 9 or later. Updates, re-downloads, download onto other devices are not counted. Family sharing downloads are included for free apps, but not for paid apps. -- **`impressions`** - The number of times the app was viewed in the Featured, Categories, Top Charts and Search Sections of the App Store. Also includes views of the product page. -- **`impressions_unique_device`** - The number of times the app was viewed in the Featured, Categories, Top Charts and Search Sections of the App Store by unique device. Also includes views of the product page. -- **`product_page_views`** - Number of times the app's product page has been viewed on devices iOS 8 and tvOS 9 or later. Includes both App Store app and store kit API -- **`product_page_views_unique_device`** - Number of times the app's product page has been viewed on devices iOS 8 and tvOS 9 or later by unique device. Includes both App Store app and store kit API -- **`active_devices_opt_in`** - The number of devices with at least one session during the selected period. Only devices with iOS 8 and tvOS 9 or later are included. Data for this metric is “opt-in” - collected only if users have agreed to share their diagnostics and usage information with app developers. -- **`active_devices_last_30_days_opt_in`** - The number of active devices with at least one session during the previous 30 days. Data for this metric is “opt-in” - collected only if users have agreed to share their diagnostics and usage information with app developers. -- **`deletions_opt_in`** - The number of times your app was deleted on devices running iOS 12.3 or tvOS 9 or later. This data includes deletions of the app from the Home Screen and deletions of the app through Manage Storage. Data from resetting or erasing a device's content and settings is not included. Data for this metric is “opt-in” - collected only if users have agreed to share their diagnostics and usage information with app developers. -- **`installations_opt_in`** - The total number of times your app has been installed on devices with iOS 8 or tvOS 9, or later. Re-downloads on the same device, downloads to multiple devices sharing the same Apple ID, and Family Sharing installations are included. Data for this metric is “opt-in” - collected only if users have agreed to share their diagnostics and usage information with app developers. -- **`sessions_opt_in`** - The number of times the app has been used for at least two seconds. If the app is in the background and is later used again, that counts as another session. Data for this metric is “opt-in” - collected only if users have agreed to share their diagnostics and usage information with app developers. +- `moz-fx-data-shared-prod.app_store_v2_syndicate.app_store_download_detailed_daily` - Daily metrics to understand to understand your total number of downloads, including first-time downloads, redownloads, updates, and more. +- `moz-fx-data-shared-prod.app_store_v2_syndicate.app_store_download_standard_daily` - Daily metrics to understand to understand your total number of downloads, including first-time downloads, redownloads, updates, and more. -## Background and Caveats +- `moz-fx-data-shared-prod.app_store_v2_syndicate.app_store_installation_and_deletion_detailed_daily` - Daily metrics to estimate the number of times people install and delete your App Store apps. +- `moz-fx-data-shared-prod.app_store_v2_syndicate.app_store_installation_and_deletion_standard_daily` - Daily metrics to estimate the number of times people install and delete your App Store apps. -The data is received from Apple with only one dimension per metric. As a result, we are unable to do multi-dimensional analysis. i.e. we can tell how each storefront is performing but we can’t see how specific platforms or sources are contributing to it. - -## Accessing the Data - -Access the data at [`moz-fx-data-marketing-prod.apple_app_store`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&folder=&organizationId=&p=moz-fx-data-marketing-prod&d=apple_app_store&page=dataset) +- `moz-fx-data-shared-prod.app_store_v2_syndicate.app_store_review` - Pulls app reviews from the App Store. +- `moz-fx-data-shared-prod.app_store_v2_syndicate.apple_store_territory_report` - Breakdown of app impressions, installs, and downloads by source (App Store search or App Store browse) and territory (country). # Data Reference ## Schema -``` -metrics_by_[dimension] tables -root - |- date: date - |- app_name: string - |- [app_referrer | app_version | campaign | platform | platform_version | region | source | storefront | web_referrer]: string - |- app_units: integer - |- impressions: integer - |- impressions_unique_device: integer - |- product_page_views: integer - |- product_page_views_unique_device: integer - |- active_devices_opt_in: integer - |- active_devices_last_30_days_opt_in: integer - |- deletions_opt_in: integer - |- installations_opt_in: integer - |- sessions_opt_in: integer -``` - -## Example Queries - -### Calculate Apple App Store Activity for a given day by app - -```sql -SELECT - date, - app_name, - SUM(impressions_unique_device) as unique_device_impressions, - SUM(product_page_views_unique_device) as unique_device_page_views, - SUM(app_units) as installs, - SAFE_DIVIDE(SUM(product_page_views_unique_device), SUM(impressions_unique_device)) as unique_device_page_view_rate, - SAFE_DIVIDE(SUM(app_units), SUM(product_page_views_unique_device)) as install_rate -FROM - `moz-fx-data-marketing-prod.apple_app_store.metrics_total` -WHERE - date = "2020-08-20" -GROUP BY - date, app_name -``` - -[`STMO#74291`](https://sql.telemetry.mozilla.org/queries/74291/source) - -### Calculate Apple App Store Activity for a given day and app by source - -```sql -SELECT - date, - app_name, - source, - SUM(impressions_unique_device) as unique_device_impressions, - SUM(product_page_views_unique_device) as unique_device_page_views, - SUM(app_units) as installs, - SAFE_DIVIDE(SUM(product_page_views_unique_device), SUM(impressions_unique_device)) as unique_device_page_view_rate, - SAFE_DIVIDE(SUM(app_units), SUM(product_page_views_unique_device)) as install_rate -FROM - `moz-fx-data-marketing-prod.apple_app_store.metrics_by_source` -WHERE - date = "2020-08-20" - AND app_name = "Firefox" -GROUP BY - date, app_name, source -``` - -[`STMO#74290`](https://sql.telemetry.mozilla.org/queries/74290/source) +A schema and field information can be found in the [dbt_apple_store docs](https://fivetran.github.io/dbt_apple_store/#!/source_list/apple_store). ## Scheduling -The job to retrieve the raw data from the Apple App Store can be found in [the `app-store-analytics-export` repository](https://github.com/mozilla/app-store-analytics-export). The exported results are individual metrics grouped by a single dimension. These exports are initially loaded into the [`apple_app_store_exported`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store_exported&page=dataset) data source. The exports are scheduled in [`airflow`](https://github.com/mozilla/telemetry-airflow/blob/master/dags/app_store_analytics.py). The job to create the derived tables found in [`moz-fx-data-marketing-prod.apple_app_store`](https://console.cloud.google.com/bigquery?project=moz-fx-data-marketing-prod&p=moz-fx-data-marketing-prod&d=apple_app_store&page=dataset) can be found in [`bigquery-etl` under `apple_app_store`](https://github.com/mozilla/bigquery-etl/tree/master/sql/moz-fx-data-marketing-prod/apple_app_store). +The raw data is retrieved from the Apple App Store using a [Fivetran Apple App Store connector](https://fivetran.com/dashboard/connections/branches_bunches/status?groupId=loading_reputation&service=itunes_connect&syncChartPeriod=1%20Day). This data is then also transformed by a [Fivetran transformation](https://fivetran.com/dashboard/transformations/quickstart/refill_regaining?groupId=loading_reputation) and made available inside a BigQuery dataset [moz-fx-data-bq-fivetran.firefox_app_store_v2](https://console.cloud.google.com/bigquery?project=moz-fx-data-shared-prod&ws=!1m8!1m3!3m2!1smoz-fx-data-shared-prod!2sapp_store!1m3!3m2!1smoz-fx-data-shared-prod!2sapp_store_v2_syndicate) from where downstream ETL processes consume the data for deriving insights (example: [app_store_funnel_v1](https://github.com/mozilla/bigquery-etl/blob/17172bf4a46483966d1971d30b9bce60e41e9d88/sql/moz-fx-data-shared-prod/firefox_ios_derived/app_store_funnel_v1/query.sql)). From af01fc3cfea7149be5d08303aaf31bb140a7ad6c Mon Sep 17 00:00:00 2001 From: kik-kik Date: Wed, 14 Jan 2026 19:51:26 +0100 Subject: [PATCH 2/3] feat: rename dataset apple_app_store to app_store_v2_syndicate --- src/SUMMARY.md | 2 +- src/datasets/non_desktop.md | 2 +- .../{apple_app_store => app_store_v2_syndicate}/reference.md | 0 3 files changed, 2 insertions(+), 2 deletions(-) rename src/datasets/non_desktop/{apple_app_store => app_store_v2_syndicate}/reference.md (100%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 968451d63..f09a4c72d 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -131,7 +131,7 @@ - [Non-Desktop Datasets](datasets/non_desktop.md) - [Day 2-7 Activation](datasets/non_desktop/day_2_7_activation/reference.md) - [Google Play Store](datasets/non_desktop/google_play_store/reference.md) - - [Apple App Store](datasets/non_desktop/apple_app_store/reference.md) + - [Apple App Store](datasets/non_desktop/app_store_v2_syndicate/reference.md) - [Other Datasets](datasets/other.md) - [hgpush](datasets/other/hgpush/reference.md) - [Stub installer ping](datasets/other/stub_installer/reference.md) diff --git a/src/datasets/non_desktop.md b/src/datasets/non_desktop.md index 917808d5f..ec06add4a 100644 --- a/src/datasets/non_desktop.md +++ b/src/datasets/non_desktop.md @@ -6,4 +6,4 @@ Non-Desktop data includes data for all mobile browser products as well as non br - [Day 2-7 Activation](./non_desktop/day_2_7_activation/reference.md) - Used to calculate the [Day 2-7 Activation metric](../metrics/metrics.md#day-2-7-activation), a key result in 2020 for non-desktop products. - [Google Play Store Data](./non_desktop/google_play_store/reference.md) - Used to understand the acquisition performance for non-desktop products on the Google Play Store. -- [Apple App Store Data](./non_desktop/apple_app_store/reference.md) - Used to understand the acquisition performance for the non-desktop products on the Apple App Store. +- [Apple App Store Data](./non_desktop/app_store_v2_syndicate/reference.md) - Used to understand the acquisition performance for the non-desktop products on the Apple App Store. diff --git a/src/datasets/non_desktop/apple_app_store/reference.md b/src/datasets/non_desktop/app_store_v2_syndicate/reference.md similarity index 100% rename from src/datasets/non_desktop/apple_app_store/reference.md rename to src/datasets/non_desktop/app_store_v2_syndicate/reference.md From 94d6f7eaa433e082f386467da7295191ce487f79 Mon Sep 17 00:00:00 2001 From: kik-kik Date: Wed, 14 Jan 2026 20:15:34 +0100 Subject: [PATCH 3/3] feat: add words to .spelling --- .spelling | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.spelling b/.spelling index a6f581ce3..433815da9 100644 --- a/.spelling +++ b/.spelling @@ -501,4 +501,9 @@ unmute is_lineage_mode schemaFilter client_scalar_probe_counts -20Definition \ No newline at end of file +20Definition +redownloads +dbt_apple_store +moz-fx-data-bq-fivetran +firefox_app_store_v2 +app_store_funnel_v1 \ No newline at end of file