From d33aa6b7cc59e13bf213d1d6c93a5c7806588466 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 25 Sep 2024 12:47:46 +0100 Subject: [PATCH 1/6] add custom granularities --- .../docs/docs/build/metricflow-time-spine.md | 78 +++++++++++++++++++ .../docs/docs/dbt-versions/release-notes.md | 1 + 2 files changed, 79 insertions(+) diff --git a/website/docs/docs/build/metricflow-time-spine.md b/website/docs/docs/build/metricflow-time-spine.md index 7fd196f7f7c..3316977f90f 100644 --- a/website/docs/docs/build/metricflow-time-spine.md +++ b/website/docs/docs/build/metricflow-time-spine.md @@ -20,6 +20,7 @@ To see the generated SQL for the metric and dimension types that use time-spine - You only need to configure time-spine models that the Semantic Layer should recognize. - At a minimum, define a time-spine table for a daily grain. - You can optionally define a time-spine table for a different granularity, like hourly. +- If your organization uses [custom calendars](#custom-calendar) (like fiscal years) or custom granularities (like `retail_month`), you can configure those as well. Learn how to [add custom calendars and granularities](#add-custom-granularities) further down this page to help support specific business needs. - Note that if you don’t have a date or calendar model in your project, you'll need to create one. - If you're looking to specify the grain of a time dimension so that MetricFlow can transform the underlying column to the required granularity, refer to the [Time granularity documentation](/docs/build/dimensions?dimension=time_gran) @@ -36,13 +37,18 @@ If you don’t have a date dimension table, you can still create one by using th + ```yaml models: - name: time_spine_hourly + description: "my favorite time spine" time_spine: standard_granularity_column: date_hour # column for the standard grain of your table + custom_granularities: + - name: fiscal_year + column_name: fiscal_year_column columns: - name: date_hour granularity: hour # set granularity at column-level for standard_granularity_column @@ -54,6 +60,29 @@ models: granularity: day # set granularity at column-level for standard_granularity_column ``` + + + + + +```yaml +models: + - name: time_spine_hourly + description: "my favorite time spine" + time_spine: + standard_granularity_column: date_hour # column for the standard grain of your table + columns: + - name: date_hour + granularity: hour # set granularity at column-level for standard_granularity_column + - name: time_spine_daily + time_spine: + standard_granularity_column: date_day # column for the standard grain of your table + columns: + - name: date_day + granularity: day # set granularity at column-level for standard_granularity_column +``` + + For an example project, refer to our [Jaffle shop](https://github.com/dbt-labs/jaffle-sl-template/blob/main/models/marts/_models.yml) example. @@ -230,3 +259,52 @@ where date_day > dateadd(year, -4, current_timestamp()) and date_hour < dateadd(day, 30, current_timestamp()) ``` + + +## Custom calendar + + + +Being able to configure custom calendars, such as like a fiscal calendar, is available in [dbt Cloud Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) or dbt Core [v1.9 and above](/docs/dbt-versions/core). + +To access this feature, [upgrade to Versionless](/docs/dbt-versions/versionless-cloud) or dbt Core v1.9 and above. + + + + +If you use a custom calendar in your organization, such as a fiscal calendar, you can configure it in MetricFlow using its date and time operations. + +This is useful for calculating metrics based on a custom calendar, such as fiscal quarters or weeks. Use the `custom_granularities` key to define a non-standard time period for querying data, such as a `retail_month` or `fiscal_week`, instead of standard options like `day`, `month`, or `year`. This feature provides more control over how time-based metrics are calculated. + + +### Why use a custom calendar model? +Custom date transformations can be complex, and organizations often have unique needs that can’t be easily generalized. Creating a custom calendar model allows you to define these transformations in SQL, offering more flexibility than native transformations in MetricFlow. This approach lets you map custom columns back to MetricFlow granularities, ensuring consistency while giving you control over the transformations. + +### Calendar model requirements + +To use a custom calendar model, ensure it meets the following requirements: + +- The model must be at a daily grain and joinable to other date columns using `date_day`. +- Columns must align with expected data types for a given granularity (for example, `quarter` should be a date). +- Support period-over-period calculations. +- Support cumulative metric windows using custom granularities (for example, cumulative fiscal year-to-date bookings). + +### Add custom granularities + +To add custom granularities, the Semantic Layer supports custom calendar configurations that allow users to query data using non-standard time periods like `fiscal_year` or `retail_month`. You can define these custom granularities by modifying your model's YAML configuration like this: + + + +```yaml +models: + - name: my_time_spine + description: "my favorite time spine" + time_spine: + standard_granularity_column: date_day + custom_granularities: + - name: fiscal_year + column_name: fiscal_year_column +``` + + + diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index f3a0c53844e..c885533a832 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -20,6 +20,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo ## September 2024 +- **New**: The dbt Semantic Layer supports custom calendar configurations in MetricFlow that allow you to query data using non-standard time periods like `fiscal_year` or `retail_month`. Refer to [custom calendar](/docs/build/metricflow-time-spine#custom-calendar) to learn how to define these custom granularities in your MetricFlow timespine YAML configuration. - **New:** There are two new [environment variable defaults](/docs/build/environment-variables#dbt-cloud-context) — `DBT_CLOUD_ENVIRONMENT_NAME` and `DBT_CLOUD_ENVIRONMENT_TYPE`. - **New:** The [Amazon Athena warehouse connection](/docs/cloud/connect-data-platform/connect-amazon-athena) is available as a public preview for dbt Cloud accounts that have upgraded to [`versionless`](/docs/dbt-versions/versionless-cloud). From cf6500a84ab1c19b7e5941e6d1ee39f753b64dfd Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 25 Sep 2024 12:53:31 +0100 Subject: [PATCH 2/6] fix space --- website/docs/docs/build/metricflow-time-spine.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/website/docs/docs/build/metricflow-time-spine.md b/website/docs/docs/build/metricflow-time-spine.md index 3316977f90f..af0baf5cb4b 100644 --- a/website/docs/docs/build/metricflow-time-spine.md +++ b/website/docs/docs/build/metricflow-time-spine.md @@ -13,10 +13,9 @@ MetricFlow requires you to define a time-spine table as a model-level configurat - [Metric offsets](/docs/build/derived#derived-metric-offset) - [Conversion metrics](/docs/build/conversion) - [Slowly Changing Dimensions](/docs/build/dimensions#scd-type-ii) -- [Metrics](/docs/build/metrics-overview) with the `join_to_timespine` configuration set to true -To see the generated SQL for the metric and dimension types that use time-spine joins, refer to the respective documentation or add the `compile=True` flag when querying the Semantic Layer to return the compiled SQL. +- [Metrics](/docs/build/metrics-overview) with the `join_to_timespine` configuration set to `true`. To see the generated SQL for the metric and dimension types that use time-spine joins, refer to the respective documentation or add the `compile=True` flag when querying the Semantic Layer to return the compiled SQL. -#### Configuring time-spine +### Configuring time-spine - You only need to configure time-spine models that the Semantic Layer should recognize. - At a minimum, define a time-spine table for a daily grain. - You can optionally define a time-spine table for a different granularity, like hourly. From 07816ee6b81c2ebf1b30bf12b68157c6f0ec3696 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 25 Sep 2024 12:55:49 +0100 Subject: [PATCH 3/6] fix order --- website/docs/docs/build/metricflow-time-spine.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/website/docs/docs/build/metricflow-time-spine.md b/website/docs/docs/build/metricflow-time-spine.md index af0baf5cb4b..2f15f0b2923 100644 --- a/website/docs/docs/build/metricflow-time-spine.md +++ b/website/docs/docs/build/metricflow-time-spine.md @@ -271,13 +271,13 @@ To access this feature, [upgrade to Versionless](/docs/dbt-versions/versionless- -If you use a custom calendar in your organization, such as a fiscal calendar, you can configure it in MetricFlow using its date and time operations. - -This is useful for calculating metrics based on a custom calendar, such as fiscal quarters or weeks. Use the `custom_granularities` key to define a non-standard time period for querying data, such as a `retail_month` or `fiscal_week`, instead of standard options like `day`, `month`, or `year`. This feature provides more control over how time-based metrics are calculated. +Custom date transformations can be complex, and organizations often have unique needs that can’t be easily generalized. Creating a custom calendar model allows you to define these transformations in SQL, offering more flexibility than native transformations in MetricFlow. This approach lets you map custom columns back to MetricFlow granularities, ensuring consistency while giving you control over the transformations. +For example, if you use a custom calendar in your organization, such as a fiscal calendar, you can configure it in MetricFlow using its date and time operations. -### Why use a custom calendar model? -Custom date transformations can be complex, and organizations often have unique needs that can’t be easily generalized. Creating a custom calendar model allows you to define these transformations in SQL, offering more flexibility than native transformations in MetricFlow. This approach lets you map custom columns back to MetricFlow granularities, ensuring consistency while giving you control over the transformations. +- This is useful for calculating metrics based on a custom calendar, such as fiscal quarters or weeks. +- Use the `custom_granularities` key to define a non-standard time period for querying data, such as a `retail_month` or `fiscal_week`, instead of standard options like `day`, `month`, or `year`. +- This feature provides more control over how time-based metrics are calculated. ### Calendar model requirements From 15e86c2acc2b0eab3960c2439e3928edaef0b0f9 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Wed, 25 Sep 2024 13:01:59 +0100 Subject: [PATCH 4/6] add headers --- website/docs/docs/build/metricflow-time-spine.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/website/docs/docs/build/metricflow-time-spine.md b/website/docs/docs/build/metricflow-time-spine.md index 2f15f0b2923..1e5154789c3 100644 --- a/website/docs/docs/build/metricflow-time-spine.md +++ b/website/docs/docs/build/metricflow-time-spine.md @@ -30,7 +30,7 @@ For example, given the following directory structure, you can create two time sp :::tip Previously, you had to create a model called `metricflow_time_spine` in your dbt project. Now, if your project already includes a date dimension or time spine table, you can simply configure MetricFlow to use that table by updating the `model` setting in the Semantic Layer. -If you don’t have a date dimension table, you can still create one by using the code snippet below to build your time spine model. +If you don’t have a date dimension table, you can still create one by using the following code snippet to build your time spine model. ::: @@ -85,9 +85,10 @@ models: For an example project, refer to our [Jaffle shop](https://github.com/dbt-labs/jaffle-sl-template/blob/main/models/marts/_models.yml) example. -Now, break down the configuration above. It's pointing to a model called `time_spine_daily`. It sets the time spine configurations under the `time_spine` key. The `standard_granularity_column` is the lowest grain of the table, in this case, it's hourly. It needs to reference a column defined under the columns key, in this case, `date_hour`. Use the `standard_granularity_column` as the join key for the time spine table when joining tables in MetricFlow. Here, the granularity of the `standard_granularity_column` is set at the column level, in this case, `hour`. - +#### Understanding time-spine and granularity +The previous configuration demonstrates a time-spine model called `time_spine_daily`. It sets the time spine configurations under the `time_spine` key. The `standard_granularity_column` is the lowest grain of the table, in this case, it's hourly. It needs to reference a column defined under the columns key, in this case, `date_hour`. Use the `standard_granularity_column` as the join key for the time spine table when joining tables in MetricFlow. Here, the granularity of the `standard_granularity_column` is set at the column level, in this case, `hour`. Additionally, the `custom_granularities` field lets you specify non-standard time periods like `fiscal_year` or `retail_month` that your organization may use. +#### Creating a time-spine table If you need to create a time spine table from scratch, you can do so by adding the following code to your dbt project. The example creates a time spine at a daily grain and an hourly grain. A few things to note when creating time spine models: * MetricFlow will use the time spine with the largest compatible granularity for a given query to ensure the most efficient query possible. For example, if you have a time spine at a monthly grain, and query a dimension at a monthly grain, MetricFlow will use the monthly time spine. If you only have a daily time spine, MetricFlow will use the daily time spine and date_trunc to month. From e7c5156f0391aebeb7c708a47ab7ea8bf891e7c3 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Thu, 26 Sep 2024 15:18:25 +0100 Subject: [PATCH 5/6] fold feedback --- .../docs/docs/build/metricflow-time-spine.md | 24 +++++++------------ 1 file changed, 9 insertions(+), 15 deletions(-) diff --git a/website/docs/docs/build/metricflow-time-spine.md b/website/docs/docs/build/metricflow-time-spine.md index eabb5268a0b..a76ccbbd264 100644 --- a/website/docs/docs/build/metricflow-time-spine.md +++ b/website/docs/docs/build/metricflow-time-spine.md @@ -46,7 +46,7 @@ models: - name: time_spine_hourly description: "my favorite time spine" time_spine: - standard_granularity_column: date_hour # column for the standard grain of your table + standard_granularity_column: date_hour # column for the standard grain of your table, must be date time type." custom_granularities: - name: fiscal_year column_name: fiscal_year_column @@ -88,7 +88,9 @@ models: For an example project, refer to our [Jaffle shop](https://github.com/dbt-labs/jaffle-sl-template/blob/main/models/marts/_models.yml) example. #### Understanding time-spine and granularity -The previous configuration demonstrates a time-spine model called `time_spine_daily`. It sets the time spine configurations under the `time_spine` key. The `standard_granularity_column` is the lowest grain of the table, in this case, it's hourly. It needs to reference a column defined under the columns key, in this case, `date_hour`. Use the `standard_granularity_column` as the join key for the time spine table when joining tables in MetricFlow. Here, the granularity of the `standard_granularity_column` is set at the column level, in this case, `hour`. Additionally, the `custom_granularities` field lets you specify non-standard time periods like `fiscal_year` or `retail_month` that your organization may use. +The previous configuration demonstrates a time-spine model called `time_spine_daily`. It sets the time spine configurations under the `time_spine` key. The standard_granularity_column is the column that maps to one of our standard granularities [list or link to options here]. The grain of this column must be finer or equal in size to the granularity of all custom granularity columns in the same model. In this case, it's hourly. It needs to reference a column defined under the `columns` key, in this case, `date_hour`. MetricFlow will use the `standard_granularity_column` as the join key when joining the time spine table to other source table. Here, the granularity of the `standard_granularity_column` is set at the column level, in this case, `hour`. + +Additionally, [the `custom_granularities` field](#custom-calendar), available in dbt v1.9 and higher) lets you specify non-standard time periods like `fiscal_year` or `retail_month` that your organization may use. #### Creating a time-spine table If you need to create a time spine table from scratch, you can do so by adding the following code to your dbt project. @@ -282,34 +284,26 @@ Custom date transformations can be complex, and organizations often have unique For example, if you use a custom calendar in your organization, such as a fiscal calendar, you can configure it in MetricFlow using its date and time operations. - This is useful for calculating metrics based on a custom calendar, such as fiscal quarters or weeks. -- Use the `custom_granularities` key to define a non-standard time period for querying data, such as a `retail_month` or `fiscal_week`, instead of standard options like `day`, `month`, or `year`. +- Use the `custom_granularities` key to define a non-standard time period for querying data, such as a `retail_month` or `fiscal_week`, instead of standard options like `day`, `month`, or `year`. +- Ensure the the `standard_granularity_column` is a date time type. - This feature provides more control over how time-based metrics are calculated. -### Calendar model requirements - -To use a custom calendar model, ensure it meets the following requirements: - -- The model must be at a daily grain and joinable to other date columns using `date_day`. -- Columns must align with expected data types for a given granularity (for example, `quarter` should be a date). -- Support period-over-period calculations. -- Support cumulative metric windows using custom granularities (for example, cumulative fiscal year-to-date bookings). - ### Add custom granularities -To add custom granularities, the Semantic Layer supports custom calendar configurations that allow users to query data using non-standard time periods like `fiscal_year` or `retail_month`. You can define these custom granularities by modifying your model's YAML configuration like this: +To add custom granularities, the Semantic Layer supports custom calendar configurations that allow users to query data using non-standard time periods like `fiscal_year` or `retail_month`. You can define these custom granularities (all lower cased) by modifying your model's YAML configuration like this: ```yaml models: - name: my_time_spine - description: "my favorite time spine" + description: my favorite time spine time_spine: standard_granularity_column: date_day custom_granularities: - name: fiscal_year column_name: fiscal_year_column -``` +``` From 3c0bd332184142eb17aa998eb94b56b375c1dfa5 Mon Sep 17 00:00:00 2001 From: mirnawong1 Date: Thu, 26 Sep 2024 15:45:28 +0100 Subject: [PATCH 6/6] fix --- .../docs/docs/build/metricflow-time-spine.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/website/docs/docs/build/metricflow-time-spine.md b/website/docs/docs/build/metricflow-time-spine.md index a76ccbbd264..2e5da2dcad8 100644 --- a/website/docs/docs/build/metricflow-time-spine.md +++ b/website/docs/docs/build/metricflow-time-spine.md @@ -87,17 +87,26 @@ models: For an example project, refer to our [Jaffle shop](https://github.com/dbt-labs/jaffle-sl-template/blob/main/models/marts/_models.yml) example. -#### Understanding time-spine and granularity -The previous configuration demonstrates a time-spine model called `time_spine_daily`. It sets the time spine configurations under the `time_spine` key. The standard_granularity_column is the column that maps to one of our standard granularities [list or link to options here]. The grain of this column must be finer or equal in size to the granularity of all custom granularity columns in the same model. In this case, it's hourly. It needs to reference a column defined under the `columns` key, in this case, `date_hour`. MetricFlow will use the `standard_granularity_column` as the join key when joining the time spine table to other source table. Here, the granularity of the `standard_granularity_column` is set at the column level, in this case, `hour`. + -Additionally, [the `custom_granularities` field](#custom-calendar), available in dbt v1.9 and higher) lets you specify non-standard time periods like `fiscal_year` or `retail_month` that your organization may use. +- The previous configuration demonstrates a time-spine model called `time_spine_daily`. It sets the time spine configurations under the `time_spine` key. +- The `standard_granularity_column` is the column that maps to one of our [standard granularities](/docs/build/dimensions?dimension=time_gran). The grain of this column must be finer or equal in size to the granularity of all custom granularity columns in the same model. In this case, it's hourly. +- It needs to reference a column defined under the `columns` key, in this case, `date_hour`. +- MetricFlow will use the `standard_granularity_column` as the join key when joining the time spine table to other source table. +- Here, the granularity of the `standard_granularity_column` is set at the column level, in this case, `hour`. + +Additionally, [the `custom_granularities` field](#custom-calendar), (available in dbt v1.9 and higher) lets you specify non-standard time periods like `fiscal_year` or `retail_month` that your organization may use. + + + + -#### Creating a time-spine table If you need to create a time spine table from scratch, you can do so by adding the following code to your dbt project. The example creates a time spine at a daily grain and an hourly grain. A few things to note when creating time spine models: * MetricFlow will use the time spine with the largest compatible granularity for a given query to ensure the most efficient query possible. For example, if you have a time spine at a monthly grain, and query a dimension at a monthly grain, MetricFlow will use the monthly time spine. If you only have a daily time spine, MetricFlow will use the daily time spine and date_trunc to month. * You can add a time spine for each granularity you intend to use if query efficiency is more important to you than configuration time, or storage constraints. For most engines, the query performance difference should be minimal and transforming your time spine to a coarser grain at query time shouldn't add significant overhead to your queries. * We recommend having a time spine at the finest grain used in any of your dimensions to avoid unexpected errors. i.e., if you have dimensions at an hourly grain, you should have a time spine at an hourly grain. + ## Example time-spine tables