Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add custom granularities to mf timespine #6145

Open
wants to merge 9 commits into
base: current
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 84 additions & 2 deletions website/docs/docs/build/metricflow-time-spine.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ To see the generated SQL for the metric and dimension types that use time-spine
- You only need to configure time-spine models that the Semantic Layer should recognize.
- At a minimum, define a time-spine table for a daily grain.
- You can optionally define a time-spine table for a different granularity, like hourly.
- If your organization uses [custom calendars](#custom-calendar) (like fiscal years) or custom granularities (like `retail_month`), you can configure those as well. Learn how to [add custom calendars and granularities](#add-custom-granularities) further down this page to help support specific business needs.
- Note that if you don’t have a date or calendar model in your project, you'll need to create one.
- If you're looking to specify the grain of a time dimension so that MetricFlow can transform the underlying column to the required granularity, refer to the [Time granularity documentation](/docs/build/dimensions?dimension=time_gran)

Expand All @@ -31,17 +32,44 @@ For example, given the following directory structure, you can create two time sp
:::tip
Previously, you had to create a model called `metricflow_time_spine` in your dbt project. Now, if your project already includes a date dimension or time spine table, you can simply configure MetricFlow to use that table by updating the `model` setting in the Semantic Layer.

If you don’t have a date dimension table, you can still create one by using the code snippet below to build your time spine model.
If you don’t have a date dimension table, you can still create one by using the following code snippet to build your time spine model.

:::

<Lightbox src="/img/time_spines.png" title="Time spine directory structure" />

<VersionBlock firstVersion="1.9">
<File name="models/_models.yml">

```yaml
models:
- name: time_spine_hourly
description: "my favorite time spine"
time_spine:
standard_granularity_column: date_hour # column for the standard grain of your table, must be date time type."
custom_granularities:
- name: fiscal_year
column_name: fiscal_year_column
columns:
- name: date_hour
granularity: hour # set granularity at column-level for standard_granularity_column
- name: time_spine_daily
time_spine:
standard_granularity_column: date_day # column for the standard grain of your table
columns:
- name: date_day
granularity: day # set granularity at column-level for standard_granularity_column
```
</File>
</VersionBlock>

<VersionBlock lastVersion="1.8">
<File name="models/_models.yml">

```yaml
models:
- name: time_spine_hourly
description: "my favorite time spine"
time_spine:
standard_granularity_column: date_hour # column for the standard grain of your table
columns:
Expand All @@ -55,17 +83,30 @@ models:
granularity: day # set granularity at column-level for standard_granularity_column
```
</File>
</VersionBlock>

For an example project, refer to our [Jaffle shop](https://github.com/dbt-labs/jaffle-sl-template/blob/main/models/marts/_models.yml) example.

Now, break down the configuration above. It's pointing to a model called `time_spine_daily`. It sets the time spine configurations under the `time_spine` key. The `standard_granularity_column` is the lowest grain of the table, in this case, it's hourly. It needs to reference a column defined under the columns key, in this case, `date_hour`. Use the `standard_granularity_column` as the join key for the time spine table when joining tables in MetricFlow. Here, the granularity of the `standard_granularity_column` is set at the column level, in this case, `hour`.
<Expandable alt_header="Understanding time-spine and granularity">

- The previous configuration demonstrates a time-spine model called `time_spine_daily`. It sets the time spine configurations under the `time_spine` key.
- The `standard_granularity_column` is the column that maps to one of our [standard granularities](/docs/build/dimensions?dimension=time_gran). The grain of this column must be finer or equal in size to the granularity of all custom granularity columns in the same model. In this case, it's hourly.
- It needs to reference a column defined under the `columns` key, in this case, `date_hour`.
- MetricFlow will use the `standard_granularity_column` as the join key when joining the time spine table to other source table.
- Here, the granularity of the `standard_granularity_column` is set at the column level, in this case, `hour`.

Additionally, [the `custom_granularities` field](#custom-calendar), (available in dbt v1.9 and higher) lets you specify non-standard time periods like `fiscal_year` or `retail_month` that your organization may use.

</Expandable>

<Expandable alt_header="Creating a time-spine table">

If you need to create a time spine table from scratch, you can do so by adding the following code to your dbt project.
The example creates a time spine at a daily grain and an hourly grain. A few things to note when creating time spine models:
* MetricFlow will use the time spine with the largest compatible granularity for a given query to ensure the most efficient query possible. For example, if you have a time spine at a monthly grain, and query a dimension at a monthly grain, MetricFlow will use the monthly time spine. If you only have a daily time spine, MetricFlow will use the daily time spine and date_trunc to month.
* You can add a time spine for each granularity you intend to use if query efficiency is more important to you than configuration time, or storage constraints. For most engines, the query performance difference should be minimal and transforming your time spine to a coarser grain at query time shouldn't add significant overhead to your queries.
* We recommend having a time spine at the finest grain used in any of your dimensions to avoid unexpected errors. i.e., if you have dimensions at an hourly grain, you should have a time spine at an hourly grain.
</Expandable>

## Example time-spine tables

Expand Down Expand Up @@ -234,3 +275,44 @@ where date_day > dateadd(year, -4, current_timestamp())
and date_hour < dateadd(day, 30, current_timestamp())
```
</File>


## Custom calendar

<VersionBlock lastVersion="1.8">

Being able to configure custom calendars, such as like a fiscal calendar, is available in [dbt Cloud Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) or dbt Core [v1.9 and above](/docs/dbt-versions/core).

To access this feature, [upgrade to Versionless](/docs/dbt-versions/versionless-cloud) or dbt Core v1.9 and above.
</VersionBlock>

<VersionBlock firstVersion="1.9">

Custom date transformations can be complex, and organizations often have unique needs that can’t be easily generalized. Creating a custom calendar model allows you to define these transformations in SQL, offering more flexibility than native transformations in MetricFlow. This approach lets you map custom columns back to MetricFlow granularities, ensuring consistency while giving you control over the transformations.

For example, if you use a custom calendar in your organization, such as a fiscal calendar, you can configure it in MetricFlow using its date and time operations.

- This is useful for calculating metrics based on a custom calendar, such as fiscal quarters or weeks.
- Use the `custom_granularities` key to define a non-standard time period for querying data, such as a `retail_month` or `fiscal_week`, instead of standard options like `day`, `month`, or `year`.
- Ensure the the `standard_granularity_column` is a date time type.
- This feature provides more control over how time-based metrics are calculated.

### Add custom granularities

To add custom granularities, the Semantic Layer supports custom calendar configurations that allow users to query data using non-standard time periods like `fiscal_year` or `retail_month`. You can define these custom granularities (all lower cased) by modifying your model's YAML configuration like this:

<File name="models/_models.yml">

```yaml
models:
- name: my_time_spine
description: my favorite time spine
time_spine:
standard_granularity_column: date_day
custom_granularities:
- name: fiscal_year
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

presume this is case sensitive, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea it should be lowercase

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spec looks good but linking back to the source of truth just to be safe: dbt-labs/dbt-semantic-interfaces#338 (comment)

column_name: fiscal_year_column
```
</File>

</VersionBlock>
1 change: 1 addition & 0 deletions website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo

## September 2024

- **New**: The dbt Semantic Layer supports custom calendar configurations in MetricFlow that allow you to query data using non-standard time periods like `fiscal_year` or `retail_month`. Refer to [custom calendar](/docs/build/metricflow-time-spine#custom-calendar) to learn how to define these custom granularities in your MetricFlow timespine YAML configuration.
- **Enhancement**: You can now run [Semantic Layer commands](/docs/build/metricflow-commands) commands in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud). The supported commands are `dbt sl list`, `dbt sl list metrics`, `dbt sl list dimension-values`, `dbt sl list saved-queries`, `dbt sl query`, `dbt sl list dimensions`, `dbt sl list entities`, and `dbt sl validate`.
- **New**: Microsoft Excel, a dbt Semantic Layer integration, is now generally available. The integration allows you to connect to Microsoft Excel to query metrics and collaborate with your team. Available for [Excel Desktop](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100&rs=en-US&correlationId=4132ecd1-425d-982d-efb4-de94ebc83f26) or [Excel Online](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100&rs=en-US&correlationid=4132ecd1-425d-982d-efb4-de94ebc83f26&isWac=True). For more information, refer to [Microsoft Excel](/docs/cloud-integrations/semantic-layer/excel).
- **New**: [Data health tile](/docs/collaborate/data-tile) is now generally available in dbt Explorer. Data health tiles provide a quick at-a-glance view of your data quality, highlighting potential issues in your data. You can embed these tiles in your dashboards to quickly identify and address data quality issues in your dbt project.
Expand Down
Loading