forked from openzfs/zfs
-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
In a4b21ea we added the zap_micro_max_size tuneable to raise the size at which "micro" (single-block) ZAPs are upgraded to "fat" (multi-block) ZAPs. Before this, a microZAP was limited to 128KiB, which was the old largest block size. The side effect of raising the max size past 128KiB is that it be stored in a large block, requiring the large_blocks feature. Unfortunately, this means that a backup stream created without the --large-block (-L) flag to zfs send would split the microZAP block into smaller blocks and send those, as is normal behaviour for large blocks. This would be received correctly, but since microZAPs are limited to the first block in the object by definition, the entries in the later blocks would be inaccessible. For directory ZAPs, this gives the appearance of files being lost. This commit adds a feature flag, large_microzap, that must be enabled for microZAPs to grow beyond 128KiB, and which will be activated the first time that occurs. This feature is later checked when generating the stream and if active, the send operation will abort unless --large-block has also been requested. Changing the limit still requires zap_micro_max_size to be changed. The state of this flag effectively sets the upper value for this tuneable, that is, if the feature is disabled, the tuneable will be clamped to 128KiB. A stream flag is also added to ensure that the receiver also activates its own feature flag upon receiving the stream. This is not strictly necessary to _use_ the received microZAP, since it doesn't care how large its block is, but it is required to send the microZAP object on, otherwise the original problem occurs again. Because it's difficult to reliably distinguish a microZAP from a fatZAP from outside the ZAP code, and because it seems unlikely that most users are affected (a fairly niche tuneable combined with what should be an uncommon use of send), and for the sake of expediency, this change activates the feature the first time a microZAP grows to use a large block, and is never deactivated after that. This can be improved in the future. This commit changes nothing for existing pools that already have large microZAPs. The feature will not be retroactively applied, but will be activated the next time a microZAP grows past the limit. Don't use large_blocks feature for enable/disable tests. The large_microzap depends on large_blocks, so it gets enabled as a dependency, breaking the test. Instead use feature "longname", which has the exact same feature characteristics. Sponsored-by: Klara, Inc. Sponsored-by: Wasabi Technology, Inc. Reviewed-by: Allan Jude <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Alexander Motin <[email protected]> Signed-off-by: Rob Norris <[email protected]> Closes openzfs#16593
- Loading branch information
Showing
16 changed files
with
162 additions
and
22 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,6 +30,7 @@ | |
| * Portions Copyright 2010 Robert Milkowski | ||
| * Copyright (c) 2021, Colm Buckley <[email protected]> | ||
| * Copyright (c) 2022 Hewlett Packard Enterprise Development LP. | ||
| * Copyright (c) 2024, Klara, Inc. | ||
| */ | ||
|
|
||
| #ifndef _SYS_FS_ZFS_H | ||
|
|
@@ -1631,6 +1632,7 @@ typedef enum { | |
| ZFS_ERR_CRYPTO_NOTSUP, | ||
| ZFS_ERR_RAIDZ_EXPAND_IN_PROGRESS, | ||
| ZFS_ERR_ASHIFT_MISMATCH, | ||
| ZFS_ERR_STREAM_LARGE_MICROZAP, | ||
| } zfs_errno_t; | ||
|
|
||
| /* | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,6 +30,7 @@ | |
| * Copyright 2016 Igor Kozhukhov <[email protected]> | ||
| * Copyright (c) 2018, loli10K <[email protected]>. All rights reserved. | ||
| * Copyright (c) 2019 Datto Inc. | ||
| * Copyright (c) 2024, Klara, Inc. | ||
| */ | ||
|
|
||
| #include <assert.h> | ||
|
|
@@ -2828,7 +2829,12 @@ zfs_send_one_cb_impl(zfs_handle_t *zhp, const char *from, int fd, | |
| case EROFS: | ||
| zfs_error_aux(hdl, "%s", zfs_strerror(errno)); | ||
| return (zfs_error(hdl, EZFS_BADBACKUP, errbuf)); | ||
|
|
||
| case ZFS_ERR_STREAM_LARGE_MICROZAP: | ||
| zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, | ||
| "source snapshot contains large microzaps, " | ||
| "need -L (--large-block) or -w (--raw) to " | ||
| "generate stream")); | ||
| return (zfs_error(hdl, EZFS_BADBACKUP, errbuf)); | ||
| default: | ||
| return (zfs_standard_error(hdl, errno, errbuf)); | ||
| } | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,12 +14,11 @@ | |
| .\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your | ||
| .\" own identifying information: | ||
| .\" Portions Copyright [yyyy] [name of copyright owner] | ||
| .\" Copyright (c) 2019, Klara Inc. | ||
| .\" Copyright (c) 2019, 2023, 2024, Klara, Inc. | ||
| .\" Copyright (c) 2019, Allan Jude | ||
| .\" Copyright (c) 2021, Colm Buckley <[email protected]> | ||
| .\" Copyright (c) 2023, Klara Inc. | ||
| .\" | ||
| .Dd February 14, 2024 | ||
| .Dd October 2, 2024 | ||
| .Dt ZPOOL-FEATURES 7 | ||
| .Os | ||
| . | ||
|
|
@@ -706,6 +705,24 @@ are destroyed. | |
| Large dnodes allow more data to be stored in the bonus buffer, | ||
| thus potentially improving performance by avoiding the use of spill blocks. | ||
| . | ||
| .feature com.klarasystems large_microzap yes extensible_dataset large_blocks | ||
| This feature allows "micro" ZAPs to grow larger than 128 KiB without being | ||
| upgraded to "fat" ZAPs. | ||
| .Pp | ||
| This feature becomes | ||
| .Sy active | ||
| the first time a micro ZAP grows larger than 128KiB. | ||
| It will only be returned to the | ||
| .Sy enabled | ||
| state when all datasets that ever had a large micro ZAP are destroyed. | ||
| .Pp | ||
| Note that even when this feature is enabled, micro ZAPs cannot grow larger | ||
| than 128 KiB without also changing the | ||
| .Sy zap_micro_max_size | ||
| module parameter. | ||
| See | ||
| .Xr zfs 4 . | ||
| . | ||
| .feature com.delphix livelist yes extensible_dataset | ||
| This feature allows clones to be deleted faster than the traditional method | ||
| when a large number of random/sparse writes have been made to the clone. | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -25,7 +25,7 @@ | |
| * Copyright (c) 2014, Joyent, Inc. All rights reserved. | ||
| * Copyright 2014 HybridCluster. All rights reserved. | ||
| * Copyright (c) 2018, loli10K <[email protected]>. All rights reserved. | ||
| * Copyright (c) 2019, Klara Inc. | ||
| * Copyright (c) 2019, 2024, Klara, Inc. | ||
| * Copyright (c) 2019, Allan Jude | ||
| * Copyright (c) 2019 Datto Inc. | ||
| * Copyright (c) 2022 Axcient. | ||
|
|
@@ -593,6 +593,9 @@ recv_begin_check_feature_flags_impl(uint64_t featureflags, spa_t *spa) | |
| if ((featureflags & DMU_BACKUP_FEATURE_LARGE_DNODE) && | ||
| !spa_feature_is_enabled(spa, SPA_FEATURE_LARGE_DNODE)) | ||
| return (SET_ERROR(ENOTSUP)); | ||
| if ((featureflags & DMU_BACKUP_FEATURE_LARGE_MICROZAP) && | ||
| !spa_feature_is_enabled(spa, SPA_FEATURE_LARGE_MICROZAP)) | ||
| return (SET_ERROR(ENOTSUP)); | ||
|
|
||
| /* | ||
| * Receiving redacted streams requires that redacted datasets are | ||
|
|
@@ -994,6 +997,24 @@ dmu_recv_begin_sync(void *arg, dmu_tx_t *tx) | |
| numredactsnaps, tx); | ||
| } | ||
|
|
||
| if (featureflags & DMU_BACKUP_FEATURE_LARGE_MICROZAP) { | ||
| /* | ||
| * The source has seen a large microzap at least once in its | ||
| * life, so we activate the feature here to match. It's not | ||
| * strictly necessary since a large microzap is usable without | ||
| * the feature active, but if that object is sent on from here, | ||
| * we need this info to know to add the stream feature. | ||
| * | ||
| * There may be no large microzap in the incoming stream, or | ||
| * ever again, but this is a very niche feature and its very | ||
| * difficult to spot a large microzap in the stream, so its | ||
| * not worth the effort of trying harder to activate the | ||
| * feature at first use. | ||
| */ | ||
| dsl_dataset_activate_feature(dsobj, SPA_FEATURE_LARGE_MICROZAP, | ||
| (void *)B_TRUE, tx); | ||
| } | ||
|
|
||
| dmu_buf_will_dirty(newds->ds_dbuf, tx); | ||
| dsl_dataset_phys(newds)->ds_flags |= DS_FLAG_INCONSISTENT; | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.