Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Change SharedArbitrator extra config string based #10685

Closed
wants to merge 1 commit into from
Closed

Change SharedArbitrator extra config string based #10685

wants to merge 1 commit into from

Conversation

tanjialiang
Copy link
Contributor

Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 7, 2024
@netlify Netlify
Copy link

netlify bot commented Aug 7, 2024 •
edited
Loading

✅ Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit 5406019
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/66bd85c220004600085cc018

@facebook-github-bot
Copy link
Contributor

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@tanjialiang tanjialiang force-pushed the arb_config_2 branch 2 times, most recently from a16ec0a to 62cd2dc Compare August 7, 2024 19:30
xiaoxmeng
xiaoxmeng reviewed Aug 7, 2024
View reviewed changes
Copy link
Contributor

@xiaoxmeng xiaoxmeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tanjialiang thanks for the refactor % nits.

velox/common/base/ConfigUtil.h Outdated
PETABYTE
};

double toBytesPerCapacityUnit(CapacityUnit unit);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add unit tests for these? Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I can

velox/common/base/ConfigUtil.cpp Outdated

#include "velox/common/base/ConfigUtil.h"

namespace facebook::velox::config {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume there is no functional change? Why not put this into velox/common/config/config.h?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a completely different ConfigBase class that is only used by dwrf and has nothing to do with this. But it will be nice to do some followup cleanup for our configs.

velox/common/memory/Memory.cpp
extraArbitratorConfigs["memory-pool-reserved-capacity"] =
folly::to<std::string>(options.memoryPoolReservedCapacity);
folly::to<std::string>(options.memoryPoolReservedCapacity) + "B";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use utility for this conversion instead of attaching a unit string which seems to be a bit hack. Do we already have such utilities in Prestissimo or Velox? thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is temporary. It will be removed very soon

@facebook-github-bot
Copy link
Contributor

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

xiaoxmeng
xiaoxmeng approved these changes Aug 14, 2024
View reviewed changes
Copy link
Contributor

@xiaoxmeng xiaoxmeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tanjialiang LGTM. Thanks!

@facebook-github-bot
Copy link
Contributor

@tanjialiang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@tanjialiang @facebook-github-bot
Change SharedArbitrator extra config string based (facebookincubator#…
5406019 
…10685)

Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D60920773

@facebook-github-bot
Copy link
Contributor

@tanjialiang merged this pull request in fe9605f.

@conbench-facebook conbench-facebook
Copy link

Conbench analyzed the 1 benchmark run on commit fe9605f7.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 16, 2024
@tanjialiang
Change SharedArbitrator extra config string based (facebookincubator#…
01d6fe5 
…10685)

Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 16, 2024
@tanjialiang
Change SharedArbitrator extra config string based (facebookincubator#…
9aec261 
…10685)

Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 16, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
62ae656 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 16, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
fa1fbaa 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 16, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
ef745e7 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 16, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
096c3b3 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 16, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
865eb72 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 16, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
9118f57 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 16, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
67bb943 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 17, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
9cde856 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 17, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
f3f7dee 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 18, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
57740e8 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 18, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
a66fe94 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 18, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
1a76bbd 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 18, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
5fdb70c 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 18, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
5d2d8c3 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
tanjialiang added a commit to tanjialiang/velox-1 that referenced this pull request Aug 19, 2024
@tanjialiang
Add shrink target adjustment mechanism in memory arbitration
774f19d 
Summary:
Switch to string based config for extra configs in shared arbitrator for better consistency and fewer back and forth conversions.

Pull Request resolved: facebookincubator#10685

Reviewed By: xiaoxmeng

Differential Revision: D60920773

Pulled By: tanjialiang

fbshipit-source-id: 215d4e45d89c0b319f883a7c4b1ed27abd1c3e9f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiaoxmeng xiaoxmeng xiaoxmeng approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tanjialiang @facebook-github-bot @xiaoxmeng