Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AIP-157 Partial response implementation #717

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

sayan-biswas
Copy link
Contributor

@sayan-biswas sayan-biswas commented Feb 21, 2024

In the present implementation, a user cannot filter the data in the response message, a feature provided by modern API implementations. This is particularly needed for the List APIs as these APIs return the whole object in the list. In scenario where the list and object definition are big, there's a lot network bandwidth wasted because the response of the List API is mostly used to display a list of resources and the Get API is then used to fetch the individual resource.
This feature provides an easy and basic way to filter the response message across all APIs, which reduces the size of the payload and hence allows faster transfer over network.

Changes

This feature add capabilities to filter the response message from across all APIs.
AIP detail:
https://google.aip.dev/157

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you review them:

  • Has Docs included if any changes are user facing
  • Has Tests included if any functionality added or changed
  • Tested your changes locally (if this is a code change)
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including functionality, content, code)
  • Has a kind label. You can add a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
  • Release notes block below has been updated with any user-facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
  • Release notes contain the string "action required" if the change requires additional action from users switching to the new release

Release Notes

API: Request message allows an additional parameter `fields` to filter the response message.

@sayan-biswas sayan-biswas added the kind/feature Categorizes issue or PR as related to a new feature. label Feb 21, 2024
@sayan-biswas sayan-biswas self-assigned this Feb 21, 2024
@tekton-robot
Copy link

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@tekton-robot tekton-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Feb 21, 2024
@tekton-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from sayan-biswas after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Feb 21, 2024
@tekton-robot tekton-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 28, 2024
@sayan-biswas sayan-biswas force-pushed the partial-response branch 2 times, most recently from 32027d4 to 08b524b Compare February 28, 2024 19:57
@sayan-biswas
Copy link
Contributor Author

/test all

@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
internal/fieldmask/fieldmask.go Do not exist 70.1%

@sayan-biswas sayan-biswas marked this pull request as ready for review February 28, 2024 20:45
@tekton-robot tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 28, 2024
@sayan-biswas
Copy link
Contributor Author

/test all

@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
internal/fieldmask/fieldmask.go Do not exist 70.1%

@sayan-biswas
Copy link
Contributor Author

/test pull-tekton-results-integration-tests

1 similar comment
@sayan-biswas
Copy link
Contributor Author

/test pull-tekton-results-integration-tests

Copy link
Contributor

@gabemontero gabemontero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

finally got around to taking a pass at this @sayan-biswas

of course, being brand new / a novice to all this, consider that perspective when reviewing my feedback ;-)

docs/api/README.md Outdated Show resolved Hide resolved
docs/api/README.md Outdated Show resolved Hide resolved
mask := make(FieldMask)
for _, path := range paths {
current := mask
fields := strings.Split(path, ".")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put a comment above this split with some sample paths to illustrate why we are splitting on .

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's not make the coder have to do up to your README change to undersatand what is going on

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added reference and an example in godoc format.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome thanks

internal/fieldmask/fieldmask.go Outdated Show resolved Hide resolved
internal/fieldmask/fieldmask_test.go Show resolved Hide resolved
internal/fieldmask/fieldmask.go Show resolved Hide resolved
internal/fieldmask/fieldmask.go Show resolved Hide resolved
reflect.Set(fd, protoreflect.ValueOfBytes(b))
}
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feels like we should have a default case to at least log unexpected messages, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say no, because this is simply filtering. Any unknown cases will just be ignored.
And for logging, there's are some restrictions here. We cannot throw error because then the client will receive the error. We cannot bring logger from the context, then the implementation will have dependency and can't be used as a library.
Although we can still log with normal golang's fmt or logger library, but that will break structured logging.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

valid reasoning

I still wonder about being able to debug in the future "why did my filter not catch 'X'"

could we emit events for the errors in this switch?

could we emit events for the default case? Or are there too many types besides MessageKind and BytesKind?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Emit the events where? You mean kubernetes events?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes k8s events .... api server has a k8s client so in theory could emit events to help with debug, and you don't have the logging pain you noted earlier

cmd/api/main.go Outdated
@@ -169,6 +171,7 @@ func main() {
grpc_zap.UnaryServerInterceptor(grpcLogger, zapOpts...),
grpc_auth.UnaryServerInterceptor(determineAuth),
prometheus.UnaryServerInterceptor,
fieldmask.UnaryServerInterceptor(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we have a feature enablement check of some sort ? presumably a client could send the extra headers or encoded data and it is OK to ignore ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added a feature flag implementation to handle all features control.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feature gate looks good @sayan-biswas good think .... in looking at it though I raised some questions around changes you made to my recently added tuning options

I'll submit a separate review for those

@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
internal/fieldmask/fieldmask.go Do not exist 70.1%

@sayan-biswas sayan-biswas force-pushed the partial-response branch 2 times, most recently from 0acbbac to 840e4cc Compare May 2, 2024 22:40
@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
cmd/api/main.go 9.3% 9.1% -0.2
internal/fieldmask/fieldmask.go Do not exist 70.1%
pkg/api/server/config/config.go Do not exist 0.0%
pkg/api/server/config/config.go Do not exist 0.0%
pkg/api/server/features/features.go Do not exist 85.3%

This feature add capabilities to filter the response message from all the APIs.
AIP detail:
https://google.aip.dev/157
@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
cmd/api/main.go 9.3% 9.1% -0.2
internal/fieldmask/fieldmask.go Do not exist 70.1%
pkg/api/server/config/config.go Do not exist 0.0%
pkg/api/server/config/config.go Do not exist 0.0%
pkg/api/server/features/features.go Do not exist 85.3%

Copy link
Contributor

@gabemontero gabemontero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a doc bit I think @sayan-biswas

@@ -6,13 +6,7 @@ DB_NAME=tekton-results
DB_SSLMODE=disable
DB_SSLROOTCERT=
DB_ENABLE_AUTO_MIGRATION=true
DB_MAX_IDLE_CONNECTIONS=10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm assuming you did this @sayan-biswas because I didn't need to set these defaults when I added the tuning options ? If so, thanks for the correction.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or was this accidental? ...If it was, in hindsight, I does probably make sense to not set these, even to the defaults. (though 10 does not line up with the default as described in the doc.

Users can always add these to their config map.

| DB_ENABLE_AUTO_MIGRATION | Auto-migrate the database on startup (create/update schemas). For further details, refer to <https://gorm.io/docs/migration.html> | true (default) |
| PROFILING | Enable profiling server | false (default) |
| PROFILING_PORT | Profiling Server Port | 6060 (default) |
| DB_MAX_IDLE_CONNECTIONS | The number of idle database connections to keep open | 2 (default for golang, but specific database drivers may have settings for this too) |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You deleted all my tuning options here in the README @sayan-biswas .... we need to minially document them, even if they are not specified in the config/base/env/config file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ohh.. Not sure why how this was missed. I'll rebase again to include the changes.

cmd/api/main.go Outdated
@@ -169,6 +171,7 @@ func main() {
grpc_zap.UnaryServerInterceptor(grpcLogger, zapOpts...),
grpc_auth.UnaryServerInterceptor(determineAuth),
prometheus.UnaryServerInterceptor,
fieldmask.UnaryServerInterceptor(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feature gate looks good @sayan-biswas good think .... in looking at it though I raised some questions around changes you made to my recently added tuning options

I'll submit a separate review for those

@gabemontero
Copy link
Contributor

gabemontero commented May 3, 2024

I don't seem to have permission to resolve comment threads @sayan-biswas

Feel free to resolve all of mine except #717 (review) and the one about submitting k8s events (at least until you respond to my clarification there).

@gabemontero
Copy link
Contributor

bump @sayan-biswas on my last round of comments - #717 (review)

Copy link
Contributor

@khrm khrm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sayan-biswas Can you rebase it again without removing Gabe's performance tuning changes?

@tekton-robot tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 15, 2024
@tekton-robot
Copy link

@sayan-biswas: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@sayan-biswas
Copy link
Contributor Author

@sayan-biswas Can you rebase it again without removing Gabe's performance tuning changes?

Sorry! I was caught up with other activities. I will rebase soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants