Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: usage of ReadDataFromJson in array tensors #7624

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

v-shobhit
Copy link

@v-shobhit v-shobhit commented Sep 18, 2024

What does the PR do?

The generate and generate_stream endpoints did not seem to work when directly querying TRTLLM backend with input tokens. This is because the HTTPAPIServer::GenerateRequestClass::ExactMappingInput does not send the correct size of an array input to ReadDataFromJson.

This PR also fixes triton-inference-server/tensorrtllm_backend#369

Checklist

  • I have read the Contribution guidelines and signed the Contributor License
    Agreement
  • PR title reflects the change and is of format <commit_type>: <Title>
  • Changes are described in the pull request.
  • Related issues are referenced.
  • Populated github labels field
  • Added test plan and verified test passes.
  • Verified that the PR passes existing CI.
  • I ran pre-commit locally (pre-commit install, pre-commit run --all)
  • Verified copyright is correct on all changed files.
  • Added succinct git squash message before merging ref.
  • All template sections are filled out.
  • Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

  • build
  • ci
  • docs
  • feat
  • fix
  • perf
  • refactor
  • revert
  • style
  • test

Related PRs:

Where should the reviewer start?

Test plan:

Added a new test case to L0_http job.
Internal CI pipeline id: 18800660

Caveats:

Background

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

  • closes GitHub issue: #xxx

@v-shobhit v-shobhit changed the title fix usage of ReadDataFromJson fix: usage of ReadDataFromJson in array tensors Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Infer failed: Unable to parse 'data': Shape does not match true shape of 'data' field in generate endpoint
4 participants