Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update TESTING.md #1283

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Conversation

aramaraju
Copy link
Contributor

Test plan

Added test plans for verifying JBs client works for Enterprise customers, specifically Cody Gateway and AWS Bedrock deployments

TESTING.md Outdated Show resolved Hide resolved
TESTING.md Outdated Show resolved Hide resolved
…ce and one for AWS Bedrock. Removed the validation on the configuration page for the Enterprise testing, the team will simply test against whatever configuration is on the instances
Copy link
Contributor

@dominiccooney dominiccooney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TL;DR:

  • Pointer to where to get creds.
  • Remove duplication by instructing testers to do a set of steps for each kind of instance.
  • More specificity in what those steps are.

TESTING.md Outdated Show resolved Hide resolved
#### Completions

1. Login to the Cody Enterprise Cody Gateway Instance
2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go).
2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go). A completion should appear.

@aramaraju

  • Document the keyboard shortcut to press or the name of the command so the testers can look up the keyboard shortcut.
  • This test should specify having a particular repo cloned and open, and mention specific files to open. It will make testing easier and make reproducing any bugs much easier.

@pkukielka does the tester's setup include other autocomplete providers? If so we need to ensure the completions are coming from Cody specifically somehow.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

opt \ is the manual trigger hotkey for autocomplete.

The QA testers have experience with this, I don't think we need to assume they have another completion provider. The Cody logo also "spins/loads" when it triggers a completion, so theres indication from the UI that it's coming from Cody

1. Login to Cody Enterprise account
2. Create new chat
3. User should see the default model in the dropdown but is unable to change it
1. Login to the Cody Enterprise Gateway Instance
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where's the credentials for this, if I wanted to perform these tests myself? Ditto below for "Cody Enterprise Cody Gateway Instance". Is that a different instance?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are they getting the credentials today? I thought they were already shared outside of this doc. @kalanchan consolidated them into 1Password

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"where can I get the credentials" kind of information should live in the handbook, but we can also add a link to 1password here. For the sake of completeness, let's just add it here 😄

Comment on lines +502 to +521
### Cody Enterprise - AWS Bedrock Instance

#### Chat

1. Login to the Cody Enterprise AWS Bedrock Instance
2. Create a new chat
3. User should not be able to change the LLM model on the chat window
4. Ask Cody a question in the new chat window.

#### Commands

1. Login to the Cody Enterprise AWS Bedrock Instance
2. Go to commands panel
3. Trigger command
4. Command should be executed

#### Completions

1. Login to the Cody Enterprise AWS Bedrock Instance
2. Generate a completion using the manual-trigger key binding on any primary languages (Javascript, Typescript, TypescriptReact, Python, Go).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be better if you removed the duplication and said something like: There are these kinds of instances (list them) and then had one set of instructions, like "For each kind of instance: ..."

Copy link
Contributor Author

@aramaraju aramaraju Apr 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing to note is that we are evaluating the LLM dropdown for Enterprise customers using Cody Gateway, which might not work as elegantly if combine the instance work. I agree combining them reads better, but I'm not sure if the team is going through a check list and having it split up by instances is easier in that case. If we do consolidate, shouldn't this also apply to the entire document? Do the following with these instances: Free, Pro, Enterprise 1, and 2? Will require a rewrite of the whole thing and I don't know why we didn't do that in the first place.

TESTING.md Outdated Show resolved Hide resolved
Co-authored-by: Dominic Cooney <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants