feat: Gemini Access via Vertex AI

[parhammmm]

@marcusschiesser here's the Vertex implementation. It's a little different than the others because with vertex there isn't any api key auth.

To authenticate for local development:

npm install @google-cloud/vertexai
gcloud auth application-default login

For production the preferred method is via a service account, more details: https://cloud.google.com/docs/authentication/ . The GeminiVertexSession accepts all the params supported by import { VertexAI } from "@google-cloud/vertexai";. e.g.

new GeminiVertexSession({
  location: "",
  project: "",
  googleAuthOptions: {...},
})

Google Auth Options takes all the values from GoogleAuthOptions in https://github.com/googleapis/google-auth-library-nodejs/blob/main/src/auth/googleauth.ts incase someone wants to directly pass their own keyFile or it's values via credentials

Once logged in you can test via:

export GOOGLE_VERTEX_PROJECT="" GOOGLE_VERTEX_LOCATION="us-central1" && npx ts-node ./gemini/chatVertex.ts

Some notes:

The @google-cloud/vertexai sdk is lagging behind @google/generative-ai, main differences affecting this PR:

• embeddings aren't supported. I've opened an issue Will this sdk support the equivalent of GenerativeModel.embedContent from @google/generative-ai googleapis/nodejs-vertexai#340
• vertexai doesn't have EnhancedGenerateContentResponse yet so added the missing pieces here too

[changeset-bot]

🦋 Changeset detected

Latest commit: a5114b7

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
llama-index-ts-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 17, 2024 4:24pm

[parhammmm]

Not sure I quite follow why the example checks are failing, would you have any ideas?

[marcusschiesser]

@parhammmm you have to run pnpm type-check to get more details about what is failing

[marcusschiesser]

Looks great thanks @parhammmm! Please fix the builld, then we can merge it

[marcusschiesser]

i guess that's not something that should ever happen - throw an exception?

[parhammmm]

makes sense, sure!

[marcusschiesser]

can you update https://github.com/run-llama/LlamaIndexTS/blob/main/apps/docs/docs/modules/llms/available_llms/gemini.md and add it there?

[parhammmm]

will do!

[parhammmm]

@marcusschiesser I found the issue; it's because Vertex can only be used via the backend and here webpack is attempting to build a node package https://github.com/googleapis/google-auth-library-nodejs for the browser.

Have you dealt with such cases before?

Update: if someone stumbles on this in the future, the solution is to add the package name to packages/core/src/next.ts

Update 2: also you'll have to make sure the export of the said module is done in packages/core/src/index.ts and not anywhere else so that the cloudflare worker doesn't break

[parhammmm]

@marcusschiesser done. I had to restructure the code a little bit to make sure node specific code doesn't break the browser/cloudflare builds.... on to bedrock implementation (pretty close)!

[parhammmm]

@marcusschiesser @himself65 how do I use exported items from packages/core/src/index.ts seems like they aren't actually exported during the build... I've tried with HuggingFaceEmbedding (and GeminiVertexSession) and in nextjs I get:

Attempted import error: 'HuggingFaceEmbedding' is not exported from 'llamaindex' (imported as 'HuggingFaceEmbedding').

UPDATE: Seems like this is only an issue when pnpm run build & pnpm run dev are ran locally the official release seems to work fine.

[marcusschiesser]

@parhammmm the exports that you add here are only used for NodeJS - so this change means VertexGemini does only work with NodeJS (not edge for example) - if that' ok, nothing to do - otherwise you can send a PR and move this to src/llm/index.ts

[marcusschiesser]

@parhammmm looks great thanks!