How to change the gpt4v model in the code to another model? Such as Gemini pro vision?

[Linermao]

As the title says. I'd really appreciate it if you have any thoughts or success codes!

[Linermao]

After a few days of exploration, I have some alternative methods to Gemini that can be referenced.

But there are still a few minor issues, although it is now able to open files and carry out your requests.

If you have any suggestions for modifications, feel free to discuss them here.

1. Use Gemini API

First, change all openai to gemini, don't forget config.yaml.

# In model.py, before use ask_gpt4v(), we should 
 
configs = load_config()
configs["GEMINI_API_BASE"] = f"https://generativelanguage.googleapis.com/v1/models/{configs["GEMINI_API_MODEL"]}:generateContent?key={configs["GEMINI_API_KEY"]}"
def ask_gpt4v(content):
    xxxx
    xxxx

Solutions to Some Possible Issues in China:

# If you get 503 error, try this. You should run your VPN device first.

#cd /home/{USERNAME}/miniconda3/envs/{YOUR_CONDA_ENV_NAME}/lib/python3.12/site-packages/google/ai/generativelanguage_v1beta/services/generative_service/transports

#find grpc.py and grpc_asyncio.py

#GenerativeServiceGrpcTransport ---> init ---> if not self._grpc_channel: ---> options ---> add "('grpc.http_proxy', 'http://127.0.0.1:7890')"  to use localhost VPN

refer to : https://zhuanlan.zhihu.com/p/673100444

2. Change aks_gpt4v() in model.py

2.1 change "header" and "payload"

# delete "Authorization"
    headers = {
        "Content-Type": "application/json",
    }

# change structure
    payload = {
        "contents": content,
        "generationConfig": {
            "temperature": configs["TEMPERATURE"],
            "maxOutputTokens": configs["MAX_TOKENS"]
        }
    }

2.2 delete spare code

# delete it
if "error" not in response.json():
        usage = response.json()["usage"]
        prompt_tokens = usage["prompt_tokens"]
        completion_tokens = usage["completion_tokens"]
        print_with_color(f"Request cost is "
                         f"${'{0:.2f}'.format(prompt_tokens / 1000 * 0.01 + completion_tokens / 1000 * 0.03)}",
                         "yellow")

3. Change content in self_explorer.py

# line 114
content = [
        {
            "parts": [
                {"text": prompt},
                {
                    "inline_data": {
                        "mime_type": "image/jpeg",
                        "data": base64_img_before
                        }
                }
            ]
        },
    ]

# line 205
content = [
        {
            "parts": [
                {"text": prompt},
                {
                    "inline_data": {
                        "mime_type": "image/jpeg",
                        "data": base64_img_before
                        },
                    "inline_data": {
                        "mime_type": "image/jpeg",
                        "data": base64_img_after
                        }
                }
            ]
        },
    ]

4. Change msg in model.py

def parse_explore_rsp(rsp): 
    try:
        #msg = rsp["choices"][0]["message"]["content"]
        msg = rsp['candidates'][0]['content']['parts'][0]['text']

def parse_grid_rsp(rsp):
    try:
        # msg = rsp["choices"][0]["message"]["content"]
        msg = rsp['candidates'][0]['content']['parts'][0]['text']

def parse_reflect_rsp(rsp):
    try:
        #msg = rsp["choices"][0]["message"]["content"]
        msg = rsp['candidates'][0]['content']['parts'][0]['text']