IndexError: The shape of the mask [1, 304] at index 1 does not match the shape of the indexed tensor [1, 624, 256] at index 1

[MM-Huang]

Hi, I run with the code ' CUDA_VISIBLE_DEVICES=0 python app.py --version=LISA-13B-llama2-v1 --load_in_4bit', but got this error, how can I fix them?

Traceback (most recent call last):
File "/home/xin/anaconda3/envs/lisa/lib/python3.8/site-packages/gradio/queueing.py", line 501, in call_prediction
output = await route_utils.call_process_api(
File "/home/xin/anaconda3/envs/lisa/lib/python3.8/site-packages/gradio/route_utils.py", line 258, in call_process_api
output = await app.get_blocks().process_api(
File "/home/xin/anaconda3/envs/lisa/lib/python3.8/site-packages/gradio/blocks.py", line 1684, in process_api
result = await self.call_function(
File "/home/xin/anaconda3/envs/lisa/lib/python3.8/site-packages/gradio/blocks.py", line 1250, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/xin/anaconda3/envs/lisa/lib/python3.8/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/home/xin/anaconda3/envs/lisa/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "/home/xin/anaconda3/envs/lisa/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 851, in run
result = context.run(func, *args)
File "/home/xin/anaconda3/envs/lisa/lib/python3.8/site-packages/gradio/utils.py", line 750, in wrapper
response = f(*args, **kwargs)
File "app.py", line 274, in inference
output_ids, pred_masks = model.evaluate(
File "/home/xin/PycharmProjects/LISA/model/LISA.py", line 383, in evaluate
pred_embeddings = last_hidden_state[seg_token_mask]
IndexError: The shape of the mask [1, 304] at index 1 does not match the shape of the indexed tensor [1, 624, 256] at index 1

[ShashankHuang]

I met the same error when I trained on reasonseg dataset.

[dreamingaa]

i met the same error . anybody fix it?

[pengfeiZhao1993]

any suggestion for this problem?

[min99830]

In my case <image> token is not in prompt.

Make sure your prompt contain <image> token

[pengfeiZhao1993]

@min99830
` prompt = input("Please input your prompt: ")
prompt = DEFAULT_IMAGE_TOKEN + "\n" + prompt
if args.use_mm_start_end:
replace_token = (
DEFAULT_IM_START_TOKEN + DEFAULT_IMAGE_TOKEN + DEFAULT_IM_END_TOKEN
)
prompt = prompt.replace(DEFAULT_IMAGE_TOKEN, replace_token)

    conv.append_message(conv.roles[0], prompt)
    conv.append_message(conv.roles[1], "")
    prompt = conv.get_prompt() `

Token will be added here. right?

and final prompt seems as below,

"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <im_start><im_end>\nPlease segment Lisa in this figure. ASSISTANT:"

[pengfeiZhao1993]

@min99830
seg_token_mask = output_ids[:, 1:] == self.seg_token_idx # hack for IMAGE_TOKEN_INDEX (we suppose that there is only one image, and it is in the front) seg_token_mask = torch.cat( [ torch.zeros((seg_token_mask.shape[0], 255)).bool().cuda(), seg_token_mask, ], dim=1, )