飞桨AI Studio可以玩多模态了？MiniGPT4实战演练！

MiniGPT4是基于GPT3的改进版本，它的参数量比GPT3少了一个数量级，但是在多项自然语言处理任务上的表现却不逊于GPT3。项目作者以MiniGPT4-7B作为实战演练项目。

创作者：衍哲

一键fork

fork该项目并运行，运行环境建议至少选择A100(40G)及以上配置

安装相关模块

import

os
2os

.system(

"pip install --pre --upgrade paddlenlp -f https://www.paddlepaddle.org.cn/whl/paddlenlp.html"

) # 安装nlp分支最新包

3os

.system(

"pip install paddlepaddle-gpu==0.0.0.post112 -f https://www.paddlepaddle.org.cn/whl/linux/gpu/develop.html"

)

4os

.system(

"pip install tqdm"

)

!pip install ipywidgets

引用相关模块

%%capture

os.environ[

"CUDA_VISIBLE_DEVICES"

] =

"0"
3

os.environ[

"FLAGS_use_cuda_managed_memory"

] =

"true"
4import

requests

5from

PIL

import

Image

6import

gradio

7from

tqdm

import

tqdm

8import

ipywidgets

widgets

9from

IPython.display

import

display

10import

csv

11from

itertools

import

islice

12from

paddlenlp.transformers

import

MiniGPT4ForConditionalGeneration, MiniGPT4Processor

下载miniGPT4权重或配置文件

1!mkdir minigpt4

%%capture

os.system(

"wget -O minigpt4/model_config.json https://bj.bcebos.com/v1/ai-studio-online/924ed883c17b4b8b88b4a1f98e24d34b3b00160ac9bd4b3ba478aff6974e0e9d?responseContentDisposition=attachment%3B%20filename%3Dmodel_config.json "

)

!wget -O ./minigpt4/model_state.pdparams https:

//bj

.bcebos.com/v1/ai-studio-online/

bd53eaa2854263ba31fb4d75f31a5f0d38421a6da64525bff6da230389fc36?responseContentDisposition=attachment%3B%20filename%3Dmodel_state.pdparams

!wget -O ./minigpt4/generation_config.json https:

//bj

.bcebos.com/v1/ai-studio-online/f0b2129d6a934a97abcaa139ac1f28e33a6940004c7a4c859737f282640cf332?responseContentDisposition=attachment%3B%20filename%3Dgeneration_config.json

!wget -O ./minigpt4/preprocessor_config.json https:

//bj

.bcebos.com/v1/ai-studio-online/

748

c332837d34f389d762f487470b1a7221edd36ccb5484b913bd2d3855ee9f6?responseContentDisposition=attachment%3B%20filename%3Dpreprocessor_config.json

!wget -O ./minigpt4/sentencepiece.bpe.model https:

//bj

.bcebos.com/v1/ai-studio-online/

013

9a1bfcdf84058b77cea4631837340ea94f5fcc37445929a3414f05d07579b?responseContentDisposition=attachment%3B%20filename%3Dsentencepiece.bpe.model

!wget -O ./minigpt4/special_tokens_map.json https:

//bj

.bcebos.com/v1/ai-studio-online/

b16a96d4f94200ab417b39dcf3bce4ddef5885625c4d0c8e70b3f659cb6993?responseContentDisposition=attachment%3B%20filename%3Dspecial_tokens_map.json

!wget -O ./minigpt4/tokenizer.json https:

//bj

.bcebos.com/v1/ai-studio-online/e877a685eb86499cb87e1c4cbf85353856506d12e9a841a292e780aa4a9e188a?responseContentDisposition=attachment%3B%20filename%3Dtokenizer.json

!wget -O ./minigpt4/tokenizer_config.json https:

//bj

.bcebos.com/v1/ai-studio-online/f93064db167c4075b1f86d6878cac9303fb8df418f7a42a7900785a6e188cc44?responseContentDisposition=attachment%3B%20filename%3Dtokenizer_config.json

2023

2710

-- https:

//bj

.bcebos.com/v1/ai-studio-online/

924

ed883c17b4b8b88b4a1f98e24d34b3b00160ac9bd4b3ba478aff6974e0e9d?responseContentDisposition=attachment%3B%20filename%3Dmodel_config.json

Resolving bj.bcebos.com (bj.bcebos.com)...

182.61

200.195

182.61

200.229

2409

c04:

1001

1002

:ff:b001:

368

Connecting to bj.bcebos.com (bj.bcebos.com)|

182.61

200.195

443

... connected.

HTTP request sent, awaiting response...

200

Length:

5628

(

5.5

K) [application/octet-stream]

Saving to:

'minigpt4/model_config.json'

实例化miniGPT4模型和处理器

model_path ='./minigpt4'

model = MiniGPT4ForConditionalGeneration.from_pretrained(model_path)

model.eval()

processor = MiniGPT4Processor.from_pretrained(model_path)

模型推理

输入图像url+prompt(单张图片+单轮对话)

另有本地上传图像形式，请进入项目查看

1defpredict_per_url_prompt(url=None,text=None):
2if

url==

None

url =

"https://paddlenlp.bj.bcebos.com/data/images/mugs.png"
4

image = Image.open(requests.get(url, stream=

True

).raw)

5if

text==

None

text =

"describe this image"
7
8

prompt =

"Give the following image: <Img>ImageContent</Img>. You will be able to see the image once I provide it to you. Please answer my questions.###Human: <Img><ImageHere></Img> <TextHere>###Assistant:"
9
10

inputs = processor([image], text, prompt)

11
12

generate_kwargs = {

13"max_length"

300

14"num_beams"

15"top_p"

1.0

16"repetition_penalty"

1.0

17"length_penalty"

18"temperature"

19"decode_strategy"

"greedy_search"

20"eos_token_id"

: [[

835

], [

2277

29937

]],

}

outputs = model.generate(**inputs, **generate_kwargs)

msg = processor.batch_decode(outputs[

])

24return

msg[

][

-5

]

将图像上传到本地后的file_path+prompt(多张图片+单轮对话)

def predict_dir_and_one_prompt_out_list(dir_path=None,text=None):

import

os 3assertos

path

.isdir(dir_path),

(

'请输入文件夹路径,而不是图像路径'

)

4output

= []

5for

per_image_name

tqdm (

.listdir(dir_path)):

image = Image.

open

(

path

.join(dir_path,per_image_name))

7if

text== None:

text =

"describe this image" 9else

text = text

1112

prompt =

inputs = processor([image], text, prompt)

1516

generate_kwargs = {

17"max_length"

300

18"num_beams"

19"top_p"

1.0

20"repetition_penalty"

1.0

21"length_penalty"

22"temperature"

23"decode_strategy"

"greedy_search"

24"eos_token_id"

[[835], [2277, 29937]]

}

outputs = model.generate(**inputs, **generate_kwargs)

msg = processor.batch_decode(outputs[

])

28output

.append(msg[

][

-5

])

29returnoutput

效果展示

输入：描述这张图片，使用中文

输出：这张图片显示了一个女性角色，穿着红色和白色的服装，手持一根金色的剑。她的头发是白色的，眼睛是红色的。她站在一张草地上，手持剑的柄子。这个角色看起来像是一个英雄，她的服装和装备显示出她的力量和勇气

predict_per_url_prompt(url=

'https://ai-studio-static-online.cdn.bcebos.com/d283b05404bd44b69b9be868fddb67616296858284bf4ad587e29432de66e930'

,text=

"描述这张图片,使用中文"

)

2'这张图片显示了一个女性角色，穿着红色和白色的服装，手持一根金色的剑。她的头发是白色的，眼睛是红色的。她站在一张草地上，手持剑的柄子。这个角色看起来像是一个英雄，她的服装和装备显示出她的力量和勇气'

更多玩法，可一键fork该项目进行模型微调。

点击底部阅读原文即可立即体验更多大模型应用。

关注【飞桨PaddlePaddle】公众号

获取更多技术内容~

继续阅读

阅读原文