--- language: - zh - en tags: - qwen pipeline_tag: text-generation inference: false --- # Qwen-VL-Chat-Int4


Qwen-VL 🤗 🤖  | Qwen-VL-Chat 🤗 🤖  (Int4: 🤗 🤖 ) | Qwen-VL-Plus 🤗 🤖  | Qwen-VL-Max 🤗 🤖 
Web   |    API   |    WeChat   |    Discord   |    Paper   |    Tutorial


**Qwen-VL** 是阿里云研发的大规模视觉语言模型(Large Vision Language Model, LVLM)。Qwen-VL 可以以图像、文本、检测框作为输入,并以文本和检测框作为输出。Qwen-VL 系列模型性能强大,具备多语言对话、多图交错对话等能力,并支持中文开放域定位和细粒度图像识别与理解。 **Qwen-VL** (Qwen Large Vision Language Model) is the visual multimodal version of the large model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud. Qwen-VL accepts image, text, and bounding box as inputs, outputs text and bounding box. The features of Qwen-VL include: 目前,我们提供了Qwen-VL和Qwen-VL-Chat两个模型,分别为预训练模型和Chat模型。如果想了解更多关于模型的信息,请点击[链接](https://github.com/QwenLM/Qwen-VL/blob/master/visual_memo.md)查看我们的技术备忘录。本仓库为Qwen-VL-Chat的量化模型Qwen-VL-Chat-Int4仓库。 We release Qwen-VL and Qwen-VL-Chat, which are pretrained model and Chat model respectively. For more details about Qwen-VL, please refer to our [technical memo](https://github.com/QwenLM/Qwen-VL/blob/master/visual_memo.md). This repo is the one for Qwen-VL-Chat-Int4.
## 安装要求 (Requirements) * python 3.8及以上版本 * pytorch2.0及以上版本 * 建议使用CUDA 11.4及以上 * python 3.8 and above * pytorch 2.0 and above are recommended * CUDA 11.4 and above are recommended
## 快速开始 (Quickstart) 我们提供简单的示例来说明如何利用 🤗 Transformers 快速使用Qwen-VL-Chat-Int4。 在开始前,请确保你已经配置好环境并安装好相关的代码包。最重要的是,确保你满足上述要求,然后安装相关的依赖库。 Below, we provide simple examples to show how to use Qwen-VL-Chat-Int4 with 🤗 Transformers. Before running the code, make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries. ```bash pip install -r requirements.txt pip install optimum git clone https://github.com/JustinLin610/AutoGPTQ.git & cd AutoGPTQ pip install -v . ``` 接下来你可以开始使用Transformers来使用我们的模型。关于视觉模块的更多用法,请参考[教程](TUTORIAL.md)。 Now you can start with Transformers. More usage aboue vision encoder, please refer to [tutorial](TUTORIAL_zh.md). #### 🤗 Transformers To use Qwen-VL-Chat-Int4 for the inference, all you need to do is to input a few lines of codes as demonstrated below. However, **please make sure that you are using the latest code.** ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch torch.manual_seed(1234) # Note: The default behavior now has injection attack prevention off. tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-VL-Chat-Int4", trust_remote_code=True) # use cuda device model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-VL-Chat-Int4", device_map="cuda", trust_remote_code=True).eval() # 1st dialogue turn query = tokenizer.from_list_format([ {'image': 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'}, {'text': '这是什么'}, ]) response, history = model.chat(tokenizer, query=query, history=None) print(response) # 图中是一名年轻女子在沙滩上和她的狗玩耍,狗的品种可能是拉布拉多。她们坐在沙滩上,狗的前腿抬起来,似乎在和人类击掌。两人之间充满了信任和爱。 # 2nd dialogue turn response, history = model.chat(tokenizer, '输出"击掌"的检测框', history=history) print(response) # 击掌(517,508),(589,611) image = tokenizer.draw_bbox_on_latest_picture(response, history) if image: image.save('1.jpg') else: print("no box") ```


## 量化 (Quantization) ### 效果评测 (Performance) 我们列出不同精度下模型在评测基准 **[TouchStone](https://github.com/OFA-Sys/TouchStone)** 上的表现,并发现量化模型并没有显著性能损失。结果如下所示: We illustrate the model performance of both BF16 and Int4 models on the benchmark **[TouchStone](https://github.com/OFA-Sys/TouchStone)**, and we find that the quantized model does not suffer from significant performance degradation. Results are shown below: | Quantization | ZH. | EN | | ------------ | :--------: | :-----------: | | BF16 | 401.2 | 645.2 | | Int4 | 386.6 | 651.4 | ### 推理速度 (Inference Speed) 我们测算了在输入一张图片(即258个token)的条件下BF16和Int4的模型生成1792 (2048-258) 和 7934 (8192-258) 个token的平均速度。 We measured the average inference speed (tokens/s) of generating 1792 (2048-258) and 7934 (8192-258) tokens with the context of an image (which takes 258 tokens) under BF16 precision and Int4 quantization, respectively. | Quantization | Speed (2048 tokens) | Speed (8192 tokens) | | ------------ | :-----------------: | :-----------------: | | BF16 | 28.87 | 24.32 | | Int4 | 37.79 | 34.34 | 推理速度测算是在单卡 A100-SXM4-80G GPU上运行,使用PyTorch 2.0.1及CUDA 11.4。 The profiling runs on a single A100-SXM4-80G GPU with PyTorch 2.0.1 and CUDA 11.4. ### GPU显存占用 (GPU Memory Usage) 我们还测算了在一张图片输入的条件下BF16和Int4模型生成1792 (2048-258) 和 7934 (8192-258) 个token所需显存。结果如下所示: We also profile the peak GPU memory usage for encoding 1792 (2048-258) tokens (including an image) as context (and generating single token) and generating 7934 (8192-258) tokens (with an image as context) under BF16 or Int4 quantization level, respectively. The results are shown below. | Quantization | Peak Usage for Encoding 2048 Tokens | Peak Usage for Generating 8192 Tokens | | ------------ | :---------------------------------: | :-----------------------------------: | | BF16 | 22.60GB | 28.01GB | | Int4 | 11.82GB | 17.23GB | 上述速度和显存测算使用[此脚本](https://qianwen-res.oss-cn-beijing.aliyuncs.com/profile_mm.py)完成。 The above speed and memory profiling are conducted using [this script](https://qianwen-res.oss-cn-beijing.aliyuncs.com/profile_mm.py).
## 评测 我们从两个角度评测了两个模型的能力: 1. 在**英文标准 Benchmark** 上评测模型的基础任务能力。目前评测了四大类多模态任务: - Zero-shot Caption: 评测模型在未见过数据集上的零样本图片描述能力; - General VQA: 评测模型的通用问答能力,例如判断题、颜色、个数、类目等问答能力; - Text-based VQA:评测模型对于图片中文字相关的识别/问答能力,例如文档问答、图表问答、文字问答等; - Referring Expression Compression:评测模型给定物体描述画检测框的能力; 2. **试金石 (TouchStone)**:为了评测模型整体的图文对话能力和人类对齐水平。我们为此构建了一个基于 GPT4 打分来评测 LVLM 模型的 Benchmark:TouchStone。在 TouchStone-v0.1 中: - 评测基准总计涵盖 300+张图片、800+道题目、27个类别。包括基础属性问答、人物地标问答、影视作品问答、视觉推理、反事实推理、诗歌创作、故事写作,商品比较、图片解题等**尽可能广泛的类别**。 - 为了弥补目前 GPT4 无法直接读取图片的缺陷,我们给所有的带评测图片提供了**人工标注的充分详细描述**,并且将图片的详细描述、问题和模型的输出结果一起交给 GPT4 打分。 - 评测同时包含英文版本和中文版本。 评测结果如下: We evaluated the model's ability from two perspectives: 1. **Standard Benchmarks**: We evaluate the model's basic task capabilities on four major categories of multimodal tasks: - Zero-shot Caption: Evaluate model's zero-shot image captioning ability on unseen datasets; - General VQA: Evaluate the general question-answering ability of pictures, such as the judgment, color, number, category, etc; - Text-based VQA: Evaluate the model's ability to recognize text in pictures, such as document QA, chart QA, etc; - Referring Expression Comprehension: Evaluate the ability to localize a target object in an image described by a referring expression. 2. **TouchStone**: To evaluate the overall text-image dialogue capability and alignment level with humans, we have constructed a benchmark called TouchStone, which is based on scoring with GPT4 to evaluate the LVLM model. - The TouchStone benchmark covers a total of 300+ images, 800+ questions, and 27 categories. Such as attribute-based Q&A, celebrity recognition, writing poetry, summarizing multiple images, product comparison, math problem solving, etc; - In order to break the current limitation of GPT4 in terms of direct image input, TouchStone provides fine-grained image annotations by human labeling. These detailed annotations, along with the questions and the model's output, are then presented to GPT4 for scoring. - The benchmark includes both English and Chinese versions. The results of the evaluation are as follows: Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has a more comprehensive coverage in terms of capability range.

### 零样本图像描述 & 通用视觉问答 (Zero-shot Captioning & General VQA)
Model type Model Zero-shot Captioning General VQA
NoCaps Flickr30K VQAv2dev OK-VQA GQA SciQA-Img
(0-shot)
VizWiz
(0-shot)
Generalist
Models
Flamingo-9B - 61.5 51.8 44.7 - - 28.8
Flamingo-80B - 67.2 56.3 50.6 - - 31.6
Unified-IO-XL 100.0 - 77.9 54.0 - - -
Kosmos-1 - 67.1 51.0 - - - 29.2
Kosmos-2 - 66.7 45.6 - - - -
BLIP-2 (Vicuna-13B) 103.9 71.6 65.0 45.9 32.3 61.0 19.6
InstructBLIP (Vicuna-13B) 121.9 82.8 - - 49.5 63.1 33.4
Shikra (Vicuna-13B) - 73.9 77.36 47.16 - - -
Qwen-VL (Qwen-7B) 121.4 85.8 78.8 58.6 59.3 67.1 35.2
Qwen-VL-Chat 120.2 81.0 78.2 56.6 57.5 68.2 38.9
Previous SOTA
(Per Task Fine-tuning)
- 127.0
(PALI-17B)
84.5
(InstructBLIP
-FlanT5-XL)
86.1
(PALI-X
-55B)
66.1
(PALI-X
-55B)
72.1
(CFR)
92.53
(LLaVa+
GPT-4)
70.9
(PALI-X
-55B)
- 在 Zero-shot Caption 中,Qwen-VL 在 Flickr30K 数据集上取得了 **SOTA** 的结果,并在 Nocaps 数据集上取得了和 InstructBlip 可竞争的结果。 - 在 General VQA 中,Qwen-VL 取得了 LVLM 模型同等量级和设定下 **SOTA** 的结果。 - For zero-shot image captioning, Qwen-VL achieves the **SOTA** on Flickr30K and competitive results on Nocaps with InstructBlip. - For general VQA, Qwen-VL achieves the **SOTA** under the same generalist LVLM scale settings. ### 文本导向的视觉问答 (Text-oriented VQA)
Model type Model TextVQA DocVQA ChartQA AI2D OCR-VQA
Generalist Models BLIP-2 (Vicuna-13B) 42.4 - - - -
InstructBLIP (Vicuna-13B) 50.7 - - - -
mPLUG-DocOwl (LLaMA-7B) 52.6 62.2 57.4 - -
Pic2Struct-Large (1.3B) - 76.6 58.6 42.1 71.3
Qwen-VL (Qwen-7B) 63.8 65.1 65.7 62.3 75.7
Specialist SOTAs
(Specialist/Finetuned)
PALI-X-55B (Single-task FT)
(Without OCR Pipeline)
71.44 80.0 70.0 81.2 75.0
- 在文字相关的识别/问答评测上,取得了当前规模下通用 LVLM 达到的最好结果。 - 分辨率对上述某几个评测非常重要,大部分 224 分辨率的开源 LVLM 模型无法完成以上评测,或只能通过切图的方式解决。Qwen-VL 将分辨率提升到 448,可以直接以端到端的方式进行以上评测。Qwen-VL 在很多任务上甚至超过了 1024 分辨率的 Pic2Struct-Large 模型。 - In text-related recognition/QA evaluation, Qwen-VL achieves the SOTA under the generalist LVLM scale settings. - Resolution is important for several above evaluations. While most open-source LVLM models with 224 resolution are incapable of these evaluations or can only solve these by cutting images, Qwen-VL scales the resolution to 448 so that it can be evaluated end-to-end. Qwen-VL even outperforms Pic2Struct-Large models of 1024 resolution on some tasks. ### 细粒度视觉定位 (Referring Expression Comprehension)
Model type Model RefCOCO RefCOCO+ RefCOCOg GRIT
val test-A test-B val test-A test-B val-u test-u refexp
Generalist Models GPV-2 - - - - - - - - 51.50
OFA-L* 79.96 83.67 76.39 68.29 76.00 61.75 67.57 67.58 61.70
Unified-IO - - - - - - - - 78.61
VisionLLM-H 86.70 - - - - - - -
Shikra-7B 87.01 90.61 80.24 81.60 87.36 72.12 82.27 82.19 69.34
Shikra-13B 87.83 91.11 81.81 82.89 87.79 74.41 82.64 83.16 69.03
Qwen-VL-7B 89.36 92.26 85.34 83.12 88.25 77.21 85.58 85.48 78.22
Qwen-VL-7B-Chat 88.55 92.27 84.51 82.82 88.59 76.79 85.96 86.32 -
Specialist SOTAs
(Specialist/Finetuned)
G-DINO-L 90.56   93.19 88.24 82.75 88.95 75.92 86.13 87.02 -
UNINEXT-H 92.64 94.33 91.46 85.24 89.63 79.79 88.73 89.37 -
ONE-PEACE 92.58 94.18 89.26 88.77 92.21 83.23 89.22 89.27 -
- 在定位任务上,Qwen-VL 全面超过 Shikra-13B,取得了目前 Generalist LVLM 模型上在 Refcoco 上的 **SOTA**。 - Qwen-VL 并没有在任何中文定位数据上训练过,但通过中文 Caption 数据和 英文 Grounding 数据的训练,可以 Zero-shot 泛化出中文 Grounding 能力。 我们提供了以上**所有**评测脚本以供复现我们的实验结果。请阅读 [eval/EVALUATION.md](eval/EVALUATION.md) 了解更多信息。 - Qwen-VL achieves the **SOTA** in all above referring expression comprehension benchmarks. - Qwen-VL has not been trained on any Chinese grounding data, but it can still generalize to the Chinese Grounding tasks in a zero-shot way by training Chinese Caption data and English Grounding data. We provide all of the above evaluation scripts for reproducing our experimental results. Please read [eval/EVALUATION.md](eval/EVALUATION.md) for more information. ### 闲聊能力测评 (Chat Evaluation) TouchStone 是一个基于 GPT4 打分来评测 LVLM 模型的图文对话能力和人类对齐水平的基准。它涵盖了 300+张图片、800+道题目、27个类别,包括基础属性、人物地标、视觉推理、诗歌创作、故事写作、商品比较、图片解题等**尽可能广泛的类别**。关于 TouchStone 的详细介绍,请参考[touchstone/README_CN.md](touchstone/README_CN.md)了解更多信息。 TouchStone is a benchmark based on scoring with GPT4 to evaluate the abilities of the LVLM model on text-image dialogue and alignment levels with humans. It covers a total of 300+ images, 800+ questions, and 27 categories, such as attribute-based Q&A, celebrity recognition, writing poetry, summarizing multiple images, product comparison, math problem solving, etc. Please read [touchstone/README_CN.md](touchstone/README.md) for more information. #### 英语 (English) | Model | Score | |---------------|-------| | PandaGPT | 488.5 | | MiniGPT4 | 531.7 | | InstructBLIP | 552.4 | | LLaMA-AdapterV2 | 590.1 | | mPLUG-Owl | 605.4 | | LLaVA | 602.7 | | Qwen-VL-Chat | 645.2 | #### 中文 (Chinese) | Model | Score | |---------------|-------| | VisualGLM | 247.1 | | Qwen-VL-Chat | 401.2 | Qwen-VL-Chat 模型在中英文的对齐评测中均取得当前 LVLM 模型下的最好结果。 Qwen-VL-Chat has achieved the best results in both Chinese and English alignment evaluation.
## 常见问题 (FAQ) 如遇到问题,敬请查阅 [FAQ](https://github.com/QwenLM/Qwen-VL/blob/master/FAQ_zh.md)以及issue区,如仍无法解决再提交issue。 If you meet problems, please refer to [FAQ](https://github.com/QwenLM/Qwen-VL/blob/master/FAQ.md) and the issues first to search a solution before you launch a new issue.
## 使用协议 (License Agreement) 研究人员与开发者可使用Qwen-VL和Qwen-VL-Chat或进行二次开发。我们同样允许商业使用,具体细节请查看[LICENSE](https://github.com/QwenLM/Qwen-VL/blob/master/LICENSE)。如需商用,请填写[问卷](https://dashscope.console.aliyun.com/openModelApply/qianwen)申请。 Researchers and developers are free to use the codes and model weights of both Qwen-VL and Qwen-VL-Chat. We also allow their commercial use. Check our license at [LICENSE](LICENSE) for more details.
## 引用 (Citation) 如果你觉得我们的论文和代码对你的研究有帮助,请考虑:star: 和引用 :pencil: :) If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil: :) ```BibTeX @article{Qwen-VL, title={Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities}, author={Bai, Jinze and Bai, Shuai and Yang, Shusheng and Wang, Shijie and Tan, Sinan and Wang, Peng and Lin, Junyang and Zhou, Chang and Zhou, Jingren}, journal={arXiv preprint arXiv:2308.12966}, year={2023} } ```
## 联系我们 (Contact Us) 如果你想给我们的研发团队和产品团队留言,请通过邮件(qianwen_opensource@alibabacloud.com)联系我们。 If you are interested to leave a message to either our research team or product team, feel free to send an email to qianwen_opensource@alibabacloud.com. ``` ```