InternLM-XComposer2-4KHD

[💻Github Repo](https://github.com/InternLM/InternLM-XComposer) [Paper](https://arxiv.org/abs/2401.16420)

**InternLM-XComposer2-4KHD** is a general vision-language large model (VLLM) based on [InternLM2](https://github.com/InternLM/InternLM), with the capability of 4K resolution image understanding. ### Import from Transformers To load the InternLM-XComposer2-4KHD model using Transformers, use the following code: ```python import torch from transformers import AutoTokenizer, AutoModelForCausalLM ckpt_path = "internlm/internlm-xcomposer2-4khd-7b" tokenizer = AutoTokenizer.from_pretrained(ckpt_path, trust_remote_code=True).cuda() # Set `torch_dtype=torch.floatb16` to load model in bfloat16, otherwise it will be loaded as float32 and might cause OOM Error. model = AutoModelForCausalLM.from_pretrained(ckpt_path, torch_dtype=torch.bfloat16, trust_remote_code=True).cuda() model = model.eval() ``` ## Quickstart We provide a simple example to show how to use InternLM-XComposer with 🤗 Transformers. ```python import torch from transformers import AutoModel, AutoTokenizer torch.set_grad_enabled(False) # init model and tokenizer model = AutoModel.from_pretrained('internlm/internlm-xcomposer2-4khd-7b', torch_dtype=torch.bfloat16, trust_remote_code=True).cuda().eval() tokenizer = AutoTokenizer.from_pretrained('internlm/internlm-xcomposer2-4khd-7b', trust_remote_code=True) ############### # First Round ############### query1 = 'Illustrate the fine details present in the image' image = './example.webp' with torch.cuda.amp.autocast(): response, his = model.chat(tokenizer, query=query, image=image, hd_num=55, history=[], do_sample=False, num_beams=3) print(response) # The image is a vibrant and colorful infographic that showcases 7 graphic design trends that will dominate in 2021. The infographic is divided into 7 sections, each representing a different trend. # Starting from the top, the first section focuses on "Muted Color Palettes", highlighting the use of muted colors in design. # The second section delves into "Simple Data Visualizations", emphasizing the importance of easy-to-understand data visualizations. # The third section introduces "Geometric Shapes Everywhere", showcasing the use of geometric shapes in design. # The fourth section discusses "Flat Icons and Illustrations", explaining how flat icons and illustrations are being used in design. # The fifth section is dedicated to "Classic Serif Fonts", illustrating the resurgence of classic serif fonts in design. # The sixth section explores "Social Media Slide Decks", illustrating how slide decks are being used on social media. # Finally, the seventh section focuses on "Text Heavy Videos", illustrating the trend of using text-heavy videos in design. # Each section is filled with relevant images and text, providing a comprehensive overview of the 7 graphic design trends that will dominate in 2021. ############### # Second Round ############### query1 = 'what is the detailed explanation of the third part.' with torch.cuda.amp.autocast(): response, _ = model.chat(tokenizer, query=query1, image=image, hd_num=55, history=his, do_sample=False, num_beams=3) print(response) # The third part of the infographic is about "Geometric Shapes Everywhere". It explains that last year, designers used a lot of # flowing and abstract shapes in their designs. However, this year, they have been replaced with rigid, hard-edged geometric # shapes and patterns. The hard edges of a geometric shape create a great contrast against muted colors. ``` ### Open Source License The code is licensed under Apache-2.0, while model weights are fully open for academic research and also allow free commercial usage. To apply for a commercial license, please fill in the application form (English)/申请表（中文）. For other questions or collaborations, please contact internlm@pjlab.org.cn.