Multi-Modal | Parasail

Multimodal

Multimodal (or Multi-Modal) refers to systems or methods that can process or integrate multiple types of data, typically from different modalities such as text, image, audio, video, and other sensory inputs. Multi-modal systems aim to understand and generate outputs based on this diverse, merged information, mimicking human perception, where we simultaneously use different senses (for example, seeing, hearing, and reading) to interpret the world.

In machine learning and AI, multi-modal models combine data from different modalities to improve the performance of tasks like classification, generation, captioning, and more. For example, an AI model that processes both images and text could interpret and respond to visual and written information in a cohesive and context-aware manner.

Quickstart

The simplest way to use Vision Language Model is using OpenAI-compatible API and embed image input as base64 encoded data URLs.

Below is an example using Qwen2.5-VL 72 B model with Parasail's serverless API. GLM 4.5V also works with this example.

!pip install openai > /dev/null

import base64
import openai
from IPython.display import Image, display

try:
  from google.colab import userdata
  API_KEY = userdata.get('PARASAIL_API_KEY')
except:
  import os
  API_KEY = os.getenv('PARASAIL_API_KEY')

!wget https://upload.wikimedia.org/wikipedia/commons/thumb/a/a5/Flower_poster_2.jpg/1540px-Flower_poster_2.jpg -O flower.jpg &> /dev/null
display(Image("flower.jpg", width=500))

def data_url(image):
    with open(image, "rb") as f:
        return f"data:image/jpeg;base64," + base64.b64encode(f.read()).decode("ascii")

oai_client = openai.OpenAI(base_url="https://api.parasail.io/v1", api_key=API_KEY)

messages = [{
    "role": "user",
    "content": [
        { "type": "image_url", "image_url": { "url": data_url("flower.jpg")}},
        {"type": "text", "text": "What's in the image?"},
    ],
}]

chat_completion = oai_client.chat.completions.create(
    model="parasail-qwen25-vl-72b-instruct", messages=messages)

print(chat_completion.choices[0].message.content)

The image is a collage of twelve different types of flowers, each labeled with its common name and scientific name. Here is a list of the flowers:

St Bernard's Lily (Anthericum liliago)
Bermuda Buttercup (Oxalis pes-caprae)
Oleander (Nerium oleander)
Lantana (Lantana camara)
Scarlet Pimpernel (Anagallis arvensis)
Verbascum (Verbascum sinuatum)
Common Mallow (Malva sylvestris)
Spanish Oyster (Scolymus hispanicum)
Stork's bill (Erodium malacoides)
Bindweed (Convolvulus arvensis)
Blue Gem (Hebes × franciscana)
Calla Lily (Zantedeschia aethiopica)

Each flower is depicted with vibrant colors and detailed features, showcasing the diversity and beauty of the flora.

Vision Capabilities

Our public serverless model can support up to 8 input images for multimodal LLMs. For private dedicated deployments, the model supports 2 input images for RTX4090s and 8 input images for A100s, H100s, and H200s. If you need a different configuration than this than please contact us.

Multiple Images: Document Visual Question Answering Example

To process multiple images, you can simply add multiple image_url items into the content list. The images are referenced in the order they are present in the content list and in the prompt.

!apt-get install -qq -y poppler-utils &> /dev/null
!pip install pdf2image &> /dev/null
import pdf2image

!wget https://arxiv.org/pdf/1706.03762 -O transformer.pdf &> /dev/null

# Use multiple images as input
messages = []
for i, page in enumerate(pdf2image.convert_from_path("transformer.pdf", last_page=4)):
    page.save(f"page_{i}.jpg", 'JPEG')
    messages.append({"type": "image_url", "image_url": { "url": data_url(f"page_{i}.jpg")}})

messages = [{
    "role": "user",
    "content": messages + [{
        "type": "text",
        "text": "How many of encoder layers were used and where in the paper it's mentioned?"
    }]
}]

chat_completion = oai_client.chat.completions.create(
    model="parasail-qwen25-vl-72b-instruct", messages=messages)

display(Image("page_0.jpg", width=500))
print(chat_completion.choices[0].message.content)

Video Understanding

Qwen-2-VL and Qwen-2.5-VL support the video_url content type, where the data URL is a series of encoded base64 PNG frames separated by a comma. The following notebook shows how to load an MP4 video, reorder the channels correctly in a Torch format, resize the video, and compile the data URL.

!pip install torch torchvision opencv-python &> /dev/null
from IPython.display import Video

import os
import cv2
import gc
import math
import openai
import textwrap
import torch
import torchvision

FPS = 2  # Video frame sample rate
MIN_FRAMES = 4
MAX_FRAMES = 110  #
FRAME_FACTOR = 2  #

IMAGE_FACTOR = 28  # Image patch size
VIDEO_MIN_PIXELS = 128 * 28 * 28  # Min number of pixels of a single video frame
VIDEO_MAX_PIXELS = 768 * 28 * 28  # Max number of pixels of a single video frame
VIDEO_TOTAL_PIXELS = 12000 * 28 * 28  # Max number of video pixels in a request
MAX_RATIO = 200  # Aspect ratio limit
MAX_GATEWAY_STRING_LEN = 20_000_000

... # Additional function definitions and processing here...

print(f"Processing snowman.webm...")
Video("snowman.webm", embed=True)
video_url, frames = process_video("snowman.webm")

Object Detection and Localization

Models like Qwen2.5-VL series and PaliGemma2 are trained with object detection and localization capabilities. Simply prompt the model with the description of the objects of interest, the models can output object bounding boxes in the specified format.

import json
import PIL
import PIL.ImageDraw

messages = [{
    "role": "user",
    "content": [
        { "type": "image_url", "image_url": { "url": data_url("page_0.jpg")}},
        {"type": "text", "text": "Locate the abstract section, output its bbox coordinates using JSON format."},
    ],
}]

chat_completion = oai_client.chat.completions.create(
    model="parasail-qwen25-vl-72b-instruct", messages=messages)
print(chat_completion.choices[0].message.content)

json_string = chat_completion.choices[0].message.content.removeprefix("```json").removesuffix("```")
data = json.loads(json_string)

img = PIL.Image.open("page_0.jpg")

for bbox in data:
  PIL.ImageDraw.Draw(img).rectangle(bbox["bbox_2d"], outline='red')

img.save("page_0_bbox.jpg")
display(Image("page_0_bbox.jpg", width=500))