Feature Request: Add Paligemma support

[nischalj10]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Its a really solid model and has a lot of requests in the discussions

Motivation

Pulls way above its weight and has really good ocr capabilites

Possible Implementation

No response

[iamlemec]

Yup! Work in progress at #7553.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[TalonBvV]

Hey there, I hope all is well, any luck getting the model to run?

I'm quite curious, I managed to use Clip.cpp quantization to quantize phi-3 vision's projector and Llama.cpp quantization to quantize the language model component, the result was a pretty useful VLM with a total size of 1.5GB.
While this is great, because it proves that VLM models can be smaller and still accurate, but in terms of overall functionality Paligemma is a much better choice, it has baked in bounding box abilities which on it's own has limitless possibilities.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.