Enhancement: We need CogVLM support - extremely good image and text analysis, feels like a multi generational step forward.

[cmp-nct]

Discussed in #4350

^{Originally posted by cmp-nct December 7, 2023}
I've just seen CovVLM which is a Vicuna 7B language model behind a 9B vision tower (laion/CLIP-ViT-bigG-14-laion2B-39B-b160k) on a opensource license.
I've compared it with llava-1.5 (not even compareable) and Qwen-VL and it beats Qwen-VL by a margin in OCR abilities, detection of details and no or almost no hallucinations.
It understands handwritten as well as typed letters, context, fine details, background graphics
It can also locate tiny visual targets with pixel coordinates
I'm quite blown away that I didn't know it before..

I believe that this is what we need, it has similarities to llava but adds an additional expert model, so that's not super quick to implement.
In addition the ViT needs K-type quantization support.
Definitely worth a close look

URL: https://github.com/THUDM/CogVLM
Webdemo: http://36.103.203.44:7861/
Paper: https://github.com/THUDM/CogVLM/blob/main/assets/cogvlm-paper.pdf

Look at this example, I asked for a JSON representation - not cherry picked, it can actually extract all of the content with minimal errors:

The image appears to be a driver's license from California. Here's a JSON representation:
{
  "License_Number": "DL 11234568",
  "Expiration_Date": "08/31/2014",
  "Class": "C",
  "License_Holder": "Iva Cardholder",
  "Address": "2570 24th Street, Anytown, CA 95818",
  "Veteran": "YES",
  "Sex": "F",
  "Hair_Color": "Brown",
  "Eyes": "Brown",
  "Height": "125 lb",
  "Weight": "125 lb",
  "Issue_Date": "08/31/2009",
  "Birth_Date": "08/31/1977",
  "VISUALS": {
      "left": "Iva Cardholder",
      "right": "Iva Cardholder",
      "top": "California Bear",
      "bottom": "Gold Rush Miner"
    }
}

Here is what QWEN-VL does:

{
  "id": "123456",
  "issue_date": "08/31/2014",
  "expiration_date": "08/31/2024",
  "cardholder": {
    "name": "Ina Cordero",
    "gender": "F",
    "race": "A",
    "eye_color": "B",
    "hair_color": "R",
    "eyebrow_color": "N",
    "height": "5'6"",
    "weight": "126 lb"
  },
  "class": "C",
  "type": "DRIVER LICENSE",
  "state": "CA",
  "country": "USA"
}

Here is llava1.5-13B:

{
"image": "https://i.imgur.com/39vZv.jpg",
"description": "A California driver's license with a woman's picture on it. The license is blue and white and has a picture of a bear on it. The license number is 11324567890."
}

I've not yet looked into architectural challenges. But this is literally game changer..
That's seriously good OCR and its image detection abilities are beyond anything I've remotely seen from llava 1.5/ShareGPT4V

@monatis @FSSRepo

[mirek190]

I know about that model from some time and also mentioned here but NO ONE CARE ... why I do not know ....

[ericruleman]

+1 CogVLM is the best open source vision model currently available. Having a super powerful multi-modal LLM that's easy to run locally is a game changer.

I know that Ollama is looking to add CogVLM support, but they need llama.cpp to support it first.

[truebit]

+1 CogVLM/CogAgent is amazing on mobile UI dection and UI object detection.

[mirek190]

Is better than GPT4:V

[darkacorn]

+1 we need it ! asap

[Foul-Tarnished]

MobileVLM might be even better

[cmp-nct]

MobileVLM might be even better

There is no demo of it online, it uses the same vision encoder and similar projection as llava just a tiny llm instead of the 7B vicuna.
It is trained on llava-1.5 data (so lacks a lot of sharegptv4 knowledge) but claiming that tiny thing is at eye level with GPT4V needs some more evidence than 5 words. The benchmarks do not support it.

I didn't test it on llama.cpp but my guess is that it requires minimal changes to get the language model supported - the projection has small changes as well (normalization)
Regarding support: the authors actually already had it working in llama.cpp according to the paper (mentioned using Q4 on the llm) but didn't release the changes as a fork or PR for some reason ?

I'm not saying it is not what you claim - just from what I've seen at a first view I find it highly unlikely. Would be a huge development in showcasing what the small CLIP can do despite everyone else not being able to do the same.

I believe MobileVLM is worthy of support, it's tiny and appears to be a little bit worse than llava-1.5 but of course much faster.
Should not distract from CogVLM being the best open source one

[darkacorn]

cogvlm is far better then llava - llava already works on most - so please lets stick with cogvlm if anyone would embark on that - as it takes about 80g vram here in fp16 .. and bnb isnt cutting it

[darkacorn]

pong - how does that not get any traction ?!
the main example given is worthless as thats simple ocr - but cogvlm is so much better then llava on every vertical

[dtiarks]

I started looking into it. But have many on my schedule currently.

[husnoo]

@darkacorn, I'd like to test it, what's your branch called?

…

On Wed, 10 Jan 2024, 09:41 darkacorn, ***@***.***> wrote: pong - how does that not get any traction ?! — Reply to this email directly, view it on GitHub <#4387 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAWJWSS6SVTNMMUKMI23ULDYNZO25AVCNFSM6AAAAABANM23OOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBUGUYDCNBVG4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[darkacorn]

https://github.com/THUDM/CogVLM no branch not affiliated with them either

[darkacorn]

I started looking into it. But have many on my schedule currently.

understandable - i talked to turboderp(exllama) and casper (autoawq) too .. apparently its quite a bit of work to get a quant / inference going outside of the regular transformer arch

[dtiarks]

Jip that's also my feeling. To make the deep feature fusion work you have to provide an additonal mask as input. That's quite different from the usual stuff.