Closed
Description
Since #5971 I tried requantizing IQ1_S of this model, using the same imatrix as before, however, where the following worked as expected 75% of the time (and the rest of the time it just gave the wrong output):
./main --log-disable --no-display-prompt -t 7 -ngl 35 -m gorilla-openfunctions-v2.IQ1_S.gguf --color -c 16384 --temp 0 -p "You are an AI programming assistant, utilizing the Gorilla LLM model, developed by Gorilla LLM, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer."$'\n''### Instruction: <<function>>[{"name":"get_current_weather","description":"Get the current weather in a given location","parameters":{"type":"object","properties":{"location":{"type":"string","description":"The city and state, e.g. San Francisco, CA",},"unit":{"type":"string","enum":["celsius","fahrenheit"]},},"required":["location"]}}]'$'\n'"<<question>>What's the weather like in Oslo?"$'\n'"### Response: "The newly quantized version just outputs gibberish like this, every time:
45° CelsiusIEEEeqnarray---classvrtexmalinkmalinkndefinedndefinedndefinedndefined---Título:Taxonomia螃---Título:Taxonomia
Which seems like a pretty massive regression, any idea what's going on?