Requesting Support for phi-1_5 by Microsoft

[aiaicode]

It's a 1.3B SOTA model and competes with < 10B models.

https://huggingface.co/microsoft/phi-1_5
https://huggingface.co/microsoft/phi-1

Would be blazing fast with Llama.cpp.

[gardner]

Attempting to convert the pytorch model bin:

$ python3 convert.py ~/models/microsoft/phi-1
Loading model file /Users/gardner/models/microsoft/phi-1/pytorch_model.bin
Traceback (most recent call last):
  File "/Users/gardner/src/llama.cpp/convert.py", line 1208, in <module>
    main()
  File "/Users/gardner/src/llama.cpp/convert.py", line 1157, in main
    params = Params.load(model_plus)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gardner/src/llama.cpp/convert.py", line 288, in load
    params = Params.loadHFTransformerJson(model_plus.model, hf_config_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/gardner/src/llama.cpp/convert.py", line 203, in loadHFTransformerJson
    n_embd           = config["hidden_size"]
                       ~~~~~~^^^^^^^^^^^^^^^
KeyError: 'hidden_size'

From the model card:

Remark. In the generation function, our model currently does not support beam search (num_beams >1) and `attention_mask' parameters. Furthermore, in the forward pass of the model, we currently do not support outputing hidden states or attention values, or using custom input embeddings (instead of the model's).

[x4080]

Yes, this will be very cool if implemented with llama cpp

[matankley]

+1

[ggerganov]

Here is an example how to integrate it in llama.cpp:

#3187

• convert script
• build graph
• tokenizer (if necessary)

[gardner]

The phi-1 tokenizer is CodeGenTokenizer. This appears to be the same tokenizer used by the 1st CodeGen models from SalesForce.

Existing issue: #1299
Existing discussion: #2137

[aiaicode]

@wsxiaoys , Could you please help with this model as well? Thanks

[v3ss0n]

Is this supported now ?

[RozanskiT]

+1

[tom-adsfund]

This fine-tune (https://huggingface.co/Open-Orca/oo-phi-1_5) has better ARC than llama2 7b, and is trained on an Orca dataset that also allows ChatML contextualizing. Altogether very powerful. Would be a great addition to the llama.cpp ecosystem!

[rodas-j]

Still waiting for support here

[aiaicode]

@monatis If you have time to do your magic here.

[goerch]

@rodas-j , @aiaicode : do you think these are helpful comments?

[aiaicode]

@goerch Comments just show that people are still waiting for a solution after a month and that the issue needs attention.

[goerch]

the issue needs attention.

I'm happy that it has your attention and am waiting for your contribution then.