Skip to content

[Feature Request] Add API for manually setting logits. #1024

Closed
@shaunabanana

Description

@shaunabanana

Prerequisites

Please answer the following questions for yourself before submitting an issue.

  • I am running the latest code. Development is very rapid so there are no tagged versions as of now.
  • I carefully followed the README.md.
  • I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
  • I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Add a public API (e.g., llama_set_logits(ctx, logits)) that allows to manually set the logits. This would open the door to bridging llama.cpp with use cases like FUDGE (code) and Grounded Decoding, which guides the generation process by manipulating the probabilities without the need for fine-tuning.

Current Behavior

I understand that there is currently an API function for getting the current logits (llama_get_logits(ctx)), but no way to modify the logits through a public API.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions