Minor fix: `dff8` reference design

[luca-valente]

Hello,

I found a minor issue with the dff8 reference design.

The reference solution implements a

  initial
    q = 8'h0;

which makes the test fail when the LLM's answer is simply

module top_module (
  input clk,
  input [7:0] d,
  output reg [7:0] q
);

  always @(posedge clk)
    q <= d;

endmodule

which in my opinion is correct.

Thank you for your attention!!

Luca

[npfet]

Hi Luca, good catch- I will add this to a list of fixes to get in before an upcoming update. We have a handful of consistency fixes for the VerilogEval v2 release as well (I'd expect it to be pushed somewhere in the next month or two... we're mostly finalizing benchmarking SOTA LLMs against v2 internally before pushing the fixes). We will only maintain an errata list for the v1 release since we want to have a consistent dataset to compare against prior papers.

Thanks,

Nate

[luca-valente-sony]

Thank you for the feedback 😄

[npfet]

Sorry for the delay on this- this has been fixed along with some prompt refinements too.

[ggcr]

@npfet Are these changes that recently got added correspond to the ones stated in the VerilogCoder paper?

(https://arxiv.org/abs/2408.08927)

[npfet]

Hi @ggcr, yes- this is now complete. I was hoping to point over to the updated VerilogEval v2 arXiv paper as well for the updated tables and plots, but it's still under review there. I'd expect it to be up by the end of the week. I'll post here when it is available. The benchmark results will also be published in an upcoming ACM ToDAES journal issue.

[npfet]

@ggcr The new arXiv preprint is now available here: https://arxiv.org/abs/2408.11053