Problem and in-context learning prompt fixes

Author: npfet

dataset_code-complete-iccad2023/Prob034_dff8_ref.sv

@@ -6,7 +6,7 @@ module RefModule (
 );
 
   initial
-    q = 8'h0;
+    q = 8'hx;
 
   always @(posedge clk)
     q <= d;

pass_rate_to_csv.py

@@ -17,7 +17,7 @@
 data = result.stdout
 
 # Parse the output data
-pattern = re.compile(r'\./build_(\S+)_(\S+)_shots(\d+)_n(\d+)/summary\.txt:pass_rate\s+=\s+(\d+\.\d+)')
+pattern = re.compile(r'\./build_([^_\s]+)_(\S+)_shots(\d+)_n(\d+)/summary\.txt:pass_rate\s+=\s+(\d+\.\d+)')
 matches = pattern.findall(data)
 
 # Organize data into a dictionary

scripts/sv-generate

@@ -78,6 +78,7 @@ openai_models = [
   "gpt-3.5-turbo",
   "gpt-4",
   "gpt-4-turbo",
+  "gpt-4o",
 ]
 
 nim_chat_models = [
@@ -95,7 +96,9 @@ nim_chat_models = [
 ]
 
 manual_models = [
-  'manual-rtl-coder'
+  'manual-rtl-coder',
+  'manual-deepseek-coder-6.7b',
+  'manual-deepseek-coder-33b'
 ]
 
 model_aliases = {
@@ -137,7 +140,7 @@ prompts = {}
 
 prompts['code-complete-iccad2023'] = {
   'system_msg' : """
-You only complete chats with syntax correct Verilog code. End the Verilog module code completion with 'endmodule'. Do not include module, input and output definitions
+You only complete chats with syntax correct Verilog code. End the Verilog module code completion with 'endmodule'. Do not include module, input and output definitions.
 """,
   'prompt_prefix' : """
 // Implement the Verilog module based on the following description. Assume that signals are positive clock/clk triggered unless otherwise stated.
@@ -146,8 +149,7 @@ You only complete chats with syntax correct Verilog code. End the Verilog module
 
 prompts['spec-to-rtl'] = {
   'system_msg' : """
-You are a Verilog RTL designer that only writes code using correct
-Verilog syntax.
+You are a Verilog RTL designer that only writes code using correct Verilog syntax.
 """,
   'prompt_prefix' : ""
 }
@@ -161,17 +163,17 @@ Verilog syntax.
 prompt_rules_suffix="""
 Here are some additional rules and coding conventions.
 
- - declare all ports and signals as logic; do not to use wire or reg
+ - Declare all ports and signals as logic; do not to use wire or reg.
 
- - for combinational logic with an always block do not explicitly specify
-   the sensitivity list; instead use always @(*)
+ - For combinational logic with an always block do not explicitly specify
+   the sensitivity list; instead use always @(*).
 
- - all sized numeric constants must have a size greater than zero
-   (e.g, 0'b0 is not a valid expression)
+ - All sized numeric constants must have a size greater than zero
+   (e.g, 0'b0 is not a valid expression).
 
- - an always block must read at least one signal otherwise it will never
+ - An always block must read at least one signal otherwise it will never
    be executed; use an assign statement instead of an always block in
-   situations where there is no need to read any signals
+   situations where there is no need to read any signals.
 
  - if the module uses a synchronous reset signal, this means the reset
    signal is sampled with respect to the clock. When implementing a

scripts/verilog-example-prefix_code-complete-iccad2023_1-shot.txt

@@ -1,4 +1,4 @@
-// Implement the Verilog module based on the following description. Assume that sigals are positive clock/clk triggered unless otherwise stated.
+// Implement the Verilog module based on the following description. Assume that signals are positive clock/clk triggered unless otherwise stated.
 //
 // The module should implement an incrementer which increments the input by
 // one and writes the result to the output. Assume all values are encoded as

scripts/verilog-example-prefix_code-complete-iccad2023_2-shot.txt

@@ -1,4 +1,4 @@
-// Implement the Verilog module based on the following description. Assume that sigals are positive clock/clk triggered unless otherwise stated.
+// Implement the Verilog module based on the following description. Assume that signals are positive clock/clk triggered unless otherwise stated.
 //
 // The module should implement an incrementer which increments the input by
 // one and writes the result to the output. Assume all values are encoded as
@@ -16,7 +16,7 @@ module TopModule
 
 endmodule
 
-// Implement the Verilog module based on the following description. Assume that sigals are positive clock/clk triggered unless otherwise stated.
+// Implement the Verilog module based on the following description. Assume that signals are positive clock/clk triggered unless otherwise stated.
 //
 // The module should implement an 8-bit registered incrementer with an
 // active-high synchronous reset. The 8-bit input is first registered and

scripts/verilog-example-prefix_code-complete-iccad2023_3-shot.txt

@@ -1,4 +1,4 @@
-// Implement the Verilog module based on the following description. Assume that sigals are positive clock/clk triggered unless otherwise stated.
+// Implement the Verilog module based on the following description. Assume that signals are positive clock/clk triggered unless otherwise stated.
 //
 // The module should implement an incrementer which increments the input by
 // one and writes the result to the output. Assume all values are encoded as
@@ -16,7 +16,7 @@ module TopModule
 
 endmodule
 
-// Implement the Verilog module based on the following description. Assume that sigals are positive clock/clk triggered unless otherwise stated.
+// Implement the Verilog module based on the following description. Assume that signals are positive clock/clk triggered unless otherwise stated.
 //
 // The module should implement an 8-bit registered incrementer with an
 // active-high synchronous reset. The 8-bit input is first registered and
@@ -58,7 +58,7 @@ module TopModule
 
 endmodule
 
-// Implement the Verilog module based on the following description. Assume that sigals are positive clock/clk triggered unless otherwise stated.
+// Implement the Verilog module based on the following description. Assume that signals are positive clock/clk triggered unless otherwise stated.
 //
 // Build a finite-state machine that takes as input a
 // serial bit stream and outputs a one whenever the bit stream contains two

scripts/verilog-example-prefix_code-complete-iccad2023_4-shot.txt

@@ -1,4 +1,4 @@
-// Implement the Verilog module based on the following description. Assume that sigals are positive clock/clk triggered unless otherwise stated.
+// Implement the Verilog module based on the following description. Assume that signals are positive clock/clk triggered unless otherwise stated.
 //
 // The module should implement a XOR gate.
 
@@ -13,7 +13,7 @@ module TopModule
 
 endmodule
 
-// Implement the Verilog module based on the following description. Assume that sigals are positive clock/clk triggered unless otherwise stated.
+// Implement the Verilog module based on the following description. Assume that signals are positive clock/clk triggered unless otherwise stated.
 //
 // The module should implement an 8-bit registered incrementer. The 8-bit
 // input is first registered and then incremented by one on the next cycle.
@@ -54,7 +54,7 @@ module TopModule
 
 endmodule
 
-// Implement the Verilog module based on the following description. Assume that sigals are positive clock/clk triggered unless otherwise stated.
+// Implement the Verilog module based on the following description. Assume that signals are positive clock/clk triggered unless otherwise stated.
 //
 // The module should implement an n-bit registered incrementer where the
 // bitwidth is specified by the parameter nbits. The n-bit input is first
@@ -99,7 +99,7 @@ module TopModule
 endmodule
 
 
-// Implement the Verilog module based on the following description. Assume that sigals are positive clock/clk triggered unless otherwise stated.
+// Implement the Verilog module based on the following description. Assume that signals are positive clock/clk triggered unless otherwise stated.
 //
 // Build a finite-state machine that takes as input a
 // serial bit stream and outputs a one whenever the bit stream contains two

scripts/verilog-example-prefix_spec-to-rtl_1-shot default.txt

@@ -16,11 +16,11 @@ Assume all sequential logic is triggered on the positive edge of the
 clock. The reset input is active high synchronous and should reset the
 output to zero.
 
-Enclose your code with <CODE> and </CODE>. Only output the code snippet
+Enclose your code with [BEGIN] and [DONE]. Only output the code snippet
 and do NOT output anything else.
 
 Answer:
-<CODE>
+[BEGIN]
 module TopModule
 #(
   parameter nbits
@@ -55,4 +55,4 @@ module TopModule
   assign out = temp_wire;
 
 endmodule
-</CODE>
+[DONE]

scripts/verilog-example-prefix_spec-to-rtl_2-shot.txt

@@ -43,7 +43,7 @@ reset to zero when the reset input is one. Assume all values are encoded
 as two's complement binary numbers. Assume all sequential logic is
 triggered on the positive edge of the clock.
 
-Enclose your code with <CODE> and </CODE>. Only output the code snippet
+Enclose your code with [BEGIN] and [DONE]. Only output the code snippet
 and do NOT output anything else.
 
 Answer:

scripts/verilog-example-prefix_spec-to-rtl_3-shot.txt

@@ -43,7 +43,7 @@ reset to zero when the reset input is one. Assume all values are encoded
 as two's complement binary numbers. Assume all sequential logic is
 triggered on the positive edge of the clock.
 
-Enclose your code with <CODE> and </CODE>. Only output the code snippet
+Enclose your code with [BEGIN] and [DONE]. Only output the code snippet
 and do NOT output anything else.
 
 Answer:
@@ -100,7 +100,7 @@ Assume all sequential logic is triggered on the positive edge of the
 clock. The reset input is active high synchronous and should reset the
 finite-state machine to an appropriate initial state.
 
-Enclose your code with <CODE> and </CODE>. Only output the code snippet
+Enclose your code with [BEGIN] and [DONE]. Only output the code snippet
 and do NOT output anything else.
 
 Answer:

scripts/verilog-example-prefix_spec-to-rtl_4-shot.txt

@@ -9,11 +9,11 @@ All input and output ports are one bit unless otherwise specified.
 
 The module should implement a XOR gate.
 
-Enclose your code with <CODE> and </CODE>. Only output the code snippet
+Enclose your code with [BEGIN] and [DONE]. Only output the code snippet
 and do NOT output anything else.
 
 Answer:
-<CODE>
+[BEGIN]
 module TopModule
 (
   input  logic in0,
@@ -24,7 +24,7 @@ module TopModule
   assign out = in0 ^ in1;
 
 endmodule
-</CODE>
+[DONE]
 
 Question:
 Implement a hardware module named TopModule with the following interface.
@@ -42,11 +42,11 @@ Assume all sequential logic is triggered on the positive edge of the
 clock. The reset input is active high synchronous and should reset the
 output to zero.
 
-Enclose your code with <CODE> and </CODE>. Only output the code snippet
+Enclose your code with [BEGIN] and [DONE]. Only output the code snippet
 and do NOT output anything else.
 
 Answer:
-<CODE>
+[BEGIN]
 module TopModule
 (
   input  logic       clk,
@@ -79,7 +79,7 @@ module TopModule
   assign out = temp_wire;
 
 endmodule
-</CODE>
+[DONE]
 
 Question:
 Implement a hardware module named TopModule with the following interface.
@@ -99,11 +99,11 @@ Assume all sequential logic is triggered on the positive edge of the
 clock. The reset input is active high synchronous and should reset the
 output to zero.
 
-Enclose your code with <CODE> and </CODE>. Only output the code snippet
+Enclose your code with [BEGIN] and [DONE]. Only output the code snippet
 and do NOT output anything else.
 
 Answer:
-<CODE>
+[BEGIN]
 module TopModule
 #(
   parameter nbits
@@ -138,7 +138,7 @@ module TopModule
   assign out = temp_wire;
 
 endmodule
-</CODE>
+[DONE]
 
 Question:
 Implement a hardware module named TopModule with the following interface.
@@ -158,11 +158,11 @@ Assume all sequential logic is triggered on the positive edge of the
 clock. The reset input is active high synchronous and should reset the
 finite-state machine to an appropriate initial state.
 
-Enclose your code with <CODE> and </CODE>. Only output the code snippet
+Enclose your code with [BEGIN] and [DONE]. Only output the code snippet
 and do NOT output anything else.
 
 Answer:
-<CODE>
+[BEGIN]
 module TopModule
 (
   input  logic clk,
@@ -212,4 +212,4 @@ module TopModule
   end
 
 endmodule
-</CODE>
+[DONE]