This still doesn't really pin the llm down, because it can still later produce an odd or even number. It's not like asking this question forces it to commit to a number.
If you really want to see if it lies, in the sense of knowingly producing output that is objectively untrue, you can ask it to produce a number and hide it in a <hidden></hidden> block, and tell it that your user interface will mask such a block so that you can't see it. Then see if the revealed number later matches that or not.
> Is the number strictly greater than six?
This still doesn't really pin the llm down, because it can still later produce an odd or even number. It's not like asking this question forces it to commit to a number.
If you really want to see if it lies, in the sense of knowingly producing output that is objectively untrue, you can ask it to produce a number and hide it in a <hidden></hidden> block, and tell it that your user interface will mask such a block so that you can't see it. Then see if the revealed number later matches that or not.
That’s true. But I thought it might in fact commit to a number, in which case you might see different behaviour.
Seems like experimental economics with AI is a fun subfield….