Agreed. Many of the plugins label their first input/output as “left” and the second as “right” - having the headphone output match this would avoid a lot of confusion!
E.g. hook up a stereo mixer with its output 1 connected to Dwarf output 1, output 2 to Dwarf output 2 - pan a channel to the left and the sound moves to the right in the headphones!