At least with Ristretto 0.6.3, there's a preference for that. In the menu, select “Edit” → “Preferences”, go to the “Behaviour” tab, and untick the “Maximize window on startup when opening an image” checkbox. Ristretto remembers its last window size, so if it started maximized when you last opened it, it might occupy the whole screen (but not maximized) the next time; resize the window, and when you start it again it should stick at this smaller size.
The setting is stored in ~/.config/xfce4/xfconf/xfce-perchannel-xml/ristretto.xml
. You can use this file to preset it to your desired window size.
<?xml version="1.0" encoding="UTF-8"?>
<channel name="ristretto" version="1.0">
<property name="window" type="empty">
<property name="height" type="uint" value="600"/>
<property name="width" type="uint" value="400"/>
<property name="maximize-on-startup" type="bool" value="false"/>
</property>
</channel>