Obviously I have a problem with the RGB to luminance conversion in RSpec already at the level of the raw file.
Tom Field, the developper of RSpec, has been following this thread and he has analysed the phenomenon in depth after I had send him the raw file. First of all he has offered me a free download of the latest version of RSpec (I was using a quite old version): Great customer service! Thanks Tom!
The rotation of a RGB image with a low signal to noise ratio can also contribute to this phenomenon and Tom did send me an insightful video on this topic.
Obviously the use of a DSLR camera for spectroscopy is ok for the first steps (low price and a large sensor; the Canon 450 Da I'm employing is a used one that I bought for 100 USD and did modify myself) but for diving deeper into the topic I have to get a cooled CCD (though the prices for CCD's with an APS-C sized sensor really knocked me out....)