Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

How do you stack Sony, Pentax etc RAW files

  • Please log in to reply
22 replies to this topic

#1 moxican

moxican

    Messenger

  • -----
  • topic starter
  • Posts: 453
  • Joined: 13 Nov 2011
  • Loc: Toronto, Canada

Posted 08 July 2018 - 09:16 PM

I am trying to put together a lightweight grab & go set up and want to buy a mirrorless camera. There are a few options Sony, Panasonic, Samsung, Pentax, Olympus. The problem is that they all have their own RAW file and I do not know if stacking software are compatible with them. I use Nebulosity4 now, and I know it works with Canon and Nikon Raws.

Anyone using any cameras from the above mentioned list? If yes, how do you register and calibrate your exposures? Images of examples are also welcome.



#2 james7ca

james7ca

    Fly Me to the Moon

  • *****
  • Posts: 5410
  • Joined: 21 May 2011
  • Loc: San Diego, CA

Posted 08 July 2018 - 09:39 PM

I stack my Sony NEX images using PixInsight. Note that many of the stacking packages use the open-source decoder dcraw, which handles many different cameras.

 

  https://en.wikipedia.org/wiki/Dcraw



#3 moxican

moxican

    Messenger

  • -----
  • topic starter
  • Posts: 453
  • Joined: 13 Nov 2011
  • Loc: Toronto, Canada

Posted 08 July 2018 - 09:45 PM

Thanks James, but the issue with Dcraw that it "develops" the raw files into Tif format, meaning that it is no longer raw file. If I remember right I've read it somewhere on this forum that it is not a good method. People complained of outcome images. 

Once the raw file is turned into Tif format it is a color image. So, how does any stacking software would subtract Bias, Darks and Flats?

I do not have Pixinsight, does it also use dcraw?


Edited by moxican, 08 July 2018 - 09:49 PM.


#4 james7ca

james7ca

    Fly Me to the Moon

  • *****
  • Posts: 5410
  • Joined: 21 May 2011
  • Loc: San Diego, CA

Posted 08 July 2018 - 09:52 PM

dcraw has options that allow different formats for its output, it doesn't have to be TIFF.

 

Here is a brief summary on the dcraw commands to output to different formats:

 


In versions 8.25 and later, "dcraw -T" writes TIFF output with metadata. To write other formats:

dcraw -c crw_0001.crw | pnmtofits > crw_0001.fits
dcraw -c crw_0001.crw | pnmtopng > crw_0001.png
dcraw -c crw_0001.crw | ppmtobmp > crw_0001.bmp
dcraw -c crw_0001.crw | cjpeg > crw_0001.jpeg

 

However, you'll have to check on what options are allowed by various dcraw front-ends. The actual command-line options may be different.

 

Here is the command line that PixInsight uses to open a Sony NEX RAW file:

 

  dcraw -4 -o 0 -r 1 1 1 1 -D -t 0 -k 0 -H 1


Edited by james7ca, 09 July 2018 - 06:05 AM.

  • bobzeq25 likes this

#5 whwang

whwang

    Vanguard

  • *****
  • Posts: 2113
  • Joined: 20 Mar 2013

Posted 08 July 2018 - 10:29 PM

I believe you can use DSS to stack Sony and Pentax raw files, and it's free.  I used DSS for Sony A7R and Pentax 645z files.  It works with no problems.

 

DCRaw convert raw files into TIFF, but this doesn't mean the TIFF are not raw.  It's just a different format.  You can ask DCRaw to maintain all the original information (data linearity etc) in the output TIFF file, like what James said above.  As far as I can tell, Canon's CR2 file itself is also based on the TIFF format, but packed with multiple layers of resized images for quick preview, and with compression.  So, stop thinking that TIFF cannot be raw.  As long as the pixel values are unaltered, it's raw.

 

Cheers,

Wei-Hao


  • seigell and bobzeq25 like this

#6 bobzeq25

bobzeq25

    Voyager 1

  • *****
  • Posts: 12238
  • Joined: 27 Oct 2014

Posted 08 July 2018 - 11:11 PM

To emphasize the above.  Your "dcraw mangles the data" would apply to Canons and Nikons as well.  Yet a great many very serious imagers use PiXinsight to stack their data from those.

 

I don't think they're either foolish or incompetent.

 

By the way, deviations from "pure raw" often occur inside the camera, perhaps on the sensor.  Noise reduction, compression, etc.


Edited by bobzeq25, 08 July 2018 - 11:22 PM.


#7 SKYGZR

SKYGZR

    Surveyor 1

  • *****
  • Posts: 1963
  • Joined: 13 Aug 2009
  • Loc: Planet Earth, Milky Way Galaxy, Speeding towards the Virgo Supercluster

Posted 09 July 2018 - 12:08 AM

To confirm, DSS stacks Pentax .dng files just fine



#8 bulrichl

bulrichl

    Sputnik

  • -----
  • Posts: 38
  • Joined: 27 May 2018
  • Loc: La Palma (Canary Islands)

Posted 09 July 2018 - 04:32 AM

There is a new RAW module for PixInsight (since february 2018) that is based on libraw (instead of dcraw):

 

https://pixinsight.c...p?topic=12126.0

 

This RAW module is not yet part of the current PI version, but can be installed as a replacement of the DSLR_RAW module.

 

Bernd



#9 Tonk

Tonk

    Cosmos

  • *****
  • Posts: 8616
  • Joined: 19 Aug 2004
  • Loc: Leeds, UK, 54N

Posted 09 July 2018 - 05:06 AM

Most good stacking software does the calibration steps on each light frame *before* stacking - and the debayer step is only done on the final output image. This is certainly the case with DSS (and the debayer is a linear conversion).

 

A common mistake is when people pre treat light, dark, flat and bias RAW files and debayer them to full colour images with colour balance and gamma corrections applied (often the automatic default) before feeding them to a stacking program. Under these circumstances calibration does not work as expected and the rssults are compromised.


Edited by Tonk, 09 July 2018 - 05:07 AM.

  • sharkmelley, 17.5Dob and bsavoie like this

#10 james7ca

james7ca

    Fly Me to the Moon

  • *****
  • Posts: 5410
  • Joined: 21 May 2011
  • Loc: San Diego, CA

Posted 09 July 2018 - 05:32 AM

Most good stacking software does the calibration steps on each light frame *before* stacking - and the debayer step is only done on the final output image. This is certainly the case with DSS (and the debayer is a linear conversion).

 

A common mistake is when people pre treat light, dark, flat and bias RAW files and debayer them to full colour images with colour balance and gamma corrections applied (often the automatic default) before feeding them to a stacking program. Under these circumstances calibration does not work as expected and the rssults are compromised.

This is true, but dcraw doesn't have to debayer the image before it is output to the file container.

 

In fact, for complete coverage here is the dcraw command to perform a pure RAW conversion from a Sony NEX file (.ARW):

 

  dcraw -4 -o 0 -r 1 1 1 1 -D -t 0 -k 0 -H 1

 

and here is the output to debayer that same file:

 

  dcraw -4 -o 0 -w -q

 

And, here is a link to the command-line options for dcraw:

 

  https://www.cybercom...aw/dcraw.1.html



#11 Tonk

Tonk

    Cosmos

  • *****
  • Posts: 8616
  • Joined: 19 Aug 2004
  • Loc: Leeds, UK, 54N

Posted 09 July 2018 - 08:15 AM

I wasn't disputing what dcraw can do (I didnt even mention it) instead I was highlighting a common pre stacking mistake that people can make and that modern stacking programs are designed to avoid this mistake if you stick to feeding in raw data (in contrast some now well respected stacking programs in their early incarnations circa 2004 did debayer before calibration - dcraw was not involved - and yes you got strange artifacts as a result)


Edited by Tonk, 09 July 2018 - 08:16 AM.

  • sharkmelley and bsavoie like this

#12 gbeaton

gbeaton

    Messenger

  • -----
  • Posts: 407
  • Joined: 06 Oct 2016
  • Loc: Montreal

Posted 10 July 2018 - 12:47 PM

Just have your camera output is DNG. Having said that, unfortunately Nebulosity still doesn’t handle DNGs from the Pentax. I just discovered Siril. It’s pretty good at stacking DNG, including Pentax but be warned it’s fits file may be a mirror image compared to what you use now.

#13 lock042

lock042

    Siril astronomical image processing software

  • -----
  • Vendors
  • Posts: 107
  • Joined: 03 Nov 2015

Posted 11 July 2018 - 02:53 AM

 it’s fits file may be a mirror image compared to what you use now.

As discussed on the Siril forum, I do not agree with this statement.


Edited by lock042, 11 July 2018 - 02:53 AM.


#14 ImNewHere

ImNewHere

    Surveyor 1

  • *****
  • Posts: 1544
  • Joined: 29 Mar 2017
  • Loc: Nashville, IN

Posted 11 July 2018 - 03:39 AM

Save the Pentax files in .dng format and use the latest 64 bit version of DSS.



#15 lock042

lock042

    Siril astronomical image processing software

  • -----
  • Vendors
  • Posts: 107
  • Joined: 03 Nov 2015

Posted 11 July 2018 - 04:09 AM

Siril uses libraw that is able to natively read pentax files.

 

I've tested this file: http://www.rawsample...X_K10D_SRGB.DNG

Here the result

Capture_du_2018-07-11_11-07-02.png


Edited by lock042, 11 July 2018 - 05:54 AM.


#16 17.5Dob

17.5Dob

    Soyuz

  • *****
  • Posts: 3863
  • Joined: 21 Mar 2013
  • Loc: Colorado,USA

Posted 11 July 2018 - 09:40 PM

Most good stacking software does the calibration steps on each light frame *before* stacking - and the debayer step is only done on the final output image. This is certainly the case with DSS (and the debayer is a linear conversion).

 

A common mistake is when people pre treat light, dark, flat and bias RAW files and debayer them to full color images with color balance and gamma corrections applied (often the automatic default) before feeding them to a stacking program. Under these circumstances calibration does not work as expected and the results are compromised.

+1

DSS, PI, Nebulosity, Sequator, etc. etc.  all work exactly same way and use the same RAW conversion engine (outside of the new module for PI).

There are only two very vocal people in the AP community that advocate pre processing your RAWS in ACR or LR and then stacking the TIF's.

The other several 10,000' s ++++  just stack RAWS using any of the above programs.

 


  • Tonk likes this

#17 Tonk

Tonk

    Cosmos

  • *****
  • Posts: 8616
  • Joined: 19 Aug 2004
  • Loc: Leeds, UK, 54N

Posted 11 July 2018 - 11:37 PM

Hallas is an unfortunate distraction in this regard. Who is the other one (a Hallas desciple?)?



#18 sharkmelley

sharkmelley

    Surveyor 1

  • -----
  • Posts: 1642
  • Joined: 19 Feb 2013

Posted 12 July 2018 - 11:57 AM

Hallas is an unfortunate distraction in this regard. Who is the other one (a Hallas desciple?)?

ClarkVision?


  • Tonk likes this

#19 17.5Dob

17.5Dob

    Soyuz

  • *****
  • Posts: 3863
  • Joined: 21 Mar 2013
  • Loc: Colorado,USA

Posted 12 July 2018 - 09:15 PM

ClarkVision?

Bingo waytogo.gif


  • Tonk likes this

#20 gbeaton

gbeaton

    Messenger

  • -----
  • Posts: 407
  • Joined: 06 Oct 2016
  • Loc: Montreal

Posted 13 July 2018 - 05:39 AM

Bingo waytogo.gif

Really? I have to check that out (about Clarkvision).

 

One thing I'm beginning to wonder about is this green cast in the output of stackers, including PI - as I've seen on a youtube tutorial. It didnt make sense to me that it would be like that. Now I'm beginning to believe that they just aren't converting properly. I dont think they are applying the colour conversion matrix. It seems they are just applying a simple 1:1:1 conversion of the RAW pixel value to RGB. This will not produce a faithful preproduction of the scene since there is overlapping light from the green spectrum into the red, for example. Am I right in this conclusion - i.e. the stackers like DSS (and PI it seems) are using a 1:1:1 colour conversion?


Edited by gbeaton, 13 July 2018 - 05:41 AM.


#21 james7ca

james7ca

    Fly Me to the Moon

  • *****
  • Posts: 5410
  • Joined: 21 May 2011
  • Loc: San Diego, CA

Posted 13 July 2018 - 10:39 AM

There are options in the PixInsight format converter to use a white balance from the DSLR camera. Otherwise, if you do a pure RAW conversion there is no color adjustment made to the pixel values. Look under PI's DSLR_RAW format converter (go to the Format Converter tab and double click on the DSLR_RAW entry to view the options). Generally, however, you probably don't want to apply any color correction with the format conversion debayering on files that have had bias/DF calibrations and you can always change the color balance during the post processing.

 

Here is a simple shot of the blue sky that I took using a Sony NEX camera. The image in the background was opened using the camera white balance setting in DSLR_RAW while the foreground image was opened with no white balance (they are the same source file). After each file was opened I applied an auto screen transfer stretch to the image. You can see that the version that received no white balance is somewhat green or cyan, while the camera color balance produced a deep blue (maybe too blue, but a greater stretch on the linear source would likely make it appear closer to what you might consider normal).

 

And, the DSLR_RAW options dialog in PI. Note, if you change any of these setting make certain that you go back to "Pure RAW" before you do any processing of your astro subs.

Attached Thumbnails

  • Camera versus No White Balance or Blue Sky.jpg
  • DSLR_RAW Options.png

Edited by james7ca, 13 July 2018 - 10:56 AM.


#22 gbeaton

gbeaton

    Messenger

  • -----
  • Posts: 407
  • Joined: 06 Oct 2016
  • Loc: Montreal

Posted 13 July 2018 - 12:18 PM

Thanks. I figured PI had a setting. But my point is that colour balance is only part of colour conversion. Without the colour matrix applied (pure raw it seems to be called), the colour is not correct because each pixel has to be adjusted for slight green and slight blue components in the red cell, for example.

#23 greasemonkey

greasemonkey

    Lift Off

  • -----
  • Posts: 11
  • Joined: 05 Nov 2017

Posted 15 July 2018 - 10:33 PM

To confirm, DSS stacks Pentax .dng files just fine

 

There is a new RAW module for PixInsight (since february 2018) that is based on libraw (instead of dcraw):

 

https://pixinsight.c...p?topic=12126.0

 

This RAW module is not yet part of the current PI version, but can be installed as a replacement of the DSLR_RAW module.

 

Bernd

 

I concur,  I have used DSS for my pentax k70 and seems to work fine

 

As far as PI it seems to process my pentax raw files just fine, but just downloaded the trial today and still exploring it. Spent my entire Sunday just playing around with it confused1.gif  but looks promisingsmile.gif




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics