Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Sony A7RII lossless RAW

This topic has been archived. This means that you cannot reply to this topic.
38 replies to this topic

#26 t_image

t_image

    Gemini

  • -----
  • Posts: 3,499
  • Joined: 22 Jul 2015

Posted 28 October 2015 - 03:40 PM

....

That said...IF you wanted to use a full and proper calibration process (which in some circumstances may be required...even with it's low dark current, during the summer with an uncooled camera, it could still be problematic enough to require dark calibration; vignetting could be bad enough to require flat calibration in some situations), then black clipping would pose a problem. For that matter, so would any kind of SLog as well.

 

Why? Because for bias and dark subtraction and flat division to work, the signal has to be truly linear. Any non-linear processing to any of those frames before calibration occurs will render the calibration incorrect at the very least, and useless or even damaging at worst. Black clipping will mess that up, although again, I would like to know how much the black point is actually clipped, as if it is only clipped a small amount, it might not be totally damaging (or it might be totally damaging, just depends on what the offset is, and if the offset is fixed or dynamic.) I think SLog is probably worse than black point clipping though, as that by design renders the signal totally non-linear.

......

FWIW, in video, any log-like SLOG2/SLOG3 is only meant to be a container for the imaged colorspace information that is more than any display can show including the dynamic range.

One can use Davinci Resolve and the ACES colorspace workflow, that allows one to work with the information in a total linear fashion.

However, the final data must have a LUT (look up table) applied to it so it will render into the limited gamut displays have to show the image/video.

 

Given, there is the assumption that RAW is linear.

However, even if Sony "cooks" their ARW with a curve to package the colorspace and dynamic range (that no display can fully render),

I would think it is then converted to a linear form to be processed before it is "stretched" as you AP'ers call it?

Can someone address if the ARW cannot be rendered to a linear form?

I do think there is a problem trying to look at a rendered image on a display and decide whether the blacks are clipped since the gamut necessarily has to be shaped into the gamut your display needs to render the image, and therefore is already altered.

 

Please watch this video (start at 2:00) to understand the value of log demonstrated:

https://www.youtube....h?v=fUF8xkjOnAI

 

________________________________________________________________

Additionally all:

Something I've noticed with the a7s [No one has ever mentioned this]

In the Picture Profiles

http://helpguide.son...0000435736.html

Black level, Gamma, Knee, Mode, Auto Set, Color Mode, Saturation, Color Phase, Color Depth, Detail can be changed.

Black level, Black Gamma, Knee, Color Depth do not apply with RAW.

Obviously most we set to off, etc.

But could there be an adjustment that changes some of what is discussed?

Could you set a picture profile (most things off) but change the detail setting for your RAW stills and have different results?

 

http://helpguide.son.../level1_02.html

________________________________________________________________

If someone is smart about this and can speak to this, please help inform me?


Edited by t_image, 28 October 2015 - 10:32 PM.


#27 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 26,034
  • Joined: 10 Jan 2014

Posted 28 October 2015 - 10:27 PM

I think trying to compare video to the kind of linear RAW data we really need for ideal astro processing is probably an exercise in futility. Video is quite different, and I don't know that it generally "requires" linear data. It requires access to all the information the camera was capable of recording, but your not generally going to be calibrating each and every frame in a video file with bias, dark and flat data. For that matter, the video data may not even be raw, it could be encoded into one of a wide variety of video formats, which is decidedly not raw, nor linear (as the sensor recorded it).

Once data is non-linear, there really isn't any converting it back to linear. "Stretching" as we call it is, in a sense, a "destructive" process in that it tends to be one way. That is why, at the moment, I don't think many A7 series users bother with things like full calibration that REQUIRE linear data. Because with the lossy compression, there definitely is that curve applied to compress the information into the smaller bit depth. I don't know that all the linear calibration is absolutely necessary with Sony cameras, though...I've seen a fair number of images from primarily the A7s that look really great, and I don't see much in the way of hot pixels, clipped or NRed stars or clipped blacks or anything like that. Vignetting is a problem in some, but that could be corrected with something like DBE in PI pretty easily.

But fundamentally, once you stretch the data, there really isn't any going back to linear. It is now non-linear. Things that REQUIRE linear data, such as calibration or deconvolution are no longer going to work optimally or at all. So a lossy compressed ARW is unlikely to ever be able to be rendered as the original linear sensor data again. An uncompressed ARW, will probably be mostly linear. If sharkmelly's example signal plot is indicative of most dark and light frame signals, then even though the camera offsets the black point, it might not actually affect most of the data. There might be a bit of a non-linear shoulder into the black point, but between that and the white point the data should be pretty linear. If any of the high ISO long exposure NR kicks in, then the signal would again probably become more non-linear. Without the lossy compression, however, the signal should be much better than it is with the lossy compression.

#28 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 7,388
  • Joined: 26 Aug 2012

Posted 28 October 2015 - 10:51 PM

I guess I am not so rigid on black point clipping. Remember that these Sony Alpha series cameras are designed for regular old photographers. They were not designed explicitly with astrophotography in mind (and technically speaking, few DSLRs ever have been...the 20Da, the 60Da, the D810a would be exceptions). For terrestrial photography, for the amount of black point offset that is necessary to hide the read noise (which is utterly minuscule, given we are talking about <5e- most of the time, possible exception A7s at low ISO) is immaterial. It certainly is not ideal for astrophotography, however the camera wasn't designed for it in the first place. As such, I think it may be a little harsh to call it a 'cheat.'

 

Most people (regular photographers here) will clip the black point anyway when they process, as most want that higher contrast. When it comes to astro processing, a lot of imagers will also clip the black during processing, for the exact same reason that Sony probably did: to eliminate the most offending deep-shadow noise. I think more seasoned processors learn to preserve as much of the background sky detail as they can...however, as someone who has had quite a bit of experience extracting every last ounce of detail from Canon DSLRs, there is always crap in the background sky that is a major pain to deal with that I've never seen in Sony data...black point clipped or not. There aren't that many people who regularly need to lift an image over four stops either. The Sony Exmor sensors are capable of handling shadow pushes significantly more than that...in my own testing, I've pushed data over seven stops before and still only had random noise, and I've never actually seen any consequences of black point clipping even then:

 

attachicon.gifExample.jpg

 

Top is just the shadow push (effective over 7 stops on the deepest shadows in both cases, between +5 exposure slider in LR, as well as increases to both shadows and blacks).

Bottom is after I processed to restore contrast. 

 

The left is the A7r and the right is the 5D III. 

 

Seven stops. That is a very considerable push. The Canon data has not been "cheated" in any way, and yet the A7r data is clearly vastly superior. That blotchy red and green noise in the Canon data is something I deal with in every one of my astro images, despite using higher ISO than this test was done at. For the intended use case, I have a hard time calling the black point clipping a cheat. For that matter, even the lossy compression did not exhibit in any fashion here, which actually surprised me when I originally did this test. Neither operations are ideal for astro, I totally agree there, however Sony is still fairly new to the game, and I honestly don't think astro was an intended, or at least not a planned-for, use case. So even the lossy compression...eh, given the IQ I've experienced with the A7r and A6000 under terrestrial use cases, I have a hard time faulting Sony for it in the grand scheme of things. Not ideal for astro, but I do think it is a little harsh to call them cheaters. Well...that's me, at least. 

 

That said, I DO fault them for hard-coding automatic long exposure NR. :p That's just...inane. :ohgeeze:

 

I have high hopes for future Sony camera models. They seem to be fairly responsive to customer feedback, something I have not really felt with Canon for a while. Not really since the 1D X and 5D III first hit (and the 5D III was a big disappointment on the noise front, as you can see above.) I think things will improve for Sony cameras in the astro arena as future models are released. For our part, we just need to be vocal in the right forums about what we need. Personally, I hope they deliver a richer SDK before anything else. Then at least we might be able to start hacking away some of these problems ourselves before Sony solves them.

 

 

Good stuff Jon, and happy birthday!!   BTW, you have good taste in chairs--I think that I have exactly the same one in my office.  Ok, now back to the subject at hand.  My wife just bought a Sony A7 and I want to try it out so this is interesting stuff.

 

John



#29 sharkmelley

sharkmelley

    Cosmos

  • *****
  • Posts: 8,085
  • Joined: 19 Feb 2013

Posted 29 October 2015 - 01:02 AM

Once data is non-linear, there really isn't any converting it back to linear. "Stretching" as we call it is, in a sense, a "destructive" process in that it tends to be one way. That is why, at the moment, I don't think many A7 series users bother with things like full calibration that REQUIRE linear data. Because with the lossy compression, there definitely is that curve applied to compress the information into the smaller bit depth. I don't know that all the linear calibration is absolutely necessary with Sony cameras, though...I've seen a fair number of images from primarily the A7s that look really great, and I don't see much in the way of hot pixels, clipped or NRed stars or clipped blacks or anything like that. Vignetting is a problem in some, but that could be corrected with something like DBE in PI pretty easily.
 

 

You appear to be implying that the data coming from Sony A7 series raw files is non-linear, which is incorrect.

 

I use a Sony A7S and I use the usual calibration frames.  The data coming from the raw files of A7 cameras is definitely linear and can be calibrated.  Yes, a tone curve is used during the data compression but that is reversed out during uncompression.  The only effect this compression/uncompression is to create varying size gaps in the histogram (sparseness) .  At the low end of the histogram (in 14bit space) the stepsize is 2 (i.e. only even numbers appear in the histogram) and the sparseness increases toward the high end of the histogram where the stepsize is 32 (only 1 data point in 32 is represented in the histogram).

 

Jpg files are different (as on any camera) and are definitely non-linear.

 

For completeness I should point out that the uncompressed files from the cameras with the firmware update have no histogram gaps in 14 bit space, so they are proper 14 bit files.

 

Mark


Edited by sharkmelley, 29 October 2015 - 01:14 AM.


#30 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 26,034
  • Joined: 10 Jan 2014

Posted 29 October 2015 - 11:12 AM

Good stuff Jon, and happy birthday!!


Thanks, and...thanks! I did not realize it was the 29th until just now...but thanks!

BTW, you have good taste in chairs--I think that I have exactly the same one in my office.  Ok, now back to the subject at hand.  My wife just bought a Sony A7 and I want to try it out so this is interesting stuff.
 
John


It's a nice chair! :) I actually have a couple of them, and a matching couch. Love em. Quite comfortable.

I think you will like the A7. They are good little cameras.

#31 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 26,034
  • Joined: 10 Jan 2014

Posted 29 October 2015 - 11:19 AM

Once data is non-linear, there really isn't any converting it back to linear. "Stretching" as we call it is, in a sense, a "destructive" process in that it tends to be one way. That is why, at the moment, I don't think many A7 series users bother with things like full calibration that REQUIRE linear data. Because with the lossy compression, there definitely is that curve applied to compress the information into the smaller bit depth. I don't know that all the linear calibration is absolutely necessary with Sony cameras, though...I've seen a fair number of images from primarily the A7s that look really great, and I don't see much in the way of hot pixels, clipped or NRed stars or clipped blacks or anything like that. Vignetting is a problem in some, but that could be corrected with something like DBE in PI pretty easily.

 
You appear to be implying that the data coming from Sony A7 series raw files is non-linear, which is incorrect.
 
I use a Sony A7S and I use the usual calibration frames.  The data coming from the raw files of A7 cameras is definitely linear and can be calibrated.  Yes, a tone curve is used during the data compression but that is reversed out during uncompression.  The only effect this compression/uncompression is to create varying size gaps in the histogram (sparseness) .  At the low end of the histogram (in 14bit space) the stepsize is 2 (i.e. only even numbers appear in the histogram) and the sparseness increases toward the high end of the histogram where the stepsize is 32 (only 1 data point in 32 is represented in the histogram).
 
Jpg files are different (as on any camera) and are definitely non-linear.
 
For completeness I should point out that the uncompressed files from the cameras with the firmware update have no histogram gaps in 14 bit space, so they are proper 14 bit files.
 
Mark


My point is that you cannot fully reverse what is done by the craw compression process. Even if you reverse the compression curve applied with craw, reversing it is not capable of entirely restoring what you had before. There ARE gaps. Those gaps represent missing data. The data, at least IMO, is no longer linear...it is "stepped" or quantized. It's like stairs, and worse, if what you say is correct, the depth of those stairs gets larger as you progress farther into the highlights of the data. That would be quite non-linear, even if it is represented as linear 14-bit data values. Many of those values are identical when they should not be. Non-linear, whether you reverse the curve or not, because you discarded information when you compressed.

Don't get me wrong. I'm a fan of Sony cameras. I really want to get the A7r II, but I am biding my time watching to see how the uncompressed RAW firmware works out, and to see if Sony will deliver more much-needed firmware updates (i.e. maybe remove the hard-coded application of high ISO long exposure NR). (Well, that, and I would prefer to get a CCD right now anyway. :p) I like Sony. I actually like Sony better than I like Canon...I just appreciate their innovativeness, their candor, and their responsiveness to customers.

However, I am not going to fool myself about what their craw compression does. It IS destructive to data. I guess it's a non-issue now that they are upgrading the firmware of their cameras with uncompressed RAW. If/once the A7s and A7 and A7r get the firmware update, there truly wouldn't be an issue anymore.

#32 sharkmelley

sharkmelley

    Cosmos

  • *****
  • Posts: 8,085
  • Joined: 19 Feb 2013

Posted 29 October 2015 - 02:28 PM

Linear, with variable quantization ;)

 

Mark



#33 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 26,034
  • Joined: 10 Jan 2014

Posted 29 October 2015 - 02:34 PM

If it wasn't variable quantization, I'd say linear. :p But maybe were delving into the depths of semantics at this point.  :scared:



#34 Robert York

Robert York

    Messenger

  • -----
  • Posts: 462
  • Joined: 05 May 2013

Posted 09 November 2015 - 11:34 PM

Not to interrupt the somewhat off-topic discussion, but did someone actually answer the original question yet? Does the uncompressed RAW format get rid of the 'star eater', or low-pass filter in bulb shooting? It would be nice to hear from someone with a supported camera whether or not this fixes the star eater issue, rather than just speculate.

 

I'm starting to see reports of slight differences in color rendering and other things that imply that this uncompressed RAW may actually be by-passing some of the in-body filtering that happens in most all RAW camera formats, in addition to just bypassing the compression. It would be a huge boon indeed if Sony becomes the first consumer manufacturer to actually deliver a truly (or at least, more?) RAW file.

 

(And yes, every consumer camera cooks the data to some degree in a RAW file, but I think that's pretty well understood by now.)



#35 sharkmelley

sharkmelley

    Cosmos

  • *****
  • Posts: 8,085
  • Joined: 19 Feb 2013

Posted 10 November 2015 - 12:52 AM

I have heard nothing to indicate that any fix has been bundled in with the firmware update for uncompressed raw files.  They are two completely independent problems with different causes.

 

Also, to the best of my knowledge, there is absolutely no indication that Sony see the star eater issue any kind of problem at all.

 

However, if anyone has installed the A7RII firmware update and can upload a long exposure bulb mode dark then it is very quick and easy to determine if the star eater algorithm is still active - it's simply a case of looking for the curious pairing of hot pixels in the raw file of a bulb mode exposure, which is an artefact of the algorithm applied.  It was this pixel pairing that first drew my attention to the issue because it was identical in character to the old star eating Nikon cameras.

 

post-216852-0-06847600-1430009784.jpg

 

Regards,

 

Mark


Edited by sharkmelley, 10 November 2015 - 01:00 AM.


#36 Robert York

Robert York

    Messenger

  • -----
  • Posts: 462
  • Joined: 05 May 2013

Posted 10 November 2015 - 01:24 AM

I can confirm that my Sony a6000 and Sony a7II both have the star-eater feature. When set to an exposure other than bulb, (I used 10 seconds) while under sampling the stars (IE, using it on a 400mm lens/telescope), I could see the star cluster I was trying to image with no troubles. As soon as I set it to bulb, even if I took the same time of exposure, the cluster was simply gone (along with most other stars in the image). All that was left was an odd shaped 'star' several pixels wide, where the core was.

 

An easy fix was to increase the focal length until the stars were super sampled, (IE, I bumped it up to my 1300mm telescope) and the stars passed the filter and no longer vanished. While not a terrible option, certainly not useful for wide-field shots.

 

Now that Sony is delivering the uncompressed RAW firmware to the a7II as well, I should know in a few weeks if it's helped.

 

Edit: I wasn't aware of the hot-pixel thing. I'll take some more images before I update the firmware and see if I have a similar signature.


Edited by Robert York, 10 November 2015 - 11:10 PM.


#37 charotarguy

charotarguy

    Apollo

  • *****
  • Posts: 1,356
  • Joined: 10 Sep 2014

Posted 10 November 2015 - 01:58 PM

 
I think you will like the A7. They are good little cameras.

 

 

"They are good little cameras." that sir, put a smile on my face  :lol:  :lol:  :lol: . Its a freakin $1300 camera! 



#38 Robert York

Robert York

    Messenger

  • -----
  • Posts: 462
  • Joined: 05 May 2013

Posted 10 November 2015 - 10:40 PM

 

I think you will like the A7. They are good little cameras.

 

 

"They are good little cameras." that sir, put a smile on my face  :lol:  :lol:  :lol: . Its a freakin $1300 camera! 

 

 

I would argue that they would produce easy as good, and most likely, a better picture than a $1300 astro-cam. Different sort of work to process, and you need a scope that will fill the sensor, but they gather a lot of light.



#39 sharkmelley

sharkmelley

    Cosmos

  • *****
  • Posts: 8,085
  • Joined: 19 Feb 2013

Posted 11 November 2015 - 12:50 AM

 

I think you will like the A7. They are good little cameras.

 

 

"They are good little cameras." that sir, put a smile on my face  :lol:  :lol:  :lol: . Its a freakin $1300 camera! 

 

 

That neatly sums up the problem. To pay $1300 for a camera that intentionally destroys your data (bulb mode star eater) is very frustrating.

 

Mark


Edited by sharkmelley, 11 November 2015 - 12:50 AM.



CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics