Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

The Pragmatist's Guide to Calibration in Photoshop

astrophotography dslr
  • Please log in to reply
18 replies to this topic

#1 BQ Octantis

BQ Octantis

    Viking 1

  • -----
  • topic starter
  • Posts: 928
  • Joined: 29 Apr 2017
  • Loc: Outback NT, Australia

Posted 15 April 2019 - 05:11 AM

G'day all,

 

After reading Bracken's Deep Sky Imaging Primer, I set out to improve my DSO image quality with calibration frames. His breakdown of the types of noise (Chapter 2 in particular) and the application of calibration frames (Chapter 14) is exceptional, but he gives practically no guidance on how to apply them. (Indeed, my favorite quote is "Fortunately, all of this heavy mathematical lifting is done for us behind the scenes by the calibration software." [emphasis mine]). The astroimaging world is reliant on this mythical calibration software, and in the wild, DSS and PI seem to be the automagical go-to for DSOs (and there seems to be very little appetite for Photoshop in particular). Yet when there's a problem, even the most expert of experts is stumped by something that should be solvable—even for something as simple as applying a flat. So I compiled this quick primer on Photoshop-only lights calibration.

 

I use Photoshop CS5 for MacOS X; there may be slight variations across versions and platforms, but the functions needed are about as basic as they get: layers, layer modes, layer opacity, and histogram adjustment (all present since version 3.0). The same functions exist in Gimp, but I've yet to dive into Gimp scripting.

 

I. Calibration Frames

 

I use all three calibration frame types referenced by Bracken: Master Bias, Master Darks, and Master Flats. I use Flat Darks (vice the Master Bias) to create the Master Flats; while there's minimal difference between the two, there are slight hot pixel increases in the Flat Darks that are amplified with a histogram stretch. I use Photoshop to create the stacks. I like stacks of 100 for each—the noise statistics seem to settle well enough for my purposes. I first use Adobe Camera Raw in Photoshop to adjust the Color Temperature and Tint of all the files to what I plan to use every time (and—most importantly—set the Blacks to 0). This is done as a batch by first selecting all the images in the preview pane (⌘-A) and then changing the values. Daylight defaults are probably fine; I use the numbers I derived from a gray card at high noon (e.g., Temp = 5000, Tint = -17 for ISO400). The stacking procedure is then

  1. File ⇒ Scripts ⇒ Load Files into Stack… ⇒ select 100 calibration images
  2. Starting from the bottom, set the layer opacity to 1/N, where N is the layer sequence number
  3. Layer ⇒ Flatten Image

For the second step, I like merging 10 at a time (i.e., set the opacity for 10 layers, select the layers, then Layer ⇒ Merge Layers) in the stack because the opacity sequence is easy to memorize (and I made a Photoshop Action to adjust and merge 10 layers from the current layer). The opacity sequence (from the bottom of the 10 to be merged) is 100%, 50%, 33%, 25%, 20%, 17%, 14%, 13%, 11%, 10%. Once I've created 10 sets of 10, I merge those 10 (i.e., set the opacity for 10 layers, select the layers, then Layer ⇒ Merge Layers), flatten, and save as a PSD. The merged stack of bias images is the Master Bias. The Master Dark is the dark stack with the Master Bias applied as a subtraction layer (i.e., Edit ⇒ Paste, and then select "Subtract" instead of "Normal" as the layer combine method in the drop-down menu in the Layer pane) and flattened.

 

The Master Flat is a bit trickier. I first apply the Flat Darks stack as a subtraction layer on the flat stack and flatten. I then use Levels to stretch the histogram by changing the white input clip level for each channel to the point where the histogram tapers off:

 

Screen Shot 2019-04-14 at 10.28.52 AM.png

Figure 1. Flat histogram adjustment.

 

Putting it all together, here's my all-Photoshop workflow for calibration frame creation for a given ISO, exposure time, and aperture:

 

Screen Shot 2019-04-15 at 7.15.23 PM.png

Figure 2. Photoshop-only calibration frame production.

 

II. Calibration

 

a. Bias Frames

 

Bias frames are the easiest to remove. They're just applied as a subtraction frame with no scaling whatsoever. Bias is the largest source of sensor noise on my T3i, and thankfully it's temperature independent.

 

b. Dark Frames

 

Dark frames were the hardest to dissect. To first order, they're mainly hot pixels, but there's also some background noise with low spatial frequency to its structure. They're temperature dependent, and no two are the same—the sensor temperature changes the moment I use it, and I have no evidence that steady state is ever achieved (and plenty of evidence to the contrary). Hence why I apply bias frames separately vice simply including the bias with the dark.

Before applying the Master Dark to the light, I have to find the right level to apply. I do this by selecting a light from the middle of the imaging window and subtracting the Master Bias (as a subtraction layer). I then apply the Master Dark to it as a subtraction layer and then scroll through the Master Dark layer's opacity until I find the opacity at which the hot pixels in the image just start to disappear. I also hunt for hot red pixels—they turn blue when the dark is overapplied. That's the level I then use for the full imaging window. The opacity value is typically between 30% and 50%.

 

c. Flat Frames

 

The only hard part with flats is the creation of the Master Flat. Once created, it's simply applied as a divide layer.

 

Putting it all together, here's my all-Photoshop workflow for calibration frame application for a given ISO, exposure time, and aperture:

 

Screen Shot 2019-04-15 at 7.16.04 PM.png

Figure 3. Photshop-only calibration workflow.

 

III. Automation

 

Manual production is exceedingly slow. But I successfully recorded Photoshop Actions for both calibration frame creation and lights calibration.

 

a. Changing and Selecting Layers for Stacking:

 

There's not a scriptable way to tell Photoshop to select a specific layer, so I use the Select Forward Layer (Option-] ) and Select Backward Layer keyboard commands (Option-[ ) to switch layers while recording actions. I can also multiple-select sequential layers with Shift-Option-[ or Shift-Option-]. For 10 layers in the stack, I make sure they're all visible and then make the bottom layer active. I then record the action from there with the keyboard commands for switching layers.

 

b. Changing Windows and Applying Calibration Images:

 

I initially created actions per calibration file, where the action opens the calibration file, copies it, closes it, pastes it to the image, changes the layer mode (subtract or divide), and flattens the image. But constantly opening and closing files that could remain open wastes a lot of time. So instead, the minimalist method I came up with to record an action was opening the Master Bias, Master Dark, and Master Flat files (in that order) and doing Select ⇒ Select All for each. I then opened a single sub on which to record the action. Photoshop treats windows as a stack, and when I select between windows while recording actions, Photoshop uses the current window as a reference ("Select Document ±n"). If I set the window sequence before recording, I can subsequently apply specific calibration files to a light if I open them the same way before running the action.

 

This leaves the problem of the adjusted Master Dark. This is solved easily enough—once I figure out the opacity, I use the Levels command on the open Master Flat to adjust the output whites level to the opacity level*255.

 

So the automation action (from the Light window, with the Master Bias, Master Dark, and Master Flat all open with Select All applied) is then

  • Select Master Bias Window ⇒ Copy ⇒ Select Light Window ⇒ Paste ⇒ Set layer mode to Subtract
  • Select Master Dark Window ⇒ Copy ⇒ Select Light Window ⇒ Paste ⇒ Set layer mode to Subtract
  • Select Master Flat Window ⇒ Copy ⇒ Select Light Window ⇒ Paste ⇒ Set layer mode to Divide
  • Flatten Image

This is then batch applied to the prepped (i.e., correct Temp, Tint, Blacks) light RAWs with the File ⇒ Scripts ⇒ Image Processor, with an output of TIFF with LZW compression. The TIFF output files are then ready for my favorite stacker.

 

Hope that's useful.

 

Cheers,
BQ


  • wrnchhead likes this

#2 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • Posts: 6959
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 15 April 2019 - 04:36 PM

Hi BQ,

 

ACR takes the raw files and applies a non-linear curve to them so how will calibration frames work?

 

You need a raw, linear files for calibration.

 

Why do you think you need to do this in Photoshop, since DeepSkyStacker is free and much more powerful for aligning and calibration and stacking, offering sigma-clip stacking and automatic sub-pixel alignment, which you can't do at all in Photoshop?

 

Jerry


Edited by Jerry Lodriguss, 15 April 2019 - 04:38 PM.

  • sharkmelley likes this

#3 sharkmelley

sharkmelley

    Vanguard

  • *****
  • Posts: 2131
  • Joined: 19 Feb 2013

Posted 15 April 2019 - 04:51 PM

Hi BQ,

 

Your method would work if the data were linear. 

 

But Jerry is right, a non linear curve is applied.  In fact, as soon as you open a raw file in PS/ACR/LR then it automatically subtracts the bias level and it converts the data to your chosen default colour space (e.g. sRGB or Adobe RGB) with non linear gamma (i.e. with a gamma around 2.2 instead of 1.0).

 

Calibration with flats and darks cannot work with non-linear gamma.  I'm afraid you need to use "mythical calibration software" instead.  DSS is a good suggestion.

 

Mark


Edited by sharkmelley, 15 April 2019 - 04:55 PM.


#4 BQ Octantis

BQ Octantis

    Viking 1

  • -----
  • topic starter
  • Posts: 928
  • Joined: 29 Apr 2017
  • Loc: Outback NT, Australia

Posted 15 April 2019 - 10:40 PM

Hi BQ,

 

ACR takes the raw files and applies a non-linear curve to them so how will calibration frames work?

 

You need a raw, linear files for calibration.

 

Why do you think you need to do this in Photoshop, since DeepSkyStacker is free and much more powerful for aligning and calibration and stacking, offering sigma-clip stacking and automatic sub-pixel alignment, which you can't do at all in Photoshop?

 

Jerry

Hi BQ,

 

Your method would work if the data were linear. 

 

But Jerry is right, a non linear curve is applied.  In fact, as soon as you open a raw file in PS/ACR/LR then it automatically subtracts the bias level and it converts the data to your chosen default colour space (e.g. sRGB or Adobe RGB) with non linear gamma (i.e. with a gamma around 2.2 instead of 1.0).

 

Calibration with flats and darks cannot work with non-linear gamma.  I'm afraid you need to use "mythical calibration software" instead.  DSS is a good suggestion.

 

Mark

Jerry & Mark,

 

Thanks for the feedback. But my post was not theoretical. It's documentation of what I do on a regular basis. It's now simply a previously undocumented calibration alternative—especially for those for whom DSS calibration does not work.

 

For the record, I don't recommend Photoshop for stacking (for all the reasons mentioned by Jerry) except for in unique cases. But then, this wasn't a post on stacking…

 

Cheers,

BQ



#5 whwang

whwang

    Mercury-Atlas

  • *****
  • Posts: 2553
  • Joined: 20 Mar 2013

Posted 15 April 2019 - 11:18 PM

If DSS does not work, I cannot imagine a reason why PS can work better.

#6 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • Posts: 6959
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 15 April 2019 - 11:37 PM

But my post was not theoretical. It's documentation of what I do on a regular basis. It's now simply a previously undocumented calibration alternative—especially for those for whom DSS calibration does not work.

So you are saying that you can correctly calibrate with non-linear stretched files?

 

 

Jerry


Edited by Jerry Lodriguss, 15 April 2019 - 11:37 PM.


#7 sharkmelley

sharkmelley

    Vanguard

  • *****
  • Posts: 2131
  • Joined: 19 Feb 2013

Posted 16 April 2019 - 12:12 AM

Jerry & Mark,

 

Thanks for the feedback. But my post was not theoretical. It's documentation of what I do on a regular basis. It's now simply a previously undocumented calibration alternative—especially for those for whom DSS calibration does not work.

 

 

Fair enough.  You are perfectly at liberty to use any workflow you like with your own data and to document your approach for the "benefit" of others. 

 

But other users need to be warned why this is approach is not recommended i.e. because it is not mathematically consistent.  Otherwise it will cause them endless grief.

 

If DSS is not working for you then you are better served by trying to understand the cause of your own problem.

 

Mark


Edited by sharkmelley, 16 April 2019 - 12:24 AM.


#8 whwang

whwang

    Mercury-Atlas

  • *****
  • Posts: 2553
  • Joined: 20 Mar 2013

Posted 16 April 2019 - 12:17 AM

I actually suggest beginners to use Photoshop to manually stack the calibration frames and to do the calibrations, at least for a few times.  This is a good way to see what's the actual process without just relying on black boxes for your entire astrophorography life.  This is very educational, but is also very inefficient and can often be inaccurate.  As both Jerry and Mark pointed out, the ACR conversion is nonlinear and far far away from being ideal for astrophotography.

 

If you really love Photoshop that much and feel you have to use it for calibration, then at least you should avoid using ACR for raw conversion.  You can download LibRaw and use it to linearly convert your raw files to 16bit linear TIFF, and then go to Photoshop for the rests.

 

Cheers,

Wei-Hao



#9 BQ Octantis

BQ Octantis

    Viking 1

  • -----
  • topic starter
  • Posts: 928
  • Joined: 29 Apr 2017
  • Loc: Outback NT, Australia

Posted 16 April 2019 - 02:59 AM

I haven't done apples-to-apples with ACR gamma adjustments, but per my original post, only the darks require scaling. But then, no two darks are the same. Bias and Flats seem to work just fine—the Photoshop Action is plug-and-chug. The results are quite acceptable.

 

I really like the LibRaw suggestion, but installation is even more complicated than running DSS in Wine…(do I just chmod the files in the bin folder and put them in /usr/bin?—the README files and online documentation all focus on compiling).

 

If you love DSS (and more importantly, it works for you…congrats!) and feel this workflow is a waste of time, then don't bother with it. But I'd be happy to share the Photoshop Action set with any serious Photoshop power user who wants to try them. Just PM me…

 

Cheers,

BQ


Edited by BQ Octantis, 16 April 2019 - 03:22 AM.


#10 whwang

whwang

    Mercury-Atlas

  • *****
  • Posts: 2553
  • Joined: 20 Mar 2013

Posted 16 April 2019 - 04:16 AM

Are you using MacOS or Linux?  I use MacOS, and the executable of LibRaw can be directly used immediately after download.



#11 BQ Octantis

BQ Octantis

    Viking 1

  • -----
  • topic starter
  • Posts: 928
  • Joined: 29 Apr 2017
  • Loc: Outback NT, Australia

Posted 16 April 2019 - 04:56 AM

Mac OS X. I downloaded the "Binaries and Source" for both 0.18.13 and 0.19.3 from the download page…the only binary files are the 10 in the bin folder. There is no executable per se without chmodding and running those from the command line.



#12 gatsbyiv

gatsbyiv

    Viking 1

  • *****
  • Posts: 642
  • Joined: 29 Mar 2009
  • Loc: Doylestown, PA

Posted 16 April 2019 - 06:47 AM

DSS may be a little bit of a black box, but calibration in PixInsight is very much a manual process if you don't use the BatchPreProcessing script.  If the goal is to understand how calibration and stacking work, doing it by hand in PI is a great way to fully understand each step and see the effects of each adjustment and setting as you go.  (Of course, it can be insanely tedious for regular use, too.)  You've done a thorough job creating an educational exercise to calibrate in Photoshop, so you probably have a good grasp on how things work now, but as noted above, it will not produce accurate results if the images are gamma adjusted.

 

If DSS isn't working for you, have you tried AstroPixelProcessor, which takes a similar approach?  It has some flexibility on the spectrum of completely manual to black box preprocessing.

 

Charlie


Edited by gatsbyiv, 16 April 2019 - 07:17 AM.


#13 sharkmelley

sharkmelley

    Vanguard

  • *****
  • Posts: 2131
  • Joined: 19 Feb 2013

Posted 16 April 2019 - 05:51 PM

I haven't done apples-to-apples with ACR gamma adjustments, but per my original post, only the darks require scaling. But then, no two darks are the same. Bias and Flats seem to work just fine—the Photoshop Action is plug-and-chug. The results are quite acceptable.

 

I really like the LibRaw suggestion, but installation is even more complicated than running DSS in Wine…(do I just chmod the files in the bin folder and put them in /usr/bin?—the README files and online documentation all focus on compiling).

 

If you love DSS (and more importantly, it works for you…congrats!) and feel this workflow is a waste of time, then don't bother with it. But I'd be happy to share the Photoshop Action set with any serious Photoshop power user who wants to try them. Just PM me…

 

Cheers,

BQ

I can now see where you are coming from.  If DSS in Wine is a real pain and if Libraw is also a real pain then there is a third option.

 

It is possible to create a colour space in Photoshop that is linear i.e. gamma=1.0

You can then tell ACR or Lightroom to use this bespoke linear colour space instead of sRGB or Adobe RGB or whatever else you normally use.  Also make sure you set it to 16bits instead of 8bits.  But there is still the problem that Photoshop will be applying a hidden tone curve i.e. the extra contrast and colour saturation that Adobe thinks photographers want to give their images more artistic "pop".  You can work around this by using Process Version 2010 and setting the sliders for Blacks, Brightness and Contrast to zero.  Also set the tone curve to "Linear".

 

Once you do all this then the files you open will have linear values and all the calibration operations you described in your original post will work correctly.  But this is a fairly advanced technique and is not for the fainthearted.  Also, when you have finished, convert the colour space to sRGB before saving the file otherwise other applications may misinterpret the values in your file and display them incorrectly.

 

Mark



#14 RowlandC

RowlandC

    Asterism

  • -----
  • Vendors
  • Posts: 122
  • Joined: 31 Mar 2014

Posted 16 April 2019 - 08:33 PM

Siril is a sophisticated package available for MacOS and a bunch of UNIX style platforms.

https://www.siril.org/download/

If you are familiar with PixInsight calibration options, Siril incorporates quite a few.

Tip: set the destination folder - it's an easy step to miss.

#15 whwang

whwang

    Mercury-Atlas

  • *****
  • Posts: 2553
  • Joined: 20 Mar 2013

Posted 16 April 2019 - 08:41 PM

The solution suggested by Mark is still not the most straightforward one.  It solves the nonlinearity issue, but you will still need something else for image registration and stacking after the calibration. And most programs capable for registration and stacking can do much better jobs on calibration, like PixInsight, APP, and Siril, as suggested by various people here, and all these can run natively on MacOS.

 

I am a heavy Photoshop user and I use Mac exclusively.  For many tasks that people advertise for PixInsight, I do them in Photoshop.  In today's standard, I am just a dinosaur who cannot catch up with the new advancements in astro image processing.  However, what I do know is that calibration in Photoshop isn't optimal and is time consuming.


  • sharkmelley likes this

#16 BQ Octantis

BQ Octantis

    Viking 1

  • -----
  • topic starter
  • Posts: 928
  • Joined: 29 Apr 2017
  • Loc: Outback NT, Australia

Posted 17 April 2019 - 03:40 AM

All,

 

Excellent feedback…thank you! I hadn't even heard of AstroPixelProcessor (a little steep at 145€), and I've only gotten as far as requesting a trial license for PI (even tougher to swallow at 230€). For me, at least, PS CS5 is sunk cost (purchased 9 years ago, now…dinosaur, indeed!), and developing workflows is a fun hobby that dovetails well with sipping wine. I'm excited by Siril, but I only discovered it a few weeks ago—well after I was far down this road. Maybe my next project…

 

As for stacking, I've used Lynkeos v2.10 for the past couple of years (it looks like Jean Etienne has restarted development, but v3.0 has been troublesome). It does sub-pixel registration (0.1 pixels) and has provisions for calibration, but the documentation is a little spotty. And I'm not happy with its RAW provisions—they take forever, often hang, and the results are terrible. Hence my focus on Photoshop calibration.

 

As to the gamma, ACR 6.7 in PS CS5 has a curves function that claims it's linear. And my color space options are fairly limited.

 

Screen Shot 2019-04-17 at 6.00.38 PM.png

 

But I've found that for bias frames, anyway—my biggest source of error—using the same settings for the lights as the calibration frames produces very good results. As I already shared, darks and flats are a different story…

 

BQ



#17 whwang

whwang

    Mercury-Atlas

  • *****
  • Posts: 2553
  • Joined: 20 Mar 2013

Posted 17 April 2019 - 04:57 AM

Hi,

 

I believe the curve in the screenshot you just showed is the additional curve adjustment applied to the image that has already undergone a nonlinear gamma stretch.  A straight line there does not mean linear data.



#18 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • Posts: 6959
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted Yesterday, 04:24 PM

Wei-hao is correct, the non-linear curve is already applied, dont be mislead by it saying "linear".

#19 RowlandC

RowlandC

    Asterism

  • -----
  • Vendors
  • Posts: 122
  • Joined: 31 Mar 2014

Posted Yesterday, 08:51 PM

I'm guessing you know this- but for illustration. A 'linear' histogram would ordinarily be narrower and further left if not right up against the left margin.

Screenshot_2019-04-19_11-47-43.png

The left image is linear, being a relative term applied to RAW consumer camera data. The right a not yet stretched deBayered tif file.

 

ImageMagick histogram generation.


Edited by RowlandC, Yesterday, 08:52 PM.



CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics





Also tagged with one or more of these keywords: astrophotography, dslr



Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics