Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

PixInsight Updates ... Broken BPP integration?

This topic has been archived. This means that you cannot reply to this topic.
53 replies to this topic

#1 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22364
  • Joined: 10 Jan 2014

Posted 26 February 2015 - 01:02 AM

I recently updated PixInsight, and a number of things were updated. I don't remember what they all were, however since doing so, I am having some issues integrating with BPP. One issue may or may not be caused by the updates...I think I may have been having it before, however it seems more extreme now. The other is new.

For reference, my PI version is 01.08.03.1123 Ripley (x64).

First, the new issue. Master dark integration no longer seems to be generating valid darks. When I do generate a dark, once it is used to calibrate, I get the following errors:
 

Loading calibration frame image:
E:/Astrophotography/Integrations/Leo Triplet 840mm/With Flats Up-Bottom/master/dark-BINNING_1-EXPTIME_1200.xisf
Loading image 'integration': w=5796 h=3870 n=1 Gray Float32
1 image property
Loading image 'rejection_low': w=5796 h=3870 n=1 Gray Float32
1 image property
Loading image 'rejection_high': w=5796 h=3870 n=1 Gray Float32
1 image property
* Ignoring 2 additional image(s) in master calibration frame.
22 FITS keyword(s) extracted.
** Warning: The file format reports no CFA pattern for the master dark frame, but it is being forced as per process instance parameters.

 

 

And later, when it tries to use the master on a light:

 

Writing output file: E:/Astrophotography/Integrations/Leo Triplet 840mm/With Flats Up-Bottom/calibrated/flat/5DIIIHOOD800IDASF63_FLAT_Tv1-8s_800iso_f6-3_+22c_IDAS-LPS-P2_00514stdev_20150225-21h17m32s509ms_c.xisf
Dark scaling factors:
k0 = 0.000
** Warning: No correlation between the master dark and target frames (channel 0).
Writing image: w=5796 h=3870 n=1 Gray Float32

 

 

I honestly don't know what this is doing...it's warning me, but continuing. When I look at the integrations produced, they seem to be fairly heavily banded, much more so than I was getting before. General quality does not seem to be very good. 

 

There is a worse problem, though. I believe my flats, which I have been having a problem with recently, are being applied rotated 180 degrees from the orientation of the lights. This is producing this rather annoying result:

 

yIdquME.jpg

 

Is anyone else having problems like these? 


Edited by Jon Rista, 26 February 2015 - 01:11 AM.


#2 JJK

JJK

    Gemini

  • -----
  • Posts: 3462
  • Joined: 27 Apr 2008

Posted 26 February 2015 - 08:31 AM

BPP works on the latest release on my Mac.

 

However, the Real Time Preview in HISTOGRAM TRANSFORMATION and CURVE LEVELS has been broken the last few releases.


Edited by JJK, 26 February 2015 - 08:31 AM.


#3 terry59

terry59

    Cosmos

  • *****
  • Posts: 8577
  • Joined: 18 Jul 2011

Posted 26 February 2015 - 08:44 AM

I am seeing none of these issues....interesting



#4 Madratter

Madratter

    Voyager 1

  • *****
  • Posts: 10795
  • Joined: 14 Jan 2013

Posted 26 February 2015 - 08:47 AM

I don't see them either. On the other hand, I'm not using a DSLR or OSC so the CFA stuff doesn't matter.



#5 karambit27

karambit27

    Viking 1

  • *****
  • Posts: 872
  • Joined: 15 Aug 2013

Posted 26 February 2015 - 09:05 AM

I have definitely seen the no correlation error before. There are a ton of posts if you search the pixinsight forums about it that can explain it better than I can.

 

Here's an explanation from Juan on the PI forums I found:

 

Those warning messages are telling you that the master dark frame you are using has no correlation to the light frame in terms of thermal noise. In other words, if the master dark frame is subtracted from that light frame, then the result will always have more noise than the original. Hence, a zero dark scaling factor is being applied, which is equivalent to not subtracting the master dark frame at all.

The noise evaluation routine implemented in our ImageCalibration tool is very robust, so in general my advice is trusting it. We still haven't seen a practical case were it fails to compute an optimal dark scaling factor.

 

 

Here is an explanation for the CFA error I found on the PI forums:

 

This caused by yourself (or BPP) having set the input hint to 'raw cfa' (which is correct for DSLR). 
When the input is in a raw format, PI can find out that it is indeed in CFA format, so it is happy.
When the input is in FITS format, the 'raw' part is ignored (but does not hurt), the 'cfa' part is obeyd (and must be present for some process and do not hurt for others), but PI cannot check that the file was indeed in 'cfa' mode. This information is not in a FITS header.  So it tell you: OK, you tell me it is CFA, I believe you, but I cannot check.


Edited by karambit27, 26 February 2015 - 09:17 AM.


#6 anismo

anismo

    Fly Me to the Moon

  • *****
  • Posts: 5927
  • Joined: 30 Jan 2014

Posted 26 February 2015 - 09:15 AM

Yep. I have seen the no correlation error before as well but it completes the run ok and the output seems fine too . I  have been doing manual integration rather than BPP lately.



#7 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22364
  • Joined: 10 Jan 2014

Posted 26 February 2015 - 10:28 AM

BPP is definitely not doing dark subtraction properly. It doesn't appear to be subtracting darks at all. I am getting horribly banded integrations, which don't occur when the bias and dark are subtracted properly.

 

The flats issue is making it all worse. I manually rotated my master flat, and it improved things, but did not fully correct the field. Probbly because rotating the flats is offsetting noise patterns in the flat vs. the lights, which is incorrect. Why the flats and lights have different orientations is beyond me...makes no sense at all...but that is clearly the case.

 

All of these issues are new since I updated PI, I never had them before. I always did the CFA checkbox in BPP before, but it generated proper darks (with a CFA pattern) that never gave any warnings. Right now, BPP seems to be generating demosaiced master darks, or at least not tagging the output files properly, as they do not appear to have a bayer pattern in them, and that is different than what I was experiencing before. 



#8 terry59

terry59

    Cosmos

  • *****
  • Posts: 8577
  • Joined: 18 Jul 2011

Posted 26 February 2015 - 10:33 AM

Have you tried reverting to the version that worked?



#9 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22364
  • Joined: 10 Jan 2014

Posted 26 February 2015 - 10:57 AM

I am not really sure how. I checked the installed updates...the version of PI hasn't actually changed...only updates were installed. I do not know if there is any way to uninstall those specific updates (none that I can see). If I download PI again, it's the same version...I suspect it would include all the latest updates. 

 

There were updates to DSLR_RAW and to ImageIntegration in the last two updates, as well as to the XISF file format. Not sure which, if any, of those may have caused my problems. 

 

BTW, this is my latest integration, using a manually rotated flat. It fixed the funky halo issues I was having, mostly...but it clearly offset all the dust motes, so I have both dark ones and light ones at opposing points in the frame. It made the banding problem from lack of proper dark subtraction even worse. The field is actually horrible now...blotchy and warped and...yuck. 

 

F5uaaH2.jpg


Edited by Jon Rista, 26 February 2015 - 11:01 AM.


#10 terry59

terry59

    Cosmos

  • *****
  • Posts: 8577
  • Joined: 18 Jul 2011

Posted 26 February 2015 - 11:02 AM

PM



#11 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22364
  • Joined: 10 Jan 2014

Posted 26 February 2015 - 12:20 PM

Here is a better example of what I encounter with non-rotated flats:

 

j4HNKNt.jpg

 

The motes get corrected, but my field is not flat...it appears as though the flat has an inverted form of whatever is causing the concentric rings and vignette in the lights. 

 

Anyone have any ideas why my flats, when properly oriented for dust motes, would have the opposite or inverted field structure?



#12 David Ault

David Ault

    Gemini

  • -----
  • Posts: 3424
  • Joined: 25 Sep 2010

Posted 26 February 2015 - 01:33 PM

If you are not bias or dark subtracting correctly then the flat frame will not divide correctly because neither the light or flat will have the bias pedestal removed.  That offset means that as gradients change across the field the flat can divide too much or too little causing some dust motes to be over and others to be under corrected.  You can also get over and under corrected vignetting patterns.  I just ran through a pass of BPP with my DSLR (Canon 1100D) data and didn't see either of these issues, so it may be specific to your data.  Definitely post this along with your data on the PI forum and see if you can get any traction there.

 

Regards,

David



#13 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22364
  • Joined: 10 Jan 2014

Posted 26 February 2015 - 01:43 PM

I wanted to see if the issue was just me, or if others had it, before I posted to the PI forum. I am quite sure that my darks aren't being handled properly anymore, so that is probably part of the problem with the flats. I also think that BPP may be calibrating the lights with the darks as well as the bias, in which case, if the darks aren't being handled properly, that could be double subtracting the bias or subtracting a 20-minute dark from a 1/13th second flat, etc. Either way, I am going to do a full manual integration tonight, and see if that resolves the issue. 



#14 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22364
  • Joined: 10 Jan 2014

Posted 26 February 2015 - 01:53 PM

I am going to upload my data as well, in case anyone else wants to have a crack at it. I'd be interested in knowing if anyone with different versions of PI or different updates has better (or worse) luck. I'll only upload 10 or 20 lights, as  I have over 200 of them for this particular target...and I have 10 20-min dark frames that should be scaled, 30x flats at 2/3rd histogram (which is about 50% ADU out of 2^14 max ADU), and a superbias (made from 100 bias frames).



#15 karambit27

karambit27

    Viking 1

  • *****
  • Posts: 872
  • Joined: 15 Aug 2013

Posted 26 February 2015 - 02:32 PM

I'll give it a try Jon. I have versions 1.3.0.246 for DSLR_RAW and 1.9.4.255 for ImageIntegration installed.



#16 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22364
  • Joined: 10 Jan 2014

Posted 26 February 2015 - 02:37 PM

Ok, data is uploading. The full data set is 4.2 gigs. Sorry, my 5D III cranks out huge 22.3mp files. :\

 

I am supplying two sets of light data, and two sets of dark data, though. The first is from my first batch, the subs are darker as the skies were darker. The temps are colder, and match better with the 3C darks. The second is from my third batch, the subs are lighter as the skies were lighter (lower transparency, after the first snowstorm we had last week.) The temps are a lot higher, oddly, and match better with the 23C darks. So, if you only download and process one set of data, you won't have to download the full 4.2 gigs worth of data.

 

I don't know if the lighter subs of the latest integration are any part of the problem, but that is why I am supplying both sets of subs and darks.

 

The flats are quite large. I took two sets...a brighter set and a darker set, so it's 60 individual frames. That zip file is 1.6 gigs... :\ I've tried calibrating in BPP with both...had issues both times, but the results are a little different. I'll also upload my superbias. 

 

EDIT:

 

Here is the onedrive link: http://1drv.ms/1Fz2B72

 

I would give everything about an hour to upload. It should be done sooner than that, a gig is already up, but I would give it some time to make sure everything is all the way up so you don't download a partial file of any kind.


Edited by Jon Rista, 26 February 2015 - 02:40 PM.


#17 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22364
  • Joined: 10 Jan 2014

Posted 26 February 2015 - 03:19 PM

I believe everything has finished uploading.



#18 karambit27

karambit27

    Viking 1

  • *****
  • Posts: 872
  • Joined: 15 Aug 2013

Posted 26 February 2015 - 03:51 PM

I am grabbing the first batch of darks and lights as well as the superbias and flats now. I'll try running them through BPP later tonight after work.



#19 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22364
  • Joined: 10 Jan 2014

Posted 26 February 2015 - 04:01 PM

So, a note on my flats. I always take two sets, a darker and lighter set of 30x each. I do this because I have sometimes noticed that dividing out the darker flats inverts the vignette, darkening the center a lot, brightening the corners too much. The lighter flats don't have that problem, since it's a division scaled to 0.0-1.0 on a floating point basis (the closer to 1.0 the values are, the less significant the impact from the division). 

 

I usually use the darker set of flats. They are about 2/3rds histogram, which is about 50% my maximum ADU (8500-8600 out of 16384). The lighter set of flats are more like 3/4rs to 4/5ths histogram, and I am not certain about their linearity...that's around 10,000 ADU. In terms of 8-bit RGB level, they are up round level 233-238 usually, and everything above that has no data...so I think they are fine (not clipped), but maybe not the most ideal. I just keep them in case my darker flats end up inverting my vignette. 



#20 David Ault

David Ault

    Gemini

  • -----
  • Posts: 3424
  • Joined: 25 Sep 2010

Posted 26 February 2015 - 04:51 PM

Hey Jon,

 

I'm downloading them now and will give them a try as soon as I can.  I'm running these versions:

PixInsight:  1.08.03.1123

BatchPreprocessing:   1.40

Debayer:  1.04.03.0146
DSLR_RAW:  1.03.00.0246
ImageCalibration: 1.03.00.0196
ImageIntegration: 1.09.04.0255

 

Regards,

David



#21 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22364
  • Joined: 10 Jan 2014

Posted 26 February 2015 - 05:01 PM

Thanks, guys. I appreciate the help.



#22 Ken Sturrock

Ken Sturrock

    Cardinal Ximénez - NO ONE EXPECTS THE SPANISH INQUISITION!

  • *****
  • Administrators
  • Posts: 7169
  • Joined: 26 Sep 2009

Posted 26 February 2015 - 05:06 PM

Is anyone else having problems like these? 

 

Shoot Bubba, all my pictures look like that........ :(

 

I've always just blamed it on the streetlight.

 

I tease I tease....



#23 karambit27

karambit27

    Viking 1

  • *****
  • Posts: 872
  • Joined: 15 Aug 2013

Posted 26 February 2015 - 06:23 PM

I just ran it through BPP for calibration only. I did get the CFA warning but no correlation warning when calibrating the lights. I am registering them and integrating them now and will post an update. Here is what I got. I just added the superbias to BPP and checked use master bias. Then added darks, flats, and lights. I had calibrate only checked from suing my own data and let it run. After I just dropped the calibrated and debayered frames into StarAlignment and picked a random reference frame. After I dropped all the registered frames into ImageIntegration and selected the typical values BPP uses and let it run.

Attached Thumbnails

  • integration.jpg

Edited by karambit27, 26 February 2015 - 06:38 PM.


#24 Ken Sturrock

Ken Sturrock

    Cardinal Ximénez - NO ONE EXPECTS THE SPANISH INQUISITION!

  • *****
  • Administrators
  • Posts: 7169
  • Joined: 26 Sep 2009

Posted 26 February 2015 - 08:55 PM

Jon:

 

I tried to help you out but I don't use the BPP script and it complained because I didn't choose a pixel rejection algorithm - something that I know how to do "manually" but couldn't find the option for in the script's many many tabs. Anyways, after ignoring that error, I got this sweet message:

 

                                      ImageWindow.windowById(): invalid view identifier

 

Anyway, I went ahead and tried to process the files "manually". It was slow going on my 2008 MacBook non-Pro. You've gotta get a smaller camera!

 

When I reached the point where I tried to calibrate the flats prior to integrating them into a "master flat", I got the console output below (one example) - note the warnings.

 

The master dark frame I used was an integration of the dark frames that you provided. The master bias was your master bias. Now, I could well have screwed something up but I hope it might be useful to you. This was done on PixInsight OSX 1.08.03.1123 Ripley (x64).

 

--------------------------------------------------

Calibrating target frame 59 of 60
Loading target frame:
/Users/ks/Desktop/jon_rista/5DIIIHOOD800IDASF63/2015-02-25/5DIIIHOOD800IDASF63_FLAT_Tv1-8s_800iso_f6-3_+23c_IDAS-LPS-P2_00517stdev_20150225-21h17m18s343ms.CR2

Timestamp: Wed Feb 25 22:12:53 2015
Camera: Canon EOS 5D Mark III
ISO speed: 800
Shutter: 1/8.0 sec
Aperture: f/6.4
Focal length: 840.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:  5760 x 3840
Full size:   5920 x 3950
Image size:  5796 x 3870
Output size: 3870 x 5796
Raw colors: 3
Filter pattern: RGGBRGGBRGGBRGGB
Daylight multipliers: 2.125175 0.943985 1.338680
Camera multipliers: 1945.000000 1024.000000 1631.000000 1024.000000

Invoking: dcraw -w -q 0 -t 0 -o 0 -4
Decoding Canon EOS 5D Mark III file (5796x3870 pixels, ISO=800, Exposure=1/8s): done
Loading raw image: done

Writing output file: /Users/ks/Desktop/jon_rista/out/flats/5DIIIHOOD800IDASF63_FLAT_Tv1-8s_800iso_f6-3_+23c_IDAS-LPS-P2_00517stdev_20150225-21h17m03s473ms_c.fit
Dark scaling factors:
k0 = 0.000
** Warning: No correlation between the master dark and target frames (channel 0).
k1 = 0.000
** Warning: No correlation between the master dark and target frames (channel 1).
k2 = 0.147
Gaussian noise estimates:
s0 = 2.037e-03, n0 = 0.433 (MRS)
s1 = 3.005e-03, n1 = 0.601 (MRS)
s2 = 1.862e-03, n2 = 0.433 (MRS)
34 FITS keywords embedded
Writing FITS image: 32-bit floating point, 3 channel(s), 5796x3870 pixels: done

--------------------------------------------------



#25 Jon Rista

Jon Rista

    ISS

  • *****
  • topic starter
  • Posts: 22364
  • Joined: 10 Jan 2014

Posted 26 February 2015 - 09:03 PM

Thanks guys. I appreciate your taking the time to do this. 

 

Karambit, your result looks like mine, so maybe the issue IS my data.

 

Ken, I only calibrate my flats with the bias. I uploaded both my superbias and my still-mosaiced master bias (generating a superbias eliminates the CFA pattern). If you want the most reliable form of flat calibration, I would calibrate the flats with just the master bias.

 

Feel free to integrate however you are comfortable, btw. I have no expectations for how you guys do what you do with my data...I'm very curious to see what you guys end up with by doing things however you do things. If you want to calibrate manually, please do. I am actually calibrating manually myself, and am hoping that will give me better results. I'm starting 100% from scratch though, reintegrating all the masters, doing all calibration, registration, and integration from the original frames. 




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics