8 replies to this topic

[sll74]

My current interest is studying variable stars, using a Canon DSLR + a tele lens on DIY mount (Open Astro Tracker).

My current workflow is something like the following:

1. take several sets of 10x60 seconds exposures (i don't want to saturate my target variable star) using N.I.N.A. Images are saved as raw camera format .cr2. Now I have a few hundred of .cr2 images
2. convert the raw .cr2 images to .fits, using ASTAP tools --> batch processing -->  batch convert to FITS. Now I have a bunch of .cr2.fits images
3. extract the green channel from fits, using ASTAP tools --> batch processing --> raw color reparation --> extract raw GREEN pixels and combine. Now I have a bunch of _TG.fit images
4. load 10 images in the "light" tab and stack them, using ASTAP "Avarage" mode and plate solving for the alignment. This step calibrates each image using darks and flats. Now I have reduced 10 images to a single .staked.fit image
5. repeat step 4 for 10 more images and so on, until I stacked all the _TG.fit images. Now I have a bunch of .stacked.fit images, with a total of 600 seconds exposure each.
6. load all the .stacked.fit in "photometry" tab of ASTAP and finally run the photometry to get the light curve of the variable star.

As you can see, there are a lot of manual steps involved, that make this process tedious and errors prone.

As a software developer my self, I can't not try to make some kind "script" or "API interfacing" to automate all the process.

In my mind, I would start script that converts the raw images to fits, extract the green channel and stack them 10 at the time, maybe even crop the ROI to reduce the storage required, so that everything is ready to run the final photometric analysis manually.

 

Any suggestions?

 

 

 

[skybadger]

Yes, use numpy instead for greater flexibility and wider range of actions.

[han.k]

ASTAP is not build as a collection of separate programs like some other big names. It is all in a single code with procedures for each activity. So it less tailored for scripting but makes other operations much easier.

 

Using the ASTAP batch processing is normally not required.

 

    convert the raw .cr2 images to .fits, using ASTAP tools --> batch processing -->  batch convert to FITS. Now I have a bunch of .cr2.fits images

In the stack and photometry tabs you can browse for CR2. No need to convert in the batch menu.

 

    extract the green channel from fits, using ASTAP tools --> batch processing --> raw color reparation --> extract raw GREEN pixels and combine. Now I have a bunch of _TG.fit images

 

You can do that in the photometry tab with button "raw RGB extraction"

 

    load 10 images in the "light" tab and stack them, using ASTAP "Avarage" mode and plate solving for the alignment. This step calibrates each image using darks and flats. Now I have reduced 10 images to a single .staked.fit image
    repeat step 4 for 10 more images and so on, until I stacked all the _TG.fit images. Now I have a bunch of .stacked.fit images, with a total of 600 seconds exposure each.

I would start with calibrating the RAWs before extraction of green. You can select the stack option "calibration only". Stacking the green channel is a method I haven't foreseen.  Note the calibration/stack result all and up in the Results tab. From there you can copy them back to lights tab or photometry tab. See screenshot below. What is maybe is missing is copy/move the green (extracted) channel images  from photometry tab back to the light tab for stacking. But you could also browse for them in the lights tab. I'm open for suggestions. Is stacking of green channel images required?

 

An other possibility would be to allow calibration and stacking of raw images in the photometry tab. With selection of 10 images you could allow to do all the steps by one popup menu option like "calibrate & green extract" or "stack". Again I'm open for suggestions.

 

 

 

    load all the .stacked.fit in "photometry" tab of ASTAP and finally run the photometry to get the light curve of the variable star.

 

Use the result tab to copy the images to other tabs.

 

Han

 

[gmiller123456]

Another possibility is that ASTAP is open source, so you can make it scriptable.

[han.k]

Adding a scripting language is not so easy. For most users scripting is probably not so convenient. But the processing process is pretty straightforward and I don't think a scripting flexibility is required. I liked my earlier idea to extend the photometry tab popup menu and it is almost finished:

 

 

So you can calibrate raw files and extract green channel directly using the popup menu.

 

 

I noted that the last ASTAP versions in November automatically de-mosaic when doing a calibration only. This was on request for more automation but it spoils now the calibration for photometry. So don't use these November versions. I have now created two calibration options. One without de-mosaic for photometry. This new one is used automatically by the previous mentioned calibration menu in the photometry tab.

 

A development version for Windows with these features will be ready tonight or tomorrow.

 

Han

 

[sll74]

I would start with calibrating the RAWs before extraction of green. You can select the stack option "calibration only". Stacking the green channel is a method I haven't foreseen.  Note the calibration/stack result all and up in the Results tab. From there you can copy them back to lights tab or photometry tab. See screenshot below. What is maybe is missing is copy/move the green (extracted) channel images  from photometry tab back to the light tab for stacking. But you could also browse for them in the lights tab. I'm open for suggestions.

Hello Han.
The workflow your proposed (load .cr2 images, calibrate, move to photometry, extract green channel...) is the first one I tried, but it seams that is not possible to extract a channel from not raw images; You can check the attached screenshot, this is what I have in the log:
 
 

21:44:21  Ready. Resulting files are available in tab Results and can be copied to the Blink, Photometry or Lights tab.
21:46:06  Skipped COLOUR image C:\APT_Images\Camera_1\test\V_ V417 Aur_2022-11-26_23-40-40__17c_60.00s_0322.cr2_cal.fit, Raw red, green or blue pixel extraction is only possible for raw images.

That's why I came up with the workflow I described. Is it me that I'm doing something wrong?

 

 

An other possibility would be to allow calibration and stacking of raw images in the photometry tab. With selection of 10 images you could allow to do all the steps by one popup menu option like "calibrate & green extract" or "stack". Again I'm open for suggestions.

Yes, this will be simplify a lot the process.

 

Something like: select 10 images, right click, the menu appears with a new voice "calibrate, extract channel and stack", then a popup appears to choose the channel (sometimes I want to stack the red channel) and the stack method ("average", "sigma clip average"). The final result image will appear in the "Results" tab, with a name like "10x60s.TG.stacked.fit" (or whatever is your naming convention)

 

 

Is stacking of green channel images required?

Let me explain why I'm stacking the green channel: my goal is to increase the SNR of the stars, improving the photometry results, keeping the sensor working in the linear zone (let's say less that 12000 counts for my 14 bit camera). To keep stars' shape as round as possible, I need to stay under about 120 seconds exposure (I'm shooting unguided at the moment) and gain at 800 ISO (that seams to be a nice spot for my camera sensor); so by best option is to stack a few images, but maybe I'm totally wrong

 

Simonluca

 

 
 
 
 

Attached Thumbnails

• 

[han.k]

You can not extract green from a de-mosaiced image ( colour image). You need a calibrated raw. The calibration of ASTAP was modified about a month ago and I added an auto demosaic function. I forgot the photometry tab. That's why you get this error message. This is now fixed and "calibration only. No de-mosaic" and should work now from the new popup menus. See version below.

 

I will look into stacking of an selection of files from the photometry tab. This will keep the interface clean and simple.

 

 

The development version 2022-11-29a of ASTAP for Windows is just ready but only briefly tested. It has the two new popup menus in the photometry tab as described above.:

http://www.hnsky.org...m#alpha_version

 

Tomorrow I will do some more testing so beware of any problems.

 

Before extracting the green channel check the de-mosaic pattern. In most case it should work in auto. Try it with a terrestrial image.

 

Han

[sll74]

The development version 2022-11-29a of ASTAP for Windows is just ready but only briefly tested. It has the two new popup menus in the photometry tab as described above.:

http://www.hnsky.org...m#alpha_version

 

Just to let you know that I tested this version and it seams to work fine.

 

 

Adding a scripting language is not so easy. For most users scripting is probably not so convenient. But the processing process is pretty straightforward and I don't think a scripting flexibility is required.

I see your point, and it totally makes sense. So, what about adding an option like "stack n at time" and an input field in the "Stack method" tab? For example you can flag this option and write "10" in the input field, start the stacking and the sw will stack 10 images, then 10 images, then 10 more, etc.. till all the selected images are processed. In my opinion, the UI will still clean and this will save a LOT of time, because you don't need to select 10 images, stack them, unselect the 10 images, select the next 10 images, stack them, unselect, etc..

 

-=[ SLL ]=-

[han.k]

I have added the stack option to the photometry tab. ASTAP is now released as version 2022-11-30 for Windows, Linux and Mac.

 

For OSC images you should go through these four steps in the photometry tab:

 

1) Press analyse to convert RAW images to FITS

2) Select popup menu "Calibration selected files". (Darks, flats, flat-darks should be selected in the corresponding tabs) 

3) Select popup menu "Extract green of selected files"

 

and optional

 

4) Select popup menu "Stack selected files"

 

 

For mono FITS images:

 

1) Select popup menu "Calibration selected files"". (Darks, flats, flat-darks should be selected in the corresponding tabs)

 

and optional

4) Select popup menu "Stack selected files"

 

For stacking I have considered something more automatic as you suggested, but for most users it will not work because they could observe a variable for weeks at variable intervals. You need a continuous series to make it work. If there is an inactive period during the observation, then the local average date/time will not be accurate.

 

For correctness I have also updated the reported DATE_OBS for a stacked file. It is now reported as:

 

DATE-OBS= '2021-02-28T18:54:54'/ Calculated for stack JD_AVG - EXPTIME/(2*86400)

 

So after the stack a DATE-OBS (start observation) is calculated from the average date minus the half total exposure time. In the past it was the DATE-OBS of the reference image used for stacking. You could also use the oldest date but that seems less accurate to me so that idea was skipped. Anyhow for the AAVSO report it uses JD_AVG (Julian Day average), so the DATE-OBS doesn't matter for the report.

 

Please report any bug/problem. This modification requires a long testing period.

 

Han

 

 

Edited by han.k, 01 December 2022 - 02:44 AM.