Return to the Cloudy Nights Telescope Reviews home page


Speciality Forums >> Astronomy Software & Computers

Pages: 1 | 2 | (show all)
daev
Technically Enlightened
*****

Reged: 03/10/04

Loc: On the edge of the desert
bare-bones Linux image processing new
      #5092541 - 02/26/12 04:35 PM Attachment (94 downloads)

This past week I performed my own mini economic stimulus and spent part of my tax refund on an Astrotrac (plenty of reviews out there on that one, so I'll just say it was worth every penny). I drove out to a nice dark site overlooking Palo Duro Canyon with a buddy on Friday night and set up for a night of imaging. I knew I'd be wasting a lot of time because the setup was new, and I'm rusty. ENd result I took home 22 lights and 18 darks that were suitable for stacking.

The catch here is that I don't have any stacking software, and GIMP is only 8bit (and therefore unsuited to the task). So I started digging, and found out that I did indeed have stacking software... it's hidden in plain sight in most out-of-the-box Linux distros as the Imagemagick set of tools.

So what else did I need? Something to register each of my light frames with each other, for starters. Hugin's panorama tools includes a routine called align_image_stack which fit the bill perfectly. I also needed something that I could view the images in and tweak levels on the final image in 16bit color. Krita works for that well enough. It's buggy and unstable for general use, but for opening images and setting the black point and stretching levels/curves it's functional. One last thing I needed was a RAW converter that could batch convert my images, I chose UFRAW.


How it's done:

Drop all of your RAW files into a new directory. Navigate there in a terminal and type:
Quote:

ufraw-batch --out-type=tif --out-depth=16 *.CR2


I shoot with a Canon, so that last bit of the command might be *.NEF or *.CRW etc depending on the brand of camera you use. Once this is done, move all of your RAW files to some safe-keeping location to get them out of the way.

When that's done, drop your darks in a folder made just for them. Navigate to this folder in a terminal window and issue the following command:
Quote:

convert -average *.tif darkframe.tif


This will create a 16bit/channel averaged dark frame called "darkframe.tif".

This next part is the toughie (I'm working on learning how to script this, just so I don't have to issue a command for each and every image). Drop all your lights into their own folder along with a copy of darkframe.tif. Issue the following command for each light frame:
Quote:

composite -compose minus imagename.tif darkframe.tif dsimagename.tif


The command is structured thus: "composite -compose minus" calls up imagemagick's subtraction routine, takes the imagename.tif, subtracts darkframe.tif from it and writes the result to dsimagename.tif (I chose ds for dark-subtracted). In practice, this takes only a few seconds for each image since usually you're only changing the digits in the file names so you only type the full command once and use the terminal history to recall the last command and edit it.

Next issue the following command:
Quote:

align_image_stack -a ais ds*.tif


"align_image_stack" calls Hugin's stacker. "-a" tells the program to use the next field in the command (in this case "ais") as the first set of characters in the output images' file names and increment them (so the output files will be ais0000.tif, ais0001.tif, ais0002.tif, etc). "ds*.tif" restricts the command to operating only on tif files that start with "ds". This can take a while, and will likely throw warnings, but should complete in a few minutes and generate a series of new files that are aligned and rotated to match the first image in your set of lights.
(edited to remove an extraneous command option. Thanks, lambermo, for catching that [see below])

Next command is:
Quote:

convert -average ais*.tif averagestack.tif


This is the same command we used to average the darks, we've simply specified different input and output files. "averagestack.tif" is your final stacked image, ready to open up in krita to set the black point and stretch levels to suit your taste. Save the image from krita to a 16bit/channel tif (don't forget to uncheck the alpha channel, it adds that by default). This file can be opened in GIMP and exported to 8bit/channel filetypes, or you can save from krita to such.

One of the resulting pics, 100% crop, with levels prolly stretched too far :

Edited by daev (02/26/12 05:44 PM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
lambermo
professor emeritus


Reged: 07/16/07

Loc: .nl
Re: bare-bones Linux image processing new [Re: daev]
      #5092600 - 02/26/12 05:13 PM

Very interesting !

Next add bias and flat frame handling

I think the ALIGN_IMAGE_STACK(1) -m "Useful for aligning focus stacks with slightly different magnification." should be replaced with something not dealing with rescaling.

ps. You might want to check out theli and regim as well, 2 stackers. And fitswork (single binary file) runs fine under with wine.
http://www.astro.uni-bonn.de/~theli/index.html
http://www.andreasroerig.de/regim/regim.htm
http://www.fitswork.de/software/softw_en.php


Post Extras: Print Post   Remind Me!   Notify Moderator  
daev
Technically Enlightened
*****

Reged: 03/10/04

Loc: On the edge of the desert
Re: bare-bones Linux image processing new [Re: lambermo]
      #5092647 - 02/26/12 05:40 PM

I'm interested in Theli, I just need to wrap my head around it. I did this experiment just because I thought it could be done and I was itching to stack the results of the new toy. I had some obvious issues with wind, but it's much better still than my past attempts with a rickety GEM and clock drive from the Town of Bedrock.

Quote:

I think the ALIGN_IMAGE_STACK(1) -m "Useful for aligning focus stacks with slightly different magnification." should be replaced with something not dealing with rescaling.



Whoops! You are quite correct, sir. That's what I get for copy/pasting from my notes. And those notes were tailored to focus stacks, BTW. Nice catch, and thanks. The image I posted was processed with the -m, and it's safe to leave in there, but I'll go back and edit the original post to remove it.

Flats I would love to include, but I don't know if imagemagick has a function to handle that... don't you divide lights by your flats? I'm not quite sure how flats are merged mathematically. Holler if you know, 'cause I haven't really got that bit nailed.

Bias frames: does anyone shooting with a DSLR still use these? I was under the impression that these were deprecated for DSLR use (tho still useful for CCD imaging).

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
lambermo
professor emeritus


Reged: 07/16/07

Loc: .nl
Re: bare-bones Linux image processing new [Re: daev]
      #5094376 - 02/27/12 04:35 PM

Hi Dave,

Here's a nice diagram on how flats and bias frames are to be used mathematically : http://deepskystacker.free.fr/english/theory.htm#CalibrationProcess

About the bias frames on DSLR : in use everywhere afaik. Never even heard of the 'still' part you mention. I do hear about recent cooled CCD's that are so good that people do not use darks anymore, I assume they don't use bias then either ;-)

-- Hans


Post Extras: Print Post   Remind Me!   Notify Moderator  
daev
Technically Enlightened
*****

Reged: 03/10/04

Loc: On the edge of the desert
Re: bare-bones Linux image processing new [Re: lambermo]
      #5094471 - 02/27/12 05:32 PM

oooohhhh...thank you for that! Just what I was looking for!

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
astrotrf
Not at all Mundane
*****

Reged: 09/30/07

Loc: Rodeo, NM
Re: bare-bones Linux image processing new [Re: daev]
      #5101154 - 03/02/12 02:12 PM

Script for the "composite" command (well, OK, it's C shell; sorry):

#! /bin/csh
foreach f (*.tif)
echo $f
composite -compose minus $f darkframe.tif ds$f
end

Put this into a file called, say, "composite", and protect it so it's executable. Run it with the command "./composite" if the script is in the same directory as the image files. What I like to do is create a subdirectory under my home directory called "bin" and put executable scripts there. Then you could execute the script with "~/bin/composite" from anywhere.

Hints:

To remove the ".tif" from the filename, use "$f:r". So we could change the filenames from imagename-n.tif to, say, imagename-n-ds.tiff by "$f:r-ds.tiff" (where n, of course, is the sequence number of the image).

If you want to give arguments to the "composite" script, they are accessed as $argv[1] and up. So you could change the name of the darkframe by using $argv[1] instead of darkframe.tif, as:

composite -compose minus $f $argv[1] ds$f

so your final command could be:

~/bin/composite ~/darklib/10-below-dark.tif

I hope all of the special characters come through in the posting ...


Post Extras: Print Post   Remind Me!   Notify Moderator  
daev
Technically Enlightened
*****

Reged: 03/10/04

Loc: On the edge of the desert
Re: bare-bones Linux image processing new [Re: astrotrf]
      #5101867 - 03/02/12 09:56 PM

I'll be looking into giving that a go when I get back from vacation next week! Thanks loads!

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
Harshad
super member


Reged: 10/04/10

Loc: India
Re: bare-bones Linux image processing new [Re: astrotrf]
      #5103027 - 03/03/12 04:01 PM

Nice image of the Horsehead!

Quote:

Script for the "composite" command (well, OK, it's C shell; sorry):

#! /bin/csh
foreach f (*.tif)
echo $f
composite -compose minus $f darkframe.tif ds$f
end





Here's the zsh version (should work with bash too I guess):
Code:

for f in *.tif; do echo $f; composite -compose minus $f darframe.tif ds$f; done



Or using the "find" command:
Code:

find ./ -name "*.tif" -ls -exec composite -compose minus {} darframe.tif ds{} ';'



Edit: corrected the "find" example.

Edited by Harshad (03/04/12 05:18 AM)


Post Extras: Print Post   Remind Me!   Notify Moderator  
avarakin
Carpal Tunnel
*****

Reged: 07/13/09

Loc: Parsippany NJ, USA
Re: bare-bones Linux image processing new [Re: Harshad]
      #5103958 - 03/04/12 02:48 AM

I once did flat compensation using imagemagick. I had a very bad dust particle on my sensor and I shot about 1000 images, so there was no way I would fix images one by one. So I wrote a simple script using imagemagick to apply flat. You just need to divide light image by flat image.

I used align of hugin for HDR, but how does it work for astroimages?

Showfoto is another nice program for simple postprocessing on Linux. It works in 16 bit mode.
I no longer use it though because Fitswork runs very well under wine.
Also it seems that PS 5 works well under wine.

Alex


Post Extras: Print Post   Remind Me!   Notify Moderator  
daev
Technically Enlightened
*****

Reged: 03/10/04

Loc: On the edge of the desert
Re: bare-bones Linux image processing new [Re: avarakin]
      #5113455 - 03/09/12 01:04 PM

Quote:

I used align of hugin for HDR, but how does it work for astroimages?
Alex




So far my experience with Hugin is that it works provided your images are somewhat close to registered when you feed them into the align routine. I've seen it error out/produce misaligned images if the subs are dim or offset from each other significantly. I've not dug into manual alignment with it, but since the panorama tools allow for manual creation of control points I'm thinking it must be possible...

I use Hugin for HDR as well, as a first step before I run my bracketed shots through Enfuse.

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
daev
Technically Enlightened
*****

Reged: 03/10/04

Loc: On the edge of the desert
Re: bare-bones Linux image processing new [Re: daev]
      #5113462 - 03/09/12 01:07 PM

Harshad, Astrotrf... Thanks for the scripts, I'm really looking forward to getting back from vacation to some dark skies to try these out (not to mention sit down and cross-reference all the bits so I understand what's happening).

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
daev
Technically Enlightened
*****

Reged: 03/10/04

Loc: On the edge of the desert
Re: bare-bones Linux image processing new [Re: daev]
      #5363748 - 08/11/12 06:38 PM

I'm finally getting around to doing something with the code snippets above (and thanks, folks, I really learned a lot in the past day picking that apart!)

Now I've managed to write my first useful script (whee!). I place all my raws into a directory containing a "darks" folder and a "lights" folder, then open a terminal window, enter "sh stackme" and go get lunch. Much easier.

code:
Quote:

#!/bin/bash


# This script depends on ufraw and hugin having been previously installed.
# It should be run from a directory containing 2 folders, "lights" and "darks"
# with the appropriate RAW files contained therein. It is written for some of
# the Canon cameras using the .CR2 extension. If your files have a different
# extension, edit the two occurrances of "CR2" below to reflect your filetype
# using the proper upper/lower case characters.



# convert raws in "darks" directory to tiff,
# average all frame data into darkframe.tif
cd darks
ufraw-batch --out-type=tif --out-depth=16 *.CR2
convert -average *.tif darkframe.tif
cd ..



# convert raws to tiff in "lights" directory,
# subtract darkframe.tif from each resulting tif,
# (eg. flnm.tif) write to files dsflnm.tif
cd lights
ufraw-batch --out-type=tif --out-depth=16 *.CR2

# begin loop for all .tif in directory
echo Applying darkframe to lights
for flnm in *.tif
do echo $flnm; composite -compose minus $flnm ../darks/darkframe.tif ds$flnm
done
echo Finished applying darkframe to lights

# begin alignment of dark-adjusted lights
echo Beginning alignment process, please wait.
echo This can take a while with many images.
align_image_stack -a ais ds*.tif

# average aligned images into final.tif
echo Averaging aligned files
convert -average ais*.tif final.tif
cp final.tif ..
echo Stack complete.

cd ..

# file cleanup. Comment out lines if you
# want to keep any of the converted files
echo Deleting temporary files.
cd darks
rm *.tif
cd ../lights
rm *.tif
cd ..
echo Temporary files deleted.




Results: (unadjusted, straight jpg conversion from 16bit/channel tiff output from the above script)


dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
avarakin
Carpal Tunnel
*****

Reged: 07/13/09

Loc: Parsippany NJ, USA
Re: bare-bones Linux image processing new [Re: daev]
      #5363858 - 08/11/12 08:15 PM

Daev, Interesting post. Did you try stacking other images? I am curios how the aligning works. Typical astro imaging software detects stars and uses them for aligning. I doubt that align_image_stack does it the same way so it would be interesting to see result on more images.

As far as I know, ufraw-batch uses its configuration file for conversion parameters, thus it is not certain that conversion to tiff will happen using linear algorithm without any scaling. If scaling of darks and lights is not the same, then darks will not work as expected.
I typically use dcraw -6 -T to preserve scaling of the image.


Alex


Post Extras: Print Post   Remind Me!   Notify Moderator  
daev
Technically Enlightened
*****

Reged: 03/10/04

Loc: On the edge of the desert
Re: bare-bones Linux image processing new [Re: avarakin]
      #5364012 - 08/11/12 10:34 PM

I tested this on a couple of image groups, with no issues as far as stacking goes.

Interesting point on linearity, I will check into that.

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
gonzothegreat
journeyman


Reged: 11/20/11

Re: bare-bones Linux image processing new [Re: daev]
      #5370178 - 08/15/12 01:51 PM

That's very interesting, thanks for sharing.

Post Extras: Print Post   Remind Me!   Notify Moderator  
Mike Phillips
Post Laureate
*****

Reged: 02/21/06

Loc: Swift Creek, NC - 35.682 N, 78...
Re: bare-bones Linux image processing new [Re: gonzothegreat]
      #5384867 - 08/24/12 04:18 PM

This is awesome! I'm totally going to use this! I wonder how it compares to DSS?

Thanks,

Mike


Post Extras: Print Post   Remind Me!   Notify Moderator  
jedimasterk
sage


Reged: 01/30/06

Re: bare-bones Linux image processing new [Re: Mike Phillips]
      #5387475 - 08/26/12 05:23 AM

Glad to see these opensource tools producing some excellent results. Now if you go to Kitt Peak you can tell them you use Linux as well. He He!!. Cannot wait for Gimp 2.10 to come out with floating point 32 bit color. I also remember someone here at the forums using Raw Therapee as well, with some very nice results. If you do any more please show them.

Post Extras: Print Post   Remind Me!   Notify Moderator  
daev
Technically Enlightened
*****

Reged: 03/10/04

Loc: On the edge of the desert
Re: bare-bones Linux image processing new [Re: jedimasterk]
      #5387914 - 08/26/12 12:44 PM

I've been using RawTherapee for final adjustments/general photography development after seeing it linked in a thread here. It's very capable, has a nice UI and a lot great tools but I miss the alpha slider in gimp/ps levels. Even with the new gimp, I'll still be using it for a lot of things.

Also waiting with bated breath for deep-color support in gimp.

When I get the muse going, I'm going to do a a run of images using avarakin's tip up-thread and compare. Since it's scripted, I really don't need UFRAW's batching ability, and UFRAW is a front end for DCRAW anyway.

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
AdirondackAstro
sage
*****

Reged: 06/06/11

Loc: Plattsburgh, NY
Re: bare-bones Linux image processing new [Re: daev]
      #5436374 - 09/23/12 03:31 PM

This was one of my main issues with Linux is it didn't have a good way of stacking. I may have to load up Linux and give this a shot. Thank you very much for this information. Looks quite promising judging from your images. Thanks, Daev.

Post Extras: Print Post   Remind Me!   Notify Moderator  
daev
Technically Enlightened
*****

Reged: 03/10/04

Loc: On the edge of the desert
Re: bare-bones Linux image processing [Re: AdirondackAstro]
      #5538836 - 11/25/12 02:24 PM

LOL I've been Boinged! 3400 hits to my flickr account this morning alone because of the above-posted Milky Way pic. Linky. This was quite a surprise as I'm used to getting a handful of hits from friends and the like. Logging in to find thousands had me thinking (What's wrong with flickr today?" It ain't APOD, but I'll take it!

dave


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1 | 2 | (show all)


Extra information
4 registered and 13 anonymous users are browsing this forum.

Moderator:  richard7, kkokkolis 

Print Thread

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled


Thread views: 3225

Jump to

CN Forums Home




Cloudy Nights Sponsor: Astronomics