Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Developing a tool to archive RAW files by temperature

  • Please log in to reply
25 replies to this topic

#1 ecloud

ecloud

    Mariner 2

  • -----
  • topic starter
  • Posts: 214
  • Joined: 17 Oct 2013
  • Loc: Dalian, China

Posted 21 May 2016 - 02:21 PM

Almost done on Mac.

Currently only work for Pentax PEF.

This is to archive RAW files into several sub directories named as temperature. The temperature is read from the exif data of the RAW files.

Who can give me the TAG names (by ExifTool) of CR2 and NEF to fetch the sensor temperature?

Ideally this tool is platform independent because it's python.

Will test it on windows tomorrow.

TEMPMAX    = 21
TEMPMIN    = 11
INTERVAL   = 4
DELTA      = float(INTERVAL)/2
TEMPTAGS   = ['MakerNotes::SensorTemperature', 'MakerNotes::SensorTemperature2']
FILEFORMAT = ['.pef','.PEF']

These values will control the folder structure.

srat.jpg



#2 StuartJPP

StuartJPP

    Mariner 2

  • -----
  • Posts: 270
  • Joined: 16 Oct 2015
  • Loc: Leeds, United Kingdom

Posted 21 May 2016 - 04:01 PM

Canon .CR2 files is:

 

Camera Temperature              : 18 C

 

ExifToolGUI displays it as "CameraTemperature" with no space.



#3 SKYGZR

SKYGZR

    Surveyor 1

  • *****
  • Posts: 1961
  • Joined: 13 Aug 2009
  • Loc: Planet Earth, Milky Way Galaxy, Speeding towards the Virgo Supercluster

Posted 21 May 2016 - 10:27 PM

Pentax .DNG in F^ would be "nice" Why .PEF? I always use .DNG when capturing RAW w/ my Pentax's...



#4 ecloud

ecloud

    Mariner 2

  • -----
  • topic starter
  • Posts: 214
  • Joined: 17 Oct 2013
  • Loc: Dalian, China

Posted 22 May 2016 - 01:54 AM

Canon .CR2 files is:

 

Camera Temperature              : 18 C

 

ExifToolGUI displays it as "CameraTemperature" with no space.

 

This value seems the camera temperature, not the sensor temperature

 

Could you please try like this:

exiftool -s -g xxx.CR2

I need both the TAG name and Group name



#5 nikao

nikao

    Viking 1

  • -----
  • Posts: 513
  • Joined: 14 Jun 2012

Posted 22 May 2016 - 05:07 AM

I think NEF files don't record this



#6 StuartJPP

StuartJPP

    Mariner 2

  • -----
  • Posts: 270
  • Joined: 16 Oct 2015
  • Loc: Leeds, United Kingdom

Posted 22 May 2016 - 06:32 AM

 

Canon .CR2 files is:

 

Camera Temperature              : 18 C

 

ExifToolGUI displays it as "CameraTemperature" with no space.

 

This value seems the camera temperature, not the sensor temperature

 

Could you please try like this:

exiftool -s -g xxx.CR2

I need both the TAG name and Group name

 

 

It is the CameraTemperature field that is used for Canon cameras, it falls under the "MakerNotes" section. There isn't a specific sensor temperature. Please see the attached file...

Attached Files

  • Attached File  t.txt   13.37KB   37 downloads


#7 ecloud

ecloud

    Mariner 2

  • -----
  • topic starter
  • Posts: 214
  • Joined: 17 Oct 2013
  • Loc: Dalian, China

Posted 22 May 2016 - 09:25 AM

 

 

Canon .CR2 files is:

 

Camera Temperature              : 18 C

 

ExifToolGUI displays it as "CameraTemperature" with no space.

 

This value seems the camera temperature, not the sensor temperature

 

Could you please try like this:

exiftool -s -g xxx.CR2

I need both the TAG name and Group name

 

 

It is the CameraTemperature field that is used for Canon cameras, it falls under the "MakerNotes" section. There isn't a specific sensor temperature. Please see the attached file...

 

 

Hi,

Thank you for the info.

Could you please try it on CR2 files?

You need to install python first, which can be found here: https://www.python.org/downloads/

Drop the srat.py file into the directory with the CR2 files, and run it.

The parameters can be set at line 27-29

Good luck!

Attached File  srat.py.zip   2.29KB   10 downloads



#8 StuartJPP

StuartJPP

    Mariner 2

  • -----
  • Posts: 270
  • Joined: 16 Oct 2015
  • Loc: Leeds, United Kingdom

Posted 22 May 2016 - 10:38 AM

It seems to work, it does create empty folders, not sure if you want that or not. Did try out the parameters which also seem to work.

 

Total 2 files are processed in C:\Utils\exiftool-8.96\srat.py
IMG_1874.CR2            : 21.0 C
IMG_1787.CR2            : 18.0 C



#9 Michael Covington

Michael Covington

    Author

  • *****
  • Vendors
  • Posts: 4057
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 22 May 2016 - 11:41 AM

You might find the code in EXIFLOG (which is open source,  http://www.covington...lr/exiflog.html ) useful; it does this for Canon.

 

Do Nikons record temperature in the EXIF data?   I'm about to get a Nikon D5300 and would like to add that to EXIFLOG for my own use, if it's there.



#10 ecloud

ecloud

    Mariner 2

  • -----
  • topic starter
  • Posts: 214
  • Joined: 17 Oct 2013
  • Loc: Dalian, China

Posted 22 May 2016 - 12:18 PM

It seems to work, it does create empty folders, not sure if you want that or not. Did try out the parameters which also seem to work.

 

Total 2 files are processed in C:\Utils\exiftool-8.96\srat.py
IMG_1874.CR2            : 21.0 C
IMG_1787.CR2            : 18.0 C

 

Hi,

Thank you for your test.

I will add the function to purge the empty folders in the final release when I get to know the detail about Nikon's RAW.



#11 ecloud

ecloud

    Mariner 2

  • -----
  • topic starter
  • Posts: 214
  • Joined: 17 Oct 2013
  • Loc: Dalian, China

Posted 22 May 2016 - 12:22 PM

Pentax .DNG in F^ would be "nice" Why .PEF? I always use .DNG when capturing RAW w/ my Pentax's...

 

PEF loses nothing but the file size than DNG ;)

Usually we want to save more space, so PEF is preferred format.



#12 ecloud

ecloud

    Mariner 2

  • -----
  • topic starter
  • Posts: 214
  • Joined: 17 Oct 2013
  • Loc: Dalian, China

Posted 23 May 2016 - 01:33 AM

Pentax .DNG in F^ would be "nice" Why .PEF? I always use .DNG when capturing RAW w/ my Pentax's...

Just head the K30, K500 modules do not have PEF

So I have added DNG support, you may try this

Attached File  srat.py.zip   2.48KB   4 downloads



#13 2020BC

2020BC

    Apollo

  • -----
  • Posts: 1101
  • Joined: 03 Aug 2005
  • Loc: Australia

Posted 23 May 2016 - 07:27 AM

Thankyou eCloud.

 

You are rightly to be commended and congratulated for this work.    :waytogo:  

 

Thanks again.


Edited by 2020BC, 23 May 2016 - 07:31 AM.

  • Michael Covington likes this

#14 Tonk

Tonk

    Cosmos

  • *****
  • Posts: 8616
  • Joined: 19 Aug 2004
  • Loc: Leeds, UK, 54N

Posted 23 May 2016 - 10:34 AM

There used to be a similar tool "DarkLibrary" that was knocking around a few years back but I can no longer find it via the links given here:
 

http://www.cloudynig...k-library-v12/​

 

There is also Dark Master on Source Forge

 


http://sourceforge.n...ects/darkmaster


Edited by Tonk, 23 May 2016 - 10:49 AM.


#15 ecloud

ecloud

    Mariner 2

  • -----
  • topic starter
  • Posts: 214
  • Joined: 17 Oct 2013
  • Loc: Dalian, China

Posted 23 May 2016 - 12:16 PM

The first stable release 1.01 is here:

http://www.cloudynig...by-temperature/



#16 Qkslvr

Qkslvr

    Apollo

  • -----
  • Posts: 1342
  • Joined: 23 Jun 2006
  • Loc: NE Ohio, US

Posted 27 May 2016 - 12:13 PM

Interesting thread. I might add something of value (or not). You can pull sensor temp right off the sensor, you need darks and a calibration.

PIN diode leakage is temp dependent, these cmos sensors use reverse bias pin diode as detectors. Then after charging the diode, light exposure discharges the voltage, after a bit of time they read the voltage that remains.

We take darks to measure this leakage without light (so we can subtract it), but each little pixel is also a thermometer. The leakage scales nicely with temp. I calibrated my camera by leaving the camera in the garage until cold, then turn it on and take 10 10sec darks immediately, while I recorded the air temp. I did this for a bunch of temps.

The one surprise I found was that about 1 in 10 10 sec dark had cosmic ray strikes.


  • Michael Covington likes this

#17 Michael Covington

Michael Covington

    Author

  • *****
  • Vendors
  • Posts: 4057
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 27 May 2016 - 06:11 PM

That is interesting and, of course, exactly like.  We talk about matching dark frames to the temperature.  We can equally well (with a little effort) estimate temperature from dark frames.



#18 Qkslvr

Qkslvr

    Apollo

  • -----
  • Posts: 1342
  • Joined: 23 Jun 2006
  • Loc: NE Ohio, US

Posted 06 June 2016 - 03:18 PM

That is interesting and, of course, exactly like.  We talk about matching dark frames to the temperature.  We can equally well (with a little effort) estimate temperature from dark frames.

That was why I was doing it, so I could match darks based on the sensor itself, but with my 40D, as long as it wasn't hot, I generally got better results without darks :)

Of course that could just be my processing skills......



#19 Astrofriend

Astrofriend

    Vostok 1

  • -----
  • Posts: 174
  • Joined: 12 Sep 2015
  • Loc: Stockholm, Sweden

Posted 08 June 2016 - 02:30 PM

Maybe some useful tips.

 

Earlier when I darkcalibrated my image files from my Canon camera I took help of the builtin temperature sensor. I used the program IrfanView to rename the filenames with temperature information. If I place the temperature in the beginning of the filename it's very easy to sort the darks by there temperatures.

 

You can read here in my short tutorial how I did this:

http://astrofriend.e...dslr-canon.html

 

If you use Deep Sky Stacker (DSS) to calibrate and stack the subimages you can with help of the software Dark Master pairing the temperature between image and dark files very easy and handy. You find that information also in the tutorial above.

 

You will never get this perfect because the temperature sensor what I know messaure the camera body, not the sensor temperature. And you can not have a unlimited number of temperatures. You can of course scale your darks and maybe get better result.

 

All this extra dark and bias calibration files add random noise which is not good. Better to skip them !

 

Nowadays I try an another technique.

 

Today I don't dark or bias calibrate my image files with dark and bias files, I replace them with a constant and only flat calibrate them. To get this to work you must take the images with dithering technique. Work best with modern cameras with low static pattern.

 

Here you have a tutorial about how I use the dithering technique:

http://astrofriend.e...-dithering.html

 

 

 

The latest is that I have started to make macros for the advanced and free program AstroImageJ (AIJ) to take dark and bias information from Canons overscan regions and get perfect temperture match because both image and dark come from the same image.

 

Here is my turorial about how I do this:

http://astrofriend.e...troduction.html

 

This latest step is in the beginning and it's a little bit more difficuld to do because AIJ is not so easy to handle, but my macros do it almost automatic.

 

 

 

 

What I know it's only Canon EOS with liveview (and 5D) that has a temperature sensor that you can read through EXIF data, or someone who knows better about Nikon, Sony, etc ?

 

And the overscan region that Canon has (I have seen it on 350D, 5D and 6D), when I look at Nikon and Sony files I can't see that they have that areas (a part of the sensors that is blocked from light), or someone who knows better?

 

You must setup DCRAW correctly to see the overscan area!

 

/Lars


Edited by Astrofriend, 08 June 2016 - 02:43 PM.

  • Michael Covington likes this

#20 Qkslvr

Qkslvr

    Apollo

  • -----
  • Posts: 1342
  • Joined: 23 Jun 2006
  • Loc: NE Ohio, US

Posted 21 June 2016 - 12:00 PM


You will never get this perfect because the temperature sensor what I know measure the camera body, not the sensor temperature.
 
 
 
But this is where what I was doing would be perfect. every pixel is a thermometer, you just need to figure a calibration, at which the body temp can help, you let the camera cool to air temp, turn it on and take your calibration shots, I used 10 subs with 10 sec exposures, then shut it off. Then use the body temp as your reference for that frame. You can also measure warming, by extending the number of frames, and see how the sensor warms compared to the body. And if you have some that leak enough, you can cut the exposure time as short as you want, and once you get your calibration, it's easy to figure.
 
The value recorded is the leakage current of that pixel, which is linearly dependent on temperature.
 
Now the dynamic range might not allow all pixels to be used, but the calibration will discover that, and there should be enough different ones that you can map any temp range you'd want to image at.
 
The other thing, about 1 in temp samples had cosmic ray tracks, you can to account for that when you calibrate.
 
I've thought of making a camera in a shielded open tube, but my tubing isn't big enough to fit the camera.


#21 Astrofriend

Astrofriend

    Vostok 1

  • -----
  • Posts: 174
  • Joined: 12 Sep 2015
  • Loc: Stockholm, Sweden

Posted 22 June 2016 - 03:00 AM

Hi Apollo,

Yes I read about your very interesting technique, very smart way to get the temperature and it works also for other cameras than Canon that don't have a built in temperturesensor.

 

But nowadyas I use the overscan area to calculate the mean darkcurrent and because it's a part outside the image from the same photo (the area that's blocked from light outside the image) I get perfect result (almost). Works very well with new cameras with low "static pattern" and you must take them with dithering technique. And I don't have to take separate dark images, I save my cameras lifetime and don't have to do this boring time consuming work.

 

Later I will improve it by mapping the defect hot pixels in the overscan region. But your point to compensate for cosmic ray maybe something to implement too, just how to figure out how much. But my progress is slowly because all other things other than astronomy that has to be done :-(

 

/Lars


Edited by Astrofriend, 22 June 2016 - 03:10 AM.


#22 ecloud

ecloud

    Mariner 2

  • -----
  • topic starter
  • Posts: 214
  • Joined: 17 Oct 2013
  • Loc: Dalian, China

Posted 29 June 2016 - 01:21 PM

Hi Astrofriend

Dithering is NOT to replace dark calibration, but an additional way to increase final SNR.

I use both of them and no calibration when sensor's temperature < 12*C

Canon DSLR does not record the true sensor's temperature, so it's not accurate.

Fortunately Pentax DSLR do record the sensor's temperature both before/after a picture is captured.

So it's meaningful for only Pentax to have a Dark-LIB in each 3*C like me:

屏幕快照 2016-06-30 02.02.59.png

 

And for Canon (camera temperature), I suggest at least 6*C as the interval.



#23 Qkslvr

Qkslvr

    Apollo

  • -----
  • Posts: 1342
  • Joined: 23 Jun 2006
  • Loc: NE Ohio, US

Posted 29 June 2016 - 01:56 PM

Canon DSLR does not record the true sensor's temperature, so it's not accurate.
 

It is easy enough to get, if you want it.



#24 Astrofriend

Astrofriend

    Vostok 1

  • -----
  • Posts: 174
  • Joined: 12 Sep 2015
  • Loc: Stockholm, Sweden

Posted 29 June 2016 - 04:29 PM

Hi Vostock,

For us who live north the temperature is not that big problem. We could only take deepsky images in the winter. Then the temperture is less than +5C degrees.

 

Interesting to know that also Pentax has a temperature sensor, there are not many people here in Sweden that use Pentax. Do you get the temperature in the EXIF data?

 

Earlier when I did take dark calibration images I normally skip the first 5 or more darkfiles until the camera has come in temperture equlibrium. If the EXIF temperature starts with -3C degree it normally stabilized at 0C degree after 5 images or some minutes. I normally take hundreds of them with different exposure times under nights when I couldn't take deepsky because of clouds. I get different temperature because the slowly temperature variations under the night. So even if the temperature sensor doesn't messure the sensor temperature it should not be to much error between image files and dark files temperatures. The EXIF temperature temperature differs about 5C degrees compare to the outside temperature in both dark and image files. If I setup my latest Canon 6D camera wrong the temperature difference could be much higher, as high as 20C degree if I have the liveview function activated.

 

Do you have a homepage about your software project? I think other astronomy people here in Sweden can be interested of your program, I can give a link in the Swedish forum.

 

/Lars



#25 ecloud

ecloud

    Mariner 2

  • -----
  • topic starter
  • Posts: 214
  • Joined: 17 Oct 2013
  • Loc: Dalian, China

Posted 29 June 2016 - 11:06 PM

If I live in Sweden and use a camera with Sony cmos, I will never do dark-calibration :cool:




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics