Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

What to do with old data?

Astrophotography
  • Please log in to reply
17 replies to this topic

#1 SpaceElements

SpaceElements

    Lift Off

  • -----
  • topic starter
  • Posts: 4
  • Joined: 13 Feb 2021

Posted 22 June 2022 - 04:57 PM

Hi all, I'm curious if anyone can share what they've done with their older data? I've been in to Astronomy for well over a decade but only recently been in a position where I have equipment that allows me to truly take the hobby seriously. For the first several months with this equipment (and even now, at times), I was over exposed, under exposed, off-target, below 30 degrees, etc. But I've kept all of the data from every session in relatively cheap cloud storage. I know that I'll probably never attempt to reprocess most of it, because a lot of it is just plain terrible relative to my progressive journey. I guess I could see me using some of it in a presentation to demonstrate what not to do someday, but other than that... I've just sort of thought of it like a journal, documenting everything from the beginning. I'm curious if anyone has an opinion with data hoarding and whether they regret deleting their data from their early experiences or otherwise.


Edited by SpaceElements, 22 June 2022 - 04:58 PM.


#2 alphatripleplus

alphatripleplus

    World Controller

  • *****
  • Moderators
  • Posts: 130,613
  • Joined: 09 Mar 2012
  • Loc: Georgia

Posted 22 June 2022 - 05:00 PM

Moving from EAA to Beginning Deep Sky Imaging for a better fit.


  • SpaceElements likes this

#3 JTNX01DC

JTNX01DC

    Explorer 1

  • *****
  • Posts: 77
  • Joined: 14 Mar 2016
  • Loc: MA 01463

Posted 22 June 2022 - 05:15 PM

My personal philosophy is "if in doubt, throw it out" .


  • Sheridan and SpaceElements like this

#4 KungFood

KungFood

    Apollo

  • -----
  • Posts: 1,242
  • Joined: 18 Jul 2014
  • Loc: Dallas, Tx

Posted 22 June 2022 - 05:19 PM

I keep my old data, but I make sure it understands that it is bad...bad data. We do not overexpose..bad data. Then I thump it on the nose.


  • klaussius, Tim J Fowler, Skysmacker and 3 others like this

#5 Tapio

Tapio

    Fly Me to the Moon

  • -----
  • Posts: 6,500
  • Joined: 24 Sep 2006
  • Loc: Tampere, Finland

Posted 22 June 2022 - 05:19 PM

I too have old data but in USB storage disks and even DVDs.
Sometimes I have even used old data for some purpose.
  • SpaceElements likes this

#6 RogerM

RogerM

    Viking 1

  • *****
  • Posts: 635
  • Joined: 13 Aug 2020
  • Loc: California

Posted 22 June 2022 - 05:45 PM

As I obtain more or new data on any one particular target I immediately cull out the poor subs based upon obvious errors such as noise (gradients from a passing car for example), other atmospheric disturbances and the occasional pilot in the sky.  I then filter based upon FWHM, Eccentricity, and PSFFlux.  This way I never keep unusable (new or old) data.  I do however keep an assortment of integrated images (one copy processed and another unprocessed) for sentimental reasons.  Requisite first astro image (Milky Way), first DSO...otherwise all poor subs are tossed.

 

It's hard enough to collect good data, the poor stuff is easy to replace!


  • BinoGuy and SpaceElements like this

#7 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 29,355
  • Joined: 27 Oct 2014

Posted 22 June 2022 - 05:59 PM

I save masters and stacks.  Not individual frames.  I have maybe 20TB of storage.


  • SpaceElements and Loopster like this

#8 SpaceElements

SpaceElements

    Lift Off

  • -----
  • topic starter
  • Posts: 4
  • Joined: 13 Feb 2021

Posted 22 June 2022 - 06:07 PM

As I obtain more or new data on any one particular target I immediately cull out the poor subs based upon obvious errors such as noise (gradients from a passing car for example), other atmospheric disturbances and the occasional pilot in the sky.  I then filter based upon FWHM, Eccentricity, and PSFFlux.  This way I never keep unusable (new or old) data.  I do however keep an assortment of integrated images (one copy processed and another unprocessed) for sentimental reasons.  Requisite first astro image (Milky Way), first DSO...otherwise all poor subs are tossed.

 

It's hard enough to collect good data, the poor stuff is easy to replace!

Perhaps this is a way to have my cake and eat it to (aka clean house, but still hoard a bit). Quick question - you noted the integrated images. I am assuming that you calibrate your images before filtering based on FWHM, Eccentricity, and PSFFlux. Do you keep the calibrated only images as well, or only those which meet your standards and are calibrated and integrated?

 

I'm currently shooting with an SBIG Aluma 4040 as my primary camera (though I have more that are coming online). Based on the shorter exposures seen with sensors like the GSENSE 4040, I end up with an excessive amount of images. With each sub at 32MB it really adds up. And like many others who have commented on working with this chip, I haven't been particularly successful in my attempts to take advantage of in camera stacking capabilities (e.g. Diffraction Limited / Maxim DL HighStack Pro).


Edited by SpaceElements, 22 June 2022 - 06:16 PM.


#9 RogerM

RogerM

    Viking 1

  • *****
  • Posts: 635
  • Joined: 13 Aug 2020
  • Loc: California

Posted 22 June 2022 - 06:50 PM

Perhaps this is a way to have my cake and eat it to (aka clean house, but still hoard a bit). Quick question - you noted the integrated images. I am assuming that you calibrate your images before filtering based on FWHM, Eccentricity, and PSFFlux. Do you keep the calibrated only images as well, or only those which meet your standards and are calibrated and integrated?

 

I'm currently shooting with an SBIG Aluma 4040 as my primary camera (though I have more that are coming online). Based on the shorter exposures seen with sensors like the GSENSE 4040, I end up with an excessive amount of images. With each sub at 32MB it really adds up. And like many others who have commented on working with this chip, I haven't been particularly successful in my attempts to take advantage of in camera stacking capabilities (e.g. Diffraction Limited / Maxim DL HighStack Pro).

Sub calibration only happens after culling, always.  For the sentimental images I only keep non post processed calibrated/integrated masters (for later attempts at post processing as my skill improves) otherwise just the fully calibrated and integrated final image.  For W.I.P., I keep culled raw subs, calibrated subs and calibration masters.  Keep in mind that much of my imaging time only comes in 1-2 hour chunks so I have to be patient and keep judicious notes to help me to reconstruct any calibrated subs as more data comes in.  Otherwise I keep a very limited number of calibrated subs on hand, those that have absolutely the best possible characteristics that I can acquire.  I have yet to acquire enough data on any one target to call it a completed project!

 

I have no experience with a camera such as yours, it sounds amazing, in-camera stacking will certainly reduce storage overhead...I take it it can discern and reject poor data during acquisition?


  • SpaceElements likes this

#10 dswtan

dswtan

    Apollo

  • *****
  • Posts: 1,378
  • Joined: 29 Oct 2006
  • Loc: Morgan Hill, CA

Posted 22 June 2022 - 07:57 PM

Storage is cheap. There’s no hoarding police. I keep everything. I rarely touch it, but I then I don’t have regrets. This is definitely a “there are two sorts of people in the world” thing. The two don’t understand each other, and that’s fine. You must know which you are, in your heart. 


  • pedxing, BinoGuy, SpaceElements and 1 other like this

#11 KungFood

KungFood

    Apollo

  • -----
  • Posts: 1,242
  • Joined: 18 Jul 2014
  • Loc: Dallas, Tx

Posted 22 June 2022 - 08:15 PM

Joking aside, I keep only mostly everything.

- Calibrate

- Cull

- Integrate

- Process

 

After culling I drop the good subs in a folder called "stackable". This folder I keep. All other intermediate files I delete. After the full shebang I'll have:

- All of my original raw subs, minus those that were culled, along with calibration frames for each group in folders by date, in a folder called "sessions"

- All of the subs deemed stackable after the full calibration process in the "stackable" folder

- A folder with masters of the darks/flats/etc

- A folder with raw stacks

- A working folder, for masks etc.

- A folder for finals (divided into folders for each file type saved out)

- All of the above goes into a dated folder within the target's folder, I.e. M101>Summer2022>

 

The largest size on disk is taken up by intermediate files. For my last large project the total size on disk was 150 gigs. After deleting the intermediate files, it was 20 gigs. Raw subs don't take up much room by comparison. Bunch of ways to do it, none wrong, but that's my method.


Edited by KungFood, 22 June 2022 - 08:18 PM.

  • SpaceElements likes this

#12 ChiTownXring

ChiTownXring

    Apollo

  • *****
  • Posts: 1,014
  • Joined: 12 Aug 2020
  • Loc: Sugar Grove, IL

Posted 23 June 2022 - 12:46 PM

I have been keeping my old hard drives for years and low and behold they have come in handy to store old data and just throw them in a box. Will I ever use the old data? I doubt it but it's there just in case.. 


  • dswtan and SpaceElements like this

#13 klaussius

klaussius

    Surveyor 1

  • *****
  • Posts: 1,642
  • Joined: 02 Apr 2019
  • Loc: Buenos Aires

Posted 23 June 2022 - 12:53 PM

I don't know if it's worth their entropy, but I keep all my old data, including raw subs.

 

I've been keeping them in an assortment of hard drives, and some ones I find particularly worhty I will upload to the cloud for backup.

 

When I have enough data (which is not yet), I plan to use it. To build a timelapse or something of the sort.

 

Old scientific data is being re-scanned all the time for undiscovered phenomena, so there's no reason to think my old data couldn't hold something interesting that I haven't noticed yet.

 

Maybe not, but storage is cheap, so keeping it is not a big deal for me.


  • dswtan and SpaceElements like this

#14 SpaceElements

SpaceElements

    Lift Off

  • -----
  • topic starter
  • Posts: 4
  • Joined: 13 Feb 2021

Posted 23 June 2022 - 01:24 PM

I have no experience with a camera such as yours, it sounds amazing, in-camera stacking will certainly reduce storage overhead...I take it it can discern and reject poor data during acquisition?

My understanding of the technology is that you essentially declare a sub exposure length that you could like to see on a target using a CCD camera, then when you select HighStack Pro it somehow divides the exposure you've selected in to multiple subs, then stacks them. Based on the support forum conversations I've read over at DL, it may be something slightly better than doing simple division (e.g. you select a 600 second sub exposure and it divides by 16 for 37.5 subs, then stacks them in to one FITS file). It's proprietary so I'm speculating, but it might do something like (let n = user defined sub length, x = 16 / n, p = x + 1, q = x - 1, capture 8p, capture 8x). I'm not being precise in the example, nor do I have any proof of whatever algo they use... maybe DL will pop in and share how it works someday. It has been my experience that it is not effective at reducing poor data. Aircraft and shooting stars still remain present as well as other defects. And then, should you end up with a good set of HighStack Pro subs (which are themselves already stacked), using PixInsight to assess them tends to be all over the place. That said, if anyone reading this has a different experience I would love to learn. As far as I can tell, most people either shoot on high gain only, or do some combination of high and low gain, with matching flats, dark flats, and darks (bias is generally avoided with the GSENSE 4040).



#15 SpaceElements

SpaceElements

    Lift Off

  • -----
  • topic starter
  • Posts: 4
  • Joined: 13 Feb 2021

Posted 23 June 2022 - 01:24 PM

Thanks all! This has given me some good insight.



#16 Oort Cloud

Oort Cloud

    Mercury-Atlas

  • -----
  • Posts: 2,657
  • Joined: 19 Nov 2020
  • Loc: New Jersey, USA

Posted 23 June 2022 - 01:54 PM

I keep an unprocessed linear FITS, fresh outta the stacker. I do linear processing , followed by an initial stretch in Siril and save that as a TIF. Then I finish the processing in GIMP, and save both a TIF (for future tweaks) as well as an 8-bit PNG for sharing. If I make any tweaks later, I replace the last 2 with the updated version.

If I think I might want to add more lights, I keep the calibrated lights that went into the stack.

#17 unimatrix0

unimatrix0

    Vanguard

  • *****
  • Posts: 2,300
  • Joined: 03 Jan 2021

Posted 23 June 2022 - 02:04 PM

I only got like 4 TB filled with data and they go back to 2019/2020.  I do have a lot of junk too, but I keep learning new processing tricks and thinking about preprocessing some of my old data, but of course, I never get to it. 

I also got old data that is absolutely not worth of keeping (bad stars, over exposed or just plain bad)  and yet my mouse cursor moves away from the delete menu every time I try it. I get sentimental about images taken with cameras that I don't own anymore. 


  • pedxing likes this

#18 idclimber

idclimber

    Mercury-Atlas

  • *****
  • Posts: 2,803
  • Joined: 08 Apr 2016
  • Loc: McCall Idaho

Posted 23 June 2022 - 02:28 PM

Like Bob I have a dedicated large storage drive for my images. It is a QNAP unit with 24tb of storage. 

 

Recent changes to the preprocessing script within Pixinsight has me convinced keeping uncalibrated subs is better than calibrated ones if you plan to reprocess data in the future. I keep the original subs with the Master Flats from that session and my library of darks, bias and dark flats. The new script has new weighting algorithms and analyzes the images while pre processing. I believe it is an improvement over prior methods I used just a year ago. 




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics





Also tagged with one or more of these keywords: Astrophotography



Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics