Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

EUCLID or MECLID ?

  • Please log in to reply
7 replies to this topic

#1 yuzameh

yuzameh

    Mercury-Atlas

  • -----
  • topic starter
  • Posts: 2,669
  • Joined: 13 Dec 2022

Posted 25 May 2024 - 07:06 PM

Well, as my tax quids went towards the EU up to 2020 or so or whenever final brexit happened in practice and the EU shoves money into ESA and EUCLID, and my tax quids go towards ESA coz UK is a member thereof, all from the outset of the project, and as some of of my tax quids will also have gone towards Manchester Uni funding, I can always pretend that this is a big telescope I have got a share in, that is I've spent some cash towards EUCLID so it is also MECLID (probably a massive sum total of nearly 10p over the years...).

 

If you dig around the ESA EUCLID site you'll find that the FITS files are available for download.  They come in a giant TAR file of 7.2 Gb size, so they take some downloading, and expand out to massive gzip files (or is it the other way around) and they themselves extract out to even more massive files, comprising FITS image files or around 4.8 Gb in size.

 

Now, it took me a while to get a FITS viewer that'd work with files that big.  Granted Aladin did but with large files most zoom scales are sampled and only the full zoom gives the full image detail, which was incredible quality!  However, not covering much area.  As well as hanging up when trying to save it out to other formats and not allowing to save anything but the current view.

 

Also, I've never really tried to process any fits files before, my skills at such appear to be somewhere between less than zero and null.

 

So why am I banging on about it?  Well, because hidden on the ESA EUCLID site are other object that have been released, about five or so, as well as the colour enhanced versions released the other day to the public.  It took me a while to get anywhere with them, and my results are far worse than the originals, but the files were so big installing things like siril or what have you did nothing as the files were too immense, but I did get to save them to PNG files which were only very big, manged to load those in GIMP, tweaked and cropped 'em a bit in GIMP and then saved 'em as JPG and cropped and shrunk them until they'd fit inside the 500 kb limit here.

 

They're way more abysmal than the originals, but it'll give you an idea of what other objects are available if someone can manage to play with single FITS files of nigh on 5Gb.  Incidentally, they are all VIS images (the optical to just NIR camera), NIR camera images in Y, J and H bands are also available but are much smaller and grainier and attempting to merge a VIS, Y and J as blue, green and red just looked totally abysmal with hue issues so I ignore that.

 

ESA and the EUCLID consortium get the credit, these are public release images.  I'll leave them up a while before ditching them.

 

First, and not at all done well as the images are where orginally far more details on the screen but so narrow field zoomed in not worth screen capturing, on screen I was able to get the overexposed bright bits not overexposed and showing gently arcuate and near linear wisps whilst B30 and B31 looked good and crisp and dark whilst surround by nebulosity, but not able to get both in the image at that scale at the same time.

 

Barnard 30 :-

 

[attachment=2593400:Barnard30.jpg]

 

Second, the dwarf (satellite?) galaxy Holmberg II where I can't manage to get a better darkness without loosing the stars.

 

[attachment=2593402:HolmbergII.jpg]

 

and on the next post NGC 2403, which looked far better at one point and then I went and lost the best settings trying to get it better still!



#2 yuzameh

yuzameh

    Mercury-Atlas

  • -----
  • topic starter
  • Posts: 2,669
  • Joined: 13 Dec 2022

Posted 25 May 2024 - 07:10 PM

Finally, NGC 2403.  If anyone knows how to play with gigafiles in FITS formats and make better piccies I'd sure like to see them.  Search for ESA EUCLID Early Release Observations and look for the data access section and you'll finally find the tarballs.  There're also a couple more globular clusters and unidentified nebulosity near Taurus R2 available, ones that have not been released in the enhanced colour images (ie still monochrome, but files are available for each in VIS (optical) and Y, J and H band for all.

 

[attachment=2593405:NGC2403.jpg]

 

 



#3 yuzameh

yuzameh

    Mercury-Atlas

  • -----
  • topic starter
  • Posts: 2,669
  • Joined: 13 Dec 2022

Posted 25 May 2024 - 07:15 PM

 Re-found the link

 

https://euclid.esac.esa.int/dr/ero/

 

just click on each object to find the tarballs, VIS just over 7Gb files and NIR nearly 3 Gb files (containing fits files in three passbands so the individual files are much smaller and not as detailed, usually about half a gigabyte to the VIS usually nearly 5 Gb sizes).

 

Wish they'd have done the Barnard 30 one, that'd've been nice!

 

Oh, and technically I'm supposed to include the acknowledgement

 

"This work has made use of the Early Release Observations (ERO) data from the Euclid mission of the European Space Agency (ESA), 2024, https://doi.org/10.57780/esa-qmocze3."


Edited by yuzameh, 25 May 2024 - 07:17 PM.


#4 italic

italic

    Apollo

  • *****
  • Posts: 1,108
  • Joined: 05 Apr 2022
  • Loc: San Francisco, CA

Posted 26 May 2024 - 12:34 AM

Holmberg II is the first one I'm playing with here. All of these are heavily resampled to reasonable dimensions. The VIS sensor has a resolution of 0.1 arcsecond per pixel and a total field of view of ~40 arcminutes. Cropped and rotated to get the most image on-screen, the file is 28000x24400 pixels. That's a lot of sky!

 

zX8KQWWh.jpeg

 

jSFHVqgh.jpeg

 

6uD610nh.jpeg


  • lambermo and yuzameh like this

#5 yuzameh

yuzameh

    Mercury-Atlas

  • -----
  • topic starter
  • Posts: 2,669
  • Joined: 13 Dec 2022

Posted 26 May 2024 - 11:01 AM

Glorious!  Thanks a lot!  I knew I was making a mess of it.  Unfortunately I can only give 1 like, so plus umpteen.

 

Could you mention what software was able to cope with these files?  Also / or at least, whether it is commercial or GPL / public domain.

 

It appears I need to look into manual downsample choices.  Or start to use software in linux instead of windows, I have the former but don't do much astro in it.

 

Linux possibly because so little Windows software seems to be orientated to reading the files on disk, whatever those files are, sampling amongst them and using a temp file if needs be, instead windows tries to stick everything into RAM, which I had more than enough of to cope with these files, but software that didn't choke on file size still hung up or timed out.

 

Incidentally I only used the LSB files having absolutely no idea of how to use flats or masks.



#6 italic

italic

    Apollo

  • *****
  • Posts: 1,108
  • Joined: 05 Apr 2022
  • Loc: San Francisco, CA

Posted 26 May 2024 - 11:23 PM

I use Pixinsight to process everything. That's not the magic sauce, though; that's my workstation with 256GB of memory and 36 cores. It's a pretty nice upgrade from my 10-year-old machine, even if it's not state of the art. I haven't played with siril in ages, but it might be worth trying with these large files.



#7 yuzameh

yuzameh

    Mercury-Atlas

  • -----
  • topic starter
  • Posts: 2,669
  • Joined: 13 Dec 2022

Posted 27 May 2024 - 01:44 PM

Thanks for that, appreciated!

 

Your having the square of my memory capacity and a fifth power increase in my parallel processing cores sufficiently explains everything and means I have not been missing some magical piece of software, paid for or otherwise.

 

I did try siril, but 16 Gb RAM and 4 cores weren't enough I think as it hung more than crashed and the box seized up on these unsmall files.  Reminded me of trying to multitask on a 286 whilst using Windows 3.1.

 

Glad to know such behemoths are not solely being used for 4k streaming gameplay.  Fortunately, primarily for my pocket, I only have very rare circumstances where such hardware would be necessary, and then not for any essential purposes, more for matters of interest.

 

I must pull my finger out and get a fibreoptic level bandwidth though if data files are going to be that big as standard before long.



#8 italic

italic

    Apollo

  • *****
  • Posts: 1,108
  • Joined: 05 Apr 2022
  • Loc: San Francisco, CA

Posted 27 May 2024 - 04:26 PM

It actually wasn't very expensive to buy this machine. I was in the market for a new computer anyway and I was originally looking at building just like I had before, but someone pointed to an ebay seller for previous-generation workstations that totally swamped what I was going to build. It was a no-brainer. I've been pretty happy with it and the only part I would need to upgrade is the GPU from my old machine, but it's doing perfectly fine for what I need it to do.




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics