187 replies to this topic

[lambermo]

Caught an issue after 26 hours or 28 268 images. A ZWO: getexpstatus EXP_FAILED , after which PHD2 retried it 3 times then gave up and reconnected to the camera (and did not enable cooling) but continued making images.

 

Here's the log if anyone's interested in that :

01:48:08.401 00.000 140737072191232 worker thread servicing REQUEST_EXPOSE 1000
01:48:08.401 00.000 140737072191232 Handling exposure in thread, d=1000 o=3 r=(0,0,0,0)
01:48:09.702 01.301 140737072191232 ZWO: getexpstatus EXP_FAILED, retry exposure
01:48:11.004 01.302 140737072191232 ZWO: getexpstatus EXP_FAILED, retry exposure
01:48:12.305 01.301 140737072191232 ZWO: getexpstatus EXP_FAILED, giving up
01:48:12.323 00.018 140737072191232 worker thread setting skip send exposure complete
01:48:12.323 00.000 140737072191232 worker thread queueing reconnect event to GUI thread
01:48:12.323 00.000 140737072191232 Error thrown from /home/hans/src/phd2/worker_thread.cpp:157->Capture failed
01:48:12.323 00.000 140737353747008 Alert: Lost connection to camera
PHD will make several attempts to re-connect the camera.
01:48:12.323 00.000 140737072191232 worker thread skipping SendWorkerThreadExposeComplete
01:48:12.323 00.000 140737072191232 worker thread done servicing request
01:48:12.331 00.008 140737353747008 Try camera reconnect, now = 1568936892
01:48:12.331 00.000 140737353747008 gear_dialog: ReconnectCamera
01:48:12.331 00.000 140737353747008 gear_dialog: DoConnectCamera [ZWO ASI Camera]
01:48:12.331 00.000 140737353747008 Status Line: Connecting to Camera ...
01:48:12.515 00.184 140737353747008 GetString("/profile/4/cam_hash/fb6ac968ce0ca5aa/whichCamera", "") returns ""
01:48:12.515 00.000 140737353747008 Connecting to camera [ZWO ASI Camera] id = []
01:48:12.516 00.001 140737353747008 ZWO: find camera id: [], ncams = 1
01:48:13.193 00.677 140737353747008 ZWO: using mode BPP = 8
01:48:13.194 00.001 140737353747008 ZWO: usb3 = 1, is_mini = 0, name = [ZWO ASI1600MM-Cool]
01:48:13.194 00.000 140737353747008 ZWO: selecting snap mode
01:48:13.194 00.000 140737353747008 ZWO: IsColorCam = 0
01:48:13.194 00.000 140737353747008 ZWO: supported bin 0 = 1
01:48:13.194 00.000 140737353747008 ZWO: supported bin 1 = 2
01:48:13.194 00.000 140737353747008 ZWO: supported bin 2 = 3
01:48:13.194 00.000 140737353747008 ZWO: supported bin 3 = 4
01:48:13.389 00.195 140737353747008 ZWO: camera has cooler
01:48:13.389 00.000 140737353747008 ZWO: gain range = 0 .. 600
01:48:13.390 00.001 140737353747008 ZWO: lowest RN gain = 300 (50%)
01:48:13.390 00.000 140737353747008 ZWO: frame (0,0)+(4656,3520)
01:48:13.424 00.034 140737353747008 Connected Camera: ZWO ASI1600MM-Cool
01:48:13.424 00.000 140737353747008 FullSize=(4656,3520)
01:48:13.424 00.000 140737353747008 PixelSize=3.80
01:48:13.424 00.000 140737353747008 BitsPerPixel=8
01:48:13.424 00.000 140737353747008 HasGainControl=1
01:48:13.424 00.000 140737353747008 GuideCameraGain=50
01:48:13.424 00.000 140737353747008 HasShutter=0
01:48:13.425 00.001 140737353747008 HasSubFrames=1
01:48:13.425 00.000 140737353747008 ST4HasGuideOutput=1
01:48:13.425 00.000 140737353747008 GetBoolean("/profile/4/camera/AutoLoadDefectMap", 1) returns 1
01:48:13.425 00.000 140737353747008 auto-loading defect map
01:48:13.425 00.000 140737353747008 Loading defect map file /home/hans/.phd2/darks_defects/PHD2_defect_map_4.txt
01:48:13.425 00.000 140737353747008 Defect map file not found: /home/hans/.phd2/darks_defects/PHD2_defect_map_4.txt
01:48:13.425 00.000 140737353747008 Status Line: Defect map not loaded
01:48:13.426 00.001 140737353747008 GetBoolean("/profile/4/camera/AutoLoadDarks", 1) returns 1
01:48:13.426 00.000 140737353747008 Auto-loading dark library
01:48:13.426 00.000 140737353747008 Error thrown from /home/hans/src/phd2/myframe.cpp:2313->File does not exist
01:48:13.426 00.000 140737353747008 failed to load dark frames from /home/hans/.phd2/darks_defects/PHD2_dark_lib_4.fit
01:48:13.426 00.000 140737353747008 Status Line: Darks not loaded
01:48:13.427 00.001 140737353747008 Status Line: Camera Connected
01:48:13.430 00.003 140737353747008 Camera Re-connect succeeded, resume exposures
01:48:13.430 00.000 140737353747008 ScheduleExposure(1000,3,0) exposurePending=0
01:48:13.430 00.000 140737353747008 Enqueuing Expose request
01:48:13.431 00.001 140737072191232 Worker thread wakes up
01:48:13.431 00.000 140737072191232 worker thread servicing REQUEST_EXPOSE 1000
01:48:13.431 00.000 140737072191232 Handling exposure in thread, d=1000 o=3 r=(0,0,0,0)
01:48:13.431 00.000 140737072191232 ZWO: set CONTROL_EXPOSURE 1000000
01:48:15.700 02.269 140737072191232 Exposure complete

-- Hans

[jhayes_tucson]

Hans,

Thanks!!  That is interesting...and it took longer than it typically has on my two cameras.  I wonder if this could be due to a hardware issue in the camera itself?

 

John

[CygnusBob]

John

 

It looks like a hardware failure to me.

 

The getexpstatus call is from the controlling application (PHD2, Maxim DL) to the camera driver about the status of the exposure.  The answer back to the application "EXP_FAILED" means that for some reason the camera failed to correctly complete the exposure.  The later calls look like the application is going through all the steps to reconnect with the camera.

 

This stuff is familiar to me because I have written application code like this using ZWO's SDK.

 

Is possible there is a software problem?  Not likely because it can run so long without a problem.  However you guys might look to see if there is a memory leak.  You can do this by monitoring memory usage.  If it slowly increases during the run, that is a sign of a memory leak.  It should stay roughly constant with no long term trend.  Also if the crash occurs in less time if larger images are returned.

 

 

Bob

Edited by CygnusBob, 20 September 2019 - 03:52 PM.

[kingjamez]

As a point of reference, my ASI183mm-pro has been running on PHD for 30 hours with the cooler on with no disconnects. I'm going to stop and change over to the ASI1600mm-c tonight.

 

-Jim

[jhayes_tucson]

Here's a file showing what the failure looked like when I ran PHD.  It does look like the camera just stops responding after a frame is started.  I had the "Reconnect unresponsive camera" option set to 180s so it took 3 minutes before PHD disconnected the camera and reconnected it, which can be seen near the end of this data stream.

 

Rather than post the data here, I've attached a PDF so that anyone who is interested can take a look.

 

This error does not appear to be the result of a memory leak since I've seen the error occur at a wide rand of intervals (as short as 15 minutes) and it does not crash the computer or the app.  Furthermore, almost all of my testing involves only transferring a 128 x 128 pixel array.  The PHD test involved full 16Mpx image transfers but that one ran for 7.5 hr before it hung--and again nothing crashed.  I can do more testing to confirm that conclusion 100% but I'm 99%+ certain that this problem isn't the result of a memory leak.

 

John

 

Attached Files

•  PHD ASI-1600 Failure Log.pdf   55.16KB   13 downloads

[CygnusBob]

One thing that is a mystery to me about these cameras is ZWO's claim that  they need  to "include a 256MB DDR3 memory buffer to help improve data transfer reliability."  If you are sending a 16 MB image maybe every 5 seconds why do they need to use a 256 MB buffer to ensure RELIABILITY?  This buffer is in a DSO camera where operating in the video mode is not likely to be used. I wonder if this buffer is there as a work around for some sort of reliability issue that existed in the original camera design?  Maybe I am missing something here but it does seem strange.

 

Its like in order to ensure that I make to work today I installed a 250 gallon tank in my car!

 

Bob

Edited by CygnusBob, 20 September 2019 - 05:31 PM.

[Tonk]

"include a 256MB DDR3 memory buffer to help improve data transfer reliability."

It may not be capacity that is the issue - but one of mitigating against processing latency. I.e. if overall its not able to process fast enough then an error state is reached. That's what buffers allow - queuing.

E.g. for a completely speculative example - one synchronous process is inserting data in the tail of the queue - and another asynchronous process is consuming the data from the head of the queue - the buffer is the storage intermediary between the processes. The queue space allows for limited periods where the consumer is not working fast enough to clear the items as they arrive. If the buffer fills up (possibly because the consumer is seriously delayed - for whatever reason - e.g your virus checker just kicked off is daily full memory scan) then an error state occurs.

Not saying that this is happening - but its a valid reason why your car might need a 250 gallon tank if its being continuously filled as you drive circuits around the fuel pump trying to avoid a tank overflow

[jdupton]

Bob,

 

One thing that is a mystery to me about these cameras is ZWO's claim that  they need  to "include a 256MB DDR3 memory buffer to help improve data transfer reliability."  If you are sending a 16 MB image maybe every 5 seconds why do they need to use a 256 MB buffer to ensure RELIABILITY?  This buffer is in a DSO camera where operating in the video mode is not likely to be used. I wonder if this buffer is there as a work around for some sort of reliability issue that existed in the original camera design?  Maybe I am missing something here but it does seem strange.

 

Its like in order to ensure that I make to work today I installed a 250 gallon tank in my car!

 

Bob

 

   The buffer is there for two reasons:

• To reduce Amp Glow from the sensor
• Ensure against data dropouts during image transfer

   Many / most CMOS sensors show Amp Glow when the image is being read out of the sensor. The faster you read the image out, the less Amp Glow will be seen. The fast DRAM buffer allows the sensor to be read at a much faster rate than if the sensor had to be synchronized to the transfer rate of the USB bus.

 

   Most of the CMOS sensors we use transfer data as a video stream even if it is only one frame. This is called Isochronous Transfer Mode in the USB Bus. Once an image transfer starts, if there is any hiccup in the transmission, the data (the whole frame) is lost. These cameras need to be able to work on both USB3 and USB2 connections. Even using USB3 capable systems, soemtimes the connection will "fall back" to a lower speed if errors are detected at the highest speed possible. With the Buffer, the camera can stream the data to the PC at a rate appropriate to the connection type / quality. Having the buffer present allows them to transfer the image, once read into the buffer, at whatever speed works best for the connection. 

 

 

John

[Jon Rista]

One thing that is a mystery to me about these cameras is ZWO's claim that  they need  to "include a 256MB DDR3 memory buffer to help improve data transfer reliability."  If you are sending a 16 MB image maybe every 5 seconds why do they need to use a 256 MB buffer to ensure RELIABILITY?  This buffer is in a DSO camera where operating in the video mode is not likely to be used. I wonder if this buffer is there as a work around for some sort of reliability issue that existed in the original camera design?  Maybe I am missing something here but it does seem strange.

 

Its like in order to ensure that I make to work today I installed a 250 gallon tank in my car!

 

Bob

It has to do with both reliability and glow and vertical banding variability with transfer speed. Slower image transfer, more glow & banding. With the on-camera DDR buffer, they read the entire frame out to the buffer very very quickly, which minimizes glow & banding (a lot, actually, with the Panasonic M sensor.)

 

There have been reliability issues in pre-Pro versions of the ASI1600. V1 through V3 had some issues with transfers when the USB bandwidth setting was configured less than optimally for the user's system. Poor configuration results in hung cameras or dropped frames. Which settings are optimal varies with the system. With the Pro versions and their DDR buffer, the images are always read out at maximum speed for optimal quality, then the driver is able to dynamically determine the best bandwidth configuration and change it as necessary to transfer the data from the buffer to the computer.

 

FTR, people do use the ASI1600 in video mode. Lot of people have used them for EAA, Lunar imaging, even some planet and solar imaging. In those cases the DDR buffer is also useful as with 16mp frames the amount of data you can acquire at frame rates around 15-20fps is quite high.

Edited by Jon Rista, 20 September 2019 - 11:04 PM.

[bortle2]

One thing that is a mystery to me about these cameras is ZWO's claim that  they need  to "include a 256MB DDR3 memory buffer to help improve data transfer reliability."  If you are sending a 16 MB image maybe every 5 seconds why do they need to use a 256 MB buffer to ensure RELIABILITY?

Well, you may be sending 16Mb images every 5 seconds, but you may also be sending 17 frames per second (or even 23 in 12-bit ADC mode). And even 256Mb may be not enough... Here is what ZWO INDI driver page says about ZWO cameras (their ZWO driver is one of the very few, if not the only one, that has non-empty "Known issues" section) here:

 

The ASI Cameras are very USB bandwidth hungry when running at high FPS. If you see "broken frames" with more that one of them running, keep in mind this important aspect. ASI Cameras continuously generate frames and they are buffered in the driver.

 

As an illustration: ZWO's ASI183MC and Altair's Hypercam 183C are based on the same chip (IMX183) and have very similar specs, but the former has 256Mb of DDR3, while the latter has 4Gb of DDR3 memory -- apparently, Altair recons that 256Mb is not enough to "help improve data transfer reliability".

 

Don't think any of this has much to do with John's issues (he's not running max frame rates and used ASCOM, not INDI drivers, I guess), but it's still an interesting data point on ZWO cameras.

[jhayes_tucson]

I want to report that I’ve now been using the camera for a few weeks now with the cooler temperature set to 40C to keep the camera cooler turned off and I have seen no more hangs.  The camera runs reliably as long as no power goes to the cooler.  This is very strange and it must have something to do with an unknown problem in the camera hardware.  I plan to talk to ZWO about it.  Clearly something is wrong; but, not so wrong that many people will ever notice.  If it were my design, I’d sure want to understand it better if for no other reason but to make sure that it doesn’t get propagated into any other products.

 

John

[SeymoreStars]

I want to report that I’ve now been using the camera for a few weeks now with the cooler temperature set to 40C to keep the camera cooler turned off and I have seen no more hangs.  The camera runs reliably as long as no power goes to the cooler.  This is very strange and it must have something to do with an unknown problem in the camera hardware.  I plan to talk to ZWO about it.  Clearly something is wrong; but, not so wrong that many people will ever notice.  If it were my design, I’d sure want to understand it better if for no other reason but to make sure that it doesn’t get propagated into any other products.

 

John

Yeah!!

[f430]

Thank goodness, as I enjoy your posts and pictures and would hate to see you quit!