I disagree with several of the other responders who said that the only advantage to the larger scope is increased resolution, presumably since the focal ratio is the same. This is simply not true. That's one possible advantage, but not the only one. Larger telescopes collect more photons from your target object in a given amount of time. Only the irradiance will be the same since the focal ratio is the same. If you are shooting a subject where the extra resolution matters, then the 5" scope will win on resolution for a given pixel size, with very similar SNR binned 1x1. However, if you are imaging a subject where you are less worried about resolution or where you are oversampled for the conditions, you can downsample the image from the 5" to get better SNR than the 4" can produce at the same sampling rate. With the larger telescope, you have the choice of either having the higher spatial resolution or a deeper image for a given integration time. If you were to re-sample by a factor of 0.8, the 5" scope would yield a deeper image with the same arc seconds per pixel as the 4" scope for a given pixel size.
Assuming similar optical quality and assuming the mount was capable of carrying the larger scope and assuming you didn't care about the extra mass and difficulties transporting/setting up the larger scope, and assuming cost wasn't a factor, I would absolutely take the 5" scope. It will yield either better resolution (even on large nebulae) or better SNR. The only down-side is you will lose some field of view. For some subjects that will be really annoying, while for others it will actually be beneficial. Depends on the subjects.
The biggest issue is that the assumptions I listed are rarely correct. Generally, cost matters, bulk matters, and the moment arm of inertial from the larger scope matters. Also, with the 4" scope you know the reducer/flattener was designed with the particular objective in mind, so they will be well matched and you won't need to worry about back-focus at all. With the 5" scope and the reducer, depending on the particular models the reducer/flattener may or may not be perfectly matched to the objective, and you will need to get the back-focus exactly right which can be a hassle. It's only a one-time hassle, but still a hassle.
If it helps at all with your decision... I have a 130mm refractor of excellent quality and a 110mm refractor of equally excellent quality. I use both scopes for astrophotography, though the 130mm tends to get pulled out more frequently for visual use. For astrophotography, I choose the 110mm about 95% of the time over the 130mm. It's MUCH smaller and lighter. I can carry it with its tripod and mount to my most frequent observing site in a single trip from the car. The 5" requires a hand truck and a significantly heavier mount. I know you didn't want to consider factors other than the optics themselves, so I provided that answer first.
Based on optics alone? I would take the 5" if the reducer/flattener was well matched. The only real downside is the narrower field of view.
If I had to account for portability, cost, mounting, etc.? I would take the 4" in a heartbeat. The ease of use advantage for the 4" can't be overstated.