I personally know how long it takes to grow trees, as I planted one-footer evergreens, and trust me; it's many years for them to achieve any growth.
It depends where you live. In the eastern U.S., trees are weeds; it's very hard to cut them as fast as they grow. If you don't keep on top of them, they take over in a hurry.
My grandparents bought our property in the country in 1930. When my father was growing up there, it was fields as far as the eye could see -- just lines of trees along the edges of the fields and a few stands of trees here and there left to be cut for firewood. Now it's solid forest with just a few clearings. The very biggest trees are the ones that used to line the fields. But even the younger trees are now 50+ feet tall and a couple of feet in diameter. All of that in one human lifetime.
If trees are healthy, it's a shame to destroy them.
I don't cut any tree needlessly, but I have no qualms about cutting down a few dozen to improve my view for astronomy. There are thousands more where they came from.
I saw hundreds in my area destroyed by the pine bark beetle; that area will never regrow in most of our lifetimes.
That's a different story entirely; the pine-bark beetle infestation is truly disastrous.
There are similar infestations in the East. But unlike the West, where forests tend to be monocultures, we have dozens of species in any given forest stand. So if one species diminishes, others will eagerly take over.