Disclaimer- I have not been paid for doing that many live recordings, so my opinion is that of somewhat of an amateur.
Gain riding is an art for sure. Riding faders was very common in the days before sophisticated compressors were available. In fact in older 78 classical music recordings you can sometimes hear very reactive turning up or down of the pots to try and squeeze a symphony into a less than 40dB dynamic range.
What you are saying you do is valid and probably most of us have done as well, playing "intelligent human compressor", just tweaking the level down slowly, smoothly and gently a bit at a time as you realize that "this is going to go way over zero real soon". If done right and a bit pro-actively, this is not too audible on the final recording, and IMHO less noticeable than compression from a device without a brain attached to the volume control. I have even seen pictures of people reading the musical score as a sort of "preview device".
How to know if you have stated off "too loud"? Well sometimes you have to because if you have had the benefit of some sort of rehearsal sound check and adjusted for the loudest parts, you find the quietest parts not even registering on the meters.
I can remember times when I set the opening quiet parts of a live classical recording to barely register on the meters and still had to gain ride and tweak down and down again to keep from blowing away the recording with meters fully pegged during the loudest parts. That is pretty typical actually.
It is, and always has been about skillfully artfully fitting the dynamic range of the program into the dynamic range of the media.
With analog tape, even at it's best without some sort of noise reduction, you don't want the majority of the recording "in the mud" for the sake of the loudest peaks. On the other hand, though you have some grace above zero, (unlike digital) you don't want to have the meters just pegged for any sustained periods either. The worst situation is of course very sudden dynamic contrasts (loud - soft - loud - soft) that are greater than 20-30db. Another difficult thing is an unaccompanied solo by an instrument particularly those with a higher pitch that is in the middle of a orchestral section of music. Gain riding artifacts, tape hiss, modulation noise and peak distortion all become much harder to avoid in these scenarios.
I guess for those situations I would rather be recording (gulp) digital just because of the extreme dynamic range. Though with digital I sometimes either hedge on the side of recording a bit "cold" and/or put a limiter in that activates just a tiny bit under zero because limiting is preferable to going over digital zero. The "grace" is at the low end of the meters in digital.
I say this because I never have had the budget to work with any analog machine (without noise reduction) that has a real dynamic range of even 60dB. Closer to 50 for most of the stuff I have had to work with. Many forms of live music, not just classical, are wider than that.