...what are the downfalls of manipulated hands swing?
There is no downside to Manipulated Hands Swinging. In fact, Homer Kelley specifically discouraged True Swinging due to the terrific problem of Ball Location.
Are we, therefore, saying that any hand manipulation is best done at address as opposed to during the actual swinging process?
Hand Manipulation is accomplished at Impact Fix with the Hands Location established per 7-8 and with the Clubface aligned per 2-J-1. The Feel of the selected Stroke Pattern (Total Motion) is then rehearsed in the Adjusted Address Routine of 3-F-5 -- Practice Swing, Waggle and Forward Press.