Welcome! I started caving underwater, then I started rock climbing, then I started caving (horizontally) where I didn't need a regulator in my mouth to breath, and now I am learning to do vertical caving. The transition from rock climbing to vertical caving isn't very difficult, but a thing to note about rock climbing vs. vertical caving: in rock climbing you climb *rock* and the rope is there Just In Case, but in vertical caving you (mostly) descend and climb *rope*. It might seem like a small distinction, but the two are fairly different.
There is an NSS Grotto in Columbia SC that you could hook up with: http://www.nssio.org/find_grotto_detail.cfm?gid=000359
And there are grottos in NC and GA that might be within your driving range. Check on http://www.nssio.org/Find_Grotto.cfm