I live near Huntsville and often go caving after work. There are dozens of really nice caves within a 20 minute drive of my house. On the weekends, there are thousands of really, really nice caves within a one hour drive of my house. If I want to drive for a couple hours, that number jumps even higher. The same is true of Chattanooga, Nashville, and other areas in Tennessee. Don't move to Atlanta, the traffic is horrendous and caves aren't close to the city!
If you like good vertical caving, move somewhere near Jackson County, AL (Huntsville or Chattanooga). There are many spectacular caves in that area. But the best thing about caves anywhere in TAG is that we have pretty much anything you could ever want. Wet, dry, horizontal, vertical, we have it all. Plus, we have so many caves that you could live here your whole life and still not make it to every fabulous cave in the region. And I wouldn't move to the southwest if I were you. It seems like most of the good caves out there are on federal land and they're all getting closed for WNS. Here in TAG, most of our caves are on private land, and while some great caves are closed, there are still thousands of great ones that will remain open. Also, the Grottos in TAG are generally very welcoming of new people. I'm in the Huntsville Grotto and we try to help people new to the area meet other cavers and we'd take you on trips to some nice caves. I often meet up with cavers from all over TAG on the weekends, we're a friendly bunch here.