There are caves in Southern Illinois, in the counties below St. Louis, and many caves in southern Missouri and Indiana. I do not know many specifics of that region, but you shouldn't have to go too far to go caving. Following Scott's suggestion to look up a club will be the easiest way to get into caving. Alternately, you can search for caves and negotiate with landowners on your own. This is a long term process. Online sleuthery, conversations with local people, and lots of walking are a good formula if you go this route.
Beginners kit? A couple of headlamps and a hard had will be plenty sufficient for you to get a feel for caving. No need for anything fancy, cheap lamps throw a lot of light these days. If you like caving, there will be plenty of opportunity to buy gear of all sorts, but it's best to start by focusing on yourself and the cave, not on stuff.
If you do find a cave, be cautious, and careful. Most caves are not very dangerous, but they can be very punishing to bad judgment. Finally, read this forum. There is a wealth of valuable information here.