Welcome to the forum, Andrea.
Have you connected with any local cavers/clubs in New Zealand? Click
here for a list of clubs and contact info. Experienced local cavers will be able to provide the best advice on what gear you need, and most clubs have spare gear available for new cavers to borrow.
Bivouac stocks caving basics like helmets, headlamps, and some of the gear needed for vertical caving.
Aspiring Enterprises is a Christchurch-based manufacturer that produces some caving-specific gear. Bivouac sells Aspiring equipment, but you may have to special order some items. Given the high cost of a lot of specialized (imported) caving gear in New Zealand, ordering some items directly from overseas may be worth the postage and customs costs.
Good luck!