I'll need to re-familiarize myself with the system. Do they use specular maps in addition to normal maps and diffuse maps? A lot of the definition for boob curves can come from the specular maps (in my experience anyways). This would handle things like shadowing and skin shading that helps give a sense of depth along with the normal maps.
Unfortunately however, it's very hard to create a single static set of texture that will work for all boob sizes simultaneously. This isn't helped by the lighting engine that STO uses, which is, like, SO last... 10 years ago! :P
Just do your best, find the preset that covers the most scenarios reasonably well. That's what I have to do with some of my mods for various games (e.g. TERA).
As for nipple outlines, that shouldn't be very hard to do. Just crop out the nipples you want to use, but them on a new layer, then make that layer use multiplier and put it in front of the outfit's layer. This can be done easily in either GIMP or Photoshop.