it's not the mask that make the bad quality, it is the people who make the masks...
Basically you should never ever work with any lossy (such as Jpg) image format when making images with masks, since they will produce hugely bad results.
Anyway, you could of course use GIF images, and open them along side with their masks, and use them then as a masked bitmaps in your app.
Anyway, no way for trasparency, if you completely don't want to use masks.