The Lightbox effect in NextGen Gallery is actually not working. To enable lightbox effect, there is another plugin which works for NextGen Gallery – slimbox. Just search “slimbox”, and enable this plugin. Then enable lightbox effect for under NextGen Gallery options->effects. Also to use lighbox, it’s a good idea to resize image to 800*600 by using NextGen Gallery.
After you are done with this, it will look like the following:
Plugin webpage: http://wordpress.org/extend/plugins/slimbox-plugin/