should set the memory for the video card to 8MB, and reset the video. I would recommend giving it a try before making it permenant. If it works, and if there's a desire for such, I will add it to an option rom to start before the system boots.
as far as I know, there's no provision for memory specifically for the video.
composite out should be no big deal, just find test points for the pins on the north bridge that are intended to connect to the ntsc encoder, hook up an encoder, and either force it always on or set it up to be controlled on the SMBus, this would involve writing drivers, but not anything very heavy..
The primary reason no ones got software DVD to run is that there's not enough memory for a display at 640x480x8 (307200B) + 2x DVD frame @ 640x480x24 (921600) = 2150400, a shade over 2 meg, though 4 meg is enough for dvd decoding at 800x600
All of the 3d features also heavily rely on extra video memory, there isn't even enough memory in 2 megs for a single z buffer.
NormConquest: how'd you get the TIVO video out of the Tivo??? I'd much like that... :)