You guys read my mind - I was just looking for this in the firmware the other day. Turns out you can set the mode with either of the control commands 0x31 or 0x32. (0x31 messes with one of the 8051's timers first, and 0x32 just sets the mode, but I'm not sure what exactly the timer stuff is all about.) The set-mode routine does something wierd; it subtracts 1 from whatever mode you specify (e.g. to specify mode 3, you have to send a 4). It's actually a little wierder for invalid modes, but that's the gist of it.
The mode values are as follows:
SPCA50X_CamMode_Idle = 0, (sitting on your USB port doing nothing)
SPCA50X_CamMode_DSC = 1, (Digital Still Camera; e.g. taking pictures the normal way)
SPCA50X_CamMode_VideoClip = 2,
SPCA50X_CamMode_PcCamera = 3,
SPCA50X_CamMode_Upload = 4 (bulk transferring pictures)
Anyway, if you...
usb_control_msg(cam, 0x41, 0x32, 0x0000, 0x0004, NULL, 0, 5000);
(that is, Out (host -> device), bRequest 0x32, wValue dont-care, wIndex 0x0004)
the camera will be set to Mode 3 (PC-Camera) and the 'Ready' light on the back of the cam will start blinking on and off rapidly. It will stay this way until another mode is set (or you unplug, etc.)
You can just as easily set VidClip mode (mode 2) with
usb_control_msg(cam, 0x41, 0x32, 0x0000, 0x0003, NULL, 0, 5000);
but I'm not sure how you'd actually use this, since the only apparent way to set it is thru the USB, and you can't take pictures while the cam is plugged into a computer. Away from its tether, it will revert to still-camera mode as soon as it's turned on.
Obvious question now is, how do you get at the webcam data? Some of the opensource webcam drivers I've seen use a fair amount of bit-bashing, fiddling with camera registers, etc.--it doesn't look as simple as 'slurp data off the USB' (though it still might be). I haven't tried anything yet with this, since I'm NOT a Windows driver developer (and barely even a programmer and the only way I know to touch the USB port on my machine is thru libusb-win32, which (like regular libusb) does not support isochronous transfers.
But we do have some programmer types here, right?