I have been able to boot off the USB using a USB floppy, and a USB ZIP.
Generally speaking these don't make very good "hacker" machines. They would be good if you had a particular purpose for a 32MB x86 capable machine, with the performance being somewhere around a 166 MHz Pentium class machine [GX's used in this box run at 200 MHz, but don't perform as well as a Pentium at 200 MHz].
As someone mentioned, there is a space to put down a SODIMM socket, to expand the RAM. But that might require other parts -- resistors, etc., that probably were not populated when the boards were built.
Other than that there is not much expandability in the thing. The SANDISK is soldered down, I believe it is 16 MB -- small for these days. USB is really the only external access means (other than the modem), and even 1 of the 2 ports is used by the mouse. So be prepared to buy a USB NIC (which are cheap these days, and still slow) to get 'connected'.
So if you get one you'd better be adept with a soldering iron and data books. You'll have yer work cut out for you...
With that said there might be a verticle market for these things, used for a specific purpose. I mean where else can you get an x86 machine, 32MB RAM, 16MB disk, Kbd, Mouse, and Modem, for $20?
Not sure if the Modem is a true hardware modem or a software modem? Might make the difference...