I-Appliance BBS
The Official Source for Internet Appliance Upgrades and Mods

Click Here!
BBS Main List | Sign In | Sign Up | Search | Help | Linux-Hacker.netReply to Thread | Printer |

Home / DigitalVideoRecorders / Tivo Hacks
Extract Stream
Extract Stream

New MessageExtract Stream (modified 0 times) codeman
Profile | Email
WHOO HOO
now you can extract and save you video to you pc from your tivo !!!!

http://www.9thtee.com/extractstream.html

download from 9thtee.com

codeman

06-07-2001 16:25:48

New MessageRE:Extract Stream (modified 0 times) piper
Profile
The files have been removed from 9thtee.com, any other sites hosting these files?
06-07-2001 18:00:05

New MessageRE:Extract Stream (modified 0 times) codeman
Profile | Email
try here
http://www.stampede.org/~skibum/tivo/
here is the text from marks page

codeman

FORWARD
The ExtractStream program and source code described in this document is released under the GNU GPL It is not warranted suitable for any purpose, it is the users responsibility to assure proper functionality before use.
Understand that this code is in a very raw form (neither easy to use or understand). It is released in this early stage in the spirit of Open Source, hoping that others might use and modify the code, improve it, and share their changes.

We urge users to understand the difference between "fair use" and copyright violation when using copyrighted video. The persons writing and distributing this code feel strongly that the code be used only in a legal manner. It is legal for you to view the video captured on your TiVo (or any video capture device) at any time and using any playback device, BUT, it is not legal to distribute or publicly playback copyrighted material. If you are unsure of the legality of your intended use, consult a real lawyer; this is just our interpretation.

HISTORY
ExtractStream was derived from d18c7db's PlayStream. This was truly a reverse engineering masterpiece. The changes made were mostly to compensate for TiVo v1.3 to v2.0.1 considerations. Even if you're using 2.0.1, you might want to look at the PlayStream distribution for it's README file, which is somewhat outdated, but still has some very useful information.
httpd.tcl is a slight modification of sfr's and TiVoTechie's httpd.tcl. The only change is to allow direct display of the relevant "NowShowing" information.

INSTALLATION
The "ExtractStream-0.1.tgz" file is a unix file archive containing the "ExtractStream" executable, a modified "httpd.tcl" daemon, an mpeg multiplexer "mplex", and NetCat:"nc".
If you've installed the tkp_tools, then there's a name conflict: "nc"is used as a variant of "mc", as defined in "/var/hack/root/.bashrc" (sometimes "/hack/root/.bashrc"). You can either rename the definition of "nc" in your ".bashrc", or you can rename "nc" on your system, or: always refer to NetCat by path (i.e. /hack/bin/nc).

If you've already installed "httpd.tcl", you might want to back it up, since the archive contains a new version that may overwrite your version.

Some of the capabilities described in this document require NetCat:.NetCat: needs to runon both the TiVo and your client PC. The archive contains "nc" for the TiVo. You need to get NetCat for your client OS here.

These instructions assume a network aware TiVo. You're going to need to transfer files totheTiVo. NFS, SAMBA ( smbfs.oand smbmount), rsync or an ftp daemon are your best options.

Download the ExtractStream archive here.

Transfer the archive to the TiVo (or, in the case of SAMBA or NFS, make sure the file can be accessed from the TiVo).

The archive is relative "hack/bin" If your "hack/bin" is on "/var", then "cd /var". If your "hack/bin" is on "/", then "cd /" (and make sure the root is mounted read/write: "mount -o remount,rw /"). Finally, unarchive:

bash-2.02# cat /var/tmp/mnt/ExtractStream-0.1.tgz | gzip -dc | cpio -i -H tar
If "ExtractStream: is in your path, you should be able to enter the command:
bash-2.02# ExtractStream
Without arguments, it will respond with usage information.
OTHER INSTALLATION
Along side the " ExtractStream-0.1.tgz" file, there should also be the " ExtractStream.c" source file and "TiVo-VCD.mcf" (to guide TMPGEnc processing of TiVo audio/video in Windows). There's also the "mplex.tgz" file, the source code for the "mplex" program.
USAGE
httpd.tcl
This daemon runs in the background and acts as a very unsophisticated web server. To launch it, run:
bash-2.02# httpd.tcl >/dev/null &
You can also put that line at the end of your "/etc/rc.d/rc.sysinit" file.
I'm not about to explain all the functionality of this file... only what's been added for mpeg extraction.

In the "location" or "URL" entry field of your favorite browser(except:"opera" doesn't like this server), type the IP address or machine name. It should provide you a list of options, including "NowShowing". Select thisoption.

The resulting page should list the programs available on the TiVo, their descriptions, and, most importantly, the tyStream FSID's needed by ExtractStream. Note that these are hyperlinks... but don't click on them. If you do, it will attempt to copy the entire raw tyStream to your browser. Some browser's react differently to this:they may try to display the unintelligible binary information. Worse, while this is transferring, your TiVo will have stuttering problems, not record well, and even worse: sometimes loose pieces of the tyStream. This optionis still a work in (mostly abandoned) progress, so, for now just use it for the tyStream information it provides.

Usually browsers will give you a "save link as" option if you right-mouse-click on the link. This is the safest way to use this option. But, again, beware that there might be corrupted data in the resulting raw tyStream file,and you might experience stuttering if you're watching or recording on theTiVo while downloading via the httpd.tcl interface. We do have some ideas on an alternate method that shouldn't cause stuttering.

ExtractStream
With no arguments, ExtractStream prints usage information:
bash-2.02# ExtractStream
ExtractStream [-v] [-s] [-d] [-p] <fsid>...

ExtractStream takes valid <fsid> numbers of the streams you want to play,
extracts the sectors allocated to that stream and attempts to play the
audio/video information on those sectors through the TiVo codecs.

-v Verbose output (stderr)

-s Dump raw tyStream data to stdout

-d Dump sector scan info to stderr (no mpeg dumped at all)

-p Include PES headers with data

If -s is not set, and AUDIO_OUT or VIDEO_OUT environmental variables
are set, then mpeg output will go to these files instead of the
codecs. If only VIDEO_OUT is set,then both audio and video are
written to the file (note PS streams not yet working).

If the DUMP_OUT environmental variable is set, then the raw tyStream
is copied to the named file. This is in addition to writing to
stdout if -s is set, or to the files named in VIDEO_OUT or AUDIO_OUT
if they're set.

The "<fsid>..." argument(s), which are required, are those shown by the "httpd.tcl" daemon when selecting the "NowShowing" option as described above. Note that there will be one or more tyStream FSID's associated with a program. The "httpd.tcl" daemon presents those separated by slashes ("/"). ExtractStream expects them separated by spaces.
If none of the environmental variables (DUMP_OUT , VIDEO_OUT, or AUDIO_OUT) are set, and none of the command line options: "-s" or"-d"are used, then ExtractStream acts as PlayStream does: writing the mpeg data to the proper devices. This mode hasn't been tested in a long time, it probably requires the "-p" option, and you have to do something to keep it from competing with MyWorld for mpeg attention -- which is not properly documented for TiVo 2.0.1 systems (it's not necessarily different than killing and restarting MyWorld was in TiVo v1.3, but a cursory look at the process queue would make one think that it may not be as simple as it was). Another reason to read the README file that comes with the old PlayStream..

Extracting MPEG
Currently, using ExtractStream for mpeg extraction requires splitting the mpeg stream into it's audio and video component elemental streams. To do this, the environmental variables VIDEO_OUT and AUDIO_OUT must beset to file names before running ExtractStream. If you're using NFS or SAMBA, you could pass these files across the network by specifying files on the remotely mounted partitions. For example, you might have your PC's "C:" drive mounted as "/mnt" on the TiVo, you could save the mpeg audio and video files with:
bash-2.02# export VIDEO_OUT=/mnt/program1.m2v
bash-2.02# export AUDIO_OUT=/mnt/program1.m2a
bash-2.02# ExtractStream 53476 5347753478
When complete, your "C:" drive should contain the files "program1.m2[av]".
Creating an MPEG2 Program Stream from Audio and Video Elemental Streams
With elemental audio and video streams, you must multiplex these into program streams expected by most MPEG2 players.
Note that there are two "mplex" programs discussed here: one is in the distribution, which is discussed later, and the one from the mjpegtools group, discussed here...

In Linux, the "mplex" program, available with the Mjpegtools at sourceforge ,works quite well. Assuming you have extracted the MPEG audio and video elemental streams from the TiVo, the correct incantation is:

pc# mplex-f 2 -m 2 -o program1.mpeg2 program1.m2[av]
This will create the program stream file "program1.mpeg2".
There has been a reported problem with a truncated audio packet that maynot be fixed. This can be easily fixed from mplex. If you see the mplex error: "Failed to find start of next stream at...", then, in mplex's inptstrm.c change:

syncword = (syncword<<(8-AU_start % 8));
next = getbits( &audio_bs,8-(AU_start % 8) );
syncword = syncword | next;
if ( syncword != AUDIO_SYNCWORD )
{
mjpeg_warn("Failed to find start of next stream at %lld prev %lld ! ", AU_start/8, prev_offset/8 );
break;
}
to:
do {
syncword = (syncword<<(8-AU_start % 8));
next = getbits( &audio_bs,8-(AU_start % 8) );
syncword = syncword | next;
} while ( syncword != AUDIO_SYNCWORD );
#if 0
{
mjpeg_warn("Failed to find start of next stream at %lld prev %lld ! ", AU_start/8, prev_offset/8 );
break;
}
#endif

In Windows, TMPGEnc will multiplex the audio and video data. Using TMPGEnc with the "TiVo-VCD.mcf" file will create a cleaned-up cropped (removal of left edgegreen vertical line and top closed caption data) VCD compatible MPEG file.

Playing an MPEG2 Program Stream
In Linux, "vlc" from Videolan, isa great player:
pc# vlc-V sdl -v program1.mpeg2
Transcoding MPEG2 Elemental and Program Streams toMPEG1
Note that transcoding MPEG2 to MPEG1 requires a great deal of processor horse power and disk space; it's only warranted if you're making VCD's. MPEG1 files sizes are equivalent to MPEG2 files captured with TiVo's "BASIC" quality.
The Mjpegtools at sourceforge contain a transcode script, that requires mpeg2dec, plus some patches to the output drivers provided by the mjpegtools.

I've changed the transcode to two separate files, one for audio, and one for video. The video script looks like:

cat $1.m2v | mpeg2dec -o YUVh | mpeg2enc-s -r 24 -o $1.m1v
I couldn't find the audio transcoding tools referenced in the mjpegtools"transcode", so I made my own audio transcode script:
mpg123 -s -r 44100 --wav -$1.m2a | toolame -p 2 -b 224 - $1.m1a
Find toolame here ,and mpg123 here (if you play mp3 files in Linux, you probably already have mpg123). The only reason for the audio transcode is: VCD compliance requires 44.1kHz audio, versus the33kHz audio coming off the TiVo.
Again, use mplex to make aVCD conformant MPEG1program stream:

pc# mplex -f 1 -m 1 -o program1.mpeg1 program1.m1[av]
Vcdimager is one of the many tools that can be used to create a VCD image in Linux,the incantation is:
pc# vcdimager--cdi-dir=/<where you put it>/cdi/ --cue-file=program1.cue \
--bin-file=program1.bin \
--volume-label="My first program" program1.mpeg1
It generates cdrdao cue and bin files, the VCD can be written with an incantation like:
pc# cdrdao write --device /dev/sg2 --speed 8 program1.cue
Most DVD players will playback VCD's (or SVCD's made with MPEG2 files). VCD playback on your Linux PC is best performed with "mtv". Note that "mtv" is neither Open Source nor free.
Other MPEG Extraction Methods
The AUDIO_OUT and VIDEO_OUT environmental variables can point to named pipes (Linux Fifo's made with mkfifo or mknod). Using this, no file needs to be saved at all; we can pipe the data directly into the recipient program on a PC.
First, on the TiVo, make some named pipes:

bash-2.02# mkfifo /var/tmp/aud
bash-2.02# mkfifo/var/tmp/vid
Run ExtractStream as before, but setting the environmental variables to the named pipes:
bash-2.02# export VIDEO_OUT=/var/tmp/vid
bash-2.02# export AUDIO_OUT=/var/tmp/aud
bash-2.02# ExtractStream 53476 53477 53478
In a separate shell on the TiVo, use "nc" to read from the fifo's:
bash-2.02# nc -n -w 3 192.168.0.1 1235 </var/tmp/vid &
bash-2.02# nc -n-w 3 192.168.0.1 1236 </var/tmp/aud&
Where "192.168.0.1" is replaced with the IP address of your PC. Actually, you need to start these after you've started "nc" on your PC, but this discussion will flow more smoothly if I show these first.
On your Linux PC, capture the data:

pc# nc -l -p 1236 >program1.m2a &
pc# nc -l -p 1235 >program1.m2v &
The net effect here does exactly the same thing as the first example where NFS or SAMBA was used to transfer the file across the network... it was just a lot trickier, requiring many more commands :)
Actually, there is a reason for this. The "nc"'s running on the PC side, rather than writing a file, can be placed into players or transcoders.

For example, the receiving "nc"'s on the PC might start a player:

pc# nc -l-p 1236 | mpeg2dec &
pc# nc -l -p 1235 | mpg123 &
This will play the video/audio on you system. Note that mpeg2dec has no mechanism for maintaining frame rate with real time. If your machine isn't fast enough, then it will playback in slow motion.
Likewise, you could be transcoding on the PC:

pc# nc -l -p 1236 | mpg123 -s -r 44100 - | \
toolame -p 2 -b 224 - program1.m1a
pc# nc -l -p 1235 | \
mpeg2dec -o YUVh | \
mpeg2enc -s -r 24 -o program1.m1v &
Using this method to transfer the mpeg data can save a great deal of disk space.
Extracting Raw tyStreams
Extracting raw tyStreams is useful for archiving a TiVo tyStream on yourPC in a format that would be easily restored back onto the TiVo for viewing at a later date. We don't know how to restore the files yet, but we can at least archive them.
The "-s" command line option in ExtractStream causes the program to not split audio and video streams, and to merely output the raw tyStream to "stdout" (standard output). You can pipe the data directly across the net using a SAMBA or NFS mounted partition,as in:

bash-2.02# ExtractStream 53476 53477 53478 >/mnt/program1.rawtivo
Or use netcat for extracting the raw tyStream across the network to a receiving program. For example, on the PC side:
pc# nc-l -p 1234 >/mnt/program1.rawtivo
On the TiVo side, run:
bash-2.02# ExtractStream -s 498441 498445 | nc -n -w 3 192.168.0.1 1234
Where, again, you need to replace "192.168.0.1" with the IP address of your PC.
Remember the problem noted in the httpd.tcl description above ? This is probably the best solution:

On the client side, in your browser, associate a mime type with the receiving "nc" command. On the TiVo side, have httpd.tcl spawn the above ExTractStream command, without any further TCL or http handling.
MPLEX
The "mplex" program, provided in this distribution, for the TiVo, is based on the mplex program in mjpegtools(discussed above). The basic difference is: it only takes one pass over theA/V input. The big difference is it's been royally hacked.
This means: it can be used to directly generate an mpeg2 file, or playback video from the TiVo without creating any file at all.

The precompiled TiVo version is in the distribution "ExtractStream-0.1.tgz", the source, "mplex.tgz", is also available. When compiling on the TiVo, make sure to use the "-msoft-float" option.

This program has performance problems on the TiVo. It needs to be profiled. It probably has to do with it's use of floating point numbers, when integers would do. For example:

(int)ceil((double)foo/8.0)
Is the same as:
(foo+7)/8
Currently, this program is limited to use on "basic" quality TiVo programs, because:
It can't process higher bitrates (better than "BASIC" quality) fast enough (in "real time"). This is not a problem if you're just writing an mpeg2 file, only if you're trying to playback in realtime.
The bitrates have been hardcoded for "BASIC" quality. Mplex needs three command line arguments to overcome this: the buffer size for video, the buffer size for audio, and the video bitrate. Refer to the "COMPBITRATE_MAX" case in multiplex.cpp.
ExtractStream too buffers data as a function of mplex's input buffers (this doesn't keep it from working for other bitrates). It too should take audio and video buffer sizes as command line arguments.
Even multiplexing basic quality causes stuttering on the TiVo's TV output (it doesn't effect "mplex"'s output).
Creating an MPEG2 Program Stream
Our mplex can run on either a Linux workstation (you need to compile it from source), or your TiVo (in the distribution). You won't get TiVo stuttering if you run it on your workstation. Here's an example running our "mplex" on a Linux workstation:
pc# mkfifo vid
pc# mkfifo aud
pc# export VIDEO_OUT=./vid
pc# export AUDIO_OUT=./aud
pc# nc -l -p 1238 >$VIDEO_OUT &
pc# nc -l -p 1239 >$AUDIO_OUT &
Then, run mplex. For output to a file, use:
pc# mplex > foo.mpeg2
For output to video, run:
pc# mplex | (while true; do vlc -V sdl -v /dev/stdin; done)
The "while" loop is to handle "vlc" crashes... it currently has some audio problems that cause a segfault, the "while" loop will keep the video streaming where it left off to the new incarnation of "vlc".
On the TiVo, start the extraction

bash-2.02# mkfifo /var/tmp/vid
bash-2.02# mkfifo /var/tmp/aud
bash-2.02# export VIDEO_OUT=/var/tmp/vid
bash-2.02# export AUDIO_OUT=/var/tmp/aud
bash-2.02# nc -w 3 -n <workstation IP address> 1238 <$AUDIO_OUT
bash-2.02# nc -w 3 -n <workstation IP address> 1239 <$VIDEO_OUT
bash-2.02# ExtractStream <tyStream FSID's>
The video should start playing or writing to the file (depending on the "mplex" incantation you used).
If you're willing to allow stuttering on your TiVo, then you can run "mplex" there. If you want realtime video playback on your workstation, run:

pc# nc -l -p 1255 | (while true; do vlc --width=200 --height=200 -V sdl -v /dev/stdin; done)
On the TiVo, run:
bash-2.02# mkfifo /var/tmp/vid
bash-2.02# mkfifo /var/tmp/aud
bash-2.02# export VIDEO_OUT=/var/tmp/vid
bash-2.02# export AUDIO_OUT=/var/tmp/aud
bash-2.02# mplex | nc -n -w 3 <workstation IP address> 1255 &
bash-2.02# ExtractStream <tyStream FSID's>
If you just want to create an mpeg2 file, then don't run the "vlc" line on the workstation, and, on the TiVo, replace:
bash-2.02# mplex | nc -n -w 3 <workstation IP address> 1255 &
With
bash-2.02# mplex >foo.mpeg2

06-07-2001 21:14:14

New MessageRE:Extract Stream (modified 0 times) piper
Profile
Found the files here.
06-07-2001 21:15:01

New MessageRE:Extract Stream (modified 0 times) laserfan
Profile
It says you need NetCat but I have not gotten the links to work at

http://www.l0pht.com/~weld/netcat/

nor have I been able to find it anywhere. Does anyone here have it to post?

06-08-2001 04:30:47

New MessageRE:Extract Stream (modified 0 times) laserfan
Profile
OK, now it's working. The link is actually here:

http://www.atstake.com/research/tools/index.html

06-08-2001 10:56:03

Reply to Thread | Printer |
All times are PSTPowered by UltraBoard v1.62



Copyright © 2000, Netmake Inc. All Rights Reserved.
See Terms and Conditions for more information.




i-opener opener laptop notebook computer help drivers dll free windows dos repair fix linux mac macintosh 2000 95 98 nt pc configure hardware software sound video netscape explorer network networking lan wan software cmos fat bios printer card mouse modem ide scsi cd rom controllers scanner tape hard drive cgi scripts source code mp3