I-Appliance BBS
The Official Source for Internet Appliance Upgrades and Mods
Amazon Honor System Click Here to Pay Learn More
BBS Main List | Sign In | Sign Up | Search | Help | Linux-Hacker.netReply to Thread | Printer |

Home / MISC Areas / Cameras
1 2 many - RAW Decode
RAW decompression code release

New Message1 2 many - RAW Decode (modified 0 times) sailpix
Profile
I have posted an initial version of my .RAW decompress code to the SourceForge forum: http://sourceforge.net/projects/pv2devkit/

It reads .RAW, decompresses and (if present) compares to .BYR file with same file base name.

Results are written to .txt file using RAW file's base name. So, if you run it with a command line argument of IMG_0001.raw then it will dump information to IMG_0001.txt and will compare the decompressed CFA data to IMG_0001.byr (if that file is present).

So the main current use to a non-programmer would be to dump the header information for a .RAW file. A number of fields in the headers are still a complete mystery - any clues are welcome.

Decode isn't correct for images where the SMALKEY_IMGR_ROWMASK header field has a value other than 0 - I know which logic to add to address this issue.

Decode isn't correct for some other images too - like, sunnycad.raw. Not sure what's going wrong there yet, but I hope we'll find it someday.

License is zlib/libpng - so, anyone can use this code freely. (Unlike RadioShack FlatFoto 2 drivers...)


- sailpix _/)
04-10-2005 23:26:52

New MessageRE:1 2 many - RAW Decode (modified 0 times) mnemonic
Profile
I would like to help doing smal reviev thru code, but archive seems to be broken. (BTW. It's the only way I can help - here in Poland there is no place you can obtain these cameras). Anyway it's exciting watching group of people united in effort of hacking those toys :). BTW. Sorry for my english...
04-11-2005 02:20:36

New MessageRE:1 2 many - RAW Decode (modified 0 times) daBass
Profile
Hello sailpix,

I've been looking forward to this release for a couple of weeks now, but trying to donwload it through sourceforge
(and several of it's mirrors) I keep coming up with a corrupt zip file. RAR can read the filetable, but I cannot
extract anything. Could you update the file on Sourceforge ?

DaBass

04-11-2005 02:47:16

New MessageRE:1 2 many - RAW Decode (modified 0 times) sailpix
Profile
I uploaded again - looks like it got truncated the first time, sorry.
Correct file size is 2210K. New file name is pv2raw_095a.zip

Though... I did add the source code into the CVS repository connected
to this SF project. So if you are impatient (like me) you _could_ have
pulled it from there. Also, I may be checking in work-in-progress
between ZIPped file releases - so having a configured CVS client will let
you see things earlier.

Yes - please review this code! Though, this is - mostly - a translation
from disassembled machine language in Che-ez Foxz2 TWAIN driver. IOW, I can't
answer for all the logic you'll see.

Where I could figure out the conceptual flow of things I have named variables
appropriately or rewritten the code. However in some areas - specifically the
arithmetic decoding handled between Next_Symbol() and DecodeAC() - there is
still lots of confusion left to sort out.

The original US Patent application (20040135903) is still the best overview of how
this compression works - but there are many "hidden" details that're only in the code.


- sailpix _/)
04-11-2005 08:34:16

New MessageRE:1 2 many - RAW Decode (modified 0 times) chrisfly
Profile
hey has anyone used their camera to the max, like all 25 pics- and gotten all 25 back?

chrisfly=noobie

04-11-2005 11:35:54

New MessageRE:1 2 many - RAW Decode (modified 0 times) sailpix
Profile
Yes - I shot a "full roll" last weekend. The screen scrambled after taking that many shots though...

Che-ez Foxz2 driver downloaded all 25 images. PV2Tool could only retrieve first 13 RAW files.


- sailpix _/)
04-11-2005 11:57:37

New MessageRE:1 2 many - RAW Decode (modified 0 times) daBass
Profile
Sailpix, you're brilliant! Why aren't you working in the Space program? (or are you? ).

Tried the code, just had to make a couple of minor modifications to make it compile under MinGW, and I got back
perfect Bayer images from my raw files.

I noticed right away that my imager has 3 dead pixels on it though.

Thanx,

daBass

04-12-2005 16:09:15

New MessageRE:1 2 many - RAW Decode (modified 0 times) sailpix
Profile
Thanks!

I'd rather work in private industry - not so much politics here!

What are you doing with the Bayer data after my code converts it? My avenue is gonna be .BYR -> .DNG -> App.

Send me your changes for MinGW and I'll work them back into the source. Also, I'll be glad to post your makefile (or whatever). I tried to avoid Microsoft-specific stuff, but you never really know till you try it with another compiler...

Also, I had thought already of adding some mechanism so you could identify your dead pixels and we would (I suppose) interpolate from the data around them. Actually got the idea of ID-ing dead pix from the DCRAW code. I haven't noticed any on my camera, yet.


- sailpix _/)
04-12-2005 17:16:42

New MessageRE:1 2 many - RAW Decode (modified 0 times) daBass
Profile
Hello sailpix,

> I'd rather work in private industry - not so much politics here!

> What are you doing with the Bayer data after my code converts it? My avenue is gonna be .BYR -> .DNG -> App.
I display it in an OpenGL window after I do a simple debayer (that needs a rewrite to a proper debayer routine).
See http://www.digitalfluff.net/pv2 .

> Send me your changes for MinGW and I'll work them back into the source. Also, I'll be glad to post your makefile (or whatever).
> I tried to avoid Microsoft-specific stuff, but you never really know till you try it with another compiler...

The code is also available above, I did some more changes, and renamed a couple of things. Haven't tried it on a different
OS yet, but it should all be fairly portable.

Let me know what you think,

daBass.

04-13-2005 06:36:19

New MessageRE:1 2 many - RAW Decode (modified 0 times) brite_eye
Profile
To view a raw, all I have to do is drag and drop it on RAWGL.exe (unzipped daBass' WIN32 app above), and in less than 2 seconds I have a view window showing my decompressed debayered raw. That speed is very impressive. Hopefully all the pieces can be eventually integrated into a single download app that flashes a page of 25 thumbnails and allows quick viewing and selection of keepers for further manipulation by standard photo apps (probably using DNG). Such a single app would not need modified firmware if it just used an automated a sequence to reset and unlock camera. I believe someone will provide such an application in less than 2 months. For anyone creating such an application, please give an expanded credit for all the pieces including efforts by morcheeba, Drmn4ea, daBass, ForkBoy, Sailpix. I have been trying to follow Sailpix's arithmetic decode along with other arithmetic examples and while the code looks short and simple there is nothing obvious about how it all works. A great mental challenge for an old bit twiddler trying to understand logic involving variables like bits_to_follow. I'll need an incredible amount of wind before I catch up to sailpix and only have a few "wtf???s".
04-13-2005 20:32:27

New MessageRE:1 2 many - RAW Decode (modified 0 times) brite_eye
Profile
daBass, how did you identify dead pixels? Would they show up as 00 in an all white image of FFs? Can dead mean a constant other than 00?
04-14-2005 04:42:52

New MessageRE:1 2 many - RAW Decode (modified 0 times) sailpix
Profile
My guess is that a "dead" CFA element could be black or white or... My laptop has one $#!% red pixel right in the middle of the screen. (Fortunately it's dark red and doesn't grab my eye too often.)

I suppose we could write an app which would look at a series of Bayer/CFA data files and "notice" that a single element just isn't changing between images. If we fed it a series of varying images (i.e. black, white, etc.) then it could automatically map out your camera's dead spots. This actually could be useful for any digital camera - hmmm...

For a format, I would automatically copy the ".badpixels" file that DCRAW uses. No need for us to go in a different direction.

daBass - Do you want to check your source code into the PV2DevKit SourceForge project? Have you looked at copying the DCRAW source code to get a better demosaicing algorithm?

REQUEST: Can anyone post (or send me) an actual Adobe .DNG file? (Or something that easily converts to .DNG, like .CRW?) Thanks!


- sailpix _/)
04-14-2005 10:57:39

New MessageRE:1 2 many - RAW Decode (modified 0 times) dcarr
Profile
I tried daBass's decompression wrapper app and it works just fine under wine.
The code will need a few modifications to compile natively under linux though.

-DC

04-14-2005 11:18:15

New MessageRE:1 2 many - RAW Decode (modified 0 times) kiram
Profile
Wow! very nice work! I cant wait till (hopefully) its integrated into PV2tool.
BTW you guys are genuises!

About dead pixels, they usually die in and either open or shorted state. but the ones that are all "white" are what our eyes make out most of the time. I would try doing some error checking in the debayerizing where if pixel is all white in a group of dark pixels, it will discard that pixel and use the information from the average of some of the others!

I just finished building a battery pack for my laptop, so now i can work on my recharging hacks for my cameras.

04-14-2005 13:13:48

New MessageRE:1 2 many - RAW Decode (modified 0 times) mconsidine
Profile
Just a thought here : A lot of these image processing issues (converting formats, reading webcam chips, handling dead pixels) are also run into in astrophotography. So some of the programs out there for which source exists might have alot of this dealt with. Unfortunately, I don't have a good survey of the software that would be most useful for this chipset. A start might be here :
http://www.astrosurf.com/astrobond/ebrawe.htm
http://www.astrosurf.com/astrobond/Using-RAW-Mode.pdf
http://3demi.net/astro/qastrocam/doc/
http://www.astrosurf.org/astropc/qcam/programme.html
Some of it's in French - sorry. And apologies if this isn't much help.
04-14-2005 14:01:22

New MessageRE:1 2 many - RAW Decode (modified 0 times) daBass
Profile
@sailpix:
>daBass - Do you want to check your source code into the PV2DevKit SourceForge project?
I never worked with sourceforge before, but I'm willing to give it a try, contact me offline
(i.e. not in the forum) about this.

>Have you looked at copying the DCRAW source code to get a better demosaicing algorithm?
No, but I found a couple of other interesting routines (See the new version on my page
http://www.digitalfluff.net/pv2 ).

Will have a look at DCRAW, see if there is some interesting routines to be salvaged .

@brite_eye:
Maybe you can find some brights now ?

> daBass, how did you identify dead pixels? Would they show up as 00 in an all white image of FFs?
> Can dead mean a constant other than 00?
I get white spots in my allblack raw, and in my first raw image where it is supposed to be black at the same spot
in both images.

@DC:
> The code will need a few modifications to compile natively under linux though.
Let me know if you have problems compiling this version, I'll try to make it Linux proof over
the weekend, help is always appreciated.

@anyone:
How sure are we that the CMOS sensor in the PV2 has an RGGB bayer filter ? Might the second G value be an intensity
value ?

04-14-2005 16:38:54

New MessageRE:1 2 many - RAW Decode (modified 0 times) dcarr
Profile
I've sucessfully ported daBass's RawGL .RAW viewer to linux.
Its online at http://oscar.dcarr.org/dd/

-DC

04-14-2005 17:02:21

New MessageRE:1 2 many - RAW Decode (modified 0 times) sailpix
Profile
@daBass: I don't have any offline mechanism to reach you. Shoot me an email to rtaylor187@users.sourceforge.net and mention PV2 in the subject so I notice that it's not spam.

Image viewer looks most excellent!

After I figured out that "demosaic" is the key word to use (pun recognized...), Google gave me a bunch of leads for algorithms.

Best looking stuff I could find was here: http://www-ise.stanford.edu/~tingchen/main.htm
And, the best algorithm is already implemented in the DCRAW code - as he notes ("Why are there false colors along edges within the image?") on the main page: http://www.cybercom.net/~dcoffin/dcraw/


- sailpix _/)
04-14-2005 21:06:46

New MessageRE:1 2 many - RAW Decode (modified 0 times) Mehoff
Profile
sailpix, any luck getting a .dng or .crw file? If not, I have a Canon G2 and can send a .crw file. Post here or email me ( mercyman at verizon dot net ) and let me know if you do and what you would like for subject of picture, if that matters.
04-15-2005 13:11:18

New MessageRE:1 2 many - RAW Decode (modified 0 times) sailpix
Profile
Yep, got it... Shot .CR2 on fiancee's camera then used Adobe's converter to make .DNG.

Thanks!


- sailpix _/)
04-15-2005 14:20:45

New MessageRE:1 2 many - RAW Decode (modified 1 times) dcarr
Profile
Updated linux version of daBass's RawGL viewer 1.1 complete with new GUI.

http://oscar.dcarr.org/dd/

-DC

04-16-2005 16:28:26

Reply to Thread | Printer |
All times are PSTPowered by UltraBoard v1.62



Copyright © 2000, Netmake Inc. All Rights Reserved.
See Terms and Conditions for more information.




i-opener opener laptop notebook computer help drivers dll free windows dos repair fix linux mac macintosh 2000 95 98 nt pc configure hardware software sound video netscape explorer network networking lan wan software cmos fat bios printer card mouse modem ide scsi cd rom controllers scanner tape hard drive cgi scripts source code mp3