more de-chunking (chaosreader patch)

ok, so i went ahead and wrote a quick patch to chaosreader to handle chunk-encoded transfers. files extracted from http sessions should be automagically dechunked. not sure if \r\n vs. \n line ends make any difference.

5359,5375d5358
<            ### Chunk Check
<            if ( $http_header =~ /Transfer-Encoding: chunked/ ) {
<              my $new_http_data="";
<              my $chunksize=-1;
<              my $pos=0;
<              until ($chunksize==0) {
<                my $eolpos=index($http_data,"\r\n",$pos);
<                $chunksize=hex(substr($http_data,$pos,$eolpos - $pos));
<                $pos=($eolpos+2);
<                if ($chunksize > 0) {
<                  $new_http_data.=substr($http_data,$pos,$chunksize);
<                }
<                $pos+=($chunksize+2);
<              }
<              $http_data=$new_http_data;
<            }
<

download link

dechunking - simple but potentially confusing

So say you've got some http data to analyze, but the content isn't lining up right - binary extractionfrom the http server response only produces corrupt files when using trace carving tools like chaosreader. Check the server response headers to see if the Transfer-Encoding header is present and set to "chunked". If it is, you will need to reassemble the response body from the chunks before subsequent decoding/decompression.

download link

#!/bin/env ruby
require 'stringio'
end_of_headers="\r\n\r\n"
outbuf=""
chunksize=-1

filename=ARGV[0]

f=File.new(filename,"r")

everything=f.gets(nil)

bodystart=everything.index(end_of_headers)
sio=StringIO.new(everything)

header=sio.read(bodystart + end_of_headers.length)
if header.match(/Transfer-Encoding: chunked/i)
until chunksize==0
  chunksize=sio.gets("\r\n").rstrip.hex
  outbuf+=sio.read(chunksize)
  2.times {sio.getc}
end
else
warn("#{filename} doesn't have a Transfer-Encoding: chunked header")
exit(1)
end

print outbuf