how to Encrypt/Decrypt compressed streams?

[paul_nicholls]

Hi all,
I have made my own packfile format, which uses zlib compression behind the scenes.

I would like to release the source codes for the TPackFileReader and TPackFileWriter classes, but I can't till I add in some password protection - I am using it in my PGAnnual competition entry to project some music files...

So I tried adding simple password protection, but I have some issues...

I am currently doing this (writing packfiles):


load file to be packed
encrypt it
compress into packfile


and this when reading packfiles:


decompress file from packfile
decrypt it


This DOES work, but because I encrypt the files PRIOR to the compression, the packfile size blows out way larger (around 3 x larger!) because of the 'garbled' file data from encrypting it.

I would now somehow like to compress the data and THEN encrypt it prior to storing in the packfile (writing to packfiles), and then decrypt and decompress reading from packfiles).

Below are my two routines I use to compress and decompress streams using the zlib library:

Procedure CompressStream(Const SrcStream,DstStream : TStream);
Var
CompressionStream : TCompressionStream;
Begin
CompressionStream := TCompressionStream.Create(clMax,DstStream);
Try
CompressionStream.CopyFrom(SrcStream,0);
Finally
CompressionStream.Free;
End;
End;
{................................................. .............................}

{................................................. .............................}
Procedure DecompressStream(Const SrcStream,DstStream : TStream;
Const Size : Int64);
Var
DecompressionStream : TDecompressionStream;
Begin
DecompressionStream := TDecompressionStream.Create(SrcStream);
Try
DstStream.CopyFrom(DecompressionStream,Size);
Finally
DecompressionStream.Free;
DstStream.Seek(0,soFromBeginning);
End;
End;

I want to somehow put in a intermediate buffer (TMemoryStream, Dynamic Array, etc) in the routines I can compress to, then encrypt, and write, or read from, decrypt and then decompress to make the final file size smaller.

I hope I'm making sense? :)

any ideas?

Below you can find the attempt at myself doing this, but I am getting read stream errors:


Procedure CompressStream(Const SrcStream,DstStream : TStream);
Var
CompressionStream : TCompressionStream;
Buffer : TMemoryStream;
BufferSize : Int64;
Begin
Buffer := TMemoryStream.Create;
Try
CompressionStream := TCompressionStream.Create(clMax,Buffer);
Try
CompressionStream.CopyFrom(SrcStream,0);
Buffer.Seek(0,soFromBeginning);
BufferSize := Buffer.Size;
DstStream.Write(BufferSize,SizeOf(BufferSize));
DstStream.CopyFrom(Buffer,0);
Finally
CompressionStream.Free;
End;
Finally
Buffer.Free;
End;
End;
{................................................. .............................}

{................................................. .............................}
Procedure DecompressStream(Const SrcStream,DstStream : TStream;
Const Size : Int64);
Var
DecompressionStream : TDecompressionStream;
Buffer : TMemoryStream;
BufferSize : Int64;
Begin
Buffer := TMemoryStream.Create;
Try
SrcStream.Read(BufferSize,SizeOf(BufferSize));
Buffer.CopyFrom(SrcStream,BufferSize);

DecompressionStream := TDecompressionStream.Create(SrcStream);
Try
Buffer.CopyFrom(DecompressionStream,BufferSize);
Buffer.Seek(0,soFromBeginning);
DstStream.CopyFrom(Buffer,0);
Finally
DecompressionStream.Free;
DstStream.Seek(0,soFromBeginning);
End;
Finally
Buffer.Free;
End;
End;

Once I get these routines working (with buffer), I can then easily add in the encryption/decryption on the buffers :)

cheers,
Paul

[User137]

# Buffer := TMemoryStream.Create;
# Try
# CompressionStream := TCompressionStream.Create(clMax,Buffer);
# Try
# CompressionStream.CopyFrom(SrcStream,0);
# Buffer.Seek(0,soFromBeginning);
# BufferSize := Buffer.Size;

If i understood right, BufferSize will always be 0? TCompressionStream.Create doesn't have access to SrcStream or does it?

Edit: Also i noticed the use of Copyfrom()...
http://www.componentace.com/help/ecl_ref/TStream_CopyFrom.htm
According to this, last parameter is "count". You call Copyfrom with count of 0 directly also.

[paul_nicholls]

# Buffer := TMemoryStream.Create;
# Try
# CompressionStream := TCompressionStream.Create(clMax,Buffer);
# Try
# CompressionStream.CopyFrom(SrcStream,0);
# Buffer.Seek(0,soFromBeginning);
# BufferSize := Buffer.Size;

If i understood right, BufferSize will always be 0? TCompressionStream.Create doesn't have access to SrcStream or does it?

Edit: Also i noticed the use of Copyfrom()...
http://www.componentace.com/help/ecl_ref/TStream_CopyFrom.htm
According to this, last parameter is "count". You call Copyfrom with count of 0 directly also.


The stream that I am passing into TCompressionStream is the destination stream that will get written to with the compressed data, ie. in this case, the buffer...

Not sure what you are saying with regards to CopyFrom() though as I am using it right now...

cheers,
Paul

[User137]

Not sure what you are saying with regards to CopyFrom() though as I am using it right now...
EDIT: I was wrong about "nothing" as i read Delphi help:
"If Count is 0, CopyFrom sets Source position to 0 before reading and then copies the entire contents of Source into the stream. If Count is greater than or less than 0, CopyFrom reads from the current position in Source."

And the first thing was, i'll comment from code

Buffer := TMemoryStream.Create; // At this point Buffer is empty
Try
CompressionStream := TCompressionStream.Create(clMax,Buffer);
// Buffer should still be empty, but CompressionStream exists
// Unless you fill something in buffer at this point
Try
CompressionStream.CopyFrom(SrcStream,0);
// Here ScrStream is copied into CompressionStream
// It should still do nothing for Buffer, unless CopyFrom is internally modified

Buffer.Seek(0,soFromBeginning); // Buffer should still be empty
BufferSize := Buffer.Size; // Size = 0
DstStream.Write(BufferSize,SizeOf(BufferSize)); // DstStream gets 0 written in

[paul_nicholls]

Thanks for the comments User137, I think a "light-globe" has gone off in my head - the TCompressionStream.Free needs to be called before it will finalise the compressed output to the buffer I think from memory (and similar for TDecompressionStream too)...

So I probably would need to do this sort of thing (untested):

Compressing streams:

1. Create buffer
2. create compression stream (linked to buffer in Create() method)
3. copy SrcStream to compression stream
4. free compression stream
5. write size of buffer to DstStream
6. write buffer to DstStream
7. free buffer

decompressing streams:

1. create buffer
2. read buffer size from SrcStream
3. copy buffer from SrcStream using buffer size
4. create decompression stream (linked to buffer in Create() method)
5. decompress buffer to DstStream using size passed into DecompressStream method
6. free decompression stream
7. free buffer

cheers,
Paul