Add support for Uncompress(source, sink). Various changes to allow
Uncompress(source, sink) to get the same performance as the different variants of Uncompress to Cord/DataBuffer/String/FlatBuffer. Changes to efficiently support Uncompress(source, sink) -------- a) For strings - we add support to StringByteSink to do GetAppendBuffer so we can write to it without copying. b) For flat array buffers, we do GetAppendBuffer and see if we can get a full buffer. With the above changes we get performance with ByteSource/ByteSink that is very close to directly using flat arrays and strings. We add various benchmark cases to demonstrate that. Orthogonal change ------------------ Add support for TryFastAppend() for SnappyScatteredWriter. Benchmark results are below CPU: Intel Core2 dL1:32KB dL2:4096KB Benchmark Time(ns) CPU(ns) Iterations ----------------------------------------------------- BM_UFlat/0 109065 108996 6410 896.0MB/s html BM_UFlat/1 1012175 1012343 691 661.4MB/s urls BM_UFlat/2 26775 26771 26149 4.4GB/s jpg BM_UFlat/3 48947 48940 14363 1.8GB/s pdf BM_UFlat/4 441029 440835 1589 886.1MB/s html4 BM_UFlat/5 39861 39880 17823 588.3MB/s cp BM_UFlat/6 18315 18300 38126 581.1MB/s c BM_UFlat/7 5254 5254 100000 675.4MB/s lsp BM_UFlat/8 1568060 1567376 447 626.6MB/s xls BM_UFlat/9 337512 337734 2073 429.5MB/s txt1 BM_UFlat/10 287269 287054 2434 415.9MB/s txt2 BM_UFlat/11 890098 890219 787 457.2MB/s txt3 BM_UFlat/12 1186593 1186863 590 387.2MB/s txt4 BM_UFlat/13 573927 573318 1000 853.7MB/s bin BM_UFlat/14 64250 64294 10000 567.2MB/s sum BM_UFlat/15 7301 7300 96153 552.2MB/s man BM_UFlat/16 109617 109636 6375 1031.5MB/s pb BM_UFlat/17 364438 364497 1921 482.3MB/s gaviota BM_UFlatSink/0 108518 108465 6450 900.4MB/s html BM_UFlatSink/1 991952 991997 705 675.0MB/s urls BM_UFlatSink/2 26815 26798 26065 4.4GB/s jpg BM_UFlatSink/3 49127 49122 14255 1.8GB/s pdf BM_UFlatSink/4 436674 436731 1604 894.4MB/s html4 BM_UFlatSink/5 39738 39733 17345 590.5MB/s cp BM_UFlatSink/6 18413 18416 37962 577.4MB/s c BM_UFlatSink/7 5677 5676 100000 625.2MB/s lsp BM_UFlatSink/8 1552175 1551026 451 633.2MB/s xls BM_UFlatSink/9 338526 338489 2065 428.5MB/s txt1 BM_UFlatSink/10 289387 289307 2420 412.6MB/s txt2 BM_UFlatSink/11 893803 893706 783 455.4MB/s txt3 BM_UFlatSink/12 1195919 1195459 586 384.4MB/s txt4 BM_UFlatSink/13 559637 559779 1000 874.3MB/s bin BM_UFlatSink/14 65073 65094 10000 560.2MB/s sum BM_UFlatSink/15 7618 7614 92823 529.5MB/s man BM_UFlatSink/16 110085 110121 6352 1027.0MB/s pb BM_UFlatSink/17 369196 368915 1896 476.5MB/s gaviota BM_UValidate/0 46954 46957 14899 2.0GB/s html BM_UValidate/1 500621 500868 1000 1.3GB/s urls BM_UValidate/2 283 283 2481447 417.2GB/s jpg BM_UValidate/3 16230 16228 43137 5.4GB/s pdf BM_UValidate/4 189129 189193 3701 2.0GB/s html4 A=uday R=sanjay
This commit is contained in:
20
snappy.h
20
snappy.h
@@ -84,6 +84,18 @@ namespace snappy {
|
||||
bool Uncompress(const char* compressed, size_t compressed_length,
|
||||
string* uncompressed);
|
||||
|
||||
// Decompresses "compressed" to "*uncompressed".
|
||||
//
|
||||
// returns false if the message is corrupted and could not be decompressed
|
||||
bool Uncompress(Source* compressed, Sink* uncompressed);
|
||||
|
||||
// This routine uncompresses as much of the "compressed" as possible
|
||||
// into sink. It returns the number of valid bytes added to sink
|
||||
// (extra invalid bytes may have been added due to errors; the caller
|
||||
// should ignore those). The emitted data typically has length
|
||||
// GetUncompressedLength(), but may be shorter if an error is
|
||||
// encountered.
|
||||
size_t UncompressAsMuchAsPossible(Source* compressed, Sink* uncompressed);
|
||||
|
||||
// ------------------------------------------------------------------------
|
||||
// Lower-level character array based routines. May be useful for
|
||||
@@ -164,6 +176,14 @@ namespace snappy {
|
||||
bool IsValidCompressedBuffer(const char* compressed,
|
||||
size_t compressed_length);
|
||||
|
||||
// Returns true iff the contents of "compressed" can be uncompressed
|
||||
// successfully. Does not return the uncompressed data. Takes
|
||||
// time proportional to *compressed length, but is usually at least
|
||||
// a factor of four faster than actual decompression.
|
||||
// On success, consumes all of *compressed. On failure, consumes an
|
||||
// unspecified prefix of *compressed.
|
||||
bool IsValidCompressed(Source* compressed);
|
||||
|
||||
// The size of a compression block. Note that many parts of the compression
|
||||
// code assumes that kBlockSize <= 65536; in particular, the hash table
|
||||
// can only store 16-bit offsets, and EmitCopy() also assumes the offset
|
||||
|
||||
Reference in New Issue
Block a user