compression: additional limit for effective dict

[stevemilk]

#11921

[AskAlexSharov]

jump outside of mutex section - it's deadlock

[stevemilk]

oh sry, just notice you fix it

[AskAlexSharov]

compression ratio seems impacted:

2.4G	v1-060700-060800-transactions.seg.compressed_s1_dict_8k_min20_max128_sof1_dictInt256k_old
3.1G	v1-060700-060800-transactions.seg.compressed_dict_8k_min20_max128_sof1_dictInt128k_new

i did debug deeply yet. not sure - do we drop "part of dictionary" which has low-score or also can drop high-scored patterns.

i used next command:

go run ./cmd/erigon snapshots uncompress /erigon-data/snapshots/v1-060700-060800-transactions.seg | DictReducerSoftLimit=1000000 MinPatternLen=20 MaxPatternLen=128 SamplingFactor=1 MaxDictPatterns=262144 EffectiveDictLimit=65536 go run ./cmd/erigon snapshots compress --datadir=/erigon-data/erigon3/  /erigon-backup/v1-060700-060800-transactions.seg.compressed_s1_dict_8k_min20_max128_sof1_dictInt256k_new --pprof --pprof.port=6061 > ~/log3.txt 2>&1 &

[stevemilk]

MaxDictPatterns = MaxDictPatterns * 2
EffectiveDictLimit = MaxDictPatterns / 2

With above config, init-dict will include more patterns than before, and may exclude some high-score patterns during dict reduction because patterns are hit randomly, low-score pattern may occupy the space of effective dict in advance.

will re-consider the solution.

[stevemilk]

The previous version incorrectly included some unused patterns
I'v updated , however, the compression ratio will still be impacted to some extent.

Here's the reason:
In the current design, init-dict is reduced to reduced-dict after compression, retaining only the used patterns. For example, if len(init-dict) = 1024 and len(reduced-dict) = 512, with a compression ratio of 0.3, setting effectiveDictLimit to 512 keeps the ratio the same. However, if effectiveDictLimit is less than len(reduced-dict), some patterns will be dropped, potentially increasing the compression ratio.

I believe effectiveDictLimit serves as a hard limit to prevent the resulting dictionary from becoming too large, rather than a means to improve the compression ratio. I conducted some investigation but would appreciate your insights or suggestions for a better solution. @AskAlexSharov

[AskAlexSharov]

Tnx. I will trst another day