ETCM-1061 Remove caching from ArchiveStateStorage and ReferenceCountedStateStorage

[leo-bogastry]

Description

Mantis is experiencing some bugs related to the state in caches and in the storage not being syncronized.
RocksDB promisses to offer fast writes compared to other more tradicional DB's.

Proposed Solution

Depending on the chosen pruning mode, StateStorage will return an ArchiveStateStorage, an ReferenceCountedStateStorage or an CachedReferenceCountedStateStorage.
In this PR the caching layer of ArchiveStateStorage and ReferenceCountedStateStorage where removed, but
you will see that method forcePersist is still part of the StateStorage trait. This is because CachedReferenceCountedStateStorage is deeply linked to caching (pruning mode inmemory).

Would it make sense to remove support to an inmemory pruning mode and further simplify the StateStorage?

Testing

Number of ETS tests failing is still the same as in develop branch
It was testing with ETC, for a more lengthy test is desirable yet
I am exploring using RocksDB statistics to help with testing these kind of changes

[AurelienRichez]

I don't really know what we should do with this test. So basically removing the cache seems to have solved a bug.

To sum up:

• we store ancestorForValidation which is the genesis block (number 0)
• we save block 1 to 4
• we save a new branch with block 2 and 3 and better difficulty to simulate a reorganisation
• we set the best block number to 4 in memory (which is false), in order to simulate an inconsistency problem

After this the behaviour changes:

• before: as we did not find the block 4 in storage, and the app state storage contained nothing (because of the caching, nothing was persisted yet), we fetched block 0 (ancestorForValidation) because it is the default valuev when nothing is stored
• Now because there is no caching, the app state storage is actually more consistent than the in memory we actually return the right block

It is not really clear what this test is actually testing now. I wonder if we should not just remove it.

[leo-bogastry]

If it is simulating a bug that has currently disappeared, leaving the test is not a bad idea, because it will make sure that the bug does not come back in the following refactors

[AurelienRichez]

Well, it's not simulating a bug in the sense that it check that the bug does not happens. It simulate the bug actually happening by putting the app in an inconsistent state and check that it returned the genesis (quite arbitrarily). But that is not the behavior anymore.
What worries me here is that it looks like we just changed the expected value to make the test pass. So I am not sure what the test is really testing now. But when we remove the BlockchainMetadata part it should become clearer, so let's leave it at that for now.