Skip to content

3813 add selected hooks for INT, BOOL, MAP, LIST #3828

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 25 commits into from
May 3, 2024

Conversation

jberthold
Copy link
Member

@jberthold jberthold commented Apr 24, 2024

Context: #3813

Adds a number of hooks (as per documentation in https://github.com/runtimeverification/haskell-backend/blob/master/docs/hooks.md) and splits the Booster.Builtin module into several parts.

  • Bool : all hooked operations implemented
  • Int: selected binary and unary operations and comparisons
  • List: get (indexing) and size
  • Map: all hooked operations except removeAll and keys (which would require using internal sets)

Comment on lines 195 to 200
intTerm :: Integer -> Term
intTerm = DomainValue SortInt . pack . show

readIntTerm :: Term -> Maybe Integer
readIntTerm (DomainValue SortInt val) = readMaybe (unpack val)
readIntTerm _other = Nothing
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Obviously these hooks would be faster if we had an internal type holding an Integer, instead of reading and showing the values for every operation.
Currently the IMP-sum-to-n test is still 30% faster with the LLVM backend than it is without.

@jberthold
Copy link
Member Author

A tiny bit of slowdown in KEVM (if not measurement noise)

Test 3813-map-update-hook-and-hook-refactorin time master-68bea10ad time (3813-map-update-hook-and-hook-refactorin/master-68bea10ad) time
kontrol/test-expectreverttest-test_expectrevert_message-0-spec.k 51.29 64.47 0.795563828137118
kontrol/test-allowchangestest-testallow_fail-0-spec.k 113.46 109.62 1.0350301039956211
kontrol/test-allowchangestest-testfailallowchangestostorage-0-spec.k 119.37 111.6 1.0696236559139787
kontrol/test-allowchangestest-testallow-0-spec.k 46.75 42.6 1.0974178403755868
kontrol/test-allowchangestest-testfailallowcallstoaddress-0-spec.k 47.72 41.19 1.158533624666181
kontrol/test-arithmetictest-test_max1_broken-uint256-uint256-0-spec.k 44.96 38.5 1.1677922077922078
kontrol/test-arithmetictest-test_wdiv_rounding-uint256-uint256-0-spec.k 55.43 46.55 1.1907626208378088
kontrol/test-expectreverttest-test_expectrevert_bytes4-0-spec.k 82.34 67.95 1.2117733627667402
TOTAL 561.32 522.48 1.0743377736946869

@jberthold
Copy link
Member Author

With one more (previously-missing) hook MAP.updateAll added, no actual slowdown (measurement noise) in KEVM.

Test 3813-map-update-hook-and-hook-refactorin time master-c391d6b87 time (3813-map-update-hook-and-hook-refactorin/master-c391d6b87) time
erc20/hkg/totalSupply-spec.k 49.02 65.28 0.7509191176470589
kontrol/test-allowchangestest-testfailallowchangestostorage-0-spec.k 109.68 124.88 0.8782831518257528
kontrol/test-expectreverttest-testfail_expectrevert_failandsuccess-0-spec.k 79.49 83.86 0.947889339375149
mcd/vat-muluu-pass-spec.k 155.28 161.26 0.9629170284013395
mcd/vat-slip-pass-rough-spec.k 89.77 77.7 1.1553410553410552
erc20/hkg/balanceOf-spec.k 53.22 40.41 1.3170007423904975
TOTAL 536.46 553.39 0.9694067475017619

Results for MX Backend are showing improvements:

Test 3813-map-update-hook-and-hook-refactorin time master-c391d6b87 time (3813-map-update-hook-and-hook-refactorin/master-c391d6b87) time
/tmp/tmp.xmCW3Z6C2u/mx-backend/kmxwasm/src/tests/booster/data/bigIntAdd.json 39.63 43.25 0.9163005780346821
test_call_add_less-spec.json 139.46 149.08 0.9354708881137644
/tmp/tmp.xmCW3Z6C2u/mx-backend/kmxwasm/src/tests/booster/data/bigIntMul2.json 39.37 41.5 0.9486746987951806
/tmp/tmp.xmCW3Z6C2u/mx-backend/kmxwasm/src/tests/booster/data/bigIntEncodeDecode.json 43.07 45.04 0.9562611012433393
/tmp/tmp.xmCW3Z6C2u/mx-backend/kmxwasm/src/tests/booster/data/checkNumArguments.json 40.63 42.4 0.9582547169811322
test_fund-spec-1k.json 221.29 208.07 1.0635363098957082
TOTAL 523.45 529.34 0.9888729361091173

(The test_fund_spec-1k test was actually failing on the missing hook before)

@jberthold jberthold force-pushed the 3813-map-update-hook-and-hook-refactoring branch from 503e19d to 34a5867 Compare April 29, 2024 05:32
Update the key-value association of an internal `KMap` with a new
value if the key is (syntactically) present (assumes bottom-up
evaluation).

Otherwise:
* If the map has an opaque part, no result is returned;
* If the map has unevaluated key data, no result is returned;
* Otherwise, the key is certain to be absent in the map. A new
  association pair is added to the list
@jberthold jberthold force-pushed the 3813-map-update-hook-and-hook-refactoring branch from 34a5867 to e095e9c Compare April 30, 2024 04:11
@jberthold jberthold force-pushed the 3813-map-update-hook-and-hook-refactoring branch from 92e7232 to e86bae4 Compare April 30, 2024 06:12
@@ -2,88 +2,6 @@
{"tag":"simplification","result":{"tag":"success","rule-id":"f310443b160236f56d095576a7500c11d1e98ee7af349b0bbf7a34b5e444d300"},"origin":"booster"}
{"tag":"simplification","result":{"tag":"failure","reason":"RuleNotPreservingDefinedness","rule-id":"3b51b2998e8ebe79b88550ea9d27ca95003744019b16b93d17a55f6f9bcaa12a"},"origin":"booster"}
{"tag":"simplification","result":{"tag":"failure","reason":"RuleNotPreservingDefinedness","rule-id":"1170b8c2f6d770eecb12b2ffb28275ed94ae1451d0ae3f2b42e570cd198099fa"},"origin":"booster"}
{"tag":"simplification","result":{"tag":"failure","reason":"Failed match","rule-id":"c2682318552299808618ea1106c22cf9df82485521eaf5e58844a16e0d4550e4"},"origin":"booster"}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I right to understand that we are evaluating certain things with the hooks, whereas before we attempted to apply rules from domains.md?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, the hook is tried first (if one exists), and if the hook returns a result, no equations are tried.

@jberthold
Copy link
Member Author

kontrol performance figures (parallelism = 3, will re-run with parallelism = 1)

Test 3813-map-update-hook-and-hook-refactorin time master-c42b8f0f9 time (3813-map-update-hook-and-hook-refactorin/master-c42b8f0f9) time
BlockParamsTest.testWarp(uint256)-trace_options2 27.05 38.98 0.6939456131349411
ArithmeticContract.add_sub_external(uint256,uint256,uint256) 57.63 64.37 0.8952928382787012
EmitContractTest.testExpectEmitLessTopics() 63.03 65.9 0.9564491654021243
SymbolicStorageTest.testFail_SymbolicStorage1(uint256) 109.11 113.74 0.9592931246703007
ChainIdTest.test_chainid_setup() 48.18 49.91 0.965337607693849
StartPrankTestMsgSender.test_startprank_msgsender_setup() 87.9 90.99 0.9660402242004618
Identity.applyOp(uint256) 90.9 87.66 1.0369609856262836
ConstructorTest.run_constructor() 58.06 55.99 1.0369708876585104
CounterTest.testSetNumber(uint256) 132.26 127.43 1.0379031625205994
StartPrankTestOrigin.test_startprank_origin_setup() 90.68 87.33 1.0383602427573573
EmitContractTest.testExpectEmitDoNotCheckData() 66.49 63.95 1.0397185301016418
BlockParamsTest.testBlockNumber() 20.64 19.84 1.0403225806451613
ExpectRevertTest.testFail_expectRevert_false() 96.61 92.77 1.0413926916028888
ExpectRevertTest.test_ExpectRevert_increasedDepth() 150.95 144.72 1.0430486456605859
BMCLoopsTest.test_countdown_symbolic(uint256) 119.01 113.67 1.0469780944840328
ArithmeticTest.test_max1(uint256,uint256) 62.75 59.89 1.0477542160627817
ExpectRevertTest.testFail_ExpectRevert_failAndSuccess() 97.79 93.33 1.0477874209793208
AssertTest.checkFail_assert_false() 29.08 27.3 1.0652014652014652
ExpectRevertTest.testFail_expectRevert_empty() 40.23 37.76 1.0654131355932204
StoreTest.testGasStoreColdVM() 30.18 28.32 1.0656779661016949
BytesTypeTest.test_bytes4(bytes4) 26.06 23.99 1.0862859524802
src/tests/integration/test_foundry_prove.py::test_foundry_xml_report 20.85 19.16 1.0882045929018789
AddrTest.test_addr_true()-trace_options1 39.13 35.64 1.0979236812570146
ExpectRevertTest.testFail_expectRevert_multipleReverts() 159.53 144.85 1.101346220227822
StoreTest.testStoreLoadNonExistent() 44.43 39.52 1.124240890688259
ArithmeticContract.add(uint256,uint256) 30.42 24.47 1.2431548835308541
Identity.identity(uint256) 32.53 14.85 2.190572390572391
TOTAL 1831.48 1766.33 1.0368843874021274

Comment on lines +76 to +78
readIntTerm :: Term -> Maybe Integer
readIntTerm (DomainValue SortInt val) = readMaybe (unpack val)
readIntTerm _other = Nothing
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

definitely not the scope of this PR, but perhaps we should make an issue about making ints an internalised value?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3848 created to track this

@jberthold jberthold self-assigned this May 1, 2024
@jberthold jberthold marked this pull request as ready for review May 2, 2024 06:00
Copy link
Contributor

@geo2a geo2a left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Truly monumental work with all those tests!

@jberthold
Copy link
Member Author

Re-tested with latest version of mx-backend

Test 3813-map-update-hook-and-hook-refactorin time master-602850806 time (3813-map-update-hook-and-hook-refactorin/master-602850806) time
/tmp/tmp.xhFhUfv6og/mx-backend/kmxwasm/src/tests/booster/data/bigIntAdd.json 43.95 47.2 0.9311440677966102
test_call_add-spec.json 151.64 160.41 0.945327598030048
/tmp/tmp.xhFhUfv6og/mx-backend/kmxwasm/src/tests/booster/data/bigIntEncodeDecode.json 48.08 50.22 0.9573874950219036
/tmp/tmp.xhFhUfv6og/mx-backend/kmxwasm/src/tests/booster/data/bigIntMul2.json 43.86 45.74 0.9588981198076082
TOTAL 287.53 303.57 0.9471621042922552

@jberthold jberthold merged commit a0907df into master May 3, 2024
@jberthold jberthold deleted the 3813-map-update-hook-and-hook-refactoring branch May 3, 2024 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants