WL#14449 - Integration of Rapid with the hypergraph join optimizer

Bug#34486254 - WL#14449: Mysqld crash - sig11 at rpd::ConstructQkrnExprCond
Bug#34381109 - Hypergraph offload Issue : LEFT JOIN test cases failing in i_subquery tests
Bug#34432230: Enabling testcase along with BM failures in order_by_limit_extended_mixed_varlen and scgts
Bug#34408615 - Assertion failure `args->ctx->m_lirid != kInvalidRelId' in ha_rpd_qkrn_expr.cc
Bug#34471424 - HYPERGRAPH HeatWave Visible fields not populated
Bug#34472083 - WL#14449: Mysqld crash - Assertion `extract_ctx.cmn_expr_or == nullptr' failed
Bug#34395166 - HYPERGRAPH BUG: QUERIES not offloading on heatwave in MTR SUITE
Bug#34056849 - WL#14449: Offload issue with dictionary encoding
Bug#34381126 - Hypergraph offload Issue : QCOMP test file non offload bug
Bug#34450394 - Hypergraph result mismatch
Bug#34472373 WL#14449: Mysqld crash - Assertion `args->ctx->m_lirid != kInvalidRelId' failed
Bug#34472354 WL#14449: Mysqld crash - sig11 at rpdrqce_check_const_cols_rpdopn
Bug#34472069 WL#14449: Mysqld crash - Assertion `n < size()' failed
Bug#34472058 WL#14449: Mysqld crash - sig11 at rpdrqc_construct_phyopt_bvfltr
Bug#34143535 - WL#14449: task formation error
Bug#34356273 - HYPERGRAPH BUG: CAST binary having issues with DICTIONARY ENCODING
Bug#34381303 - Hypergraph offload Issue : LIMIT DUAL not offloading
Bug#34356238 - HYPERGRAPH BUG: CAST DATE WITH DOUBLE_PRECISION
Bug#34448736 - Hypergraph Result mismatch:Result mismatch with user variables
BUG#34388727: Enabling testcases
Bug#34413698 - Hypergraph Union issues
Bug#34432241 - Hypergraph out of stack memory issue in rapid.qcomp_bugs_debug_notubsan_notasan
Bug#34369934 - Hypergraph Performance : TPCDS q93 qcomp issue -2
Bug#34399991 - HYPERGRAPH BUG: crash in cp_i_subquery_dict MTR file
Bug#34057893 - Fixing MTR timeout by reducing the partial JOIN search space
Bug#33321588 Hypergraph Result Mismatch : Cannot process QEXEC JSON document expected for each HeatWave node in query [no-close]
Bug#34395166 - HYPERGRAPH BUG: QUERIES not offloading on heatwave in MTR SUITE
Bug#34086457 - Hypergraph offload Issue : constant not marked correctly
Bug#34380519 BUG#33294870 BUG#34114292: Enabling testcases after these bug fixes.
BUG#34079278 : Partially enabling testcases for fixes cases.
BUG#33321588 : Fixing 'Cannot process QEXEC JSON document expected for each HeatWave node in query' Error
Bug#34360222 - HYPERGRAPH BUG: QUERIES WITH RANGE LIKE 1+1 NOT OFFLOADING WITH DICT ENCODING
Bug#34412319 - HyperGraph: sig 11 on bm mtr cp_blob_dict
Bug#34403562 - HyperGraph: Signal 6 while running rapid.cp_blob_dict testcase
Bug#34360341 - HYPERGRAPH BUG: QUERIES not offloading on heatwave with VARLEN ENCODING
Bug#34012291 - Hypergraph Offload Issue : Subquery is OOM instead of error ER_SUBQUERY_NO_1_ROW
Bug#34399868 - HYPERGRAPH BUG: Output mismatch in cp_i_index_merge
Bug#34289251 - WL#14449: post-join filters set as inner joins' extra predicates are not handled
Bug#34381354 - Hypergraph offload Issue : DATE COUNT not offloading
Bug#34119506 - Hypergraph Result Mismatch : Decimal precision issue
Bug#34399722 - HYPERGRAPH BUG: Output mismatch with mysql
Bug#34360278 - HYPERGRAPH BUG: QUERIES not offloading on heatwave
Bug#34369223 - HyperGraph: Offload failure when hypergraph is ON
Bug#34361863 - Impossible Condition Cases Failing with HyperGraph
Bug#34289797 - Hypergraph Optimizer: query projecting expression from inner side of outer join does not offload
Bug#34128728 - Hypergraph Crash : Due to ZERO ROWS
Bug#34066930 - Hypergraph Result Mismatch : Wrong result with Zero Rows
Bug#34078549 - Hypergraph Result Mismatch : Wrong result with ZERO ROWS Select_varlen test
Bug#33426211 - Hypergraph Offload issue : Due to the absence of few corner case optimizations
Bug#34086457 - Hypergraph offload Issue : constant not marked correctly
Bug#34299494 - Hypergraph : Disable commutative INNER JOIN
Bug#34299823 - Hypergraph Optimizer: Issue with projection set for partial plan
Bug#33380501 - WL#14449: expected error ER_SUBQUERY_NO_1_ROW but query succeeds
Bug#33811377 - WL#14449: SUBQUERY item in JOIN's extra predicates is not detected in partial plan
Bug#33410257 - Hypergraph Offload Issue :  Rapidside ENUM issue

* Add new file ha_rpd_estimate_ap.cc for costing AccessPath trees
  using the new Hypergraph Optimizer.

* Rework function CollectAllBaseTables() to not return any size
  information -- instead it can simply be computed by iterating
  over the map passed to it.

* Add member to Qkrn_context to store a pointer to the Hypergraph
  Optimizer object. When that is set we have a partial plan,
  otherwise it's the final plan.

* Add a couple of new timers for Hypergraph-based costing

* Replace all occurences of JOIN::having_for_explain with
  JOIN::having_cond, because the former one is not always populated
  correctly anymore.

* Ignore SQL SELECT_BIG_RESULT hint as it does not have any meaning
  for  RAPID.

* Set flags for handlerton::secondary_engine_flags

* Add new function Rapid_execution_context::IsQueryPushable() for
  partial plans.

* Currently, the patch contains a fix in the costing code which
  enables costing of any temporary table. This is ported forwards
  from the change for bug 34162247.

* Allow dumping partial plans by appending the overal partial
  plan ID  for item dump and QKRN dumps.

* Some const-ness fixes / improvements.

* Add code in ha_rpd_qkrn_ap.cc to extract the projection list
  for the root translate state of a partial plan.

* More fixes to partial plan projection set computation:
  In function ExtractGroupByHypergraph(), reuse function
  ExtractProjectionItemHypergraph() to extract sub-expressions correctly
  such that they math the current state_map.
  In function ExtractWindowHypergraph() take into account the current
  state_map, which was missing before and then for a base TABLE_SCAN we
  could pick up expressions from another base TABLE_SCAN, which led to
  offload errors.

* In HashJoinPullJoinConditionFromExtraPredicate(), remove a superfluous
  check whether Item_func::type() == Item::FUNC_ITEM.

* In TranslateHashJoin(), where we check the extra predicates also for
  inner joins, since those represent post join filters, initially we
  used UpdateItemListHashJoin(), which would project whole expressions
  from the post join filter, which leads to larger projection lists of
  the join and its children.
  For instance, for a query like
    SELECT t1.a, t2.b FROM t1 JOIN t2 ON ... WHERE t1.a > 5 OR t2.b > 10
  the WHERE condition ends up as post-join filter. Then, with the
  previous approach, t1 would project "t1.a" and "t1.a > 5" and t2 would
  project "t2.b" and "t2.b > 10" and then the post join filter would
  degrade into "qkrcol(t1.a > 5) != 0 OR qkrcol(t2.b > 10) != 0".
  Change to use UpdateItemList() which extracts only Item_fields, i.e.
  base columns, to match the behavior from the old optimizer.

* In ExtractProjectionItemHypergraph(), project all expressions for the
  final partial plan of a query block. This is necessary when e.g. a
  child query block is of the form
    SELECT 1, 2, 3, t1.a FROM t1 LIMIT 5
  Then, if we always ignore all constants in the TABLE_SCAN then when
  creating the TOPK node for the LIMIT we'll try to project the
  constants from there, which is not supported.

* Add TPC-H/DS perf job queries without STRAIGHT JOIN hints.

* Dump hypergraph performance profile timers to QEXEC_pp JSON dump.

* In Mtoq_table::GetCell() an earlier patch introduced a small
  optimization which was intended to skip an mrow_item in the case of
  aggregation Items (ITEM_SUM_FUNCs). The idea is that when one is
  <aggregate>(DISTINCT) and the other is just <aggregate>(), then
  they cannot be the same. Also, when the number of arguments differ,
  or when they are just different aggregates, then we don't need
  to call AreItemsEqual() after the special ITEM_SUM_FUNC code path.
  However, the "optimization" was wrong and skipped too many Items
  such that some were not found at all in the MtoQ table anymore.

* Add more state to the hypergraph optimizer to remove code from
   HeatWave side. In particular:
  * To decide whether ORDER BY LIMIT OFFSET can be handled we only
     need to check a new flag.
  * Whether there are multiple base tables in a query block is also
    tracked through a hypergraph optimizer flag.
  * Add handling for hypergraph filters on top of derived tables and
    joins.

Improve speed of Mtoq_table::GetCell
========================================
 Before, we called Qkrn_context::GetQueryBlock() in each loop
 iteration over the cells for the given query block. That is,
 however, already done before the loop once and we can re-use
 that retrieved query block.

 For DISTINCT aggregation Items (SUM_FUNC_ITEM) we have a special
 code path to compare them as equal when e.g.

 Also, improve code of AreDistinctItemListEqual():
 * Use only prefix incrementor.
 * Use copy operator to populate vector "it", which will reserve
   space to avoid multiple re-allocations.
 * Rename "it" to "first_copy" and "it1" to "iter".

Bug 34289251
============
Since the Hypergraph optimizer, post-join filters can be encoded
as join extra predicates, which was not yet considered in
HeatWave's AccessPath translation. This may result in wrong
results, as in the bug's case, where the query is quite simple:

  SELECT * FROM t1 NATURAL JOIN t2 WHERE t1.b > t2.b;

Here, the post-join filter "t1.b > t2.b" is encoded as a join extra
predicate.

The fix is to "simply" also consider those for inner joins, not
only for outer joins and semi joins and ensure that an intermediate
qkrnts is created for the post-join filter. Care had to be taken
to also ensure that all of the predicates' sub-expressions are
added to both projected and required items for the join itself.

Additionally added a few comments to function calls for constants
and added checks for return value of calls to
UpdateItemListHashJoin().

Bugs 33380501, 33811377:
========================
* Reset Qkrn_context::m_leaf_table_count before counting the leaf
  tables

* In CountLeafTables(): The hypergraph optimizer counts derived
  tables and common table expressions as leaf tables in their
  parent query blocks, so do not traverse any further.

* Bug 33380501: patch by Kajori Banerjee. Adds function
  rpdrqcj_rejects_multiple_rows() which recursively checks
  JOIN key expressions whether they contain a qkrcol which has
  reject_multiple_rows_qkrcol == true.

* Bug 33811377:
  Root cause: through IsQueryPushable we don't identify all cases
  where a subquery item is involved. In this case it is a join's
  extra predicate whose string representation is
    <in_optimizer>(8,<exists>(select #3))
  When we try to translate the expression it is inside an
  Item_cond_and
  and since it is const_for_exection(), we call Item::is_null(),
  which then executes the subquery and then leads to a mysqld
  crash. The fix is to also check evaluate_during_optimization()
  which identifies such a case. This is added in several places
  where we call a function which potentially evaluates the
  Item (like is_null()). By that, we avoid interpreting the
  subquery item as a constant and we will try to fully translate
  this subquery item, which will then hit the default
  (unsupported) case in ConstructQkrnExprTreeSwitch() by
  which we bail out of query translation.

* Avoid assertions in rapidjson due to calling IsQueryPushable()
  for every subplan. All calls to PrintOffloadTrace passed
    abort_query = true
  which however is wrong for partial plans, because we may get a
  supported one later.

* Split cp_join_prepart into dict and varlen tests.

* Make the Extract***Hypergraph functions for partial plans only
  return bool (false=success, true=error).

* Fix extraction of required items of GROUP BY keys and aggregation
  expressions for partial plans. This must be done similar to
  AccessPath style extraction using CompileItem. Therefore,
  refactored function ExtractRequiredItemsForGroupBy() to use
  the same functionality for partial plan extraction, too.

* In RapidEstimateQueryCostsAP() when we catch a RapidException,
  for now always print and offload trace and dump to screen the
  type and string message.

* Pick up more test changes from the original branch.

* Extract projection expressions from all WINDOWs' partition by
  and order by lists.

* Re-introduce handling aggregate items in UpdateItemList, which was
  removed during code coverage-related clean-up for worklog 14344
  (AP uptake).

* We were not aware that for semi joins we need to always project
  anexpression from the inner side due to some QCOMP issue.
  Filed bug 34252350 to track this issue.

* Fix issue with correlated semi joins where an OR condition was
  attached to the semi join as an additional condition
  (extra predicate) but was ignored because any non-Item::FUNC_ITEM
  was ignored when inspecting the extra predicates. Fix is to add
  Item::COND_ITEMs, too.

* In the AP cost estimation code, print the QCOMP exception if one
  occurs to console (for debugging, will be removed later).

* In TranslateDerivedTable(): When a derived table MATERIALIZE
  AccessPath is the root node and we're translating a partial plan,
  then when re-initializing the Qkrn_context for the parent query block
  (which contains the derived table's MATERIALIZE) we need to call
    InitializeQkrnContextPartialPlan()
  instead of
    InitializeQkrnContext().

* Fix issues when WITH ROLLUP modifier is present and when extracting
 expressions for the root AccessPath for partial plans.

* The Qkrn_context flags m_has_gby, m_has_ftag, m_has_windowfunc,
  and m_has_oby were used wrongly: they only indicate whether the
  current query block has one of those, but they don't indicate
  whether corresponding qkrn nodes were created.
* One issue with ENUM columns and WITH ROLLUP: when the ENUM column
  is inside an Item_rollup_group_item then do
* Remove argument "is_outer_query_block" from TransformQkrn().
  It is not needed as we can be sure that transformation is only
  done once perpartial plan or query fragment.

Bug 33410257 - Hypergraph Offload Issue :  Rapidside ENUM issue
===============================================================
The algorithm for detecting whether there are string operations on an
enum column in ContainsEnumOpn() was incomplete. It needs to keep
track of the parent operations, similar to how
ContainsUnsupportedCastEnumToChar() works.

Bug 34471424 - HYPERGRAPH HeatWave Visible fields not populated
===============================================================
For few partial plans in the bug's query, which contain a common
table expression, the hidden bitmap is not populated properly which
leads to an empty initial projection set for the first AccessPath
inside the CTE.

Fix is to ensure for partial plans that the projection items of the
CTE are all marked as not hidden in the hidden bitmap.

Note that there *may* be a different root cause which is specific
to some rewrite where we have a CTE inside another derived table.

Bug 34472083:
=============
The code for extracting potential join predicates from parent
FILTER AccessPaths was very strict by asserting that OR
conditionals are not nested. This can be, however, the case and
we should be more graceful. Especially for directly-nested ORs we
can simply pretend as if those ORs had been merged already by the
resolver/optimizer and proceed. For more complex nested OR/AND
filter or join predicates now just completely skip trying to
extract any predicates.

Added test wl14449_bugs with dict and varlen variants.

Bug 34395166:
=============
* scgts: The query had multiple partial plans failing because the
  hidden_map for HeatWaveVisibleFields adapter was not correct for
  some Common Table Expressions. As a quick fix, this is now
  corrected on the HeatWave side and in the meantime will be
  discussed with the MySQL optimizer team whether this is actually
  a bug on their side. The issue seems to be the following: The
  affected SCGTS query has a quite deep nesting of CTEs and derived
  tables and UNIONs and one of the CTE ismactually merged into
  another one.
  In the query, CTE
    snapshot_inventory_item_sub3
  is merged into CTE
    snapshot_inventory_item_sub4.
  while more CTEs may have been merged, we use only this example
  for illustration. Then, for computing the hidden_map, we check
  for each CTE instance its parent query block and search for all
  occurrences of Item_fields from that CTE. For that search, we use
  amongst others Query_block::top_join_list.
  Now in this case, the top_join_list however only contains the
  tables from the query block BEFORE the other CTE (and its tables
  and joins) was merged into that query block. In this case, the
  list only contains tables
    * po_octg_line
    * sub3b
  while it should contain tables
    * po_octg_line
    * snapshot_inventory_item_sub2 AS sub2
    * state_prov
    * invtry_item
    * pt_bol_date
  Those tables can be found in Query_block::leaf_tables, but not in
  Query_block::top_join_list.
  The (temporary?) fix is to check both Query_block::leaf_tables
  and Query_block::top_join_list.
* astro: The query is successfully offloaded again, re-enabled the
  test case and re-recorded the result file.

Other Changes:
==============
* ha_rpd_estimate_ap.cc: Correctly set qcomp_ret upon errors during
                         QCOMP.
* cloud_variables.result: Update due to latest bm pre-merge job.
* qcomp_bugs_debug_notubsan_notasan.result: Update due to latest bm
                                            pre-merge job.

Bug 34056849 - WL#14449: Offload issue with dictionary encoding
 ================================================================
1. The test cases in bug_poc_dict are rigtfully not offloaded given
   MYSQL's type input. On mysql-8.0 there are implicit casts
   (e.g. string to double) which enable these test cases to offload
   on that branch, but MySQL behavior is not consistent across all
   comparison functions and BETWEEN.
   Limitations on the HyperGraph branch are consistent (see the newly
   added test cases) and are in agreement with known HeatWave
   limitations with dictionary encoding.

2. Marked test cases in bushy_replaylog_dict are offloaded on the latest
   branch.

3. Test cases in join_outer_dict are rightfully not offloaded given
   the AP plan: the join condition requires comparison of different
   dictionary encoded columns (coming from a different base column).
   With natural join the plan is different: the join condition is
   pushed into both tables and the join is a cartesian product - hence
   there's no issue with comparing dict-encoded columns.
   On 8.0 both plans contain a cartesian product with two pushed-down
   filters - hence the offload issue does not exist.

Bug 34381126 :
================
Recording the testcases since bug not reproducible anymore

Bug 34450394 :
==================
Mismatch was due to insert into + select.
Changing test case to use order by in select
ensures the output is deterministic.

Bug#34472373
===================
Partial plan 4:
======================

-> Inner hash join prefix over( SUBQUERY2_t1_SUBQUERY2_t2):
   * Projection set = []
 -> FILTER prefix:
       * table_map = 0x0001
       * Original filter =
            (unhex('') = concat_ws('0',SUBQUERY2_t1.col_varchar_key
       * Projection set = [unhex('')]
       * Required items =
            [unhex(''), concat_ws('0',SUBQUERY2_t1.col_varchar_key)
       * DummyFields = [(none)(=0), (none)(=0)]
        -> TABLE_SCAN prefix on SUBQUERY2_t1:
           * Actual filter =
          (unhex('') = concat_ws('0',SUBQUERY2_t1.col_varchar_key)
           * Projection set =
          [unhex(''), SUBQUERY2_t1.col_int_key]

Problem :
===================
ExtractRequiredItemsForFilter adds the constant to the
required items.

Bug#34472354
===================
Same fix as above resolved the issue.

Bug#34472069 and Bug#34472058  : fixed in the latest branch

sum_distinct-big is timing out Job 1094368

Bug#34143535 - WL#14449: task formation error
 ==============================================
1) The distinct inside an inner query block of a partial plan was
notapplied. Updated the function ReadyToHandleDistinctAndOrderBy.

2) Problem :
Earlier all the partial plans were dumped irrespective of any
system variable or flag.

Solution :
Dump hypergraph partial plan only when system variable
qkrn_print_translate_states is set to true.

Bug 34356238
==============
Closed since diff was because we only match MySQL upto 5
decimal places.

Bug 34448736 :
===============
Closed since the diff matches with MySQL output.

Bug 34413698 - Hypergraph Union issues
======================================
Problem: Union with hypergraph cases failed due to
proper projection with special case of derived tables.

Solution: Update Projection Populate such that with this
special case of derived table + union, we appropriately
populate projection set.

Bug#34432241:
==============
Unnecessary call to ItemToString in IsConstValueItem causing the
issue. Removing the redundant call to ItemToString.

Bug#34369934
============
Join pattern changed due to the change in the order of the join
keys with the hypergraph.

Bug 34399991 - HYPERGRAPH BUG: crash in cp_i_subquery_dict MTR file
===================================================================
Ensure filter items do not have subquery in them

Bug#34395166 -P6
==================
Query :
SELECT * FROM t1 WHERE c_id=@min_cid OR c_id=@max_cid;

Has ZERO rows. But it was not getting offloaded from
IsQueryPushable.

Solution :
Do not check for IsQueryPushable for ZERO_ROWS, FAKE_SINGLE_ROW and
ZERO_ROWS_AGGREGATED.

Bug#34395166 - P5
===================
Query :

SELECT *
FROM t3
WHERE col_varchar_10_latin1_key IN (
   SELECT alias1.col_varchar_10_latin1_key
   FROM t1 AS alias1
   LEFT JOIN t1 AS alias2
   JOIN t2 AS alias3
   ON alias2.col_varchar_10_latin1_key
   ON alias1.col_varchar_1024_utf8_key
   WHERE alias1.pk AND alias1.pk < 3 OR alias1.pk AND alias3.pk);

The BV filter has the condition alias1.pk<>0 AND alias1.pk < 3.

The hash join node projects the expression (alias1.pk < 3) instead
of the individual columns. This later creates a problem during
qkrn transformation of the BV filter as one child of the condition
(AND) is a column instead of an expression.

Bug#34395166 - P2
====================
Query:
SELECT          RANK() OVER (ORDER BY AVG(id)) FROM t1;

The item_map of (RANK() OVER (ORDER BY AVG(id))) = 0
Hence it was not projected from the window function of the
partial plan 4. As a result the window node projected the dummy
column leading to an offload issue.

BUG#33321588 :
==============
- Root cause for this issue is, the ordering of the query result is
  not in expected format.
- Heatwave doesn't ensure ordering without explicit ORDER BY clause.
- ORDER BY clause is added at required place.

Bug 34086457 - Hypergraph offload Issue : constant not marked correctly
=======================================================================
Problem: coalesce(col1, col2) is not marked as Item type ITEM_CACHE
which leads to false value returned from isConstValue

Solution: check for function with string constants in isConstValue

Bug#34399868 - HYPERGRAPH BUG: Output mismatch in cp_i_index_merge
=================================================================
This is actually *NOT* a HyperGraph bug!

There are 2 test cases which are resulting in result mismatch on
mysql-8.0 as well. A bug has been open for them.

This did not show up before because cp_i_index_merge.test relies
only on randomized data. That is also risking unstable mtr.

A temporary solution here is to just put explain for all test cases
that might be unstable - which is what this patch is doing.

In the meantime the underlying bug 34423734 needs to be fixed on
8.0 and those concrete test cases (without randomness) will be
appended to this test file.

Bug#34399722 - HYPERGRAPH BUG: Output mismatch with mysql
==============================================================
The ROW_NUMBER() is a window function or analytic function
that assigns a sequential number to each row in the result set.
The first number begins with one. Since there is no ordering
metioned in the window function, the row numbers
might be assigned in a different order.

Bug#34361863 - Impossible Condition Cases Failing with HyperGraph
==================================================================
for hypergraph Rapid_execution_context fields like m_subquery_type
and m_best_outer_plan_num are not updated during join order
enumeration. Hence checking them in HasOnlyInvalidPlans is not
relevant.

BUG#34289797 :
========================
Not reproducible, thus enabling the relevant testcase
and re-recording the test file. Additionally, more queries offload
with hypergraph, changing offload correspondingly for
bushy_bugs_varlen and bushy_bugs_dict test files.

BUG 34128728, Bug 34066930 Bug 34078549
Fix for Zero rows induced problems with Hypergraph Optimizer
==================================================================
Problem: Hypergraph Optimizer (may) sometime propose JOIN plans
with Zero Row in Inner/Outer Child. In this case, we handle by
inserting Filter and ZeroRowAggregate replacing the ZeroRow AP.
However, the PopulateProjectionSet does not pick up the correct
state map of tables under the new Filter AP. TranslateHash join
now does special handling for OuterJOIN and ZeroRows, but this was
now being done for inner join as well, which is incorrect.

Solution: We introduce m_zero_row_state_map member in
Translate_State::filter struct, which is used to resolve the state
map when we replace the Zero Row AP with Filter AP. In subsequent
calls to GetUsedTables() from PopulateProjectionSet() for
new Filter() AP, we are able to resolve the correct state_map,
thus handling the projection correctly.
Secondly, in TranslateHashJoin(), the special processing of zero
rows in outer join case has additional check to ensure it's invoked
 only when outerjoin is present.

Bug#34299494 - Hypergraph : Disable commutative INNER JOIN
============================================================
Say Query is
SELECT * FROM t1, t2 WHERE t1.a=t2.a;

At present the hypergraph optimizer explores both join orders
(t1 t2) and (t2 t1). Since both are same with respect to
HeatWave perspective, it does not makes sense to explore
(t2 t1) when ( t1 t2) is already explored.

Bug#34299823 - Hypergraph Optimizer:
Issue with projection set for partial plan
==============================================================
* In ExtractProjectionItemHypergraph(), also project expressions
which do not reference any tables. This fixes queries where
partial plans would otherwise have no SELECT list at all, for
instance
    SELECT isnull(gen_rnd_ssn()) FROM ...
* In TranslateHashJoin(), for extra predicates which are used as
  post-join filters, use UpdateItemListHashJoin() instead of
  UpdateItemList, because it updates local variables
  "has_from_inner" and "has_from_outer" which may be important for
  dummy projection.
  Example query is e.g. from data_masking.test:267:
    SELECT isnull(gen_rnd_ssn()), gen_rnd_pan(gen_range(1,1))
    FROM t1 JOIN t1 AS t2 ON t1.e > t2.e
    GROUP BY gen_rnd_ssn();
  For that query the GROUP BY key and the SELECT list do not
  contain any expressions over base table columns. Hence, when
  reaching a plan with the AGGREGATE AccessPath, we don't have
  any required items for it, but it needs at least one required
  item for dummy projection (see TranslateGroupBy() for
  step case 1).

Further fixes:
* Correctly handle the post-join filter retrieved through
  GetHyperGraphFilterCond(). First, all sub-expressions must be
  projected from the join node itself and then the post-join filter
  must be added to the intermediate qkrnts' filter, too.
* Add function DumpItemToString() which simply prints the result of
  ItemToString() for the given Item to the console. This is just a
  small debug utility function because my gdb does not print that
  output anymore.

Bug#34490127 - Always enforce Hypergraph Optimizer for HeatWave
Bug#34303379 - mysqld got signal 6 in rpd::ExtractCommonExpression at ha_rpd_qkrn_ap.cc
Bug#34242831 - [Sonny] Cost estimation: AttachToCommonParent: Expected a join node
Bug#34151831 - Query offload fails at ExtractItemFromDerivedTable
Bug#33659812 - LEFT JOIN OFFLOAD ISSUE: AttachToCommonParent: Expected a join node

Bug 34490127:
=============
* Set optimizer switch for using the hypergraph optimizer in the
  constructor of Rapid_execution_context to use always it for HeatWave
  going forward.
* Added an offload Check in RapidOptimize() that indeed the Hypergraph
  Optimizer is used.
* To take effect we had to move the code in sql_select.cc (function
  Sql_cmd_dml::prepare) which updates lex->using_hypergraph_optimizer to
  after the call to open_tables_for_query() such that the secondary
  engine's prepare hook is called before checking the optimizer switch,
  otherwise the first change above would stay ineffective.
* Removed adding the cmake flag WITH_HYPERGRAPH_OPTIMIZER from file
  _helper.sh again such that we can test the current behavior that the
  hypergraph optimizer is disabled by default for other engines than
  HeatWave in PERF builds. We circumvent this behavior for HeatWave by
  setting the optimizer switch programmatically.
* Remove the `--hypergraph` flag again from r_mtr.json.
* Update test files to not source files not_hypergraph.inc or
  have_hypergraph.inc, because they test for hypergraph being enabled or
  not by setting the optimizer_switch, which is now not possible on PERF
  builds anymore and we don't need that behavior anymore with the above
  changes.
* partition_dict.test and partition_varlen.test : Added missing
  enforcement of secondary engine.
* Revert result for test autodb_hcs_console_queries
* Re-enable tests from disabled.def : rapid.autodb_hcs_console_queries
  and rapid.qcomp_cost_model_evaluation_tpch.
  Re-record result for qcomp_cost_model_evaluation_tpch.result.
* Re-record affected test results.

Bugs 34303379, 34242831, 34151831, 33659812:
============================================
* Added test cases
* Added a separate sonnys test file for all bugs which use the respective schema.

WL#14449 - Query_term support

* Implement remaining changes for proper Query_term support.
* In the qkropn dumping code which tracks the use of tables and base
  table columns, checks were missing whether RapidSchema is nullptr.
  This led to crashes in some DEBUG MTR tests.
* Stabilize and improve several tests

WL#14449 - Handle FILTER AccessPaths with FALSE predicate

Until Bug 34494609 is properly fixed, add functionality to replace
FILTER AccessPaths with only FALSE-predicate by ZERO_ROWS or
ZERO_ROWS_AGGREGATED as appropriate. The latter is used when ZERO_ROWS
can be propagated all the way up to an aggregation AccessPath in the
current Query_block.
The ZERO_ROWS will be propagated as far up the current query block as
possible. When the patch for this bug is fixed from MySQL side, this
functionality can be removed again.

Change-Id: I77e6b7a75bb9071d4ad4fbc22b445c4bd51a82c7

Author: Kajori Banerjee

sql/item.h

@@ -7025,7 +7025,6 @@ void convert_and_print(const String *from_str, String *to_str,
                        const CHARSET_INFO *to_cs);
 
 std::string ItemToString(const Item *item);
-void ExtractConditions(Item *condition, std::vector<Item *> *condition_parts);
 
 inline size_t CountVisibleFields(const mem_root_deque<Item *> &fields) {
   return std::count_if(fields.begin(), fields.end(),

sql/join_optimizer/access_path.cc

@@ -1297,9 +1297,9 @@ static void MoveFilterPredicatesIntoHashJoinCondition(
       thd->mem_root, path->filter_predicates, std::move(moved_predicates));
 }
 
-static Item *ConditionFromFilterPredicates(
-    const Mem_root_array<Predicate> &predicates, OverflowBitset mask,
-    int num_where_predicates) {
+Item *ConditionFromFilterPredicates(const Mem_root_array<Predicate> &predicates,
+                                    OverflowBitset mask,
+                                    int num_where_predicates) {
   List<Item> items;
   for (int pred_idx : BitsSetIn(mask)) {
     if (pred_idx >= num_where_predicates) break;

sql/join_optimizer/access_path.h

@@ -1766,6 +1766,14 @@ void ExpandFilterAccessPaths(THD *thd, AccessPath *path, const JOIN *join,
                              const Mem_root_array<Predicate> &predicates,
                              unsigned num_where_predicates);
 
+/**
+  Extracts the Item expression from the given “filter_predicates” corresponding
+  to the given “mask”.
+ */
+Item *ConditionFromFilterPredicates(const Mem_root_array<Predicate> &predicates,
+                                    OverflowBitset mask,
+                                    int num_where_predicates);
+
 /// Like ExpandFilterAccessPaths(), but expands only the single access path
 /// at “path”.
 void ExpandSingleFilterAccessPath(THD *thd, AccessPath *path, const JOIN *join,

sql/join_optimizer/join_optimizer.cc

@@ -163,7 +163,7 @@ AccessPath *GetSafePathToSort(THD *thd, JOIN *join, AccessPath *path,
 class CostingReceiver {
  public:
   CostingReceiver(
-      THD *thd, Query_block *query_block, const JoinHypergraph &graph,
+      THD *thd, Query_block *query_block, JoinHypergraph &graph,
       const LogicalOrderings *orderings,
       const Mem_root_array<SortAheadOrdering> *sort_ahead_orderings,
       const Mem_root_array<ActiveIndexInfo> *active_indexes,
@@ -313,7 +313,7 @@ class CostingReceiver {
   int m_num_seen_subgraph_pairs = 0;
 
   /// The graph we are running over.
-  const JoinHypergraph *m_graph;
+  JoinHypergraph *m_graph;
 
   /// Whether we have applied clamping due to a multi-column EQ_REF at any
   /// point. There is a known issue (see bug #33550360) where this can cause
@@ -2091,6 +2091,7 @@ void CostingReceiver::ProposeAccessPathForIndex(
 bool CostingReceiver::ProposeTableScan(
     TABLE *table, int node_idx, double force_num_output_rows_after_filter,
     bool is_recursive_reference) {
+  m_graph->query_block_has_multiple_base_tables = false;
   AccessPath path;
   if (is_recursive_reference) {
     path.type = AccessPath::FOLLOW_TAIL;
@@ -3047,15 +3048,12 @@ bool CostingReceiver::FoundSubgraphPair(NodeMap left, NodeMap right,
       // For inner joins and full outer joins, the order does not matter.
       // In lieu of a more precise cost model, always keep the one that hashes
       // the fewest amount of rows. (This has lower initial cost, and the same
-      // cost.) When cost estimates are supplied by the secondary engine,
-      // explore both orders, since the secondary engine might unilaterally
-      // decide to prefer or reject one particular order. (TODO: Remove this,
-      // as the only relevant secondary engine does its own flipping.)
+      // cost.)
       //
       // Finally, if either of the sides are parameterized on something
       // external, flipping the order will not necessarily be allowed (and would
       // cause us to not give a hash join for these tables at all).
-      if (is_commutative && m_secondary_engine_cost_hook == nullptr &&
+      if (is_commutative &&
           !Overlaps(left_path->parameter_tables | right_path->parameter_tables,
                     RAND_TABLE_BIT)) {
         if (left_path->num_output_rows() < right_path->num_output_rows()) {
@@ -3167,6 +3165,8 @@ void CostingReceiver::ProposeHashJoin(
     bool *wrote_trace) {
   if (!SupportedEngineFlag(SecondaryEngineFlag::SUPPORTS_HASH_JOIN)) return;
 
+  m_graph->query_block_has_multiple_base_tables = true;
+
   if (Overlaps(left_path->parameter_tables, right) ||
       Overlaps(right_path->parameter_tables, left | RAND_TABLE_BIT)) {
     // Parameterizations must be resolved by nested loop.
@@ -6597,6 +6597,8 @@ AccessPath *FindBestQueryPlan(THD *thd, Query_block *query_block,
     if (join->make_sum_func_list(*join->fields, /*before_group_by=*/true))
       return nullptr;
 
+    graph.query_block_has_aggregation_node = true;
+
     if (trace != nullptr) {
       *trace += "Applying aggregation for GROUP BY\n";
     }
@@ -6726,6 +6728,7 @@ AccessPath *FindBestQueryPlan(THD *thd, Query_block *query_block,
   }
 
   join->m_windowing_steps = !join->m_windows.is_empty();
+  graph.query_block_has_windowfunc = true;
   if (join->m_windowing_steps) {
     root_candidates = ApplyWindowFunctions(
         thd, receiver, orderings, fd_set, aggregation_is_unordered,
@@ -6739,6 +6742,7 @@ AccessPath *FindBestQueryPlan(THD *thd, Query_block *query_block,
       "Applying filter for window function in2exists conditions\n", trace,
       &root_candidates, &receiver);
 
+  graph.query_block_ready_to_handle_distinct_order_by_limit_offset = true;
   if (join->select_distinct || join->order.order != nullptr) {
     // UPDATE and DELETE must preserve row IDs through ORDER BY in order to keep
     // track of which rows to update or delete.

sql/join_optimizer/make_join_hypergraph.h

@@ -77,6 +77,25 @@ struct JoinHypergraph {
 
   hypergraph::Hypergraph graph;
 
+  /// Flag to indicate (mostly for secondary engines) that the partial plans for
+  /// the current query block have the aggregation node present.
+  bool query_block_has_aggregation_node{false};
+
+  /// Flag to indicate (mostly for secondary engines) that the current query
+  /// block contains a WINDOW AccessPath.
+  bool query_block_has_windowfunc{false};
+
+  /// Flag to indicate (mostly for secondary engines) that the partial plans for
+  /// the current query block contains more than one "base" tables. These can
+  /// include derived tables and common table expressions.
+  bool query_block_has_multiple_base_tables{false};
+
+  /// Flag to indicate (mostly for secondary engines) that for the current query
+  /// block all join ordering-related AccessPaths were proposed and the
+  /// secondary engine can handle ORDER BY and / or LIMIT OFFSET clauses when it
+  /// needs to treat them specially.
+  bool query_block_ready_to_handle_distinct_order_by_limit_offset{false};
+
   // Maps table->tableno() to an index in “nodes”, also suitable for
   // a bit index in a NodeMap. This is normally the identity mapping,
   // except for when scalar-to-derived conversion is active.

sql/query_term.h

@@ -267,7 +267,7 @@ class Query_term {
 
  public:
   /// Getter for m_parent, q.v.
-  Query_term_set_op *parent() { return m_parent; }
+  Query_term_set_op *parent() const { return m_parent; }
 
   /**
     Reset resources used.

sql/sql_lex.cc

@@ -2412,14 +2412,14 @@ void Query_expression::set_explain_marker_from(THD *thd,
   thd->unlock_query_plan();
 }
 
-ha_rows Query_block::get_offset(const THD *) {
+ha_rows Query_block::get_offset(const THD *) const {
   if (offset_limit != nullptr)
     return ha_rows{offset_limit->val_uint()};
   else
     return ha_rows{0};
 }
 
-ha_rows Query_block::get_limit(const THD *thd) {
+ha_rows Query_block::get_limit(const THD *thd) const {
   /*
     If m_use_select_limit is set in the query block, return the value
     of the variable select_limit, unless an explicit limit is set.
@@ -4291,7 +4291,7 @@ static enum_explain_type setop2result(Query_term *qt) {
   are in sql_union.cc
 */
 
-enum_explain_type Query_block::type() {
+enum_explain_type Query_block::type() const {
   Query_term *qt = master_query_expression()->find_blocks_query_term(this);
   if (qt->term_type() != QT_QUERY_BLOCK) {
     return setop2result(qt);

sql/sql_lex.h

@@ -658,7 +658,7 @@ class Query_expression {
       return m_query_term->query_block();
   }
   bool is_leaf_block(Query_block *qb);
-  Query_term *find_blocks_query_term(Query_block *qb) {
+  Query_term *find_blocks_query_term(const Query_block *qb) const {
     for (auto qt : query_terms<>()) {
       if (qt->query_block() == qb) return qt;
     }
@@ -1515,7 +1515,7 @@ class Query_block : public Query_term {
 
     @todo Integrate better with Query_expression::set_limit()
   */
-  ha_rows get_offset(const THD *thd);
+  ha_rows get_offset(const THD *thd) const;
   /**
    Get limit.
 
@@ -1525,7 +1525,7 @@ class Query_block : public Query_term {
 
    @todo Integrate better with Query_expression::set_limit()
   */
-  ha_rows get_limit(const THD *thd);
+  ha_rows get_limit(const THD *thd) const;
 
   /// Assign a default name resolution object for this query block.
   bool set_context(Name_resolution_context *outer_context);
@@ -1799,7 +1799,7 @@ class Query_block : public Query_term {
   void set_right_joins() { m_right_joins = true; }
 
   /// Lookup for Query_block type
-  enum_explain_type type();
+  enum_explain_type type() const;
 
   /// Lookup for a type string
   const char *get_type_str() { return type_str[static_cast<int>(type())]; }

sql/sql_select.cc

@@ -330,18 +330,6 @@ bool Sql_cmd_dml::prepare(THD *thd) {
   assert(!lex->unit->is_prepared() && !lex->unit->is_optimized() &&
          !lex->unit->is_executed());
 
-  lex->using_hypergraph_optimizer =
-      thd->optimizer_switch_flag(OPTIMIZER_SWITCH_HYPERGRAPH_OPTIMIZER) &&
-      (lex->sql_command == SQLCOM_SELECT || lex->sql_command == SQLCOM_DO ||
-       lex->sql_command == SQLCOM_CALL ||
-       lex->sql_command == SQLCOM_INSERT_SELECT ||
-       lex->sql_command == SQLCOM_REPLACE_SELECT ||
-       lex->sql_command == SQLCOM_INSERT ||
-       lex->sql_command == SQLCOM_DELETE_MULTI ||
-       lex->sql_command == SQLCOM_DELETE ||
-       lex->sql_command == SQLCOM_UPDATE_MULTI ||
-       lex->sql_command == SQLCOM_UPDATE);
-
   /*
     Constant folding could cause warnings during preparation. Make
     sure they are promoted to errors when strict mode is enabled.
@@ -383,6 +371,18 @@ bool Sql_cmd_dml::prepare(THD *thd) {
   if (sql_command_code() == SQLCOM_SELECT) DEBUG_SYNC(thd, "after_table_open");
 #endif
 
+  lex->using_hypergraph_optimizer =
+      thd->optimizer_switch_flag(OPTIMIZER_SWITCH_HYPERGRAPH_OPTIMIZER) &&
+      (lex->sql_command == SQLCOM_SELECT || lex->sql_command == SQLCOM_DO ||
+       lex->sql_command == SQLCOM_CALL ||
+       lex->sql_command == SQLCOM_INSERT_SELECT ||
+       lex->sql_command == SQLCOM_REPLACE_SELECT ||
+       lex->sql_command == SQLCOM_INSERT ||
+       lex->sql_command == SQLCOM_DELETE_MULTI ||
+       lex->sql_command == SQLCOM_DELETE ||
+       lex->sql_command == SQLCOM_UPDATE_MULTI ||
+       lex->sql_command == SQLCOM_UPDATE);
+
   if (lex->set_var_list.elements && resolve_var_assignments(thd, lex))
     goto err; /* purecov: inspected */