Skip to content

NH-2319 - Add ability to query collections with AsQueryable() #375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Sep 13, 2017

Conversation

hazzik
Copy link
Member

@hazzik hazzik commented Nov 19, 2014

No description provided.

@hazzik hazzik added this to the 5.0.0 milestone Nov 19, 2014
@hazzik hazzik changed the base branch from master to 5.0.x November 28, 2016 21:45
@oskarb oskarb closed this Nov 28, 2016
@hazzik hazzik changed the base branch from 5.0.x to master November 29, 2016 00:02
@hazzik hazzik reopened this Nov 29, 2016
Copy link
Member

@fredericDelaporte fredericDelaporte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for not including idbags (PersistentIdentifierBag) and lists (PersistentGenericList) in the feature? Reading now the Jira, is this PR still limited to bidi one-to-many bag and set?

A bit like Fabio, I am not much convinced by the feature. It is nice to avoid loading the full collection in some cases, but well, this can easily be achieved with a query instead. Of course it is less straight forward, but since the feature seems currently so limited, it still looks as the default way to go for me.

Otherwise, I think we need an obvious mean for knowing we are in a currently unsupported case: throw an exception instead of falling back on linq-to-object? This has the advantage of being obvious for the developer. While the fallback to linq-to-object triggers silently the full load of the collection (and I guess from batch-size others too) and may be unnoticed by the developer. But if anyone has put some pointless AsQueryable() in its current code, it will be a breaking change. Maybe add a configuration property for choosing between fallback or throw.

For supporting maps, we would have to provide a dictionary with a custom Values implementation. Maybe not worth the trouble.

My review is just preliminary, I have not thoroughly studied (and understood) how this new feature works.

@@ -46,6 +49,8 @@ public class PersistentGenericBag<T> : AbstractPersistentCollection, IList<T>, I
*/
private IList<T> _gbag;

private IQueryable<T> queryable;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tend to use _ underscore prefix for private fields, and changes fields which does not have it when I have already change a lot in the class. I see that these new queryable fields do not have it. Are we accepting both, or shall we use a single style, and which one?

I think this field needs the [NonSerialized] attribute.

Copy link
Member Author

@hazzik hazzik Apr 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a hundreds years old PR, so I don't remember/know why I didn't put underscore in front of the field name.

Agreed on [NonSerialized] part.


private IQueryable<T> InnerQueryable
{
get { return queryable ?? (queryable = new NhQueryable<T>(Session, this)); }
Copy link
Member

@fredericDelaporte fredericDelaporte Apr 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't that the job of a new Lazy<NhQueryable<T>>(() => new NhQueryable<T>(Session, this)) which would be affected to a retyped queryable in the constructor?

I know we do not need thread safety here. This is just for avoiding coding this additional property and having a field which can be null if a later coder access it instead of using InnerQueryable.

Of course, we would then have to replace InnerQueryable calls by queryable.Value.

Other consideration: I think the field must be marked as non serializable. Better having them lazily recreated after deserialization. (The current pattern better support it. With a Lazy, we would need to implement ISerializable to recreate it I fear.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lazy<> is not really required. What's required is a proper implementation of IQueryable<>, and it turns out that NhQueryable<> is doing almost everything what we need.

@@ -38,6 +40,8 @@ public class PersistentGenericSet<T> : AbstractPersistentCollection, ISet<T>
[NonSerialized]
private IList<T> _tempList;

private IQueryable<T> queryable;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same remarks as for previous class.


if (plan == null)
{
if (log.IsDebugEnabled)
{
log.Debug("unable to locate collection-filter query plan in cache; generating (" + collectionRole + " : "
+ filterString + ")");
log.Debug(string.Format("unable to locate collection-filter query plan in cache; generating ({0} : {1})", collectionRole, queryExpression.Key));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess at the time of this change, $"" formatting was not available!

I am frequently a bit surprised by those if (log.IsDebugEnabled): they are pointless if they are just about avoiding formatting without any value evaluation, aren't they? log.DebugFormat("", args) already handle that.

Copy link
Member Author

@hazzik hazzik Apr 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess at the time of this change, $"" formatting was not available!

True. However, It seems that it shall be changed to log.DebugFormat(..)

if (log.IsDebugEnabled)

It seems just to avoid doing string concatenations and win some milliseconds and object allocations (in some cases it's actually a huge win)

Haven't found anything on C#, so the links are about Java, but internals are almost the same, so

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am used to rarely need the explicit check, because I am mostly using Format suffixed methods when having to format a log. Of course sometime we need to format with computed values, or to have a more complex message construction. Then I use the explicit check. I am sensible to this because log messages are almost bloat code in my view, and I rather keep them as compact as possible. (Actually I tend to only log exceptions. I fully agree with Jeff Atwood blog on this.)



if (fromElementFound)
return (IASTNode) adaptor.Nil();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has this been done because some of the calls above would have side effects? If yes, maybe a comment would be welcome. Otherwise I would tend to move this return in the if (fromElement == null) clause.

Copy link
Member Author

@hazzik hazzik Apr 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand after a quick look-around. There are some differences in the AST between HQL and LINQ. It seems that LINQ AST already contains fromElement in the tree, and HQL does not.

We still need to do all these manipulations with the fromElement, but we do not want to add it back to the tree (because it is already there).

CheckAndUpdateSessionStatus();
queryParameters.ValidateParameters();
var plan = GetFilterQueryPlan(collection, queryExpression, queryParameters, false);

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost dup code here. I think this need to be refactored with above method.

}
}

private IQueryExpressionPlan GetFilterQueryPlan(object collection, string filter, QueryParameters parameters, bool shallow)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, lot of dup code with above method, should probably be refactored.

@@ -46,11 +46,17 @@ public NhQueryable(IQueryProvider provider, Expression expression, string entity
EntityName = entityName;
}

public NhQueryable(ISessionImplementor session, object collection)
: base(new DefaultQueryProvider(session, collection))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it use the QueryProviderFactory?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. But it wasn't here when this code was written.

@hazzik
Copy link
Member Author

hazzik commented Apr 24, 2017

Is there a reason for not including idbags (PersistentIdentifierBag) and lists (PersistentGenericList) in the feature?

It's because this feature is really just a strongly-typed wrapper on a filter collection feature which does not support these collections

@hazzik
Copy link
Member Author

hazzik commented Apr 24, 2017

Also, Hibernate is moving towards this direction as well.

They are going to:

  • deprecate session#createFilter in HHH-11390
  • shift support for "collection filtering" to Java 8 Streams (a java version of LINQ) in HHH-10962

@hazzik
Copy link
Member Author

hazzik commented Apr 24, 2017

About the differences in LINQ vs HQL:

Linq generates following AST:

( query ( select_from ( from ( range NHibernate.Test.NHSpecificTest.NH2319.Child x ) ) ( select x ) ) ( where ( == ( . x Name ) ( : p1 ) ) ) )

And HQL:

"( query ( SELECT_FROM {filter-implied FROM} ) ( where ( = ( . this Name ) 'Jack' ) ) )"

So, it's better to eliminate the initial difference in the AST tree instead of modifying CreateFromFilterElement

@fredericDelaporte
Copy link
Member

I never needed to use filters so I am not sure, but are filter supporting batch loading of collection? I tend to think yes, and then, deprecating filters for queryable collections may cause a significant performances loss, unless we find a way to maintain batching with queryable collections.

@fredericDelaporte
Copy link
Member

fredericDelaporte commented Apr 24, 2017

Hibernate team seems to want to only remove the special query on collection case. I was thinking about the more global filters functionality, declarable in mappings. Well, I never found this query on collection feature very useful, it is so easy to just write a usual query doing the same job.

@hazzik
Copy link
Member Author

hazzik commented Apr 24, 2017

Hibernate team seems to want to only remove the special query on collection case.

Yes, only this Session#createFilter feature. They want to replace HQL version with Java Streams API.

global filters functionality, declarable in mappings

Not these filters:) I know it confusing to have 2 distinct features share similar names

Well, I neither find this query on collection feature very useful, it is so easy to just write a usual query doing the same job.

Actually, [some] people are desperate to have this feature implemented. It's one of the highest voted features in the JIRA.

@hazzik
Copy link
Member Author

hazzik commented Apr 25, 2017

Later SELECT_FROM {filter-implied FROM} is replaced by HqlFilterPreprocessor with SELECT_FROM ( FROM NHibernate.DomainModel.Many this ) which makes the queries identical except the alias name (this for HQL, and whatever for LINQ)

@fredericDelaporte
Copy link
Member

fredericDelaporte commented Sep 9, 2017

Rebased. Once rebased some tests were failing when run in sequence with others, due to a trouble when a filtered query plan was retrieved from cache.
And refactored.

Copy link
Member

@fredericDelaporte fredericDelaporte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some monthes ago I had not checked the Jira proposed implementation.
Appearing more simple, it has probably way more limitations and shortcomings than the one you have made.

Maybe we should add more tests for checking that your implementation support all non bidi cases, perhaps many-to-many too, ... All cases supported by the filter feature should be supported, but I do not know much what it does support in fact.

I forgot to regen async for all commits, going to do that too.

Edit: just realized idBag were many-to-many. So it just lacking testing other unidi cases and maybe bidi with inverse.

@@ -79,7 +79,7 @@ public void Reset()
}
}
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cleanup glitch, going to fix and repush forced.

role => Factory.QueryPlanCache.GetFilterQueryPlan(filter, role, shallow, EnabledFilters));
}

private IQueryExpressionPlan GetFilterQueryPlan(object collection, QueryParameters parameters, Func<string, IQueryExpressionPlan> getPlanForRole)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I usually avoid this kind of pattern, putting code in common thanks to function pointers passed around. But here, with the function being defined for all cases right above, I think it is ok.

Copy link
Member

@fredericDelaporte fredericDelaporte Sep 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does not seem compatible with Async generator, causes the removal of GetFilterQueryPlanAsync. Now I wonder what IO are there in that function, going to check. Ok it flushes, already forgot...

// For backward compatibility, prioritize using the version without collection.
return (collection == null
? Activator.CreateInstance(session.Factory.Settings.LinqQueryProviderType, session)
: Activator.CreateInstance(session.Factory.Settings.LinqQueryProviderType, session, collection)) as INhQueryProvider;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minimal backward compatibility handling.

if (filter != null && queryExpression != null)
throw new ArgumentException($"Either {nameof(filter)} or {nameof(queryExpression)} must be specified, not both.");
if (filter == null && queryExpression == null)
throw new ArgumentException($"{nameof(filter)} and {nameof(queryExpression)} were both null.");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not like this kind of parameter semantic but do not see a better solution for avoiding code duplication and not using a pattern not handled by async generator.

@@ -73,6 +74,7 @@ public TextLogSpy(string loggerName, string pattern)
};
loggerImpl = (Logger)LogManager.GetLogger(typeof(LogsFixture).Assembly, loggerName).Logger;
loggerImpl.AddAppender(appender);
previousLevel = loggerImpl.Level;
loggerImpl.Level = Level.All;
Copy link
Member

@fredericDelaporte fredericDelaporte Sep 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This spy was used only for NHibernate.SQL, but that is an heavily logging logger. Better restore its previous log level.

@@ -684,7 +684,6 @@ void Clone_TransactionCompleted(object sender, TransactionEventArgs e)
[OneTimeSetUp]
public void TestFixtureSetUp()
{
((Logger)_log.Logger).Level = log4net.Core.Level.Info;
_spy = new LogSpy(_log);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Undue log level manual change, since LogSpy set it to Debug anyway.

<para>
If the collection is a map, call <literal>AsQueryable</literal> on its <literal>Values</literal>
property.
</para>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added support for map.

Assert.That((ICollection) country.Statistics.Keys, Is.EquivalentTo((ICollection) stats.Keys), "Keys");
Assert.That((ICollection) country.Statistics.Values, Is.EquivalentTo((ICollection) stats.Values), "Elements");
Assert.That(country.Statistics.Keys, Is.EquivalentTo(stats.Keys), "Keys");
Assert.That(country.Statistics.Values, Is.EquivalentTo(stats.Values), "Elements");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those casts were likely required for an older NUnit. Since I have not additionally implemented ICollection on the map values wrapper, they were failing. We will probably have to document that as a possible breaking change.

var filtered = parent.ChildrenMap.Values
.AsQueryable()
.Where(x => x.Name == "Piter")
.SelectMany(x => x.GrandChildren)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a map here. Supporting maps inside lambda would be another feature. (Currently already doable by pretending the map is a ICollection<TValue> with a cast, but a bit hackish.)

rc.Id(x => x.Id, m => m.Generator(Generators.GuidComb));
rc.Property(x => x.Name);
rc.Map(x => x.ChildrenMap,
map => map.Cascade(Mapping.ByCode.Cascade.All | Mapping.ByCode.Cascade.DeleteOrphans),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not found how to specify the index, although we can specify it in hbm. Missing feature of by code mapping? Anyway, works with defaults for the test.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is supposed that you use the Map overload which takes 4 lambdas, but IMapKeyRelation is missing the index part.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or probably Element will behave as you need.

@@ -1,26 +1,21 @@
<Project Sdk="Microsoft.NET.Sdk">
<Import Project="../../build-common/NHibernate.props" />

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum, probable Rider glitch... Bad point for that IDE.


public ValuesWrapper(PersistentGenericMap<TKey, TValue> map)
{
_map = map;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just storing the persistent map, not the values, since the underlying map may still be changed at that point.

@@ -22,15 +25,22 @@ namespace NHibernate.Collection.Generic
[DebuggerTypeProxy(typeof(DictionaryProxy<,>))]
public partial class PersistentGenericMap<TKey, TValue> : AbstractPersistentCollection, IDictionary<TKey, TValue>, ICollection
{
protected IDictionary<TKey, TValue> WrappedMap;
protected IDictionary<TKey, TValue> WrappedMap { get; private set; }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Protected only for one test file, looks like it should have been private but experienced a probable "rogue" change. At least switched to property with private setter.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this please be reverted? And the test needs to change to use .Entries(...) method instead.

public PersistentGenericMap() { }
public PersistentGenericMap()
{
_wrappedValues = new ValuesWrapper(this);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

x3, one for each constructor: they cannot call each over and the field cannot be initialized in place when using this.

@@ -347,8 +358,7 @@ public ICollection<TValue> Values
{
get
{
Read();
return WrappedMap.Values;
return _wrappedValues;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read() moved to ValuesWrapper ICollection<TValue> members. Otherwise, reaching Values for calling AsQueryable was loading the map.

@fredericDelaporte
Copy link
Member

I will not merge it myself since it would be a bit auto-approval of my own non cosmetic changes.

Copy link
Member

@fredericDelaporte fredericDelaporte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted changes to test csproj.

Copy link
Member

@fredericDelaporte fredericDelaporte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Just rebased.)

return GetEnumerator();
}

public void Add(TValue item)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dictionary's "Values" collection is read-only.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must still implement the interface, and I am just letting the throw be done by the underlying collection. But just for doing that, it triggers the read. Would you rather have a direct throw not reading at all? Since it is an error case anyway, I do not tend to optimize it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this actually matters much, so let it be this way.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not matter much but I have already changed it anyway, just not yet pushed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But how about "I do not tend to optimize it."?

Copy link
Member

@fredericDelaporte fredericDelaporte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested changes done.

Copy link
Member Author

@hazzik hazzik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@fredericDelaporte fredericDelaporte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dislike keeping a non private field naming on such a member, but well, if you prefer minimize changed lines count...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants