-
-
Notifications
You must be signed in to change notification settings - Fork 82
Add support for int4range, int8range, int4range[], int8range[] #330
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #330 +/- ##
==========================================
+ Coverage 41.54% 42.97% +1.43%
==========================================
Files 115 118 +3
Lines 9436 8242 -1194
==========================================
- Hits 3920 3542 -378
+ Misses 5516 4700 -816
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rausnitz Thanks so much for raising this. Do you want to also add an implementation for PostgresEncodable
, PostgresDecodable
which will be the way forward. We plan to deprecate PostgresData
sooner rather than later.
Yes, can do. |
@fabianfett The I also added support for |
One thing to note: the Postgres The part that could become an issue is that I specified the If you'd prefer me to include |
Update: The latest commit gets around this issue by adding |
The commit that I just pushed makes it straightforward to add the other Postgres range types in the future, not limited to |
private let _isLowerBoundInclusive: UInt8 = 0x02 | ||
private let _isUpperBoundInclusive: UInt8 = 0x04 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is mirroring the way these values are defined in the Postgres source code: https://github.com/postgres/postgres/blob/36f40ce2dc66f1a36d6a12f7a0352e1c5bf1063e/src/include/utils/rangetypes.h#L37-L45.
In the latest commit, I took a different approach that is more straightforward and easier to extend: I created a This PR now covers two |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for taking so long to come back to you. This already is in a nice shape. However there are a few things that I would like to bring into the same shape as the other Encodable, Decodable implementations.
throw PostgresDecodingError.Code.failure | ||
} | ||
|
||
guard let lowerBound: Bound = postgresRange.lowerBound, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we have to check if the range is inclusive here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See new commit 5aa2b19
/// Postgres automatically converts it to a canonical form. | ||
/// Types such as `int4range` get converted to upper-bound-exclusive. | ||
/// This method is needed when converting an upper bound to inclusive. | ||
var stepDown: (() -> Self)? { get } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is that a property and not a simple method? Why should this be optional?
func decrement() -> Self
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed it to a method here: 1b3d009. (I forget why I made it a property initially).
I also changed it from returning an optional value to making the function throw, which I think is clearer. The reason it can't always return a value is because some Postgres range types are "continuous" and don't have a well-defined step. This includes the built-in Postgres types numrange, tsrange, and tstzrange. Section 8.17.7 of the docs here explains it: https://www.postgresql.org/docs/current/rangetypes.html#RANGETYPES-DISCRETE.
I originally attempted to do something with Swift's Strideable
but couldn't come up with a solution. Here's essentially what I would like to do in the ClosedRange
conformance to PostgresDecodable
, though this code isn't valid Swift:
if let strideable = upperBound as? Strideable {
upperBound = strideable.advanced(by: -1)
}
|
||
// MARK: PostgresRange | ||
|
||
public struct PostgresRange<B: PostgresRangeBound>: CustomStringConvertible { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to make this type public?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I remove public
, I see build errors such as generic struct 'PostgresRange' is internal and cannot be referenced from an '@inlinable' function
on this line: https://github.com/vapor/postgres-nio/pull/330/files/61b4d5939e43d88ce92406c07645427dd561ac4e#diff-5c0292b699e71becd70812877bc88e53d4e1c18d061b195dacfcdfbb0d8f8faaR17. I could try removing @inlinable
but the other existing conformances to PostgresDecodable
and PostgresEncodable
use it so I figured I should leave it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another solution would be to add @usableFromInline
in a bunch of places, such as:
@usableFromInline
struct PostgresRange<B> {
@usableFromInline let lowerBound: B?
@usableFromInline let upperBound: B?
@usableFromInline let isLowerBoundInclusive: Bool
@usableFromInline let isUpperBoundInclusive: Bool
init(
lowerBound: B?,
upperBound: B?,
isLowerBoundInclusive: Bool,
isUpperBoundInclusive: Bool
) {
self.lowerBound = lowerBound
self.upperBound = upperBound
self.isLowerBoundInclusive = isLowerBoundInclusive
self.isUpperBoundInclusive = isUpperBoundInclusive
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes please let's go for the second option! I really, really only want to add things to the public API that we need.
|
||
// MARK: PostgresRangeBound | ||
|
||
public protocol PostgresRangeBound: PostgresDataConvertible, Comparable { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we split this into PostgresRangeEncodable
and PostgresRangeDecodable
? Why does this inherit Comparable
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just pushed a commit where I limited the usage of Comparable
: 3b29899. The reason it's needed at all is because in Swift's RangeExpression
types, the Bound
has to be Comparable
: https://developer.apple.com/documentation/swift/rangeexpression/bound.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I replaced PostgresRangeBound
with PostgresRangeEncodable
and PostgresRangeDecodable
here: 9b8b3ec.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lovely!
// MARK: PostgresData | ||
|
||
extension PostgresData { | ||
public func range<B>() -> PostgresRange<B>? | ||
where B: PostgresRangeBound | ||
{ | ||
guard var value: ByteBuffer = self.value else { | ||
return nil | ||
} | ||
|
||
return PostgresRange<B>( | ||
from: &value, | ||
format: self.formatCode | ||
) | ||
} | ||
|
||
public init<B>(range: PostgresRange<B>) | ||
{ | ||
var buffer = ByteBufferAllocator().buffer(capacity: 0) | ||
range.encode(into: &buffer) | ||
|
||
self.init( | ||
type: B.rangeType, | ||
typeModifier: nil, | ||
formatCode: .binary, | ||
value: buffer | ||
) | ||
} | ||
} | ||
|
||
// MARK: PostgresDataConvertible conformance | ||
|
||
extension PostgresRange: PostgresDataConvertible where B: PostgresRangeBound { | ||
public static var postgresDataType: PostgresDataType { return B.rangeType } | ||
public init?(postgresData: PostgresData) { | ||
guard var value = postgresData.value else { | ||
return nil | ||
} | ||
self.init(from: &value, format: .binary) | ||
} | ||
public var postgresData: PostgresData? { | ||
return PostgresData(range: self) | ||
} | ||
} | ||
|
||
extension RangeExpression where Bound: PostgresRangeBound { | ||
public static var postgresDataType: PostgresDataType { | ||
return Bound.postgresDataType | ||
} | ||
} | ||
|
||
extension Range: PostgresDataConvertible where Bound: PostgresRangeBound { | ||
public init?(postgresData: PostgresData) { | ||
guard let postgresRange: PostgresRange<Bound> = postgresData.range(), | ||
let lowerBound: Bound = postgresRange.lowerBound, | ||
let upperBound: Bound = postgresRange.upperBound, | ||
postgresRange.isLowerBoundInclusive, | ||
!postgresRange.isUpperBoundInclusive | ||
else { | ||
return nil | ||
} | ||
|
||
self = lowerBound..<upperBound | ||
} | ||
|
||
public var postgresData: PostgresData? { | ||
return PostgresData(range: PostgresRange<Bound>(range: self)) | ||
} | ||
} | ||
|
||
extension ClosedRange: PostgresDataConvertible where Bound: PostgresRangeBound { | ||
public init?(postgresData: PostgresData) { | ||
guard let postgresRange: PostgresRange<Bound> = postgresData.range(), | ||
let lowerBound: Bound = postgresRange.lowerBound, | ||
var upperBound: Bound = postgresRange.upperBound, | ||
postgresRange.isLowerBoundInclusive | ||
else { | ||
return nil | ||
} | ||
|
||
if !postgresRange.isUpperBoundInclusive, | ||
let steppedDownUpperBound = upperBound.stepDown?() | ||
{ | ||
upperBound = steppedDownUpperBound | ||
} | ||
|
||
self = lowerBound...upperBound | ||
} | ||
|
||
public var postgresData: PostgresData? { | ||
return PostgresData(range: PostgresRange<Bound>.init(closedRange: self)) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be okay for you if we don't add conformance to PostgresDataConvertible
? We have already started deprecating this API in #313.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See new commit 52f1d29
extension Range: PostgresArrayDecodable where Bound: PostgresRangeBound {} | ||
|
||
extension Range: PostgresArrayEncodable where Bound: PostgresRangeBound { | ||
public static var psqlArrayType: PostgresDataType { Bound.rangeType.arrayType! } | ||
} | ||
|
||
extension ClosedRange: PostgresArrayDecodable where Bound: PostgresRangeBound {} | ||
|
||
extension ClosedRange: PostgresArrayEncodable where Bound: PostgresRangeBound { | ||
public static var psqlArrayType: PostgresDataType { Bound.rangeType.arrayType! } | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Having the separation into PostgresRangeEncodable
and PostgresRangeDecodable
will make this much nicer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Implemented here: 9b8b3ec.
extension RangeExpression where Bound: PostgresRangeBound { | ||
public static var psqlFormat: PostgresFormat { return .binary } | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we link the postgres docs here, why this is the case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm actually not sure, though I did notice that all the other implementations of PostgresEncodable
return .binary
as the value for their psqlFormat
.
Is there a scenario where we want to enable the option of .text
even though .binary
works?
@fabianfett Thanks for the review! I'll respond to your comments in the next few days. |
I went ahead and did this in 8dc6146, but not sure what to do with |
I added a new commit aecfdc1 to handle empty ranges, since Postgres treats them as a special case:
Source: https://www.postgresql.org/docs/current/rangetypes.html#RANGETYPES-IO. This means the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Really polishing now!
extension Range: PostgresArrayEncodable where Bound: PostgresRangeEncodable { | ||
public static var psqlArrayType: PostgresDataType { Bound.psqlRangeType.arrayType! } | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm afraid this is very dangerous! Can we instead add a PostgresRangeArrayEncodable
protocol?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See new commit 71e1042
|
||
/// Postgres does not store any bound values for empty ranges, | ||
/// but Swift requires a value to initialize an empty Range<Bound>. | ||
static var defaultBoundValueForEmptyRange: Self { get } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think about this name?
fallbackValueForEmptyRange
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about valueForDecodedEmptyRange
or valueForEmptyRange
? The terms "default" and "fallback" imply that this value is only used if Postgres fails to provide a value, when we know Postgres will never provide a value.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like valueForEmptyRange
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in 884e807
} | ||
} | ||
|
||
extension PostgresRange where B: Comparable { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do we need to constrain B here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The two initializers in this extension require it.
The Bound
value in Range
and ClosedRange
must be Comparable
. Removing where B: Comparable
here causes a build error.
See https://developer.apple.com/documentation/swift/range#declaration and https://developer.apple.com/documentation/swift/closedrange#declaration.
value.encode(into: &buffer, context: .default) | ||
XCTAssertEqual(Range<Int32>.psqlType, .int4Range) | ||
XCTAssertEqual(buffer.readableBytes, 17) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we check the binary representation here? Otherwise two a bug in encode could be resolved by a bug in decode?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See new commit 41c2846. Is this what you wanted to see tests for?
@@ -341,6 +341,85 @@ final class PostgresNIOTests: XCTestCase { | |||
XCTAssertEqual(UUID(uuidString: row?[data: "id"].string ?? ""), UUID(uuidString: "123E4567-E89B-12D3-A456-426655440000")) | |||
} | |||
|
|||
func testInt4Range() throws { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome can we also add an integration test for it here?
Super tiny nit: (Please only fix if you want) We could also use the async syntax for all tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lovely. Can't believe how good this has already gotten. Sadly there is a bit more.
} | ||
|
||
/// A type that can be encoded into a Postgres range array type where it is the bound type | ||
public protocol PostgresRangeArrayEncodable { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should inherit from PostgresRangeEncodable
, shouldn't it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See new commit e3c891f
public protocol PostgresRangeArrayEncodable { | ||
static var psqlRangeArrayType: PostgresDataType { get } | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we should also add a PostgresRangeArrayDecodable
which inherits from PostgresRangeDecodable
to keep symmetry?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. Done in e3c891f.
@@ -85,6 +85,18 @@ extension UUID: PostgresArrayEncodable { | |||
public static var psqlArrayType: PostgresDataType { .uuidArray } | |||
} | |||
|
|||
extension Range: PostgresArrayDecodable where Bound: PostgresRangeDecodable {} | |||
|
|||
extension Range: PostgresArrayEncodable where Bound: PostgresRangeEncodable & PostgresRangeArrayEncodable { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
extension Range: PostgresArrayEncodable where Bound: PostgresRangeEncodable & PostgresRangeArrayEncodable { | |
extension Range: PostgresArrayEncodable where Bound: PostgresRangeArrayEncodable { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in e3c891f
@@ -85,6 +85,18 @@ extension UUID: PostgresArrayEncodable { | |||
public static var psqlArrayType: PostgresDataType { .uuidArray } | |||
} | |||
|
|||
extension Range: PostgresArrayDecodable where Bound: PostgresRangeDecodable {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we constrain to PostgresRangeArrayDecodable
here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done in e3c891f
@@ -341,6 +341,85 @@ final class PostgresNIOTests: XCTestCase { | |||
XCTAssertEqual(UUID(uuidString: row?[data: "id"].string ?? ""), UUID(uuidString: "123E4567-E89B-12D3-A456-426655440000")) | |||
} | |||
|
|||
func testInt4Range() throws { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
self.addTeardownBlock {
try await conn.close()
}
I'm enjoying the work and welcome the thorough reviews. I'll have the next batch in soon. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lovely! We should fix the one @usableFromInline
to @inlinable
. However I would love if @gwynne could take a look as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is excellent work! I had only the one concern, see below.
Co-authored-by: Fabian Fett <[email protected]>
@rausnitz this is great work! Thanks so much for pushing through with this. I know I had lot's of comments. |
@fabianfett Thanks for bearing with me through the iterations! |
This PR adds support for the
int8range
Postgres type and the corresponding Swift typeRange<Int64>
. Postgres comes with built-in support forint8range
and five other range types: https://www.postgresql.org/docs/current/rangetypes.html.The test case I created in this PR demonstrates the new functionality. In short, the range
'[-9223372036854775808, 9223372036854775807)'::int8range
gets decoded to-9223372036854775808..<9223372036854775807
and the range arrayARRAY['[0, 1)'::int8range, '[10, 11)'::int8range]
gets decoded to[0..<1, 10..<11]
.For now, I only wrote code for
int8range
andint8range[]
because that's what I personally had a need for, but I'd be happy to add support for the other range types (andClosedRange
too), once I get confirmation that there is interest in adding this stuff to PostgresNIO.Since these types are standard in Postgres, I think they belong in PostgresNIO.