Skip to content

Add support for int4range, int8range, int4range[], int8range[] #330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 37 commits into from
May 9, 2023

Conversation

rausnitz
Copy link
Contributor

@rausnitz rausnitz commented Mar 5, 2023

This PR adds support for the int8range Postgres type and the corresponding Swift type Range<Int64>. Postgres comes with built-in support for int8range and five other range types: https://www.postgresql.org/docs/current/rangetypes.html.

The test case I created in this PR demonstrates the new functionality. In short, the range '[-9223372036854775808, 9223372036854775807)'::int8range gets decoded to -9223372036854775808..<9223372036854775807 and the range array ARRAY['[0, 1)'::int8range, '[10, 11)'::int8range] gets decoded to [0..<1, 10..<11].

For now, I only wrote code for int8range and int8range[] because that's what I personally had a need for, but I'd be happy to add support for the other range types (and ClosedRange too), once I get confirmation that there is interest in adding this stuff to PostgresNIO.

Since these types are standard in Postgres, I think they belong in PostgresNIO.

@codecov-commenter
Copy link

codecov-commenter commented Mar 5, 2023

Codecov Report

Merging #330 (a6e1dd8) into main (7a816db) will increase coverage by 1.43%.
The diff coverage is 84.61%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #330      +/-   ##
==========================================
+ Coverage   41.54%   42.97%   +1.43%     
==========================================
  Files         115      118       +3     
  Lines        9436     8242    -1194     
==========================================
- Hits         3920     3542     -378     
+ Misses       5516     4700     -816     
Impacted Files Coverage Δ
...s/PostgresNIO/New/Data/Range+PostgresCodable.swift 83.82% <83.82%> (ø)
Sources/PostgresNIO/Data/PostgresDataType.swift 60.37% <100.00%> (+54.71%) ⬆️
...s/PostgresNIO/New/Data/Array+PostgresCodable.swift 71.71% <100.00%> (+0.77%) ⬆️

... and 115 files with indirect coverage changes

Copy link
Collaborator

@fabianfett fabianfett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rausnitz Thanks so much for raising this. Do you want to also add an implementation for PostgresEncodable, PostgresDecodable which will be the way forward. We plan to deprecate PostgresData sooner rather than later.

@rausnitz
Copy link
Contributor Author

Yes, can do.

@rausnitz
Copy link
Contributor Author

@fabianfett The PostgresCodable implementation is done.

I also added support for ClosedRange<Int64>. I reused my Range<Int64> functions instead of implementing binary encoding for ClosedRange<Int64>. Since Range is the canonical form for int8range (see https://www.postgresql.org/docs/current/rangetypes.html#RANGETYPES-DISCRETE), Postgres would have converted the upper bound anyway.

@rausnitz rausnitz requested a review from fabianfett March 14, 2023 20:15
@rausnitz
Copy link
Contributor Author

One thing to note: the Postgres int8range type can also map to Swift's PartialRangeFrom, PartialRangeThrough, and PartialRangeUpTo types. I didn't add support for those in this PR because I don't need it for my use-case. It wouldn't be difficult to make those types conform to PostgresCodable later without breaking anything.

The part that could become an issue is that I specified the Range type in PostgresData.init(int8Range:) and PostgresData.int8Range. If we wanted to make those generic in the future after this code has been released with Range specified, it would probably be a breaking change at that point. But it's a bit tricky to use the generic RangeExpression because of the associated type, and I didn't know if it was worth coming up with a solution if PostgresData is going away soon anyway.

If you'd prefer me to include PartialRangeFrom, PartialRangeThrough, and PartialRangeUpTo in this PR, let me know.

@rausnitz
Copy link
Contributor Author

The part that could become an issue is that I specified the Range type in PostgresData.init(int8Range:) and PostgresData.int8Range. If we wanted to make those generic in the future after this code has been released with Range specified, it would probably be a breaking change at that point. But it's a bit tricky to use the generic RangeExpression because of the associated type, and I didn't know if it was worth coming up with a solution if PostgresData is going away soon anyway.

Update: The latest commit gets around this issue by adding PostgresInt8RangeExpression.

@rausnitz
Copy link
Contributor Author

The commit that I just pushed makes it straightforward to add the other Postgres range types in the future, not limited to int8range.

Comment on lines 171 to 172
private let _isLowerBoundInclusive: UInt8 = 0x02
private let _isUpperBoundInclusive: UInt8 = 0x04
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is mirroring the way these values are defined in the Postgres source code: https://github.com/postgres/postgres/blob/36f40ce2dc66f1a36d6a12f7a0352e1c5bf1063e/src/include/utils/rangetypes.h#L37-L45.

@rausnitz rausnitz marked this pull request as draft March 25, 2023 13:57
@rausnitz rausnitz changed the title Add support for int8range and int8range[] Add support for int4range, int8range, int4range[], int8range[] Mar 28, 2023
@rausnitz rausnitz marked this pull request as ready for review March 28, 2023 22:03
@rausnitz
Copy link
Contributor Author

In the latest commit, I took a different approach that is more straightforward and easier to extend: I created a PostgresRange struct type that contains the properties necessary to represent all the Swift RangeExpression types, with any kind of associated Bound. This turned out to be a lot cleaner and simpler than using protocols to represent Postgres ranges. The new PostgresRange type handles all the encoding to and decoding from binary. So, adding support for more RangeExpression types and Bound types should be very easy, both in the old PostgresData way and the new PostgresCodable way.

This PR now covers two RangeExpression types (Range and ClosedRange) with two Bound types (Int32 and Int64). I can implement all the others supported by Postgres if @fabianfett approves of what I've got so far.

Copy link
Collaborator

@fabianfett fabianfett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for taking so long to come back to you. This already is in a nice shape. However there are a few things that I would like to bring into the same shape as the other Encodable, Decodable implementations.

throw PostgresDecodingError.Code.failure
}

guard let lowerBound: Bound = postgresRange.lowerBound,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have to check if the range is inclusive here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See new commit 5aa2b19

/// Postgres automatically converts it to a canonical form.
/// Types such as `int4range` get converted to upper-bound-exclusive.
/// This method is needed when converting an upper bound to inclusive.
var stepDown: (() -> Self)? { get }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is that a property and not a simple method? Why should this be optional?

func decrement() -> Self

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to a method here: 1b3d009. (I forget why I made it a property initially).

I also changed it from returning an optional value to making the function throw, which I think is clearer. The reason it can't always return a value is because some Postgres range types are "continuous" and don't have a well-defined step. This includes the built-in Postgres types numrange, tsrange, and tstzrange. Section 8.17.7 of the docs here explains it: https://www.postgresql.org/docs/current/rangetypes.html#RANGETYPES-DISCRETE.

I originally attempted to do something with Swift's Strideable but couldn't come up with a solution. Here's essentially what I would like to do in the ClosedRange conformance to PostgresDecodable, though this code isn't valid Swift:

if let strideable = upperBound as? Strideable {
    upperBound = strideable.advanced(by: -1)
}


// MARK: PostgresRange

public struct PostgresRange<B: PostgresRangeBound>: CustomStringConvertible {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to make this type public?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I remove public, I see build errors such as generic struct 'PostgresRange' is internal and cannot be referenced from an '@inlinable' function on this line: https://github.com/vapor/postgres-nio/pull/330/files/61b4d5939e43d88ce92406c07645427dd561ac4e#diff-5c0292b699e71becd70812877bc88e53d4e1c18d061b195dacfcdfbb0d8f8faaR17. I could try removing @inlinable but the other existing conformances to PostgresDecodable and PostgresEncodable use it so I figured I should leave it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another solution would be to add @usableFromInline in a bunch of places, such as:

@usableFromInline
struct PostgresRange<B> {
    @usableFromInline let lowerBound: B?
    @usableFromInline let upperBound: B?
    @usableFromInline let isLowerBoundInclusive: Bool
    @usableFromInline let isUpperBoundInclusive: Bool

    init(
        lowerBound: B?,
        upperBound: B?,
        isLowerBoundInclusive: Bool,
        isUpperBoundInclusive: Bool
    ) {
        self.lowerBound = lowerBound
        self.upperBound = upperBound
        self.isLowerBoundInclusive = isLowerBoundInclusive
        self.isUpperBoundInclusive = isUpperBoundInclusive
    }
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes please let's go for the second option! I really, really only want to add things to the public API that we need.


// MARK: PostgresRangeBound

public protocol PostgresRangeBound: PostgresDataConvertible, Comparable {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we split this into PostgresRangeEncodable and PostgresRangeDecodable? Why does this inherit Comparable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just pushed a commit where I limited the usage of Comparable: 3b29899. The reason it's needed at all is because in Swift's RangeExpression types, the Bound has to be Comparable: https://developer.apple.com/documentation/swift/rangeexpression/bound.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I replaced PostgresRangeBound with PostgresRangeEncodable and PostgresRangeDecodable here: 9b8b3ec.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lovely!

Comment on lines 147 to 239
// MARK: PostgresData

extension PostgresData {
public func range<B>() -> PostgresRange<B>?
where B: PostgresRangeBound
{
guard var value: ByteBuffer = self.value else {
return nil
}

return PostgresRange<B>(
from: &value,
format: self.formatCode
)
}

public init<B>(range: PostgresRange<B>)
{
var buffer = ByteBufferAllocator().buffer(capacity: 0)
range.encode(into: &buffer)

self.init(
type: B.rangeType,
typeModifier: nil,
formatCode: .binary,
value: buffer
)
}
}

// MARK: PostgresDataConvertible conformance

extension PostgresRange: PostgresDataConvertible where B: PostgresRangeBound {
public static var postgresDataType: PostgresDataType { return B.rangeType }
public init?(postgresData: PostgresData) {
guard var value = postgresData.value else {
return nil
}
self.init(from: &value, format: .binary)
}
public var postgresData: PostgresData? {
return PostgresData(range: self)
}
}

extension RangeExpression where Bound: PostgresRangeBound {
public static var postgresDataType: PostgresDataType {
return Bound.postgresDataType
}
}

extension Range: PostgresDataConvertible where Bound: PostgresRangeBound {
public init?(postgresData: PostgresData) {
guard let postgresRange: PostgresRange<Bound> = postgresData.range(),
let lowerBound: Bound = postgresRange.lowerBound,
let upperBound: Bound = postgresRange.upperBound,
postgresRange.isLowerBoundInclusive,
!postgresRange.isUpperBoundInclusive
else {
return nil
}

self = lowerBound..<upperBound
}

public var postgresData: PostgresData? {
return PostgresData(range: PostgresRange<Bound>(range: self))
}
}

extension ClosedRange: PostgresDataConvertible where Bound: PostgresRangeBound {
public init?(postgresData: PostgresData) {
guard let postgresRange: PostgresRange<Bound> = postgresData.range(),
let lowerBound: Bound = postgresRange.lowerBound,
var upperBound: Bound = postgresRange.upperBound,
postgresRange.isLowerBoundInclusive
else {
return nil
}

if !postgresRange.isUpperBoundInclusive,
let steppedDownUpperBound = upperBound.stepDown?()
{
upperBound = steppedDownUpperBound
}

self = lowerBound...upperBound
}

public var postgresData: PostgresData? {
return PostgresData(range: PostgresRange<Bound>.init(closedRange: self))
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be okay for you if we don't add conformance to PostgresDataConvertible? We have already started deprecating this API in #313.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See new commit 52f1d29

Comment on lines 88 to 98
extension Range: PostgresArrayDecodable where Bound: PostgresRangeBound {}

extension Range: PostgresArrayEncodable where Bound: PostgresRangeBound {
public static var psqlArrayType: PostgresDataType { Bound.rangeType.arrayType! }
}

extension ClosedRange: PostgresArrayDecodable where Bound: PostgresRangeBound {}

extension ClosedRange: PostgresArrayEncodable where Bound: PostgresRangeBound {
public static var psqlArrayType: PostgresDataType { Bound.rangeType.arrayType! }
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having the separation into PostgresRangeEncodable and PostgresRangeDecodable will make this much nicer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented here: 9b8b3ec.

Comment on lines 3 to 5
extension RangeExpression where Bound: PostgresRangeBound {
public static var psqlFormat: PostgresFormat { return .binary }
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we link the postgres docs here, why this is the case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually not sure, though I did notice that all the other implementations of PostgresEncodable return .binary as the value for their psqlFormat.

Is there a scenario where we want to enable the option of .text even though .binary works?

@rausnitz
Copy link
Contributor Author

@fabianfett Thanks for the review! I'll respond to your comments in the next few days.

@rausnitz
Copy link
Contributor Author

To resolve them, I can make those properties and the initializer all @usableFromInline. I'd also make _isLowerBoundInclusive and _isUpperBoundInclusive internal, which probably means I should drop the _ prefix (assuming that convention is for private properties).

I went ahead and did this in 8dc6146, but not sure what to do with let _isLowerBoundInclusive and let _isUpperBoundInclusive. The code works, but now that they're not private, should I make them part of a type instead of making them global?

@rausnitz rausnitz requested a review from fabianfett April 21, 2023 22:31
@rausnitz
Copy link
Contributor Author

I added a new commit aecfdc1 to handle empty ranges, since Postgres treats them as a special case:

-- includes no points (and will be normalized to 'empty')
SELECT '[4,4)'::int4range;

Source: https://www.postgresql.org/docs/current/rangetypes.html#RANGETYPES-IO.

This means the PostgresDecodable implementation won't find any bound values when parsing the binary. I think it's better to define a defaultBoundValueForEmptyRange than to throw an error.

Copy link
Collaborator

@fabianfett fabianfett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really polishing now!

Comment on lines 90 to 92
extension Range: PostgresArrayEncodable where Bound: PostgresRangeEncodable {
public static var psqlArrayType: PostgresDataType { Bound.psqlRangeType.arrayType! }
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid this is very dangerous! Can we instead add a PostgresRangeArrayEncodable protocol?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See new commit 71e1042


/// Postgres does not store any bound values for empty ranges,
/// but Swift requires a value to initialize an empty Range<Bound>.
static var defaultBoundValueForEmptyRange: Self { get }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about this name?

fallbackValueForEmptyRange

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about valueForDecodedEmptyRange or valueForEmptyRange? The terms "default" and "fallback" imply that this value is only used if Postgres fails to provide a value, when we know Postgres will never provide a value.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like valueForEmptyRange.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 884e807

}
}

extension PostgresRange where B: Comparable {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need to constrain B here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two initializers in this extension require it.

The Bound value in Range and ClosedRange must be Comparable. Removing where B: Comparable here causes a build error.

See https://developer.apple.com/documentation/swift/range#declaration and https://developer.apple.com/documentation/swift/closedrange#declaration.

value.encode(into: &buffer, context: .default)
XCTAssertEqual(Range<Int32>.psqlType, .int4Range)
XCTAssertEqual(buffer.readableBytes, 17)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we check the binary representation here? Otherwise two a bug in encode could be resolved by a bug in decode?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See new commit 41c2846. Is this what you wanted to see tests for?

@@ -341,6 +341,85 @@ final class PostgresNIOTests: XCTestCase {
XCTAssertEqual(UUID(uuidString: row?[data: "id"].string ?? ""), UUID(uuidString: "123E4567-E89B-12D3-A456-426655440000"))
}

func testInt4Range() throws {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome can we also add an integration test for it here?

Super tiny nit: (Please only fix if you want) We could also use the async syntax for all tests.

@rausnitz rausnitz requested a review from fabianfett April 27, 2023 11:30
Copy link
Collaborator

@fabianfett fabianfett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lovely. Can't believe how good this has already gotten. Sadly there is a bit more.

}

/// A type that can be encoded into a Postgres range array type where it is the bound type
public protocol PostgresRangeArrayEncodable {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should inherit from PostgresRangeEncodable, shouldn't it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See new commit e3c891f

public protocol PostgresRangeArrayEncodable {
static var psqlRangeArrayType: PostgresDataType { get }
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should also add a PostgresRangeArrayDecodable which inherits from PostgresRangeDecodable to keep symmetry?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Done in e3c891f.

@@ -85,6 +85,18 @@ extension UUID: PostgresArrayEncodable {
public static var psqlArrayType: PostgresDataType { .uuidArray }
}

extension Range: PostgresArrayDecodable where Bound: PostgresRangeDecodable {}

extension Range: PostgresArrayEncodable where Bound: PostgresRangeEncodable & PostgresRangeArrayEncodable {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
extension Range: PostgresArrayEncodable where Bound: PostgresRangeEncodable & PostgresRangeArrayEncodable {
extension Range: PostgresArrayEncodable where Bound: PostgresRangeArrayEncodable {

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e3c891f

@@ -85,6 +85,18 @@ extension UUID: PostgresArrayEncodable {
public static var psqlArrayType: PostgresDataType { .uuidArray }
}

extension Range: PostgresArrayDecodable where Bound: PostgresRangeDecodable {}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we constrain to PostgresRangeArrayDecodable here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e3c891f

@@ -341,6 +341,85 @@ final class PostgresNIOTests: XCTestCase {
XCTAssertEqual(UUID(uuidString: row?[data: "id"].string ?? ""), UUID(uuidString: "123E4567-E89B-12D3-A456-426655440000"))
}

func testInt4Range() throws {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self.addTeardownBlock {
    try await conn.close()
}

@rausnitz
Copy link
Contributor Author

Lovely. Can't believe how good this has already gotten. Sadly there is a bit more.

I'm enjoying the work and welcome the thorough reviews. I'll have the next batch in soon.

@rausnitz rausnitz requested a review from fabianfett April 30, 2023 11:41
Copy link
Collaborator

@fabianfett fabianfett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lovely! We should fix the one @usableFromInline to @inlinable. However I would love if @gwynne could take a look as well.

@fabianfett fabianfett requested a review from gwynne May 8, 2023 12:27
Copy link
Member

@gwynne gwynne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is excellent work! I had only the one concern, see below.

@rausnitz rausnitz requested a review from gwynne May 8, 2023 20:59
@fabianfett fabianfett merged commit 8981a23 into vapor:main May 9, 2023
@fabianfett
Copy link
Collaborator

@rausnitz this is great work! Thanks so much for pushing through with this. I know I had lot's of comments.

@rausnitz rausnitz deleted the int-range-support branch May 9, 2023 16:58
@rausnitz
Copy link
Contributor Author

rausnitz commented May 9, 2023

@fabianfett Thanks for bearing with me through the iterations!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants