Send documents as jsonlines for POST and PUT requests #306
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses #265.
The main obstacle I see is that
serde_json
does not offer a way to serialize data to jsonlines. For example, vectors are automatically serialized to json arrays byserde_json.to_string()
.A simple workaround is to iterate over the collection and serialize each item individually. Then we can join them using
\n
as the separator to obtain jsonlines:However, in the
request
function, where the data is serialized,Serialize
is the only trait bound for the data (Input
). That means we cannot iterate over the data to obtain jsonlines as in the example above.To work around this we could create another
request
function (and call itrequest_jsonlines
or whatever) which only takes in data that is iterable. Then we would also have to create a newMethod
enum. I did not want to do this because it would entail a lot of duplicate code.Another option would be to write a function that converts a json array to jsonlines. However, that would probably be less efficient than computing the jsonlines directly (as shown above).
Hence, I decided to adapt the
Method
enum, such that it can hold two different types. A type that is iterable (for jsonlines) and a type that is not necessary iterable (for all other calls torequest
).Let me know what you think about this solution. I am open to suggestions :)