form semantics

[handrews]

I'm opening this to continue the discussions around the "form" proposal from #280 and #290, which have been closed in order to split everything else from those discussions out into separate PRs (#292 and #293). This issue is for tracking @jdesrosiers concerns that are not addressed by those PRs. Also paging @dlax, @Relequestual, and @awwright.

Everything below the following line is from @jdesrosiers in #290 (comment)
(the quote to which he is respoinding is from @awwright)

---

Can you elaborate some on how clients, as you would like to see, might act differently for different values of "method" or "form"?

Honestly, I'm a little perplexed by this question. First of all, method doesn't exist in this context. "method": "post" was replaced with "form": true and "method": "get" was removed in favor of hrefSchema. So there are only two possible values. Understanding the difference is no harder than understanding HTML. How does a client act differently when it encounters an <a> opposed to encountering a <form>.

1. "form": false (default)
   The LDO is analogous to an <a>. Really, it's closer to the Link header, but that's not really the point. A client can do anything with that link that it has enough information to do. A web browser doesn't have enough information to do anything but retrieve it. An LDO with "form": false has more information than an <a> allowing a client to do more things with it based on target hints, but it is basically the same thing.

2. "form": true
   The LDO is analogous to a <form>. A <form> is a construct for sending data to a resource. A client can proceed taking that assumption into account. For example a client knows it is going to need user input and should prompt the user for that data. This is different from a Link where you would have to first select which method you want to use, then the client determines if it needs to prompt the user for input.

It's pretty obvious how a web browser behaves differently when it encounters an <a> versus when it encounters a <form>. It should be no less obvious for a hyper-schema client encountering "form": false versus "form": true.

[handrews]

I've been thinking more about this and how to explain my view that "form" is unnecessary, and that the presence or absence of "submissionSchema" conveys all the necessary information.

HTML is designed to allow browsers to present unambiguous interfaces to users based on hypermedia.

JSON Hyper-Schema (JHS) is designed to inform an automated agent about the possible uses of a hypermedia representation.

The key concept for my philosophical view here are "unambiguous" and "possibilities". These words capture the very different design imperatives behind HTML and JHS.

As a human using an HTML document via a web browser, there are only two things that you can do:

• Navigate from the current document to another statically identified document
• Submit data for processing

Behind the scenes, HTML has two ways to handle submitted data (dynamically compute the URI to which to navigate, or submit the data to a statically identified resource for processing, which may involve navigating to a new resulting document or just resetting the current view). The only way that the distinction is visible to the user is that the browser will not allow re-submitting data to a statically identified resource for processing without an explicit confirmation from the user.

Within the scope of HTML, you cannot do anything else: there are exactly two user interface possibilities, where one has two underlying mechanisms which are largely indistinguishable to the end user. While HTML's script and link tags are also hypermedia, they are invisible to the end user and therefore not relevant to the comparisons in @jdesrosiers's comment above.

It doesn't make sense for HTML to have a way to indicate multiple possible uses of a target resource, because there is no universally sensible UI for such a thing, and the entire purpose of HTML is to construct a UI out of hypermedia. So in the relatively rare case where you want the same linked resource to be used in multiple ways, you connect to it multiple times, in each way that will produce the correct UI for that use.

You also do not have generic link relation types for anchors and forms. They mean what they mean, and any additional meaning is provided by natural language text that is significant to the end user but the contents of which are not examined or used by the browser (just displayed).

Web browsers are both generic hypermedia agents and end-user applications, and HTML's design reflects that combined use case.

---

On the other hand, there are no inherent limits to how an automated agent may interact with a machine-comprehendible resource, such as one represented by a JSON Hyper-Schema + a JSON instance. There is only one link serialization syntax, which is the LDO. The set of possible semantics is infinite, and conveyed by the link relation type. No generic agent can possibly know all relation types in advance.

This makes for a two-layer approach to consuming Hyper-Schema: A generic agent (which is concerned with recognizing what a link could do, but does not have the knowledge of infinitely many relation types to handle each specifically), and an application (which knows the link relation types relevant to its purpose, and can decide based on that what operations are actually available, and knows whether and how to present those available operations to the end user, if any end user even exists).

The JSON Hyper-Schema specification is concerned with the generic agent part. It needs to be clear on everything the generic agent needs to know, including indicating validation rules for various kinds of input data.

However, JSON Hyper-Schema is not concerned with the application part. Designing a set of hypermedia resources in a way that an application can consume them is about designing link relation types, and possibly a further constrained media type (either imposing additional semantics on the instance through a custom application/foo+json media type, or constraining Hyper-Schema to a subset of its full features by declaring a more restrictive meta-schema).

---

Fundamentally, in my view the form vs anchor distinction is not relevant to the core JSON Hyper-Schema specification. Anyone could easily define a subset of JSON Hyper-Schema that only supports HTML browser presentation behavior. Or they could define link relations equivalent to HTML's anchor and form elements. But this is not a restriction that should be imposed on Hyper-Schema in general.

There are endless possible application use cases that do not map onto the anchor/form paradigm, and JSON Hyper-Schema is responsible for enabling the whole range of possibilities.

[Anthropic]

@handrews I don't like the name being form, to me that implies ui, post: true/false seems more appropriate to me, ignoring your submissionSchema suggestion if the original suggestion were considered I'd rather see a different name proposed than form.

[dlax]

@Anthropic I also agree that the name form is not appropriate. Also considering @jdesrosiers's explanation #290 (comment):

Try to view this from a different perspective. You have an LDO with a submissionSchema. The question isn't how do you know you can use it to send data to a resource (POST), the question is how do you know that you can't retrieve it (GET). @awwright wrote up a list of 4 or 5 ways an LDO can be used. That makes it sound versatile, but it is also ambiguous. Say I have an LDO that only supports sending data to an executable resource. I don't want to signal to a user-agent that there are 4 or 5 things you can do with this LDO when there is really only one. Using form allows me to signal that this LDO has one use only: sending data to an executable resource.

In my understanding this kind of restriction would be handled by a Allow: POST header in HTTP. Do we need a similar indication about this in LDO? Can't this be carried by relation name rel semantics?

[handrews]

Can't this be carried by relation name rel semantics?

In my opinion, yes, it can and should, but in cases where it's insufficient...

In my understanding this kind of restriction would be handled by a Allow: POST header in HTTP. Do we need a similar indication about this in LDO?

That's issue #73, which is my preferred solution because it's flexible unlike importing some irrelevant convention from HTML.

[handrews]

@handrews I don't like the name being form,

@Anthropic: It's not my proposal.

[handrews]

@jdesrosiers (from #290):

The web is the reference implementation for REST.

The web as a whole? Yes. HTML? No. HTML is hypermedia format designed for interactive human use, with hypermedia control semantics that are only useful in a human context. HTML is also limited to a smalls subset of HTTP that is completely inadequate for machine-oriented hypermedia, so attempting to apply its semantics in a more fully-functional system doesn't make any sense.

it would be silly to ignore it's lessons or assume they don't apply to our situation.

I'm not assuming anything. The use case of HTML and JSON Hyper-Schema are objectively extremely different.

If you want to draw analogies to the web and call that the basis of how REST should work, fine. But you have to accept the whole web, not just HTML. And that means the JavaScript that you dismiss as "hacks".

[jdesrosiers]

[We] could define link relations equivalent to HTML's anchor and form elements.

@handrews, I disagree that there is a significant difference between designing for user interaction vs machine interaction. But, I'm going to focus my response on the above statement because I believe this has the potential to move us forward. Defining link relations to give more semantic meaning to links seems like a reasonable way forward. We could define a "rel": "form" instead of "form": true and it would accomplish the same goal. HTML went the route of defining different structures for different things, but technically <script src="foo.js"> could have been defined as <link rel="script" href="foo.js">.

I don't like the name being form

@Anthropic I never thought it was a good name either. It was clear that it was necessary to have something other than method so I proposed the best name I could think of off the top of my head. I disagree that the word form implies UI, but I certainly see how people can see it that way. It's confusing at best. I am certainly open to suggestions for a better name, but you'll have to forgive me for continuing to use it until we have a better alternative.

In my understanding this kind of restriction would be handled by a Allow: POST header in HTTP.

@dlax That is correct, but it's not a good solution for several reasons. First of all an additional uncacheable OPTIONS request is necessary to determine the servers capabilities. This is at best impractical in the real world. We can address that problem by adding to Hyper-Schema an allow keyword that allows us to skip that extra call, but that is less than ideal as well. If we have to fall back on knowing what methods the server supports, it indicates that something is missing in our hypermedia system. That leads us too ...

Can't this be carried by relation name rel semantics?

Yes. But, again, there's a problem. Very few registered relations are actually usable in a generic way, so people end up having to build up their own application specific relations. This means that people have to write their own client that knows how to interpret those application specific relations. Imagine that every time you wanted to use a new web page, your browser would have to be extended to understand that application's custom relations.

That is essentially the bigger picture that I am trying to get to starting with form. One of the reasons the web has been so successful is because it has a simple and generic set of hypermedia constructs (such as <a>, <form>, etc) that any application can use to unambiguously describe itself. That way a generic web browser can run any application. Which bring us to ...

this is not a restriction that should be imposed on Hyper-Schema in general.

If these generic hypermedia constructs can be achieved solely by using a small set of standard relations, then I half agree with you ( @handrews ). That would mean that it is possible to use JSON Hyper-Schema as defined as a building block for a complete HATEOAS system. The part of me that runs around the internet screaming "DECOUPLE ALL THE THINGS!!!" thinks this is a great idea. However, I don't think JSON Hyper-Schema can truly achieve HATEOAS without these standard hypermedia constructs. Those constructs can be in a different document, but it is absolutely essential for it to exist.

[handrews]

We could define a "rel": "form" instead of "form": true and it would accomplish the same goal. HTML went the route of defining different structures for different things, but technically <script src="foo.js"> could have been defined as .

Yes, this is the correct approach. And if that is what you need, it should be done as its own RFC because it is orthogonal to JSON Hyper-Schema. As with "collection" and "item", we may want to make a note of how other relation types are to be interpreted in the context of Hyper-Schema, and while I would not use "form" myself I would have no objection to documenting its usage in the context of application/schema+json.

I would be 100% in support of any path forward that involves defining appropriate link relations to convey the semantics you need.

Very few registered relations are actually usable in a generic way

That is not the concern of JSON Hyper-Schema, except to address very generic relations such as "collection" and "item" and define how they can be used within JSON Hyper-Schema. The lack of suitable link relations means that we need to be more focused on either registering more standard link relations to cover the needed use cases, or focusing our API design on defining correct and usable custom link relations. That is by far the most important part of designing a machine-oriented hypermedia system.

That would mean that it is possible to use JSON Hyper-Schema as defined as a building block for a complete HATEOAS system.

Yes, that is the point! JSON Hyper-Schema is a media type. It is one building block in a hypermedia system. It is absolutely not responsible for taking on all roles in such a system.

I don't think JSON Hyper-Schema can truly achieve HATEOAS without these standard hypermedia constructs.

I don't know what "achieve HATEOAS" means here. JSON Hyper-Schema is responsible for providing a functional mechanism for indicating the existence of links with respect to a JSON instance document, and allowing other standard hypermedia constructs such as link relation types to play their designated role in the system. It is not responsible for ensuring that all of the additional hypermedia concepts are implemented.

It is extremely important to keep the purpose of JSON Hyper-Schema, which is a media type, separate from the purposes of other components such as link relation types, URI schemes, and protocols. Picking a favorite set of semantics that would otherwise be indicated by a relation type and hardwiring it into JSON Hyper-Schema through keywords warps the general-purpose nature of the media type and preemptively limits the design choices that should be available by sending people down a specific path instead of encouraging the choice of the correct path for a given situation.

Those constructs can be in a different document, but it is absolutely essential for it to exist.

Certainly, but as their own projects and their own RFCs. As noted, it's fine for JSON Hyper-Schema to refer to other standards (HTTP, collection/item, etc.) that are not media types and provide guidance on their usage in the context of the application/schema+json media type. That can and should bridge the gaps between the often impossibly vague general definitions and actual usage.

But JSON Hyper-Schema does not define the relation types. It simply defines how it uses them.

[jdesrosiers]

I have always thought of JSON Hyper-Schema as a general purpose hypermedia system because that is the way I have always used it. If we want to reduce that scope to simply a component of such a system, then I'm OK with that as long as we don't stop there. However, I would feel better about the distinction if JSON Hyper-Schema was only LDO.

That is not the concern of JSON Hyper-Schema, except to address very generic relations such as "collection" and "item" and define how they can be used within JSON Hyper-Schema.

I don't see any distinction between using the spec to define new relations and using the spec to add semantics to an existing relation. If defining new relations doesn't belong in the spec, then neither does extending them.

The lack of suitable link relations means that we need to be more focused on either registering more standard link relations to cover the needed use cases, or focusing our API design on defining correct and usable custom link relations.

I think we need to go further. Not only do we need to create a set of generic relations, we also need to identify a set of generic relations that all Hyper-Schema clients should understand. My APIs should be able to use the same client as your APIs and neither of our APIs should rely on custom relations because the generic client only understands the set of relations the spec says it must understand. Defining a custom relation that needs to be anything more than a label would be counter productive because it would mean you couldn't use the generic client. Later on we might be able to extend functionality with some kind of code-on-demand solution, but we should cover the basics first.

[handrews]

I don't see any distinction between using the spec to define new relations and using the spec to add semantics to an existing relation. If defining new relations doesn't belong in the spec, then neither does extending them.

So you just think it's irrelevant that RFC 6573 defines "collection" and "item" and gives as reference usages two media types that further define their usage? That is exactly how these things are supposed to work. Declaring that RFC 6573 is completely wrong in its approach is rather presumptuous.

Not only do we need to create a set of generic relations, we also need to identify a set of generic relations that all Hyper-Schema clients should understand.

That is far beyond the scope of JSON Hyper-Schema. There are many possible ways to structure a hypermedia system. If you would like to propose a link relation framework for a comprehensive system, that's fantastic and more power to you. I will cheer you on. But it is almost completely orthogonal to JSON Hyper-Schema, which is a media type. Only media type-specific usage would be part of Hyper-Schema itself (or other media types).

My APIs should be able to use the same client as your APIs

Yes, at the level of understanding generic, standardized media types, link relation types, protocols, URI schemes, etc.

neither of our APIs should rely on custom relations because the generic client only understands the set of relations the spec says it must understand.

No, there is no possible universal set of link relations that covers all scenarios. HATEOAS is not magic, and there are different places where the knowledge of how to use aspects of a hypermedia system should be encoded, and not all of them are generic.

There are link relations, such as "collection" and "item", that represent such generic interaction patterns that they are worth standardizing (in terms of their own semantics and in terms of usage within a media type) and teaching to generic clients.

But for the Cloudflare API, for instance, I will probably want a link relation type something like tag:cloudflare.com,2017:page-rules to indicate the set of page rules that are applicable to the context resource. What are page rules? It doesn't matter. They are a Cloudflare-specific concept, even if someone else has a concept that they call "page rules". To work with them, we define a Cloudflare-specific link relation type. There is nothing useful for a generic client to know about such a concept.

Extended link relations, as discussed in RFC 5988, exist for a reason. Generic clients can't and shouldn't know every possible domain-specific set of link relations. That is the responsibility of domain-specific application code, which uses the generic client, including its understanding of generic link relation types. But the understanding and usage of the domain-specific extension relation types is encoded in the domain-specific application.

---

Fundamentally, we need to clearly define what JSON Hyper-Schema is and put it in some sort of project charter. JSON Hyper-Schema is not an API description format like OpenAPI or RAML. And it is not a mega-project encompassing everything one could possibly need for the perfect hypermedia API. It is a media type. It does one thing, and it needs to do that one thing well and leave everything else to the separately defined components that provide those things.

[jdesrosiers]

So you just think it's irrelevant that RFC 6573 defines "collection" and "item" and gives as reference usages two media types that further define their usage?

That's not what I'm saying at all. I don't think it's wrong for a media type to further define a relation. I'm saying that it seems inconsistent for you to say that it isn't OK for a media type to define a relation, but it is OK for it to further define a relation. What's the difference? Why is one "far beyond the scope of JSON Hyper-Schema" while the other is perfectly natural?

There is no possible universal set of link relations that covers all scenarios.

I don't buy that. The web is a very successful counter example. I admit that I don't know yet exactly what it will look like, but the web is proof that a general purpose set of hypermedia controls is possible and can be successful. I'm not suggesting anything that has never been done.

There are link relations, such as "collection" and "item", that represent such generic interaction patterns that they are worth standardizing and teaching to generic clients.

That is exactly what I've been advocating. "collection" and "item" can be the first to be included. I want to identify / create others that define generic interaction patterns and include those as well. I want to have enough of these to be able to express almost anything without ever needing an allow hint. I'm considering execute or executable as a better name than form. It is a common and generic pattern to send data to an executable resource. I don't see any reason why that doesn't belong in the same category as "collection" and "item".

Fundamentally, we need to clearly define what JSON Hyper-Schema is and put it in some sort of project charter.

I totally agree.

[handrews]

I don't buy that. The web is a very successful counter example. I admit that I don't know yet exactly what it will look like, but the web is proof that a general purpose set of hypermedia controls is possible and can be successful.

No, you're talking about the human web, and that works because all link semantics are conveyed in natural language and interpreted by human beings (aside from stylesheets and scripts, which are machine-oriented rather than human-oriented).

HTML's anchor and form elements don't convey real semantics, just mechanisms. Which is why I am so opposed to encoding them in Hyper-Schema. Yes, there's some fundamental semantic information in the dichotomy, but the true equivalent of link relation types in HTML is the text between <a> and </a> or on the various form <input /> elements, or even just near the links or forms.

The human web works with a truly infinite number of semantic relation types, many if not most of which are imprecise because natural language. This makes it a perfect example of why you cannot define a set of link relation types that will encompass everything about an API. Using an API is a lot more than just deciding whether to use PUT, PATCH, or POST. Those are mechanics, and largely uninteresting from an API design perspective. The interesting thing is the meaning. The why of using a link, with or without data. Not the how.

I'm saying that it seems inconsistent for you to say that it isn't OK for a media type to define a relation, but it is OK for it to further define a relation. What's the difference? Why is one "far beyond the scope of JSON Hyper-Schema" while the other is perfectly natural?

It's more that a registered link relation should not be media-type specific. Some older ones are, because the whole concept of link relation types developed slowly, and a lot of it came from Atom. But the concept is solid now and has been for some time. So if we want to define a "schema" link relation with its first defined usage to be JSON Schema (not just Hyper-Schema for that example), that would be reasonable.

But most other things should work across media types and are therefore better standardized independent from any one media type. If you have a good relation type, why tie it to a media type? Get it standardized ASAP and then use it anywhere.

I'm considering execute or executable as a better name than form. It is a common and generic pattern to send data to an executable resource. I don't see any reason why that doesn't belong in the same category as "collection" and "item".

Because it's mechanics and not meaning.

[handrews]

@jdesrosiers I want to emphasize that I do agree that there are more link relations that can and should be defined to make hypermedia systems more interoperable. I just don't think that either those efforts or Hyper-Schema should be pinned to each other. For one thing, I suspect that you could get some link relation types to RFC and registered with IANA much faster than we'll finish Hyper-Schema.

Even if I disagree with some of your proposed relation types, that's really not important. If you can get other folks on board with an executable relation type as its own RFC, or as part of a package of hypermedia-system-enabling link relation types that make up their own RFC, that's great! I'm not going to get in the way of any effort of that sort.

[dlax]

@jdesrosiers

In my understanding this kind of restriction would be handled by a Allow: POST header in HTTP.

That is correct, but it's not a good solution for several reasons. First of all an additional uncacheable OPTIONS request is necessary to determine the servers capabilities. This is at best impractical in the real world.

It seems to me, most of the times, a client would already have performed a GET on the target of a relation so the Allow header would be available without any additional cost. In other cases, which I believe would not be the majority, can't we afford a HEAD on the target? Going further and considering non-trivial actions that'd be described by LDO, I even tend to believe the non-cacheable aspect is actually a feature the client should rely on (think of a state system for instance). This is mostly hypothetical thoughts (reflecting my personal experience) because it's not very clear to me what kind of actual actions you'd want to expose with this mechanism. Would you mind providing a concrete example that's not too domain specific?