Skip to content

Changed DocUrl for fragments - #61

Closed
raoashish10 wants to merge 5 commits into
HTTP-APIs:developfrom
raoashish10:link_format
Closed

raoashish10 wants to merge 5 commits into
HTTP-APIs:developfrom
raoashish10:link_format

Conversation

@raoashish10

Copy link
Copy Markdown

There is in reference to a PR in the hydrus repo

Checklist

  • My branch is up-to-date with upstream/develop branch.
  • Everything works and tested for Python 3.5.2 and above.

Description

This PR is added so that fragments of vocab can be identified. I have changed the DocUrl.
Old format of ids:
http//petstore.swagger.io/vocab#UserCollection
New format of ids:
http//petstore.swagger.io/vocab?resource=UserCollection

I also found that there were some errors with the EntryPoint id's so I tried fixing that using regex as you can see. The error was that the entrypoints were shown as http//petstore.swagger.io/vocab?resource=EntryPoint/UserCollection which I tried changing to http//petstore.swagger.io/vocab/EntryPoint/UserCollection. I am not entirely sure if this solves every problem but I did go through the doc returned and tried solving as many issues I could. Let me know if my solution needs to be changed.

Change logs

Changed

  • DocUrl was changed from:
    DocUrl.doc_url = "{}/{}#".format(urljoin(base_url, api_name), doc_name)
    to
    DocUrl.doc_url = "{}/{}?resource=".format(urljoin(base_url, api_name), doc_name)

@chrizandr

Copy link
Copy Markdown
Member

Could you generate an API Doc with this and show? I still don't think that the IDs of individual resources would be updated with the new format.

@raoashish10

Copy link
Copy Markdown
Author

Here is a sample of the generated API doc. I have removed the context because it remains the same, if you see the id's for the classes have been changed, I tried this with the hydrus hydra_doc_sample.py doc.

...

    "@id": "http://localhost:8080/serverapi/vocab",
    "@type": "ApiDocumentation",
    "description": "API Documentation for the server side system",
    "entrypoint": "http://localhost:8080/serverapi",
    "possibleStatus": [],
    "supportedClass": [
        {
            "@id": "http://localhost:8080/serverapi/vocab?resource=Drone",
            "@type": "hydra:Class",
            "description": "Class for a drone",
            "supportedOperation": [
                {
                    "@type": "http://schema.org/UpdateAction",
                    "expects": "http://localhost:8080/serverapi/vocab?resource=Drone",
                    "expectsHeader": [],
                    "method": "POST",
                    "possibleStatus": [
                        {
                            "@context": "http://www.w3.org/ns/hydra/context.jsonld",
                            "@type": "Status",
                            "description": "Drone updated",
                            "statusCode": 200,
                            "title": ""
                        }
                    ],
                    "returns": "null",
                    "returnsHeader": [],
                    "title": "SubmitDrone"
                },
                {
                    "@type": "http://schema.org/AddAction",
                    "expects": "http://localhost:8080/serverapi/vocab?resource=Drone",
                    "expectsHeader": [],
                    "method": "PUT",
                    "possibleStatus": [
                        {
                            "@context": "http://www.w3.org/ns/hydra/context.jsonld",
                            "@type": "Status",
                            "description": "Drone added",
                            "statusCode": 200,
                            "title": ""
                        }
                    ],
                    "returns": "null",
                    "returnsHeader": [],
                    "title": "CreateDrone"
                },
                {
                    "@type": "http://schema.org/FindAction",
                    "expects": "null",
                    "expectsHeader": [],
                    "method": "GET",
                    "possibleStatus": [
                        {
                            "@context": "http://www.w3.org/ns/hydra/context.jsonld",
                            "@type": "Status",
                            "description": "Drone not found",
                            "statusCode": 404,
                            "title": ""
                        },
                        {
                            "@context": "http://www.w3.org/ns/hydra/context.jsonld",
                            "@type": "Status",
                            "description": "Drone Returned",
                            "statusCode": 200,
                            "title": ""
                        }
                    ],
                    "returns": "http://localhost:8080/serverapi/vocab?resource=Drone",
                    "returnsHeader": [],
                    "title": "GetDrone"
                }
            ],
            "supportedProperty": [
                {
                    "@type": "SupportedProperty",
                    "property": "http://localhost:8080/serverapi/vocab?resource=State",
...

@raoashish10

Copy link
Copy Markdown
Author

Here is another example. I am posting it as a different comment for clarity:

{
            "@id": "http://localhost:8080/serverapi/vocab?resource=DatastreamCollection",
            "@type": "Collection",
            "description": "A collection of datastream",
            "manages": {
                "object": "http://localhost:8080/serverapi/vocab?resource=Datastream",
                "property": "http://localhost:8080/serverapi/vocab?resource=type"
            },
            "subClassOf": "http://www.w3.org/ns/hydra/core#Collection",
            "supportedOperation": [
                {
                    "@id": "_:DatastreamCollection_retrieve",
                    "@type": "http://schema.org/FindAction",
                    "description": "Retrieves all the members of DatastreamCollection",
                    "expects": null,
                    "expectsHeader": [],
                    "method": "GET",
                    "possibleStatus": [],
                    "returns": "http://localhost:8080/serverapi/vocab?resource=Datastream",
                    "returnsHeader": []
                },

Please let me know if there is anything wrong or missing. Also I had a doubt regarding what should be returned when calling the URI http://localhost:8080/serverapi/contexts/EntryPoint.jsonld because the links that were returned initially were of the form "http://localhost:8080/serverapi/Area" which do not lead to any result now. So if that needs to be changed I can do it in an another PR or maybe the same one

@chrizandr

Copy link
Copy Markdown
Member

The Entrypoint object should have contexts of its own. That needs to be returned. You can add it to the same PR.

@raoashish10

Copy link
Copy Markdown
Author

@chrizandr I want to confirm something. Is something like this expected on clicking the links in http://localhost:8080/serverapi/contexts/EntryPoint.jsonld (for example, I clicked - http://localhost:8080/serverapi/Area ) :

{
    "@context": {
        "Area": "http://localhost:8080/serverapi/vocab?resource=Area",
        "BottomRight": "http://schema.org/geo",
        "TopLeft": "http://schema.org/geo",
        "hydra": "http://www.w3.org/ns/hydra/core#",
        "members": "http://www.w3.org/ns/hydra/core#member",
        "object": "http://schema.org/object"
    }
}

@chrizandr

chrizandr commented Jan 16, 2021 •

Copy link
Copy Markdown
Member

No, that is wrong.

I don't know why that is being generated. You can see an example of what the context for EntryPoint looks like here: http://www.markus-lanthaler.com/hydra/event-api/contexts/EntryPoint.jsonld

This is already there in HydraEntrypoin.context, see

self.context = Context(

The dictionary in context variable needs to be returned.

It might be an issue with the getcontext function here:
https://github.com/HTTP-APIs/hydrus/blob/e15d18e9f7b535814ef3a2f981be443896206adb/hydrus/helpers.py#L444

Maybe we should open a separate issue for this in hydrus? I'll approve this for now.

@raoashish10

Copy link
Copy Markdown
Author

@chrizandr I tried to replicate the example provided by you and I saw that in Markus Lanthaler's Entrypoint.jsonld, the names of the classes are keys and the id's referred to the information about the class in the vocab. So I made a few changes and this is what you get on accessing the link http://localhost:8080/serverapi/contexts/EntryPoint.jsonld :

{
    "@context": {
        "Area": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=Area",
            "@type": "@id"
        },
        "Command": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=Command",
            "@type": "@id"
        },
        "CommandCollection": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=CommandCollection",
            "@type": "@id"
        },
        "Datastream": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=Datastream",
            "@type": "@id"
        },
        "DatastreamCollection": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=DatastreamCollection",
            "@type": "@id"
        },
        "Drone": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=Drone",
            "@type": "@id"
        },
        "DroneCollection": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=DroneCollection",
            "@type": "@id"
        },
        "EntryPoint": "http://localhost:8080/serverapi/vocab?resource=EntryPoint",
        "LogEntry": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=LogEntry",
            "@type": "@id"
        },
        "LogEntryCollection": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=LogEntryCollection",
            "@type": "@id"
        },
        "Message": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=Message",
            "@type": "@id"
        },
        "MessageCollection": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=MessageCollection",
            "@type": "@id"
        },
        "State": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=State",
            "@type": "@id"
        },
        "StateCollection": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=StateCollection",
            "@type": "@id"
        }
    }
}

Let me know if this makes sense and if it does then I will add the changes to the PR

@farazkhanfk7

farazkhanfk7 commented Jan 22, 2021 •

Copy link
Copy Markdown
Member

I have some doubts related to this. Please correct me if I am getting it wrong.

  • Currently the '@id' for resources at http://localhost:8080/serverapi/contexts/EntryPoint.jsonld looks like this http://localhost:8080/serverapi/vocab#EntryPoint/Area which is changed to http://localhost:8080/serverapi/vocab?resource=EntryPoint/Area after the doc_url is changed in this PR. Which I know doesn't make a lot of sense as it returns the same vocab that was previously being generated at serverapi/vocab#. This is why regex was used but isn't the EntryPoint keyword supposed to be here to indicate that it is actually the id generated from EntryPointClass and EntryPointCollection classes ?
  • Check get() function in HydraEntryPoint class.
    for item in self.entrypoint.supportedProperty:
    uri = item.id_
    if item.generate() in self.collections:
    collection_returned = item.generate()
    collection_id = uri.replace(
    "{}EntryPoint".format(DocUrl.doc_url), "{}{}".format(self.url,self.api))

    I think is used to get the EntryPoint object. In line 440 it uses replace() function to replace the "{}EntryPoint" with "{}{}".format(self.url,self.api)".
  • id for EntryPointClass/EntryPointCollection can be changed even without using regex but if replace() function won't find "EntryPoint" to replace it with something, it will return the same id, which temporarily solves the issue with /contexts/EntryPoint.jsonld but will also change the output at http://localhost:8080/serverapi/ to this :
{
"@context": "http://localhost:8080/serverapi/contexts/EntryPoint.jsonld",
"@id": "http://localhost:8080/serverapi",
"@type": "EntryPoint",
"Area": "http://localhost:8080/serverapi/vocab?resource=Area",
"Command": "http://localhost:8080/serverapi/vocab?resource=Command",
......
}

Currently, these URIs are like this http://localhost:8080/serverapi/Command and gives a message saying "The method is not allowed". I'm not sure if it is supposed to be like this, but shouldn't that also be changed to vocab?resource=Command so that it will only return Command object with context ?

@raoashish10

Copy link
Copy Markdown
Author

@farazkhanfk7 What you have pointed out is right. That is why I have changed the code locally so that temporarily the links at entrypoint.jsonld return the vocabs for the classes and the collections (vocab?resource=Command) and I am awaiting approval on that.

I agree that Entrypoint/Class or Entrypoint/Collection is necessary but for now I have just pointed to the vocabs. I might make an another PR just to solve the issues with the Entrypoints because previously no such link on dereferencing gave the intended output so I am not exactly sure how the output should be but by the example mentioned by Chris, I thought the closest thing would be to call the vocab for the resource.

I used regex because for some reason I couldn't get the result by using replace() which is very odd but there was some issue with the usage maybe.

@chrizandr

chrizandr commented Jan 29, 2021 •

Copy link
Copy Markdown
Member

I think the example you pointed out is wrong. The URL http://localhost:8080/serverapi/contexts/EntryPoint.jsonld should return those resources that have an endpoint that is present in the EntryPoint object.

In the example you provided you are pointing it to the definition of the class in the API doc itself, we instead need it to point to the entry in the EnytrPoiny class. Think of EntryPoint as a class, where the properties of the class are the endpoints that are available in the API. Not all classes in the API documentation will have endpoints, please see the context that is created for the entrypoint object here here

self.context = Context(

The URIs returned here need to be modified with the new format. Also /api/vocab?resource=EntryPoint/classname, should point to the supportedProperty in the Entrypoint class definition instead.

@chrizandr chrizandr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comment about the Entrypoint context changes.

@raoashish10

raoashish10 commented Jan 29, 2021 •

Copy link
Copy Markdown
Author

Thank you for the explanation @chrizandr. I think I have a better understanding of the Entrypoints now. I hope these were the changes required. Please let me know if I am wrong.

On accessing http://localhost:8080/serverapi/contexts/EntryPoint.jsonld:

@context": {
        "Area": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/Area",
            "@type": "@id"
        },
        "Command": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/Command",
            "@type": "@id"
        },
        "CommandCollection": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/CommandCollection",
            "@type": "@id"
        },
...

On accessing http://localhost:8080/serverapi/vocab?resource=EntryPoint/Area:

  "Area": {
        "hydra:description": "The Area Class",
        "hydra:title": "area",
        "property": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/Area",
            "@type": "hydra:Link",
            "description": "Class for Area of Interest of the server",
            "domain": "http://localhost:8080/serverapi/vocab?resource=EntryPoint",
            "label": "Area",
            "range": "http://localhost:8080/serverapi/vocab?resource=Area",
            "supportedOperation": [
                {
...

On accessing http://localhost:8080/serverapi/vocab?resource=EntryPoint/StateCollection:


    "StateCollection": {
        "hydra:description": "The StateCollection collection",
        "hydra:title": "statecollection",
        "property": {
            "@id": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/StateCollection",
            "@type": "hydra:Link",
            "description": "The StateCollection collection",
            "domain": "http://localhost:8080/serverapi/vocab?resource=EntryPoint",
            "label": "StateCollection",
            "manages": {
...

I apologise for taking time with this PR as I have been having some trouble understanding certain concepts and I appreciate your patience. I had to make some changes in the hydrus repo as well which I have now done.

@raoashish10
raoashish10 requested a review from chrizandr February 8, 2021 13:15
@raoashish10
raoashish10 changed the base branch from master to develop February 8, 2021 13:28
@Mec-iS

Mec-iS commented Mar 13, 2021 •

Copy link
Copy Markdown
Contributor

Thanks for the great piece of work to both.

@raoashish10 please remember that we cannot merge without having proper testing in place. We need to have some test cases in place for the new feature and be sure that the whole test unit runs smoothly.

@raoashish10

Copy link
Copy Markdown
Author

@Mec-iS Sure I can add some test cases in this repo as well. Though I had already added test cases in the hydrus repo for checking the format of the links but if its necessary to add tests in this repo as well then I will do that. Also I am sorry for a late reply

@Mec-iS

Mec-iS commented Mar 25, 2021

Copy link
Copy Markdown
Contributor

tests for anything should be with the target code. we can expect another repo to test the code for this repo.

@farazkhanfk7

Copy link
Copy Markdown
Member

@Mec-iS This is what I was thinking, currently the tests for doc_writer are failing. We might have to update the tests. How about creating a CI pipeline on Travis for core repo as well ?

@Mec-iS

Mec-iS commented Mar 25, 2021

Copy link
Copy Markdown
Contributor

successful tests are a pre-condition not dependant by the fact that there is or not a CI.

@farazkhanfk7

farazkhanfk7 commented Mar 26, 2021 •

Copy link
Copy Markdown
Member

@raoashish10 This looks great. I just had one small doubt. In HydraEntryPoint's get function a line was changed from uri.replace( "{}EntryPoint".format(DocUrl.doc_url), "{}{}".format(self.url,self.api)) to object_[item.name] = uri. But this would change the output at http://127.0.0.1:8080/serverapi/ to something like this :

{
      "@context": "http://localhost:8080/serverapi/contexts/EntryPoint.jsonld",
      "@id": "http://localhost:8080/serverapi",
      "@type": "EntryPoint",
      "Area": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/Area",
      "Command": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/Command",
      "Datastream": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/Datastream",
      "Drone": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/Drone",
      "LogEntry": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/LogEntry",
      "Message": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/Message",
      "State": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/State",
      "collections":  [
            {
                  "@id": "http://localhost:8080/serverapi/vocab?resource=EntryPoint/DroneCollection",
                  "@type": "Collection",
                  "manages":  {
                      "object": "http://localhost:8080/serverapi/vocab?resource=Drone",
                      "property": "http://localhost:8080/serverapi/vocab?resource=type"
                  },

The format of new links (vocab?resource=Drone) and output at http://localhost:8080/serverapi/contexts/EntryPoint.jsonld (vocab?resource=EntryPoint/Drone) looks good.
Edit :
But (at /serverapi/) this didn't change the uri for collection and classes to http://localhost:8080/serverapi/DroneCollection(This is where we can make GET request to any collection endpoint to get hydra:totalItems and search members). Can you please confirm this.

@raoashish10

Copy link
Copy Markdown
Author

@Mec-iS I have added a test which checks for the new format of links in id's for classes as well as collections. Please let me know if anything is missing or needed
As for @farazkhanfk7's question I have tested the new link formats with the agent as well and we had a discussion about it.

@sameshl

sameshl commented Mar 30, 2021

Copy link
Copy Markdown
Member

This LGTM 👍 but I think there are some pending changes requested by @chrizandr

@raoashish10

Copy link
Copy Markdown
Author

I think I have addressed the changes required in commit 28e3948. Let me know if anything is missing @sameshl

@farazkhanfk7

Copy link
Copy Markdown
Member

@raoashish10 I tested this locally by installing link_format branch of hydra_python_core and fragments branch in your hydrus repo in the same virtual env and most of the tests in hydrus failed. I apologize for confusing this a bit and not being able to raise my doubt clearly but I believe this is something that needs to be fixed. The new changes in hydra_python_core work fine separately but would result in the failure of tests in hydrus when both repositories are tested concurrently. I'll try to put this clearly this time.

Steps to reproduce :

  1. Create a venv with virtualenvwrapper and install both repos in the same environment.

  2. Install your link_format branch of hydra_python_core.

  3. Install fragments branch of hydrus.

  • You can run the server with hydrus serve and everything will work fine in both browser and agent.
  • But the problem arrives when we'll run the tests in hydrus with pytest -v

I believe this is related to the same question I raised before. Please cross-check and let me know if I am wrong.

else:
self.id_ = "{}EntryPoint/{}".format(DocUrl.doc_url, quote(self.name, safe=''))

self.id_ = "{}EntryPoint/{}".format(DocUrl.doc_url, quote(collection.path, safe=''))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a minor typo. If collection.path isn't available , self.name should be used.

@farazkhanfk7

farazkhanfk7 commented May 24, 2021 •

Copy link
Copy Markdown
Member

@Mec-iS Now that we have merged #518 in hydrus. I think we should also consider changing the format of URIs through this PR. This still needs some changes as it causes tests in hydrus to fail.

@Mec-iS
Mec-iS dismissed chrizandr’s stale review May 24, 2021 16:04

changes are good

@Mec-iS
Mec-iS requested a review from farazkhanfk7 May 24, 2021 16:10
@Mec-iS

Mec-iS commented May 24, 2021 •

Copy link
Copy Markdown
Contributor

@farazkhanfk7 please approve the PR

@farazkhanfk7

Copy link
Copy Markdown
Member

@Mec-iS The last time I checked the tests in hydrus were failing due to some changes. I'll review it again ASAP.

@Mec-iS

Mec-iS commented May 24, 2021

Copy link
Copy Markdown
Contributor

moved to #79 for additions

@Mec-iS Mec-iS closed this May 24, 2021
@Mec-iS Mec-iS mentioned this pull request May 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants