LocalGov Drupal implementation

The technical counterpart to Integrating Open Referral UK with LocalGov Drupal In the spirit of LocalGov Drupal developing in the open, I’ll document, and maybe short-circuit some of the hard work by getting feedback on the questions as we go.

1 Like

/services

Starting with the services list end point, unfiltered

URL

https://localgov.lndo.site/openreferral/v1/d919cb19-e423-4087-8155-f1bcf2b6a89a/services

In LocalGov Drupal content editors can create directories. So there are multiple, some maybe more relevant to the Open Referral API, some less: Parking and Car Parks; Libraries; Family Information Directory; etc. How these are grouped will depend on the site. It is probably practical that we could also make a single end point for all directories (in addition), but for the moment I’ve made one per directory (identified by its UUID).

Json pager

{
  "totalElements": "2",
  "totalPages": 1,
  "number": 1,
  "size": 50,
  "first": true,
  "last": true,
  "content": [...]
}

I don’t think I followed any documentation for this just output from examples.

Json Service

First up I thought I’d read somewhere about a JsonAPI style include={child_entity} with Open Referral. But I didn’t notice it again. The examples so far seemed include by default a level down (including connector ‘entity’ - by which I mean ‘service_at_locations’ and ‘locations’ and even in that case ‘physical_address’ under it).

   {
      "id": "eb77a385-e9db-416f-87df-00478d1413b6",
      "organization": {
        "id": "eb77a385-e9db-416f-87df-00478d1413b6",
        "name": "Second venue"
      },

So first up we’re cheating at the moment. There’s no concept of ‘Service’ or ‘Organisation’ so until there’s a decision here the existing ‘Directory entry’ is both, just formatted differently. My original thought had been just to make a simple Organisation entity that could be developed or not, but could be generated easily on the fly by the content editor. These requirements still stand, the easy select or generate, but I think a discusion about if Organisations themselves might also want to be first-class Directory entries needs be had first before deciding the (internal) entity implementation. So work-around at the moment. They’re both.

      "name": "Second venue",
      "description": "Body text from the entry",
      "email": null,
      "url": "https://example.com",

Pretty straightforward mapping of internal field names to the specification fields. This is hard coded at the moment. Down the line this wants to be made more flexible. Different councils implementing LocalGov Drupal Directories already add their own fields to the entries. I’d wondered about using the RDF (and RDF UI) for storing and editing these mappings, it’s already used for mapping schema.org to fields, but the non-1:1 entity:entity relationship makes me question the practicality of this. So hard coded for now. An example coming up…

      "service_at_locations": [
        {
          "id": "service:eb77a385-e9db-416f-87df-00478d1413b6:location:9bbce6b7-7583-47c3-b876-b135e5c8b8d5",
          "location": {
            "id": "9bbce6b7-7583-47c3-b876-b135e5c8b8d5",
            "name": " Katharine Street\nLondon\nCR0 1NX\nUnited Kingdom",
            "latitude": 51.372348,
            "longitude": -0.099919,
            "physical_addresses": [
              {
                "id": "address:2",
                "location_id": "9bbce6b7-7583-47c3-b876-b135e5c8b8d5",
                "address_1": " Katharine Street",
                "city": "London",
                "state_province": null,
                "postal_code": "CR0 1NX",
                "country": "GB"
              }
            ]
          }
        }
      ],

The service_at_locations part is hard coded at the internal field level. I made up an ID comprised of the service itself and the location it is pointing to. The location is the serialization of the internal geo entity that was referenced.

Taxonomy was also pretty interesting. At the moment there are no formal vocabularies imported, but two different ways of adding term to Directory Entries. One is content creator driven, dynamic on the fly creation, the other pre-configured. Obviously the latter would also work for importing an external vocabulary.

      "service_taxonomys": [
        {
          "id": "service:eb77a385-e9db-416f-87df-00478d1413b6:taxonomy:41e9e441-65fa-426c-87b3-d41616a264f8",
          "taxonomy": {
            "id": "siteid:41e9e441-65fa-426c-87b3-d41616a264f8",
            "name": "facet 1",
            "vocabulary": "facet_type_1"
          }
        },
        {
          "id": "service:eb77a385-e9db-416f-87df-00478d1413b6:taxonomy:8a8afd38-dc6e-47c7-a2ee-864ed0f899ff",
          "taxonomy": {
            "id": "siteid:8a8afd38-dc6e-47c7-a2ee-864ed0f899ff",
            "name": "facet 2:1",
            "vocabulary": "facet_type_2"
          }
        }
      ]
    },

From the OpenAPI/Swagger I’d gone for the serivice_taxonomys again with their own id and then the taxonomy entity itself within, just like location and service at location. I don’t think I found a real world example yet of this, so maybe it’s not correct.

The id is comprised of a {namespace/curie}:{id} which for all the vocabularies that aren’t externally defined would just be the namespace for the site itself. Again not to sure about this. This would also be on the /taxonomies/{id} URL. The tanonomies endpoint is probably the next to implement to look at how this is presented existing sites, and how it works with the above id structure.

@ekes great to see you’re progressing well.

Should https://localgov.lndo.site/openreferral/v1/d919cb19-e423-4087-8155-f1bcf2b6a89a/services be publicly accessible and working at present?

Nope *.lndo.site is the local docker container’s default urls. Hence cutting and pasting output, with my thoughts as I was working through it for now.

Plan is to continue, over the next couple of weeks - correcting anything people notice from the output - and adding /taxonomies and /organizations; with some subset of the filtering options. That I’ll get up on a testing site with a public url.

OK. I’ll ask a colleague to take a look. /vocabularies would be a good one to do as well to support people doing searches.

The API Query tool has some good examples and the (new) API dashboard routinely checks some live feeds. Of course you can use the validator when your feed is public.

The pager looks right.

GET methods returning lists of things like /services return properties (like the service name and many-to-ones like a service’s organization.

GET methods for individual items like /services/{id} should return fill details so all the one-to-many relationships such that you know full details of an item.

We added the include parameter for lists at Placecube’s request so that a list could include selected extra properties (e.g. service_at_locations) rather than you have to query the details of every item to get one extra property.

This all should be explained in the API Guidance page.

Here are my thoughts such as they are.

Regarding the Json Pager documentation. I did actually make some documentation on implementing the API for this but it wasn’t considered important enough to include it in the beta sprints, unfortunately.

This is correct these elements are child data and so don’t make much sense on thier own without the parent class context. As they won’t be shared with other entities. However it is possible to access locations directly, in addition to organisations, reviews, services, taxonomies and vocabularies.

I have made the suggestion before that all child data should be using unique GUIDs which helps when combining data. As the ids are strings it can support RDF references. However as this can store all types of services from all types of providers, like a group of astronomy societies. It wouldn’t be universally possible.

I am also not sure what combining the hierarchy of entities into a ID gives you as you already have that information in the child data.

Thanks

Dom

1 Like

As the ids are strings it can support RDF references.

The RDF aside was actually me suggesting (ab)using the internal Drupal RDF mapping system for the Open Referral entity types and field names. It’s an answer to the problem that site configurators can add any fields they like to the entity types, they can also add their own entity types as valid directory entries, and it is desirable to allow them also to say how these map to Open Referral output.

I am also not sure what combining the hierarchy of entities into a ID gives you as you already have that information in the child data.

The funky type:uuid:type:uuid IDs weren’t related anything other than just being a way of generating a guaranteed unique ID where it was required by OR but we don’t have an existing ID. Combining existing ones systematically guaranteed uniqueness in a way that it could be deconstructed without a look-up table back to where it was used should it ever be needed (I’m not expecting it to be needed).

For the record our entity types that are being mapped at the moment are planned as:
Multiple Directory Entry types (called things like page, venue - sites have created their own already like ‘family directory entry’ etc.) → Service
Organization (also optionally a directory entry type) → Organization
Vocabulary → Vocabulary
Taxonomy Term → Taxonomy
Location Address Type → Location (and Physical Address)
So entities of these types all have UUIDs already that can be used as IDs. They can also be looked up that way so having endpoints for them is relatively straightforward.

(While I’m rambling Locations are used in other fields elsewhere, so could also map in other contexts that service_at.
There are also Area location types which aren’t used for area covered on entries yet - but no doubt will be.
Reviews are a thing that will no doubt down the line be implemented as entities.
Having all the different directory entity types that map to the services will be the flexibility challenge).

/vocabularies would be a good one to do as well to support people doing searches

Makes sense. I’ll add that to the list.

We added the include parameter for lists at Placecube’s request … This all should be explained in the API Guidance page.

And that’s is where I saw it! And it’s also where I was clearly overlooking it after that!
I’m not sure it’s helping us at the moment anyway. If I was able to use the Drupal JsonAPI it would be perfect, as the serializers there already deal with that, but as it goes direct use wasn’t as neat as I hoped.
So for now I’ll keep implementing the depth of output much as I see it in most of the examples - if that makes sense?

Another issue I have thought about with having hierarchal identifiers is what happens if the hierarchy changes for some reason the identifiers will be different to the data unless the id is also updated but that could break other things.

Maybe I misread the ‘relational entity’. But I had in mind that:-

‘service’ { id => service_id_1 }
has
‘service_at_location’ => [ {id => service_id_1:location_id_1}, {id => service_id_1:location _id_2 }
where there location enties
‘location’ { id => location_id_1 } and { id => location_id_2 }

‘service’ { id => service_id_2 }
could have
‘service_at_location’ => [ {id => service_id_2:location_id_1} ]
where it is one of the same location entities
‘location’ { id => location_id_1 }

If a location changes on service_id_1 say to location_id_3 instead of 1 then
‘service_at_location’ => [ {id => service_id_1:location_id_3}, {id => service_id_1:location _id_2 }

So the service_at_location ‘entities’ that we don’t have are always ‘entities’ with an ID based on their two ‘field’ values the service they connect to and the location they connect to. They are a immutable and different ‘entity’ if they are connected to different services/locations.

Similarly for service_taxonomies and any of the others that have a sort of relational entity between the the property on the two entities.

That’s correct its unlikely those ids wouldn’t change. However, the taxonomy ids can and do change, services etc get renamed.

I’d have expected if a taxonomy id changes it, as my location example, would be a different thing. To clarify my mind where the mutability is, and make sure I can map to external IDs correctly when we do have external vocabularies, I figured I’d ‘quickly’ see how we’d reference the LGA term lists. Make sure the way our IDs are being implemented we’re not ruling out using these external ids, or confusing internally defined ones.

One question that I note looking at the docs and the data in the API Query Tool:-

The vocabulary name for LGA Circumstance list is esdCircumstances. I can’t quite tally the lists with https://standards.esd.org.uk/xml?uri=list/circumstances The guidance suggests two curies esdCircumstance and esdCircumstanceCollection. So is this where vocabulary id ≠ curie. I’m not missing it there isn’t an example prefixing the taxonomy_ids with curies is there?

I guess you’re referring to the Use of Taxonomies page.

This Bristol service from Placecube’s implementation uses “esdCircumstances” (should be “esdCircumstance”) in the vocabulary property of taxonomy.

Circumstance (URI = http://id.esd.org.uk/list/circumstances) is a bit of a strange one in that it is hierarchical with the top level (“collection”) representing groups of circumstances (e.g. Gender, Ethnicity, Addict or substance abuser) and the lower levels representing specific circumstances within those groups (e.g. Female, Asian, Smoker). So I’d normally expect a service to be aimed at a specific esdCircumstance, not the collection.

In all other examples I can think of, there’s just one CURIE per taxonomy.

The circumstances thing having two URI to define the Circumstance and CircustanceCollection is certainly inconvenient. So I was looking to see how people were presenting it.

I’d only looked in the API Query Tool, they all have at /vocabulary esdCircumstances (esdNeeds and esdServiceTypes); and I didn’t yet see a use of any of these as a curie style prefix to the taxonomy id, just as the value for filtering vocabulary=.

Regarding CURIEs, I think the Placecube implementation of terms like esdCircumstance in web method responses ( e.g. this Bristol service) is what we expect.

I think we’re misusing the term CURIE. What we mean is a namespace for a list, e.g. “esdService” that would resolve to “Standards | LG Inform Plus” which, when combined with an id like “242” would give a full URI like http://id.esd.org.uk/service/242.

So terminology might be:

Does that make sense to you?

So looking at the implementations I think that’s good, and pretty much what we’ve got. From the Bristol service you linked:-

  "service_taxonomys": [
    {
      "id": "9132-13",
      "taxonomy": {
        "id": "AddictSubstanceAbuser_Smoking",
        "name": "Smoker",
        "vocabulary": "esdCircumstances"
      }
    },

Where for search/filter queries on services they map to: ?taxonomy_id=AddictSubstanceAbuser_Smoking and/or ?vocabulary=esdCircumstances in the query parameters.

What I had understood would be you’d drop the ‘vocabulary’ key and use id = esdCircumstances:AddictSubstanceAbuser_Smoking in the json output.

I’m still left pondering if it’s worth trying to differentiate esdCircumstanceCollection and esdCircumstance, rather than just use esdCircumstances as seems practice. Only reason not to is that it requires making the fields based on the exception, rather than all the others where the curie / vocabulary identifier can be stored with the vocabulary itself.

I suspect you might be right but at present:

This is more of an LGA vocabulary thing than ORUK, but I have a foot in both camps so I’ll follow up.

Frankly the “Circumstances” list is the only one that uses alpha identifiers rather than number, which I think was a mistake. I’m not sure whether it’s too late to make fundamental changes. We’d certainly need to maintain deprecated identifiers.

Anyway, I can’t see a situation where the collection would be referenced against a service rather than a specific circumstance.

1 Like

Code has now been generalized: GitHub - localgovdrupal/localgov_openreferral: Open Referral integration
well there’s still more that can be done but it rolls out of the box (this is the demo content of the demo site - hence being a few councils and some web dev agencies) like http://openreferral.localgovdrupal.org/openreferral/v1

That also means that it only has a site vocabulary at the moment. In the immediate term: I’ll enable one or two external ones and add them to the demo content. I’ll also look at tidying up and testing more stuff particularly round the vocabulary I think.

Great to see it coming along. I see that https://openreferral.localgovdrupal.org/openreferral/v1/services returns a list.

Of course the list should be of services, not organizations, but I think you have a one-to-one between those.

Do let us know as it progresses and when we might add it to the dashboard and API Query tool with a suitable description like “demonstration data”.

It would be nice to see something like:
https://openreferral.localgovdrupal.org/openreferral/v1/services/c05a7f66-ffd8-4181-9129-1b1bc7f03b42-61a2c681-f994-4108-9c6c-f525cd67774a
return a result.

I’d just enabled Open Referral on the demo content directory. This exposes “Venue directory” entries as organisations and services. Those "Venue"s are those involved in the distribution. It probably helps to see the frontend presentation:-

https://openreferral.localgovdrupal.org/localgov-drupal-collaborators

I’m explaining this in detail, because it could be an issue with retrofitting Open Referral on existing sites. Directories, by default, have two content types “Entry” and “Venue”, venue the most commonly used so far and they are often Schools, or Leisure Centres etc. The default when enabling Open Referral on existing content would be to map them to Services. For other existing sites there are already other Directory Content types, like https://beta.lambeth.gov.uk/family-information-directory is a “Family Information Service” directory entry type; again I’d expect them to map these to Services. The related organisations might be manually added, or automated (if automated they’d just have the same name as the existing entry).

For new sites I think the product group might want to have a discussion about what different default types of Directory Entry there are. Obviously there’s now an ‘Organization’ one added, so an explicitly named ‘Service’ one might be desirable. On a technical level, rather than content level, the code that outputs the Open Referral data from the Directories is flexible, so it’s up to those implementing the site, or doing content design, to make sure that the content types sensibly map to the Open Referral classes.

I’ve written up notes [draft- Open Referral and LocalGov Drupal | LocalGov Drupal Docs] about the technical implementation. That’ll be getting a proper home, permanent url, on the blog soon.