Add support for external management named properties in JCloud

[ianwallen]

Update external management url

• Add {objectId} property in external management url (base64 unique identifier for the record)
• Change external management type url property {type} so that it is fixed values so that same value can be used in {objectId}

CMIS

• Fixed property names used for validation fields to be consistent with other names.

Jcloud

• Updgade from jcloud 2.3.0 to jcloud 2.5.0
• Add support for external management named properties similar to cmis
• Fix bug with deleting all resources as it was failing to identify folders correctly for azure blob.

---

After this update, it will be possible to set parameters similar to the following

JCLOUD_EXTERNAL_RESOURCE_MANAGEMENT_CHANGE_DATE_PROPERTY_NAME=changedate
JCLOUD_EXTERNAL_RESOURCE_MANAGEMENT_VALIDATION_STATUS_PROPERTY_NAME=validationstatus
JCLOUD_METADATA_UUID_PROPERTY_NAME=geonetworkcatalogueuuid

And this will set the metadata properties in the jcloud storage when uploading data.

Here is a sample of the metadata properties saved in Azure Blob storage using these property settings.

---

Unlike CMIS which contains an cmisobjectId that can be used to go identify the object, jcloud is more similar to a file system and did not have a file identifier. Therefor JCLOUD_EXTERNAL_RESOURCE_MANAGEMENT_URL was updated to support a new {objectId} was for identifying the object.
It is a base64 object which contains the following information

• type to identify type of storage - document/folder
• visibility of the resource public/private
• metadataId internal metadata id
• version identifier which can be used to directly get this version.
• resourceId or filename of the resource

Another option could be to supply the full path to the object however it seems like this could expose more information that was required and there was concern about how much information should be included. i.e. storage account, container name... The full path could always be added in the future as a new replacement variable if required. For now this new {objectId} contains all the information that already available to end users.

Also added similar option in CMIS urls

Checklist

• I have read the contribution guidelines
• Pull request provided for main branch, backports managed with label
• Good housekeeping of code, cleaning up comments, tests, and documentation
• Clean commit history broken into understandable chucks, avoiding big commits with hundreds of files, cautious of reformatting and whitespace changes
• Clean commit messages, longer verbose messages are encouraged
• API Changes are identified in commit messages
• Testing provided for features or enhancements using automatic tests
• User documentation provided for new features or enhancements in manual
• Build documentation provided for development instructions in README.md files
• Library management using pom.xml dependency management. Update build documentation with intended library use and library tutorials or documentation

[josegar74]

I have not tested the changed providers. Tested with the default filesystem provider to verify that everything looks fine with that provider and it seems to.

The code changes look fine, see some comments to clarify some of the changes.

[josegar74]

This seems a new property for CMIS. I don't fully understand this change. Is this a new property or something missing in previous implementations?

[ianwallen]

These properties are objects that can be supplied to the third party application.

As JCLOUD does not have any special ID to identify the record like CMIS did, I added a new {objectId} for JCLOUD so that the 3rd party application could use that value to identify the record.

In order to make some of these parameters consistent between storage solutions, I simply added the new {objectId} to CMIS as well so that CMIS applications could use it as well if needed.

I can remove it if you think there are issues with adding this new parameters.

[josegar74]

@ianwallen it's ok to keep it. I just wanted to have clear the reason. Thanks for the clarification.

[josegar74]

I would move the string hdi_isfolder to a constant, with some documentation.

[ianwallen]

I created a new function for isFolder and also created a constant for hdi_isfolder

[josegar74]

Ok,, but I don't see any change yet, I guess you are working on the change?

[ianwallen]

Odd that you don't see the changes

This is the new static field

core-geonetwork/datastorages/jcloud/src/main/java/org/fao/geonet/api/records/attachments/JCloudStore.java

Line 79 in d7ec517

private final static String AZURE_BLOB_IS_FOLDER_PROPERTY_NAME="hdi_isfolder";

This is the new code.

core-geonetwork/datastorages/jcloud/src/main/java/org/fao/geonet/api/records/attachments/JCloudStore.java

Lines 352 to 356 in d7ec517

	for (StorageMetadata storageMetadata : page) {
	if (!isFolder(storageMetadata)) {
	jCloudConfiguration.getClient().getBlobStore().removeBlob(jCloudConfiguration.getContainerName(), storageMetadata.getName());
	}
	}

And this is the new isFolder() function

core-geonetwork/datastorages/jcloud/src/main/java/org/fao/geonet/api/records/attachments/JCloudStore.java

Lines 503 to 506 in d7ec517

	private boolean isFolder(StorageMetadata storageMetadata) {
	// For azure Blob ADSL if the type is folder then the storage type will be BLOB and hdi_isfolder=true so we cannot only rely on StorageType.FOLDER
	return storageMetadata.getType().equals(StorageType.FOLDER) || "true".equals(storageMetadata.getUserMetadata().get(AZURE_BLOB_IS_FOLDER_PROPERTY_NAME));
	}

[josegar74]

Maybe I had an old version in the browser, I see it has been done a push force, but now I see the changes, thanks.

[josegar74]

Previously it was used "$1":"$2" instead of "folder" : "document". To check that this can not break existing setups?

[ianwallen]

Yes this would have broken existing setup.

The problem with this type is that it could be set to return custom values for the type but this was inconsistent with the type being returned in the {objectId} which was a hard coded value.

I had removed the old logic to no longer allow for custom name as I thought it could be confusing.

Looking at the code, I noticed that the old code was looking for "{type:" while the new code is supposed to looks for {type}

As they are different - I readded the old one so that it remains unchanged and I added the new {type} to be consistent with other providers.

[josegar74]

I don't know how many users use this storage, but If it can break existing setups, it might be good to add a label changelog to the pull request and a comment about any change required in existing setups, so it can be added in the release notes.

[ianwallen]

I reverted the changes and added a new one so it will not break any existing setup.

[josegar74]

Tested with the file system storage, to check that it doesn't break and looks fine.

Code changes also look good.

Check the comment about information for the release notes, in case it can be relevant.