Skip to content

Latest commit

 

History

History
106 lines (92 loc) · 4.63 KB

README.md

File metadata and controls

106 lines (92 loc) · 4.63 KB

image not found

APSViz-Archiver

Archives file created by the various APSViz applications and processes.

Licenses...

MIT License GPLv3 License RENCI License

Components and versions...

Python Linting Pylint Pytest

Build status...

Pylint and Pytest Build and push the Docker image

Description

This product is designed to perform copy/move/remove operations on file or directory entities. Additional functionality is implemented to sweep through all file/directory entities located within a directory. This product is intended to be run within a Kubernetes CronJob.

This product is rule based. Each Rule is defined in json format. A Rule can contain 1 or more Rule sets which can in turn specify 1 or more atomic Rule operations.

Atomic rules can optionally specify query criteria (age, etc.) that is paired with a predicate (equals, greater than, etc.). These query criteria are used to interrogate an entity prior to performing an operation.

A simple Rule example shown below has specified:

  • A rule definition with details including a name, version and a description.
  • A rule Set that contains rule operations.
  • 3 rule operations for copy, move and a directory sweep.
    • With each rule specifying a query criteria that will trigger the rule operation.
{
  "rule_definition_name": "APSViz data archival rule set.",
  "rule_definition_version": "0.0.0",
  "description": "Tests a complete rule.",
  "rule_sets":
  [
    {
      "rule_set_name": "Test rule set",
      "description": "Tests a rule set by making a copy then a  move request.",
      "rules": [
        {
          "name": "Test - Copy test file",
          "description": "This test copies a file from source to destination.",
          "query_criteria_type": "BY_AGE",
          "query_data_type": "INTEGER",
          "query_data_value": 30,
          "predicate_type": "GREATER_THAN_OR_EQUAL_TO",

          "action_type": "COPY",
          "data_type": "FILE",
          "source": "/source/some-data-file.ext",
          "destination": "/dest/some-new-data-location"
        },
        {
          "name": "Test - Move test file",
          "description": "This test moves a file from source to destination.",
          "query_criteria_type": "BY_AGE",
          "query_data_type": "INTEGER",
          "query_data_value": 30,
          "predicate_type": "GREATER_THAN_OR_EQUAL_TO",

          "action_type": "MOVE",
          "data_type": "FILE",
          "source": "/dest/some-new-data-location/some-data-file.ext",
          "destination": "/dest/some-newer-data-location"
        },
        {         
          "name": "Test - Sweep copy test",
          "description": "This test copies the directory contents of source to destination.",
          "query_criteria_type": "BY_AGE",
          "query_data_type": "INTEGER",
          "query_data_value": 2,
          "predicate_type": "GREATER_THAN_OR_EQUAL_TO",

          "action_type": "SWEEP_COPY",
          "data_type": "DIRECTORY",
          "source": "/source/root_directory",
          "destination": "/dest/new_directory"
        }
      ]
    }
  ]
}

There are GitHub actions to maintain code quality in this repo:

  • Pylint (minimum score of 10/10 to pass),
  • Pytest (with code coverage),
  • Build/publish a Docker image.

Helm/k8s charts for this product are available at: APSViz-Helm.