Relation Cache Updater

'relation-cache-updater' job is used to perform one time compute and cache information such as leafNodes, optionalNodes, unitsMap and ancestorsMap of a collection to Redis when the collection is published. This data is utilised by other jobs and services.

Key	Data format	Sample
<collectionId>:leafnodes	List<String>	collectionId:leafnodes: [“resource1”,”resource2”]
<collectionId>:optionalnodes	List<String>	collectionId:optionalnodes: [“resource1”,”resource2”]
<rootCollectionId>:<resourceId>:ancestors	List<String>	collectionId:resource1:ancestors: [“courseunit1”, “democourse”]

Configuration variables:

Variable	Default Value	Purpose
kafka.input.topic	{{env}}.lms.user.account.merge	Kafka topic from which messages/events are read to be processed.
kafka.groupId	{{env}}-relation-cache-updater-group	Kafka input topic group Id
lms-cassandra.keyspace	{{env}}_hierarchy_store	Cassandra keyspace name
lms-cassandra.table	content_hierarchy	Cassandra table used to read collection hierarchy.
redis.database.index	10	Redis index to which computed data like leafnodes and optionalnodes is stored
dp-redis.host	IP should be same as lp-redis host	dp-redis (data-pipeline redis) IP should be kept same as lp-redis (learning-platform) redis in order to be able to read pulished collection information
dp-redis.port	port should be same as lp-redis port	dp-redis port
dp-redis.database.index	5	Redis index to which computed data 'units map' is stored

Sample event:

{
  "eid": "BE_JOB_REQUEST",
  "ets": 1649170015935,
  "mid": "LP.1649170015935.2a7382ff-8bc9-4ff5-8620-2d467cf8990a",
  "actor": {
    "id": "Post Publish Processor",
    "type": "System"
  },
  "context": {
    "pdata": {
      "ver": "1.0",
      "id": "org.sunbird.platform"
    },
    "channel": "01272777697873100812",
    "env": "sunbirdstaging"
  },
  "object": {
    "ver": "1649169952265",
    "id": "do_21350999965318348811690"
  },
  "edata": {
    "action": "post-publish-process",
    "iteration": 1,
    "identifier": "do_21350999965318348811690",
    "channel": "01272777697873100812",
    "mimeType": "application/vnd.ekstep.content-collection",
    "contentType": "Course",
    "pkgVersion": 1,
    "status": "Live",
    "name": "CourseMTimmothy",
    "trackable": {
      "enabled": "Yes",
      "autoBatch": "No"
    }
  }
}

Source code:

data-pipeline/lms-jobs/relation-cache-updater at release-5.3.0 · Sunbird-Lern/data-pipelineGitHub