Release V 8.0.0 (Ongoing)

Document Release Version

Project	Release Date	Version
Lern		V 8.0.0

Overview

This release contains software upgrade and ownership transfer functionality.

New Features

Ownership Transfer

LR-722 - Ownership Transfer API LR-685 - Ownership Transfer Flink Job LR-748 - Ownership Transfer delete user assets report

Enhancements / Technical Tasks

LR-309 - Keycloak version is upgraded with 21.1.2 from 7.0.1. , we are supporting the existing features. Migration activity details are mentioned https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3356000303/Keycloak+Migration+7.0.1+to+21.x+Design

LR-766 - Elasticsearch upgrade 6.8.22 to 7.17.13. Steps to upgrade elasticsearch are available here.

LR-738 - Scala upgrade from 2.11 to 2.12 for userorg, course, notification and group service.

Bug Fixes

LR-759 - After deleting a user, and when tried to login to the same user immediately, we are getting "Access denied" error

Details of Released Tag

Upgrade Sunbird Lern from 7.0.0 to 8.0.0

Components	Build Jenkins Job	Build Tag	Deploy Jenkins Job	Deploy Tag	Comment
OnboardAPIs	NA	NA	Deploy/Kubernetes/OnboardAPIs	release-7.0.0	To onboard the delete user API
Cassandra Migration	Build/Core/Cassandra	release-7.0.0_RC3	Deploy/Kubernetes/Cassandra	release-7.0.0	Select the sunbird in cassandra_keyspace_to_migrate while deploying script_repo_branch_or_tag: release-7.0.0_RC3
ES mapping			Provision/Core/ESMapping	release-7.0.0	Updates the es mapping to accept dynamic framework category to index user data
UserOrg Service	Build/Core/UserOrg	release-7.0.0_RC5	Deploy/Kubernetes/UserOrg	release-7.0.0
LMS Servive	Build/Core/LMS	release-7.0.0_RC2	Deploy/Kubernetes/LMS	release-7.0.0
Group Service	Build/Core/Groups	release-7.0.0_RC1	Deploy/Kubernetes/Groups	release-7.0.0
Notification service	Build/Core/Notification	release-7.0.0_RC2	Deploy/Kubernetes/Notification	release-7.0.0
Kafka Setup	NA	NA	Deploy/Lern/KafkaSetup	release-7.0.0_RC6
DataPipeline	Build/Lern/LernFlinkJobs	release-7.0.0_RC6	Deploy/Lern/LernFlinkJobs	release-7.0.0_RC6	Define the core_vault_sunbird_keycloak_user_federation_provider_id in Lern inventory secret. Add user-deletion-cleanup and ml-user-delete into job list and deploy it.
Data Product	Build/Lern/LernDataProducts	release-7.0.0_RC5	Deploy/Lern/LernDataProducts	release-7.0.0_RC5
Discussions Middleware	Build/Core/DiscussionsMiddleware	release-7.0.0_RC1	Deploy/Kubernetes/DiscussionsMW	release-7.0.0	Removed the unwanted logs

Configurations

LR-722 - Ownership Transfer API

1. Added the below configuration in the user org service application.conf file of user org service

       "/user_ownership_transfer_actor"
             {
               router = smallest-mailbox-pool
               nr-of-instances = 5
               dispatcher = brr-usr-dispatcher
             }
           "/user_ownership_transfer_actor/*"
              {
                dispatcher = akka.actor.brr-usr-dispatcher
              }

2. Added the below property in external resource.properties related to ownership transfer kafka topic in user org service

user-ownership-transfer-topic={{env_name}}.user.ownership.transfer

3. Added the below configuration in ansible/roles/kong-api/defaults/main.yml for ownership transfer API in sunbird devops repository.

- name: ownershipTransfer
  uris: "{{ user_service_prefix }}/v1/ownership/transfer"
  upstream_url: "{{ userorg_service_url }}/v1/user/ownership/transfer"
  strip_uri: true
  plugins:
  - name: jwt
  - name: cors
  - "{{ statsd_pulgin }}"
  - name: acl
    config.whitelist:
    - userUpdate
  - name: rate-limiting
    config.policy: local
    config.hour: "{{ medium_rate_limit_per_hour }}"
    config.limit_by: credential
  - name: request-size-limiting
    config.allowed_payload_size: "{{ medium_request_size_limit }}"
  - name: opa-checks
    config.required: false
    config.enabled: false

4. Added the user ownership transfer topic to the userorgservice.env file in Sunbird DevOps repository.

user-ownership-transfer-topic={{env_name}}.user.ownership.transfer

LR-685 Ownership transfer Flink job

1. Added below partition related settings and replication_factor related settings in ansible/roles/setup-lern-kafka/defaults/main.yml file of data pipeline repository.

  - name: user.ownership.transfer
    num_of_partitions: 2
    replication_factor: 1
    
    - name: user.ownership.transfer
    retention_time: 172800000
    replication_factor: 1

2. Added below ownership transfer flinkjob related configuration in kubernetes/helm_charts/datapipeline_jobs/values.j2 file of data pipeline repository.

user-ownership-transfer:
  user-ownership-transfer: |+
    include file("/data/flink/conf/base-config.conf")
    kafka {
      input.topic = ${job.env}".user.ownership.transfer"
      groupId = ${job.env}"-user-ownership-transfer-group"
    }
    task {
      user.ownership.transfer.parallelism = {{ user_ownership_transfer_job_parallelism }}
    }
    lms-cassandra {
      course_batch.table = "{{ middleware_course_batch_table }}"
      keyspace = "{{ middleware_course_keyspace }}"
    }
    service {
         lms {
             basePath = "{{ lms_service_base_url }}"
         }
         userorg {
             basePath = "{{ userorg_service_base_url }}"
         }
    }
    user_read_api = "/private/user/v1/read/"
    batch_search_api = "/v1/course/batch/search"
    threshold.batch.write.size = {{ user_ownership_transfer_batch_write_size }}

  flink-conf: |+
    jobmanager.memory.flink.size: {{ flink_job_names['user-ownership-transfer'].jobmanager_memory }}
    taskmanager.memory.flink.size: {{ flink_job_names['user-ownership-transfer'].taskmanager_memory }}
    taskmanager.numberOfTaskSlots: {{ flink_job_names['user-ownership-transfer'].taskslots }}
    parallelism.default: 1
    jobmanager.execution.failover-strategy: region
    taskmanager.memory.network.fraction: 0.1

3. Added below ownership transfer flinkjob related configuration in kubernetes/ansible/roles/flink-jobs-deploy/defaults/main.yml of data pipeline repository

user_ownership_transfer_job_parallelism: 1
user_ownership_transfer_batch_write_size: 10


  user-ownership-transfer:
    job_class_name: 'org.sunbird.job.ownershiptransfer.task.UserOwnershipTransferStreamTask'
    replica: 1
    jobmanager_memory: 1024m
    taskmanager_memory: 1024m
    taskslots: 1
    cpu_requests: 0.3

LR-748 Ownership transfer delete user assets report:

1. Added below ownership transfer delete user assets report related configuration in ansible/roles/lern-data-products-deploy/templates/lern-model-config.j2 of data products repository.

if [ ! -z "$3" ]; then specificUserId=$3; configuredUserId=$3; fi #used as specfic user id in usercacheindxerjob,used as configured user id in deleteusersassetsreportjob
if [ ! -z "$5" ]; then refreshUserData=$5; configuredChannel=$5;fi #used as refreshUserData flag in usercacheindxerjob,used as configured channel flag in deleteusersassetsreportjob


"delete-users-assets-report-job")
echo '{"search":{"type":"none"},"model":"org.sunbird.userorg.job.report.DeletedUsersAssetsReportJob","modelParams":{"store":"{{dp_object_store_type}}","storageKeyConfig":"storage.key.config","storageSecretConfig":"storage.secret.config","storageContainer":"{{reports_container}}","storageEndpoint":"{{dp_storage_endpoint_config}}","configuredUserId":'$configuredUserId',"configuredOrganisationId":[],"configuredChannel":'$configuredChannel', "sparkCassandraConnectionHost":"{{ core_cassandra_host }}"}}'
;;

2. Added below ownership transfer delete user assets report related configuration in ansible/roles/lern-data-products-deploy/templates/lern-run-job.j2 of data products repository

"delete-users-assets-report-job") echo 'org.sunbird.userorg.job.report.DeletedUsersAssetsReportJob'
;;

3. Added below ownership transfer delete user assets report related configuration in lern-data-products/src/main/resources/application.conf of data products repository.

content.search.url="http://10.246.3.3/search/v3/search" 
course.batch.search.url="http://10.246.3.3/lms/v1/course/batch/search"
sunbird_instance_name="Sunbird" 
delete.user.cloud.objectKey="reports/"

Flink Job Configurations for Lern:

user-ownership-transfer

Flink build Jenkins job name: /Build/job/Lern/job/FlinkJobs

Flink deploy Jenkins job name:

/Deploy/job/<environment>/job/Lern/job/FlinkJobs/user-ownership-transfer

Data Product Configurations for Lern:

delete-users-assets-report-job

Data products jenkins build job name: /Build/job/Lern/job/LernDataProducts/

Data products jenkins deploy job name: /Deploy/job/<environment>/job/Lern/job/LernDataProducts/LernAnalyticsReplayJobs/delete-users-assets-report-job

Release Notes: Dependent building blocks

Sunbird-Knowlg: Release notes (V 5.7.0) Sunbird-Obsrv: Release notes (V 5.1.0) Sunbird-Ed: Release notes (V 8.0.0) Sunbird-Inquiry: Release notes (V 5.7.0) Sunbird-Telemetry: Documentation Sunbird-RC: Documentation