Release V 5.0.0

Document Release Version

Project	Release Date	Version
Lern	18 Aug 22	V 5.0.0

Details of Released Tag

Components	Tags
Batch Service	sunbird-course-service : release-5.0.0_RC6 cert-service : release-5.0.0_RC4 certficate-registry : release-5.0.0_RC1 data-pipeline : release-5.0.0_RC6
User&Org Service	sunbird-lms-service : release-5.0.0_RC5
Group Service	groups-service : release-5.0.0_RC3
Discussion Forum Service	discussions-middleware : release-5.0.0_RC2 sunbird-nodebb : release-5.0.0_RC1
Notification Service	sunbird-notification-service : release-5.0.0_RC6
Data Products	data-products : release-5.0.0_RC4

Summary of the Changes

• Migrating component repositories in to Lern organisation in GitHub as a first step in making the installation and setup easy for adopters and contributors

• Increasing code coverage and unit test cases of all the components in Lern as part of stabilising the components

• Refactoring of the provisioning and deployment scripts of Lern BB

• Making SB Lern Cloud agnostic

Details of the Changes

User&Org Service

JIRA ID	Descriptions
SB-30076	UserOrg - increase code coverage and unit test cases
SB-30067	UserOrg - Deployment and Release processes
SB-29842	UserOrg Dataproducts Migration to Lern
SB-29826	Sunbird-apimanager-util migration to Lern
SB-29825	Sunbird-auth migration to Lern
SB-29824	Sunbird-utils - DB Migration to Lern
SB-29823	UserOrg Migration to Lern
SB-29813	OrgSearch to allow partial search and fuzzy Search
LR-103	Making SB Lern Cloud agnostic : Code changes to generalise CSP support in UserOrg
LR-124	Cassandra related changes for supporting multiple Data Centers
LR-232	CSP changes for cert, learner & course service
LR-128	Data Migration related to CSP changes : Report url in Job_request table needs to be updated
LR-125	Data Migration related to CSP changes : Template url in DB Certificate Objects needs to be updated - batch tables
LR-113	Data Migration related to CSP changes : Existing reports need to be migrated
LR-112	Data Migration related to CSP changes : Credential template, context files to be stored in to new CSP
LR-111	Data Migration related to CSP changes : Old Certificates in Azure needs to be migrated
LR-110	Data Migration related to CSP changes : Template url in DB Certificate Objects needs to be updated - RC tables and ES
LR-109	Data Migration related to CSP changes:Exisisting Certificate templates needs to be migrated.

Configurations:

Sunbird-lms-service:

isMultiDCEnabled={{cassandra_isMultiDCEnabled}}
sunbird_cassandra_consistency_level={{sunbird_cassandra_consistency_level}}
sunbird_user_cert_kafka_topic={{kafka_topic_lms_user_account}}
sunbird_cloud_service_provider={{cloud_service_provider}}
sunbird_account_name={{sunbird_public_storage_account_name}}
sunbird_account_key={{sunbird_public_storage_account_key}}

Batch Service

JIRA ID	Description
SB-30075	Batch service - increase code coverage and unit test cases
SB-29843	Batch service Dataproducts Migration to Lern
SB-30068	Batch service - Deployment and Release processes
SB-29841	assessment aggregate updater Migration to Lern
SB-29840	activity aggregate updater Migration to Lern
SB-29839	course-service migration to Lern
SB-29837	enrolment-reconciliation Migration to Lern
SB-29836	relation cache updater Migration to Lern
SB-29835	colletion-certificate-generator Migration to Lern
SB-29834	collection-cert-pre-processor Migration to Lern
SB-29833	certificate processor Migration to Lern
SB-29832	certificate-registry migration to Lern
SB-29831	cert-service migration to Lern
LR-104	Making SB Lern Cloud agnostic : Code changes to generalise CSP support in BatchService
LR-105	Making SB Lern Cloud agnostic : Code changes to generalise CSP support in datapipeline
LR-106	Making SB Lern Cloud agnostic : Code changes to generalise CSP support in data products
LR-108	Signed URL Generation for old certificates - cert-service
LR-107	RC deploy helm chart changes to upload Credential template, context files

Configurations:

https://github.com/project-sunbird/sunbird-devops/blob/learn-bb/ansible/roles/stack-sunbird/templates/
Sunbird-course-service:
isMultiDCEnabled={{cassandra_isMultiDCEnabled}}

# Add proper cloud service provider (azure,aws,gcloud)
sunbird_cloud_service_provider={{cloud_service_provider}}
sunbird_account_name={{sunbird_public_storage_account_name}}
sunbird_account_key={{sunbird_public_storage_account_key}}

#deleted the below variable
sunbird_content_azure_storage_container=sunbird-content-dev
# Added below variable for supporting multiple cloud service providers
# Provide corresponding cloud service provider(azure,aws,gcloud) container name here 
sunbird_content_cloud_storage_container=sunbird-content-dev

Cert-service:
CONTAINER_NAME={{cert_service_container_name}}
# Add proper cloud service provider (azure,aws,gcloud)
CLOUD_STORAGE_TYPE={{cloud_service_provider}}
PRIVATE_CLOUD_STORAGE_SECRET={{sunbird_private_storage_account_key}}
PRIVATE_CLOUD_STORAGE_KEY={{sunbird_private_storage_account_name}}
PUBLIC_CLOUD_STORAGE_KEY={{sunbird_public_storage_account_name}}
PUBLIC_CLOUD_STORAGE_SECRET={{sunbird_public_storage_account_key}}

Certificate-registry:
isMultiDCEnabled={{cassandra_isMultiDCEnabled}}

Data-pipeline:
isMultiDCEnabled={{cassandra_isMultiDCEnabled}}
# Add proper cloud service provider (azure,aws,gcloud)
cloud_storage_type :azure/aws/gcloud

data-products:
cloud_storage_type :azure/aws/gcloud

Discussion Forum

JIRA ID	Descriptions
SB-30070	DF - Deployment and Release processes
SB-29820	DF Migration to Lern and deployment setup
SB-30073	DF - increase code coverage and unit test cases

Group Service

JIRA ID	Description
SB-30074	Group service - increase code coverage and unit test cases
SB-30069	Group service - Deployment and Release processes
SB-29819	Group-service Migration to Lern and deployment setup

Configurations:

groups-service:
File Path:
cassandra-utils/src/main/resources/cassandra.config.properties
Changes:
isMultiDCEnabled=false

File Path:
sb-utils/src/main/resources/cassandra.config.properties
Changes:
isMultiDCEnabled=false

Notification Service

JIRA ID	Description
SB-30077	Sunbird-notification-service - increase code coverage and unit test cases
SB-29828	Sunbird-notification-jobs migration to Lern
SB-29827	Sunbird-notification-service migration to Lern
SB-30071	Sunbird-notification-service - Deployment and Release processes

Configurations:

Sunbird-notification-service:
File path:
sb-utils/src/main/resources/cassandra.config.properties
Changes:
isMultiDCEnabled=false

Detailed Information is present in the JIRA list.

Configurations:

1. Nodebb upstream branch: v1.18.6

2. Jenkins build, deploy and upload related changes for Flink jobs are present in below link:

data-pipeline/kubernetes/pipelines at release-5.0.0 · Sunbird-Lern/data-pipelineGitHub

3. There is a new variable added in devops repo to configure the bb name for kafka topics of flink jobs. This variable should be appended after the env name.

data-pipeline/all.yml at release-5.0.0 · Sunbird-Lern/data-pipelineGitHub

List of Lern Flink jobs:

collection-cert-pre-processor
collection-certificate-generator
activity-aggregate-updater
relation-cache-updater
merge-user-courses
assessment-aggregator
enrolment-reconciliation
notification-job

env variable changes are listed below: 
kubernetes/ansible/roles/flink-jobs-deploy/defaults/main.yml

###  Merge User Courses Job related Vars
merge_user_courses_consumer_parallelism: 1
merge_user_courses_parallelism: 1
merge_user_courses_course_batch_parallelism: 1
merge_user_courses_course_date_format: "yyyy-MM-dd HH:mm:ss:SSSZ"

###  Notification Job related Vars
notification_job_consumer_parallelism: 1
notification_job_parallelism: 1

### assessment-aggregator related vars
assessaggregator_parallelism: 1
assessaggregator_consumer_parallelism: 1
assessaggregator_downstream_parallelism: 1
assessaggregator_scoreaggregator_parallelism: 1
middleware_cassandra_courses_keyspace: sunbird_courses
middleware_cassandra_assessment_aggregator_table: assessment_aggregator
middleware_cassandra_assessment_question_type : question
middleware_cassandra_user_enrolments_table: user_enrolments
middleware_cassandra_user_activity_agg_table: user_activity_agg
content_read_api_host: "http://dev.sunbirded.org"
content_read_api_endpoint: "/api/content/v1/read/"

merge-user-courses:
    job_class_name: 'org.sunbird.job.merge.user.courses.task.MergeUserCoursesStreamTask'
    replica: 1
    jobmanager_memory: 1024m
    taskmanager_memory: 1024m
    taskslots: 1
    cpu_requests: 0.3
  assessment-aggregator:
    job_class_name: 'org.sunbird.dp.assessment.task.AssessmentAggregatorStreamTask'
    replica: 1
    jobmanager_memory: 1024m
    taskmanager_memory: 1024m
    taskmanager_process_memory: 1700m
    jobmanager_process_memory: 1600m
    taskslots: 1
    cpu_requests: 0.3
    scale_enabled: false
  notification-job:
    job_class_name: 'org.sunbird.job.notification.task.NotificationStreamTask'
    replica: 1
    jobmanager_memory: 1024m
    taskmanager_memory: 1024m
    taskslots: 1
    cpu_requests: 0.3

Stop the existing Samza jobs - (merge-user-courses and notification-job)
Stop the assessment-aggregator job from sunbird-data-pipeline and remove it from the corresponding Jenkins job as well
All these 3 jobs will be running from LERN repo now

4. Jenkins build, deploy and upload related changes for data products are in below link:

data-products/pipelines at release-5.0.0 · Sunbird-Lern/data-productsGitHub

LERN data-products list
  exhaust / progressexhaustjob
  exhaust / responseexhaust
  exhaust / userinfoexhaust
  job / course consumption
  job / course enrolment
  job / stateadminreport
  job / stateadmingeoreport
  job / collectionsummaryjobv2
  audit / collection reconcilation
  audit / coursebatch status updater
  updater / Cassandramigrator

5. Jenkins build, deploy and upload related changes for microservices are in below link:

GitHub - project-sunbird/sunbird-devops at learn-bbGitHub

Devops config changes:

File Path: 
ansible/inventory/env/group_vars/all.yml
### Release-5.0.0 cloud service provider changes for supporting multiple providers ###
### cloud_service_provider value should be either (azure, aws, gcloud) as per cloud sdk dependency ###
cloud_service_provider: "azure"

## modified consistency levels from quorum to local_quorum for multiple data centers support
File Path: 
ansible/roles/stack-sunbird/templates/sunbird_cert-registry-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum

File Path: 
ansible/roles/stack-sunbird/templates/sunbird_groups-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum

File Path: 
ansible/roles/stack-sunbird/templates/sunbird_learner-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum
sunbird_user_cert_kafka_topic={{env_name}}{{bb}}.lms.user.account.merge

File Path: 
ansible/roles/stack-sunbird/templates/sunbird_lms-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum

File Path: 
ansible/roles/stack-sunbird/templates/sunbird_notification-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum
sunbird_notification_kafka_topic={{env_name}}{{bb}}.lms.notification

File Path:
kubernetes/helm_charts/sunbird-RC/registry/schemas/TrainingCertificate.json
Changes:
"credentialTemplate": "https://{{upstream_url}}/schema/credential_template.json"

File Path:
utils/sunbird-RC/schema/credential_template.json
Changes:
"https://{{upstream_url}}/schema/v1_context.json",
 "https://{{upstream_url}}/schema/sunbird_context.json"

Design Documentation:

Confluence

Sunbird-Lern Code, Flink Jobs and Data Products

Confluence

Lern BB repositories

Confluence

Sunbird-Lern existing jenkins jobs