Release V 5.0.0

Document Release Version

Project

Release Date

Version

Lern

18 Aug 22

V 5.0.0

Details of Released Tag

Components

Summary of the Changes

Migrating component repositories in to Lern organisation in GitHub as a first step in making the installation and setup easy for adopters and contributors
Increasing code coverage and unit test cases of all the components in Lern as part of stabilising the components
Refactoring of the provisioning and deployment scripts of Lern BB
Making SB Lern Cloud agnostic

Details of the Changes

JIRA ID

Descriptions

UserOrg - increase code coverage and unit test cases

UserOrg - Deployment and Release processes

UserOrg Dataproducts Migration to Lern

Sunbird-apimanager-util migration to Lern

Sunbird-auth migration to Lern

Sunbird-utils - DB Migration to Lern

UserOrg Migration to Lern

OrgSearch to allow partial search and fuzzy Search

Making SB Lern Cloud agnostic : Code changes to generalise CSP support in UserOrg

Cassandra related changes for supporting multiple Data Centers

CSP changes for cert, learner & course service

Configurations:

Sunbird-lms-service:

isMultiDCEnabled={{cassandra_isMultiDCEnabled}}
sunbird_cassandra_consistency_level={{sunbird_cassandra_consistency_level}}
sunbird_user_cert_kafka_topic={{kafka_topic_lms_user_account}}
sunbird_cloud_service_provider={{cloud_service_provider}}
sunbird_account_name={{sunbird_public_storage_account_name}}
sunbird_account_key={{sunbird_public_storage_account_key}}

JIRA ID

Description

Batch service - increase code coverage and unit test cases

Batch service Dataproducts Migration to Lern

Batch service - Deployment and Release processes

assessment aggregate updater Migration to Lern

activity aggregate updater Migration to Lern

course-service migration to Lern

enrolment-reconciliation Migration to Lern

relation cache updater Migration to Lern

colletion-certificate-generator Migration to Lern

collection-cert-pre-processor Migration to Lern

certificate processor Migration to Lern

certificate-registry migration to Lern

cert-service migration to Lern

Making SB Lern Cloud agnostic : Code changes to generalise CSP support in BatchService

Making SB Lern Cloud agnostic : Code changes to generalise CSP support in datapipeline

Making SB Lern Cloud agnostic : Code changes to generalise CSP support in data products

Signed URL Generation for old certificates - cert-service

RC deploy helm chart changes to upload Credential template, context files

Configurations:

https://github.com/project-sunbird/sunbird-devops/blob/learn-bb/ansible/roles/stack-sunbird/templates/
Sunbird-course-service:
isMultiDCEnabled={{cassandra_isMultiDCEnabled}}

# Add proper cloud service provider (azure,aws,gcloud)
sunbird_cloud_service_provider={{cloud_service_provider}}
sunbird_account_name={{sunbird_public_storage_account_name}}
sunbird_account_key={{sunbird_public_storage_account_key}}

#deleted the below variable
sunbird_content_azure_storage_container=sunbird-content-dev
# Added below variable for supporting multiple cloud service providers
# Provide corresponding cloud service provider(azure,aws,gcloud) container name here 
sunbird_content_cloud_storage_container=sunbird-content-dev

Cert-service:
CONTAINER_NAME={{cert_service_container_name}}
# Add proper cloud service provider (azure,aws,gcloud)
CLOUD_STORAGE_TYPE={{cloud_service_provider}}
PRIVATE_CLOUD_STORAGE_SECRET={{sunbird_private_storage_account_key}}
PRIVATE_CLOUD_STORAGE_KEY={{sunbird_private_storage_account_name}}
PUBLIC_CLOUD_STORAGE_KEY={{sunbird_public_storage_account_name}}
PUBLIC_CLOUD_STORAGE_SECRET={{sunbird_public_storage_account_key}}

Certificate-registry:
isMultiDCEnabled={{cassandra_isMultiDCEnabled}}

Data-pipeline:
isMultiDCEnabled={{cassandra_isMultiDCEnabled}}
# Add proper cloud service provider (azure,aws,gcloud)
cloud_storage_type :azure/aws/gcloud

data-products:
cloud_storage_type :azure/aws/gcloud

JIRA ID

Description

Group service - increase code coverage and unit test cases

Group service - Deployment and Release processes

Group-service Migration to Lern and deployment setup

Configurations:

groups-service:
File Path:
cassandra-utils/src/main/resources/cassandra.config.properties
Changes:
isMultiDCEnabled=false

File Path:
sb-utils/src/main/resources/cassandra.config.properties
Changes:
isMultiDCEnabled=false

JIRA ID

Description

Sunbird-notification-service - increase code coverage and unit test cases

Sunbird-notification-jobs migration to Lern

Sunbird-notification-service migration to Lern

Sunbird-notification-service - Deployment and Release processes

Configurations:

Sunbird-notification-service:
File path:
sb-utils/src/main/resources/cassandra.config.properties
Changes:
isMultiDCEnabled=false

Detailed Information is present in the JIRA list.

Configurations:

Nodebb upstream branch: v1.18.6
Jenkins build, deploy and upload related changes for Flink jobs are present in below link:

data-pipeline/kubernetes/pipelines at release-5.0.0 · Sunbird-Lern/data-pipelineGitHub

3. There is a new variable added in devops repo to configure the bb name for kafka topics of flink jobs. This variable should be appended after the env name.

data-pipeline/all.yml at release-5.0.0 · Sunbird-Lern/data-pipelineGitHub

List of Lern Flink jobs:

collection-cert-pre-processor
collection-certificate-generator
activity-aggregate-updater
relation-cache-updater
merge-user-courses
assessment-aggregator
enrolment-reconciliation
notification-job

env variable changes are listed below: 
kubernetes/ansible/roles/flink-jobs-deploy/defaults/main.yml

###  Merge User Courses Job related Vars
merge_user_courses_consumer_parallelism: 1
merge_user_courses_parallelism: 1
merge_user_courses_course_batch_parallelism: 1
merge_user_courses_course_date_format: "yyyy-MM-dd HH:mm:ss:SSSZ"

###  Notification Job related Vars
notification_job_consumer_parallelism: 1
notification_job_parallelism: 1

### assessment-aggregator related vars
assessaggregator_parallelism: 1
assessaggregator_consumer_parallelism: 1
assessaggregator_downstream_parallelism: 1
assessaggregator_scoreaggregator_parallelism: 1
middleware_cassandra_courses_keyspace: sunbird_courses
middleware_cassandra_assessment_aggregator_table: assessment_aggregator
middleware_cassandra_assessment_question_type : question
middleware_cassandra_user_enrolments_table: user_enrolments
middleware_cassandra_user_activity_agg_table: user_activity_agg
content_read_api_host: "http://dev.sunbirded.org"
content_read_api_endpoint: "/api/content/v1/read/"

merge-user-courses:
    job_class_name: 'org.sunbird.job.merge.user.courses.task.MergeUserCoursesStreamTask'
    replica: 1
    jobmanager_memory: 1024m
    taskmanager_memory: 1024m
    taskslots: 1
    cpu_requests: 0.3
  assessment-aggregator:
    job_class_name: 'org.sunbird.dp.assessment.task.AssessmentAggregatorStreamTask'
    replica: 1
    jobmanager_memory: 1024m
    taskmanager_memory: 1024m
    taskmanager_process_memory: 1700m
    jobmanager_process_memory: 1600m
    taskslots: 1
    cpu_requests: 0.3
    scale_enabled: false
  notification-job:
    job_class_name: 'org.sunbird.job.notification.task.NotificationStreamTask'
    replica: 1
    jobmanager_memory: 1024m
    taskmanager_memory: 1024m
    taskslots: 1
    cpu_requests: 0.3

Stop the existing Samza jobs - (merge-user-courses and notification-job)
Stop the assessment-aggregator job from sunbird-data-pipeline and remove it from the corresponding Jenkins job as well
All these 3 jobs will be running from LERN repo now

4. Jenkins build, deploy and upload related changes for data products are in below link:

data-products/pipelines at release-5.0.0 · Sunbird-Lern/data-productsGitHub

LERN data-products list
  exhaust / progressexhaustjob
  exhaust / responseexhaust
  exhaust / userinfoexhaust
  job / course consumption
  job / course enrolment
  job / stateadminreport
  job / stateadmingeoreport
  job / collectionsummaryjobv2
  audit / collection reconcilation
  audit / coursebatch status updater
  updater / Cassandramigrator

5. Jenkins build, deploy and upload related changes for microservices are in below link:

GitHub - project-sunbird/sunbird-devops at learn-bbGitHub

Devops config changes:

File Path: 
ansible/inventory/env/group_vars/all.yml
### Release-5.0.0 cloud service provider changes for supporting multiple providers ###
### cloud_service_provider value should be either (azure, aws, gcloud) as per cloud sdk dependency ###
cloud_service_provider: "azure"

## modified consistency levels from quorum to local_quorum for multiple data centers support
File Path: 
ansible/roles/stack-sunbird/templates/sunbird_cert-registry-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum

File Path: 
ansible/roles/stack-sunbird/templates/sunbird_groups-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum

File Path: 
ansible/roles/stack-sunbird/templates/sunbird_learner-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum
sunbird_user_cert_kafka_topic={{env_name}}{{bb}}.lms.user.account.merge

File Path: 
ansible/roles/stack-sunbird/templates/sunbird_lms-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum

File Path: 
ansible/roles/stack-sunbird/templates/sunbird_notification-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum
sunbird_notification_kafka_topic={{env_name}}{{bb}}.lms.notification

File Path:
kubernetes/helm_charts/sunbird-RC/registry/schemas/TrainingCertificate.json
Changes:
"credentialTemplate": "https://{{upstream_url}}/schema/credential_template.json"

File Path:
utils/sunbird-RC/schema/credential_template.json
Changes:
"https://{{upstream_url}}/schema/v1_context.json",
 "https://{{upstream_url}}/schema/sunbird_context.json"

Design Documentation:

PreviousRelease V 5.0.1 NextRelease V 4.10.0

Last updated 2 years ago