Migrating component repositories in to Lern organisation in GitHub as a first step in making the installation and setup easy for adopters and contributors
Increasing code coverage and unit test cases of all the components in Lern as part of stabilising the components
Refactoring of the provisioning and deployment scripts of Lern BB
https://github.com/project-sunbird/sunbird-devops/blob/learn-bb/ansible/roles/stack-sunbird/templates/
Sunbird-course-service:
isMultiDCEnabled={{cassandra_isMultiDCEnabled}}
# Add proper cloud service provider (azure,aws,gcloud)
sunbird_cloud_service_provider={{cloud_service_provider}}
sunbird_account_name={{sunbird_public_storage_account_name}}
sunbird_account_key={{sunbird_public_storage_account_key}}
#deleted the below variable
sunbird_content_azure_storage_container=sunbird-content-dev
# Added below variable for supporting multiple cloud service providers
# Provide corresponding cloud service provider(azure,aws,gcloud) container name here
sunbird_content_cloud_storage_container=sunbird-content-dev
Jenkins build, deploy and upload related changes for Flink jobs are present in below link:
3. There is a new variable added in devops repo to configure the bb name for kafka topics of flink jobs. This variable should be appended after the env name.
List of Lern Flink jobs:
collection-cert-pre-processor
collection-certificate-generator
activity-aggregate-updater
relation-cache-updater
merge-user-courses
assessment-aggregator
enrolment-reconciliation
notification-job
env variable changes are listed below:
kubernetes/ansible/roles/flink-jobs-deploy/defaults/main.yml
### Merge User Courses Job related Vars
merge_user_courses_consumer_parallelism: 1
merge_user_courses_parallelism: 1
merge_user_courses_course_batch_parallelism: 1
merge_user_courses_course_date_format: "yyyy-MM-dd HH:mm:ss:SSSZ"
### Notification Job related Vars
notification_job_consumer_parallelism: 1
notification_job_parallelism: 1
### assessment-aggregator related vars
assessaggregator_parallelism: 1
assessaggregator_consumer_parallelism: 1
assessaggregator_downstream_parallelism: 1
assessaggregator_scoreaggregator_parallelism: 1
middleware_cassandra_courses_keyspace: sunbird_courses
middleware_cassandra_assessment_aggregator_table: assessment_aggregator
middleware_cassandra_assessment_question_type : question
middleware_cassandra_user_enrolments_table: user_enrolments
middleware_cassandra_user_activity_agg_table: user_activity_agg
content_read_api_host: "http://dev.sunbirded.org"
content_read_api_endpoint: "/api/content/v1/read/"
merge-user-courses:
job_class_name: 'org.sunbird.job.merge.user.courses.task.MergeUserCoursesStreamTask'
replica: 1
jobmanager_memory: 1024m
taskmanager_memory: 1024m
taskslots: 1
cpu_requests: 0.3
assessment-aggregator:
job_class_name: 'org.sunbird.dp.assessment.task.AssessmentAggregatorStreamTask'
replica: 1
jobmanager_memory: 1024m
taskmanager_memory: 1024m
taskmanager_process_memory: 1700m
jobmanager_process_memory: 1600m
taskslots: 1
cpu_requests: 0.3
scale_enabled: false
notification-job:
job_class_name: 'org.sunbird.job.notification.task.NotificationStreamTask'
replica: 1
jobmanager_memory: 1024m
taskmanager_memory: 1024m
taskslots: 1
cpu_requests: 0.3
Stop the existing Samza jobs - (merge-user-courses and notification-job)
Stop the assessment-aggregator job from sunbird-data-pipeline and remove it from the corresponding Jenkins job as well
All these 3 jobs will be running from LERN repo now
4. Jenkins build, deploy and upload related changes for data products are in below link:
