Release V 5.3.0

Hot-fix: CSP (24-08-2023)

Component	Build Job	Build Tag	Deploy Job	Deployment	Comment
Batch Service	Build/Core/Lms	release-5.3.2_RC1	Deploy/Kubernetes/Lms	release-5.3.0-lern	CSP related changes. cloud_store_group_id: org.sunbird cloud_store_artifact_id: cloud-store-sdk cloud_store_version: 1.4.7
User&Org Service	Build/Core/Learner	release-5.3.1_RC1	Deploy/Kubernetes/Learner	release-5.3.0-lern	CSP related changes. cloud_store_group_id: org.sunbird cloud_store_artifact_id: cloud-store-sdk cloud_store_version: 1.4.7
Data pipeline	Build/Lern/FlinkJobs	release-5.3.1_RC2	Deploy/Lern/FlinkJobs	release-5.3.1_RC2	CSP related changes. cloud_store_group_id: org.sunbird cloud_store_artifact_id: cloud-store-sdk_2.12 cloud_store_version: 1.4.6
Data Products	Build/Lern/LernDataProducts	release-5.3.1_RC8	Deploy/Lern/LernDataProducts	release-5.3.1_RC8	CSP related changes. cloud_store_group_id: org.sunbird cloud_store_artifact_id: cloud-store-sdk_2.12 cloud_store_version: 1.4.6

Jenkins Configurations for csp support:

Configure the variables cloud_store_group_id, cloud_store_artifact_id and cloud_store_version with proper values in the Jenkins, it can configured in the global or to the individual service of build job. For lms, user-org, flinks-jobs, lerndataproducts build jobs configure like as we mentioned below.

Jenkins configuration

Configure the following values

Name - Default Value - Description

cloud_store_group_id - ${cloud_store_group_id} - Set the Cloud store sdk group id. e.g. org.sunbird cloud_store_artifact_id - ${cloud_store_artifact_id} - Set the Cloud store sdk artifact id. e.g. cloud-store-sdk

cloud_store_version - ${cloud_store_version} - Set the Cloud store sdk version. e.g 1.4.6

Config changes in Lern common.yaml for data-products

cloud_storage_report_verfication_bucketname: "" # default value is "report-verification"
dp_storage_endpoint_config: "" # default is "{{s3_storage_endpoint}}"

Hot-fix: 5.3.1 (05-07-2023)

Component	Build Job	Build Tag	Deploy Job	Deployment	Comment
Batch Service	Build/Core/Lms	release-5.3.1_RC1	Deploy/Kubernetes/Lms	release-5.3.0-lern	QR Codes Image download Issue fix Bug: https://project-sunbird.atlassian.net/browse/KN-889

Document Release Version

Project	Release Date	Version
Lern	27-May-2023	V 5.3.0
Lern	23-Jun-2023	V 5.3.1

Hot Fix :- ML PII Data Product (23-06-2023)

Details of Released Tag

Components	Build Jenkins Job	Build Tag	Deploy Jenkins Job	Deploy Tag	Comment
Kafka Setup			Deploy/Lern/KafkaSetup		verify if kafka topic = programuser.info is created or not
Data pipeline	Build/Lern/FlinkJobs	release-5.3.0_RC5	Deploy/Lern/FlinkJobs	release-5.3.0_RC5	Add program-user-info into job list and deploy it.
Data Products	Build/Lern/LernDataProducts	release-5.3.0_RC6	Deploy/Lern/LernDataProducts	release-5.3.0_RC6	Add program-user-exhaust into job list of Deploy/Lern/LernAnalyticsReplayJobs for running it.
Cassandra Migration	Build/Core/Cassandra	release-5.3.0_RC1	Deploy/Kubernetes/Cassandra	release-5.3.0-lern	add the sunbird_programs keyspace in Deploy Jenkins jobs
Analytics			Deploy/Kubernetes/Analytics		Deploy with release-6.0.0 branch

Summary of the Changes

Details of the Changes:

LR-491 User detail (PII) report for ML programs - Data Product LR-285 User detail (PII) report for ML programs - Flink Job

Default values for config

default config for services

sunbird.program.report.keyspace="{{ program_keyspace }}"
ml.exhaust.store.prefix="ml_reports"

Please define below variables

program_keyspace: "sunbird_programs"
ml.exhaust.store.prefix="ml_reports"

Cassandra Keyspace and Table for Program:-

https://github.com/shikshalokam/sunbird-utils/blob/release-5.2.0/sunbird-cassandra-migration/cassandra-migration/src/main/resources/db/migration/cassandra/sunbird_programs/V1.1_cassandra.cql

Flink Job Configurations for Lern:

Name of the Flink Job added
program-user-info

LR-285 - User detail flink job for ML-programs - setup/configuration details:

For this ticket, we have only done unit testing with the help of simulated events. Integration testing has not been done as the required workflows concerning this will only be enabled after Ed 6.0 release. As part of this ticket we have enabled new Flink jobs and they in no way impact any existing workflows Job name: program-user-info

The purpose of this job is to record the user's information when the user submits the program. Whenever a program is submitted, this job receives an event with the user's information as JSON data and then it parses and stores it as respective key-value pairs in Cassandra.

Keyspace name: sunbird_program

Schema of the Kafka Topic: Kafka Topic Name: {{envName}}.programuser.info Event Structure:-

{
      programId: {
        type : "ObjectId",
        required : true,
        index: true
      },
      programName: String,
      programExternalId: String,
      noOfResourcesStarted: {
        type:Number,
        index: true
        }
      userId: {
        type: String,
        index: true
      },
      requestForPIIConsent:true/false
      userProfile: Object,
      userRoleInformation: Object,
      appInformation: Object,
      createdAt: Date,
      updatedAt: Date,
      deleted:Boolean
}

Job Configurations:

kafka {
 input.topic = ${job.env}".programuser.info"
 groupId = ${job.env}"-programuser-group"
}
task {
 consumer.parallelism = 1
 downstream.parallelism = 1
 programUser{
  parallelism = 1
 }
}
ml-cassandra {
 keyspace = "sunbird_programs"
 table = "program_enrollment"
 port = "9042"
 host =
 }

Flink build Jenkins job name: /Build/job/Lern/job/FlinkJobs

Flink deploy Jenkins job name: /Deploy/job/<environment>/job/Lern/job/FlinkJobs/program-user-info

Jenkins job for building Cassandra: /Build/job/Core/job/Cassandra/

Jenkins job for deploying Cassandra: /Deploy/job/<environment>/job/Kubernetes/job/Cassandra

Data Security Policy setup

Configurations to be done by System admin:

1. Setup default 'Data Security Policy' settings using tenant preference API.

curl --location --request PATCH '{{host}}/api/org/v2/preferences/update' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "orgId": "default",
        "key": "dataSecurityPolicy",
        "data": {
            "level": "PLAIN_DATASET",
            "dataEncrypted": "No",
            "comments": "Data is not encrypted",
            "job": {
                    "userinfo-exhaust": {
                        "level": "PASSWORD_PROTECTED_DATASET",
                        "dataEncrypted": "No",
                        "comments": "Password protected file."
                    },
                    "program-user-exhaust": {
                        "level": "PASSWORD_PROTECTED_DATASET",
                        "dataEncrypted": "No",
                        "comments": "Password protected file."
                    }
                },
            "securityLevels": {
                "PLAIN_DATASET": "Data is present in plain text/zip. Generally applicable to open datasets.",
                "PASSWORD_PROTECTED_DATASET": "Password protected zip file. Generally applicable to non PII data sets but can contain sensitive information which may not be considered open.",
                "TEXT_KEY_ENCRYPTED_DATASET": "Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open.",
                "PUBLIC_KEY_ENCRYPTED_DATASET": "Data encrypted via an org provided public/private key. Generally applicable to all PII data exhaust."
            }
        }
    }
}'

Details of Released Tag

Components	Build Jenkins Job	Build Tag	Deploy Jenkins Job	Deploy Tag	Comment
Kafka Setup			Deploy/Lern/KafkaSetup	release-5.3.0_RC3
Data pipeline	Build/Lern/FlinkJobs	release-5.3.0_RC5	Deploy/Lern/FlinkJobs	release-5.3.0_RC5	Add legacy-certificate-migrator into job list and deploy it.
Data Products	Build/Lern/LernDataProducts	release-5.3.0_RC6	Deploy/Lern/LernDataProducts	release-5.3.0_RC6
Batch Service	Build/Core/Lms	release-5.3.0_RC1	Deploy/Kubernetes/Lms	release-5.3.0-lern
User&Org Service	Build/Core/Learner	release-5.3.0_RC2	Deploy/Kubernetes/Learner	release-5.3.0-lern
Analytics			Deploy/Kubernetes/Analytics		Deploy with release-6.0.0 branch

Summary of the Changes

Details of the Changes:

LR-436 OldCertificateMigration spark data-product LR-437 LegacyCertificateMigrator Flink job LR-438 Sunbird RC changes for updating schema for issued date LR-330 Certificate template font url migration LR-395, LR-465 PII data security LR-451 Local setup of Data-pipeline - Ubuntu & Mac - Github and Microsite update LR-443 Local setup of UserOrg - Ubuntu & Mac - Github and Microsite update LR-445 Local setup of LMS - Ubuntu & Mac - Github and Microsite update LR-422 Point the channel create API to content-service instead of learning-service LR-519 Textbook APIs code cleanup from Course-Batch service LR-486 Microsite update with Certificate generation flow diagram LR-520 Group service - activity type should be case insensitive LR-556 Local setup of LMS - Ubuntu & Mac - Mock service setup LR-456 Local setup of Sunbird-utils - Ubuntu & Mac - Github and Microsite update

New APIs to onboard

- name: exhaustSubmitProxyAPI
  uris: "{{ course_service_prefix }}/v1/jobrequest/submit"
  upstream_url: "{{ lms_service_url }}/v1/jobrequest/submit"
  strip_uri: true
  plugins:
  - name: jwt
  - name: cors
  - "{{ statsd_pulgin }}"
  - name: acl
    config.whitelist:
    - courseAccess
  - name: rate-limiting
    config.policy: local
    config.hour: "{{ medium_rate_limit_per_hour }}"
    config.limit_by: credential
  - name: request-size-limiting
    config.allowed_payload_size: "{{ small_request_size_limit }}"
  - name: opa-checks
    config.required: false
    config.enabled: false

- name: exhaustListProxyAPI
  uris: "{{ course_service_prefix }}/v1/jobrequest/list"
  upstream_url: "{{ lms_service_url }}/v1/jobrequest/list"
  strip_uri: true
  plugins:
  - name: jwt
  - name: cors
  - "{{ statsd_pulgin }}"
  - name: acl
    config.whitelist:
    - courseAccess
  - name: rate-limiting
    config.policy: local
    config.hour: "{{ medium_rate_limit_per_hour }}"
    config.limit_by: credential
  - name: request-size-limiting
    config.allowed_payload_size: "{{ small_request_size_limit }}"
  - name: opa-checks
    config.required: false
    config.enabled: false
    
- name: orgAddEncryptionKey
  uris: "{{ org_service_prefix }}/v1/update/encryptionkey"
  upstream_url: "{{ learning_service_url }}/v1/org/update/encryptionkey"
  strip_uri: true
  plugins:
  - name: jwt
  - name: cors
  - "{{ statsd_pulgin }}"
  - name: acl
    config.whitelist:
    - orgSuperAdmin
  - name: rate-limiting
    config.policy: local
    config.hour: "{{ medium_rate_limit_per_hour }}"
    config.limit_by: credential
  - name: request-size-limiting
    config.allowed_payload_size: "{{ small_request_size_limit }}"
  - name: opa-checks
    config.required: false
    config.enabled: false    

Env Configurations (Needs to be done before service deployment):

The below environment variable needs to be configured in the 'sunbird-lms-service.env' file dev ops repo. Ref: https://github.com/project-sunbird/sunbird-devops/blob/release-5.3.0-lern/ansible/roles/stack-sunbird/templates/sunbird_lms-service.env

Variable Name	Values	Comments
exhaust_api_base_url	{{ analytics_service_url | default('http://analytics-service:9000') }}	Obsrv exhaust API endpoint for batch service
exhaust_api_submit_endpoint	/request/submit	To submit job request from batch service
exhaust_api_list_endpoint	/request/list/	To list job request from batch service
sunbird_api_auth_token	"{{ core_vault_sunbird_api_auth_token }}"	Authentication token for APIs
content_read_url	/content/v3/read/

Exhaust Proxy API documentation

https://github.com/Sunbird-Lern/sunbird-course-service/blob/release-5.3.0/api-tests/Collection/Proxy%20Exhaust%20APIs.postman_collection.json

Data Security Policy setup

Configurations to be done by System admin:

1. Execute CURL for providing link to download "Decryption Tool". Tool reference: https://github.com/Sunbird-Lern/sunbird-utils/blob/release-5.3.0/decryption-tool/decryption-tool.zip

Please upload the tool to your public cloud location or to your repository and provide the link to the same in below system setting variable value.

curl --location --request POST '{{host}}/api/data/v1/system/settings/set' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{api_key}}' \
--header 'x-authenticated-user-token: {{user_token}}' \
--data-raw '{
    "request": {
        "id": "decryptionToolLink",
        "field": "decryptionToolLink",
        "value": "{\"link\":\"<link to download decryption tool>\", \"Comments\": \"To use this tool, run the command with encrypted file and key to decrypt\"}"
    }
}'

2. Setup default 'Data Security Policy' settings using tenant preference API.

curl --location --request POST '{{host}}/api/org/v2/preferences/create' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "orgId": "default",
        "key": "dataSecurityPolicy",
        "data": {
            "level": "PLAIN_DATASET",
            "dataEncrypted": "No",
            "comments": "Data is not encrypted",
            "job": {
                    "userinfo-exhaust": {
                        "level": "PASSWORD_PROTECTED_DATASET",
                        "dataEncrypted": "No",
                        "comments": "Password protected file."
                    }
                },
            "securityLevels": {
                "PLAIN_DATASET": "Data is present in plain text/zip. Generally applicable to open datasets.",
                "PASSWORD_PROTECTED_DATASET": "Password protected zip file. Generally applicable to non PII data sets but can contain sensitive information which may not be considered open.",
                "TEXT_KEY_ENCRYPTED_DATASET": "Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open.",
                "PUBLIC_KEY_ENCRYPTED_DATASET": "Data encrypted via an org provided public/private key. Generally applicable to all PII data exhaust."
            }
        }
    }
}'

3. Setup default 'PII data security settings' using tenant preference API.

curl --location --request POST '{{host}}/api/org/v2/preferences/create' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "orgId": "default",
        "key": "userPrivateFields",
        "data": {
            "PIIFields": [
                "email",
                "phone",
                "userName",
                "prevUsedEmail",
                "prevUsedPhone",
                "recoveryEmail",
                "recoveryPhone"
            ]
        }
    }
}'

Configurations that can be done by Tenants:

1. Use Tenant preference create API to create tenant specific 'Data Security Policy' settings similar to 'default' Data Security Policy settings but with tenant orgId.

Note: 
a. Tenant level security cannot be lower than 'default' Data Security Policy'.
b. Job Level security Policy in a Tenant specific configuration cannot be lower than Tenant Level configuration and cannot be lower than job level configuration in 'default' Data Security Policy'.
c. Below mapping shows the priority/grade of security policies 
"PLAIN_DATASET" < "PASSWORD_PROTECTED_DATASET" < "TEXT_KEY_ENCRYPTED_DATASET" < "PUBLIC_KEY_ENCRYPTED_DATASET"

2. In order to use "PUBLIC_KEY_ENCRYPTED_DATASET" security configuration for an exhaust report, tenant admin should have uploaded public pem key file using below API.

curl --location --request PATCH '{{host}}/api/org/v1/update/encryptionkey' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--form 'organisationId={{org_id}}' \
--form 'encryptionKey=@path_to_public_pem_file'

Steps to generate key pair for setting up Data Security policy configuration:

For Linux and Mac OS:

1. To generate Private Key

openssl genrsa -out private.pem 4096

2. To generate Public Key

openssl rsa -in private.pem -pubout -outform PEM -out public_key.pem

For Windows OS:

Please install GitBash: The Git installation package comes with SSH. Using Git Bash, which is the Git command line tool, you can generate SSH key pairs. Git Bash has an SSH client that enables you to connect to and interact with Triton containers on Windows.

To install Git:

1. Download and initiate the Git installer.

2. When prompted, accept the default components by clicking Next.

3. Choose the default text editor. If you have Notepad++ installed, select Notepad++ and click Next.

4. Select to Use Git from the Windows Command Prompt and click Next.

5. Select to Use OpenSSL library and click Next.

6. Select to Checkout Windows-style, commit Unix-style line endings and click Next.

7. Select to Use MinTTY (The default terminal of mYSYS2) and click Next.

8. Accept the default extra option configuration by clicking Install. When the installation completes, you may need to restart Windows.

Launching GitBash:

1. press Start+R to launch the Run dialog.

2. Type C:\Program Files\Git\bin\bash.exe and press Enter.

Generating Key pair:

1. To generate Private Key

openssl genrsa -out private.pem 4096

2. To generate Public Key

openssl rsa -in private.pem -pubout -outform PEM -out public_key.pem

Flink Job Configurations for Lern:

Name of the Flink Job added
legacy-certificate-migrator

Prerequired deployments for RC migration

LR-436 - Deploy Data-product

Data-product build Jenkins job: Build/Lern/LernDataProducts

Deploy Jenkins job: Deploy/{{env}}/Lern/LernDataProducts

LR-437 - Deploy legacy-certificate-migrator Flink job

Build Jenkins job: /Build/job/Lern/job/FlinkJobs

Deploy Jenkins job: /Deploy/job/<environment>/job/Lern/job/FlinkJobs

LR-438 - Update RC schema

Step 1 : Upload updated schema files. Deploy Jenkins job: Deploy/dev/Sunbird-RC/Upload_RC_Schema

Note: Since certificate signer service will cache the credential template. please make sure the credential template is updated in the respective path as per below file.

https://github.com/project-sunbird/sunbird-devops/blob/release-5.3.0-lern/utils/sunbird-RC/schema/credential_template.json

Step 2 : Deploy certificate signer service

Jenkins Job: Deploy/dev/Sunbird-RC/CertificateSign

Step to migrate old certificates to RC

Sunbird Lern BB is using Sunbird RC for generating & issuing e-credentials in its use cases (e.g.: course completion certificate) for all the latest completed courses (post March-2022). All the old certificates were custom generated and stored in Cassandra and cloud storage.

Once we migrate these certificates then we no longer need to store certificates in Cassandra and all the certificates will be using Sunbird RC going forward.

Reference Link: https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3117416449/LR-4+Design+of+migrating+existing+certificate+in+to+RC

Note: After migrating old certificates to RC, certificate verification of old certificates will become invalid. To support to old certificate verification, Sunbird ED building block is implementating in portal service in release 6.0. Kindly find the ticket in this link. So recommended to migrate the certificates after getting the old certification verification support as well.

Step 1

Create Kafka topic for only the purpose of this migration process

Topic name: {{env}}.legacy.certificate.migrate

Step 2

In the spark machine, update the old-certificate-migration-job model config in mount/data/analytics/scripts/lern-model-config.sh with correct values.

Sample model config:

{"search":{"type":"none"},"model":"org.sunbird.lms.audit.OldCertificateMigrationJob","modelParams":{"mode":"execute","store":"azure","sparkCassandraConnectionHost":"10.5.3.17", "cert_base_path": "https://dev.lern.sunbird.org", "cloud_storage_base_url": "https://sunbirddev.blob.core.windows.net", "cloud_store_base_path_placeholder": "CLOUD_BASE_PATH","content_cloud_storage_container": "sunbird-content-staging", "cloud_storage_cname_url": "https://obj.stage.sunbirded.org", "batchId": "01320961460024934435", "kafka_broker": "localhost:9092", "kafka_topic": "sunbirddevlern.legacy.certificate.migrate","output_file_path":"./reports/"},"parallelization":8,"appName":"OldCertificateMigrationJob"}

Note: migration job can be run single batch with "batchId": "01320961460024934435" and multiple batches with "batchId": "01320961460024934435,01220961460024934536" and for all batches with "batchId": "all" .

Step 3

Run the job with the below command in the spark machine.

/mount/data/analytics/scripts/lern-run-job.sh old-certificate-migration-job &

Note: logs can be found in below locations,

Joblog: /mount/data/analytics/scripts/logs/joblog.log

Execution log: /mount/data/analytics/logs/lern-data-products/{current_date}-job-execution.log

Note:

Verification steps can be found in the design page: https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3117416449/LR-4+Design+of+migrating+existing+certificate+in+to+RC#Verification-steps-for-the-certificate-migration-process

Steps to Font URL migration

All the templates are having dev URLs configured for Fonts in all the environments as per our observation. All these font URLs have to be migrated to the new cname URL

Note: Before font url migration, make sure all the font files are available at cname mapped account or cloud storage container. To verify, where the font files are available, open any svg template file in editor and check the font URL's host.

Please use java 11 for running the scripts

Step 1:

Download SVG file migrator and uploader jars by below command,

cd ~
mkdir svg_template_migration
cd svg_template_migration
wget "https://github.com/kumarks1122/sunbird-utils/raw/release-5.3.0-font-url-migration/svg_template_migration/template-migration/svg-migrator.jar"
wget "https://github.com/kumarks1122/sunbird-utils/raw/release-5.3.0-font-url-migration/svg_template_migration/template-upload/svg-uploader.jar"

Step 2:

Download the svg template files and update the font URLs in the template files.

java -jar svg-migrator.jar "{{ content search host }}" "0" "1000" "font_migration" "{{ Old URL }}" "{{ cname url }}"

#EXAMPLE
#java -jar svg-migrator.jar "dev.lern.sunbird.org" "0" "1000" "font_migration" "https://sunbirddev.blob.core.windows.net" "https://obj.diksha.gov.in"

Note: Before moving to next step, please verify atleast one svg file for whether the font URL got updated.

Step 3:

Upload the svg template files back to the cloud storage by below command.

java -jar svg-uploader.jar "{{ content search host }}" "0" "1000" "{{ storage key}}" "{{ storage secret }}" "{{svg file path}}" "{{storage type: (azure,..)}}" "{{ CSP endpoint (based on CSP it is optional) }}" "{{ region (based on CSP it is optional) }}"

#EXAMPLE
#java -jar svg-uploader.jar "dev.lern.sunbird.org" "0" "5" "sunbirddevbbpublic" "{{ secret }}" "/Users/{{username}}/svg_template_migration" "azure"

Configuration for making content read URL dynamic: https://project-sunbird.atlassian.net/browse/LR-579 Please define the below URL in the sunbird_lms-service.env file this will make the content read endpoint URL Configurable.

content_read_url=/content/v3/read/