Sunbird Lern
AskDot
  • LEARN
    • Overview
    • Functional Capabilities
    • Technical Architecture
      • Technical Architecture Diagram
      • Tech Stack
    • Dependencies
    • Product Roadmap
  • USE
    • Overview
    • Release Notes
      • Release V 8.0.0 (Ongoing)
      • Release V 7.0.0
      • Release V 5.4.0
      • Release V 5.3.0
      • Release V 5.2.0
      • Release V 5.1.0
      • Release V 5.0.1
      • Release V 5.0.0
      • Release V 4.10.0
      • Release V 4.9.0
      • Release V 4.8.0
      • Release V 4.7.0
    • Developer Guide
      • USER & ORG SERVICE
        • Features/Core capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Keycloak Local setup
          • Additional Installation Dependencies:
          • Source Code
          • Installation Configuration
        • Data Models
          • Cassandra
            • Data Dictionary
              • User
              • Organisation
              • System Settings
              • Role
              • Bulk Upload Process
              • Tenant Preference
              • Cassandra Migration Version
              • User Consent
              • Email Template
              • OTP
              • Page Management (LMS Service)
              • Location
              • User Notes
              • Deprecated
          • Elastic Search
          • Redis
        • APIs
          • User Management
            • User Deletion API
            • Ownership Transfer API
          • Organisation Management
          • Location Management
          • Consent Management
          • OTP Services
          • Tenant Configurations
          • Bulk Upload
          • System Settings
          • API Management Service
          • Data Sync
          • Notification APIs
        • Flink Jobs
          • User Cache Updater
          • User Deletion Cleanup Flink Job
          • Ownership Transfer Flink Job
        • Reports
          • Standard Exhaust
            • State Admin Geo report
            • State Admin Report
          • Other Jobs
            • User Cache Indexer Job
            • Delete Users Assets Report
          • Data Products Developer Installation
            • System Requirements
            • Tech Stack
            • Installation Dependencies
            • Local installation of data-products
            • Server setup Guide
            • Installation Configuration
            • Data-product creation guide
            • Troubleshooting a data-product
        • Logs, Telemetry Events
        • Configuration
          • Functional Configurations
          • System Settings
          • Email Configuration
          • SMS Configuration
          • OTP based SMS Configuration
        • Roles
          • System Roles
          • User Roles
          • RBAC
        • Dependencies
        • Keycloak on Sunbird
        • How to Guide
          • Create Organization
          • Create User
          • Add new user type & location
        • Single Sign-on with Sunbird
        • Caching and Denormalising User Metadata
          • ETLUserCacheUpdaterJob
          • UserCacheUpdaterFlinkJob
        • Unit Tests and Code Coverage
        • FAQs
      • LMS(BATCH) SERVICE
        • Features/Core Capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Source Code
        • Data Models
          • Cassandra
            • Data Dictionary
          • ElasticSearch
          • Redis
        • APIs
        • Flink Jobs
          • Merge User Courses
          • Relation Cache Updater
          • Activity Aggregate Updater
          • Assessment Aggregator
          • Enrolment Reconciliation
          • Collection Certificate Pre-Processor
          • Collection Certificate Generator
        • Reports
          • On-Demand Exhaust
            • Response Exhaust
            • User Info Exhaust
            • Progress Exhaust
          • Other Jobs
            • Collection Summary Job
            • Course Batch Status Updater Job
            • Cassandra Migrator Job
            • Score Metric Migration Job
            • Assessment Score Correction Job
            • Collection Reconciliation Job
            • Course Enrollment Job
            • Course Consumption Job
        • Logs, Telemetry Events
        • Configuration
          • System Configuration
          • Functional Configuration
        • Dependencies
        • Certificate Registry and Credentialing using Sunbird-RC
          • Configuring PublicKey in Sunbird-RC
          • Server Setup guide for Sunbird-RC
          • API Guide For Sunbird-RC
        • Certificate Flow
          • Certificates Creation and Configuration
        • FAQs
      • GROUPS
        • Features/Core Capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Source Code
          • Installation Configuration
        • Data Models
          • Cassandra
            • Data Dictionary
          • Redis
        • APIs
          • Create Group
        • Logs, Telemetry Events
          • UI Telemetry Events
          • Service Telemetry Events
        • Configuration
          • Functional Configurations
        • Design References
        • Dependencies
        • Unit Tests and Code Coverage
      • NOTIFICATION SERVICE
        • Features/Core Capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Source Code
          • Installation Configuration
        • Data Models
          • Data Dictionary
        • APIs
        • Flink Jobs
        • Logs, Telemetry Events
        • Configuration
          • Functional Configurations
        • Dependencies
        • Unit Tests and Code Coverage
      • DISCUSSION FORUM
        • Features/Core capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
            • Nodebb setup
            • Discussion Middleware Setup
            • Discussion Middleware
            • Discussion forum integration with any application
            • Discussion-UI setup along with demo application.
            • Sunbird-lern portal for DF
          • Source Code
        • APIs
          • Category APIs
          • User APIs
          • Post APIs
          • Topic APIs
          • Forum APIs
        • Context Schema
        • Configurations
          • Nodebb Admin panel settings
          • Discussion MW & Nodebb System Config
        • Telemetry Events
        • Dependencies
      • ML SERVICE
        • DATA PIPELINE (Flink Jobs)
          • Program User Info
            • Component Diagram
            • Data Model
            • Folder Structure
          • Ml User Delete
            • Component Diagram
            • Data Model
            • Folder Structure
        • DATA PRODUCTS
          • Program Exhaust
            • Component Diagram
            • Folder Structure
    • Server Installation
    • Dependency Setup
    • Deprecation
      • Release-5.4.0
    • Jenkins Jobs
    • Release Calendar
    • Learn More
      • Tech References
      • Telemetry Processing
      • Data Dictionary
        • Sample Data
      • Delete User Functionality
      • Asset Ownership Transfer
    • FAQs
  • Engage
    • Discuss
    • Contribute to Sunbird Lern
    • Raise an Issue
    • Contribution Guidelines
Powered by GitBook
On this page
  • Hot-fix: CSP (26-06-2024)
  • Hot-fix: CSP (24-08-2023)
  • Hot-fix: 5.3.1 (05-07-2023)
  • Document Release Version
  • Hot Fix :- ML PII Data Product (23-06-2023)
  • Details of Released Tag
  • Flink Job Configurations for Lern:
  • Data Security Policy setup
  • Details of Released Tag
  • Summary of the Changes
  • New APIs to onboard
  • Env Configurations (Needs to be done before service deployment):
  • Exhaust Proxy API documentation
  • Data Security Policy setup
  • Steps to generate key pair for setting up Data Security policy configuration:
  • Flink Job Configurations for Lern:
  • Prerequired deployments for RC migration
  • Step to migrate old certificates to RC
  • Steps to Font URL migration

Was this helpful?

Edit on GitHub
  1. USE
  2. Release Notes

Release V 5.3.0

Hot-fix: CSP (26-06-2024)

Component
Build Job
Build Tag
Deploy Job
Deployment
Comment

ETLJobs

Build/DataPipeline/ETLJobs

Deploy/DataPipeline/ETLJobs

  • script_to_run: DRUID_CONTENT_INDEXER

  • invoke_type: deploy

ETLDruidContentIndexer

NA

NA

Deploy/DataPipeline/ETLDruidContentIndexer

  • script_to_run: DRUID_CONTENT_INDEXER

  • invoke_type: execute-script

Data Products

Build/Lern/LernDataProducts

Deploy/Lern/LernDataProducts

CSP related changes.

  • cloud_store_group_id: org.sunbird

  • cloud_store_artifact_id: cloud-store-sdk_2.12

  • cloud_store_version: 1.4.6

Note: While deploy select set the module value as lern-dataproducts,cronjobs

Ed data product in dock env

Build/Lern/LernDataProducts

Deploy/Dock/DataPipeline/EdDataProducts

CSP related changes.

  • cloud_store_group_id: org.sunbird

  • cloud_store_artifact_id: cloud-store-sdk_2.12

  • cloud_store_version: 1.4.6

Note: While deploy select set the module value as dock-dataproducts

To run Ed related reports: Live ETB QR Code-Content Linkage Status

NA

NA

Deploy/Lern/LernAnalyticsReplayJobs

Add the etb-metrics in job_id list. - job_type: run-job - job_id: etb-metrics

To run Ed related reports: Course Adoption Report v2

NA

NA

Deploy/DataPipeline/Runreport

report_id:

  • course_adoption_by_batch

  • course_adoption_table_new

  • course_adoption_report_plays_and_time_spent

To run coKreat related report: Visitor's report

NA

NA

Deploy/DataPipeline/Runreport

report_id: vidyadaan_visitor

To run coKreat related report: Collection Level Content Gaps

NA

NA

Deploy/Dock/DataPipeline/AnalyticsReplayJobs

  • job_type: run-job

  • job_id: sourcing-metrics

To run coKreat related report: Folder Level (first level) Content Gaps

NA

NA

Deploy/Dock/DataPipeline/AnalyticsReplayJobs

  • job_type: run-job

  • job_id: sourcing-metrics

To run coKreat related report: Project level funnel report

NA

NA

Deploy/Dock/DataPipeline/AnalyticsReplayJobs

  • job_type: dock-run-job

  • job_id: funnel-report

To run coKreat related report: Content Details Report

NA

NA

Deploy/Dock/DataPipeline/AnalyticsReplayJobs

  • job_type: dock-run-job

  • job_id: content-details

Hot-fix: CSP (24-08-2023)

Component
Build Job
Build Tag
Deploy Job
Deployment
Comment

Batch Service

Build/Core/Lms

Deploy/Kubernetes/Lms

CSP related changes.

  • cloud_store_group_id: org.sunbird

  • cloud_store_artifact_id: cloud-store-sdk

  • cloud_store_version: 1.4.7

User&Org Service

Build/Core/Learner

Deploy/Kubernetes/Learner

CSP related changes.

  • cloud_store_group_id: org.sunbird

  • cloud_store_artifact_id: cloud-store-sdk

  • cloud_store_version: 1.4.7

Data pipeline

Build/Lern/FlinkJobs

Deploy/Lern/FlinkJobs

CSP related changes.

  • cloud_store_group_id: org.sunbird

  • cloud_store_artifact_id: cloud-store-sdk_2.12

  • cloud_store_version: 1.4.6

Data Products

Build/Lern/LernDataProducts

Deploy/Lern/LernDataProducts

CSP related changes.

  • cloud_store_group_id: org.sunbird

  • cloud_store_artifact_id: cloud-store-sdk_2.12

  • cloud_store_version: 1.4.6

Jenkins Configurations for csp support:

Configure the variables cloud_store_group_id, cloud_store_artifact_id and cloud_store_version with proper values in the Jenkins, it can configured in the global or to the individual service of build job. For lms, user-org, flinks-jobs, lerndataproducts build jobs configure like as we mentioned below.

Configure the following values

Name - Default Value - Description

cloud_store_group_id - ${cloud_store_group_id} - Set the Cloud store sdk group id. e.g. org.sunbird cloud_store_artifact_id - ${cloud_store_artifact_id} - Set the Cloud store sdk artifact id. e.g. cloud-store-sdk

cloud_store_version - ${cloud_store_version} - Set the Cloud store sdk version. e.g 1.4.6

Config changes in Lern common.yaml for data-products

cloud_storage_report_verfication_bucketname: "" # default value is "report-verification"
dp_storage_endpoint_config: "" # default is "{{s3_storage_endpoint}}"

Hot-fix: 5.3.1 (05-07-2023)

Component
Build Job
Build Tag
Deploy Job
Deployment
Comment

Batch Service

Build/Core/Lms

Deploy/Kubernetes/Lms

Document Release Version

Project
Release Date
Version

Lern

27-May-2023

V 5.3.0

Lern

23-Jun-2023

V 5.3.1

Hot Fix :- ML PII Data Product (23-06-2023)

Details of Released Tag

Components
Build Jenkins Job
Build Tag
Deploy Jenkins Job
Deploy Tag
Comment

Kafka Setup

Deploy/Lern/KafkaSetup

verify if kafka topic = programuser.info is created or not

Data pipeline

Build/Lern/FlinkJobs

Deploy/Lern/FlinkJobs

Add program-user-info into job list and deploy it.

Data Products

Build/Lern/LernDataProducts

Deploy/Lern/LernDataProducts

Add program-user-exhaust into job list of Deploy/Lern/LernAnalyticsReplayJobs for running it.

Cassandra Migration

Build/Core/Cassandra

Deploy/Kubernetes/Cassandra

add the sunbird_programs keyspace in Deploy Jenkins jobs

Analytics

Deploy/Kubernetes/Analytics

Deploy with release-6.0.0 branch

Summary of the Changes

Details of the Changes:

LR-491 User detail (PII) report for ML programs - Data Product LR-285 User detail (PII) report for ML programs - Flink Job

Default values for config

default config for services

sunbird.program.report.keyspace="{{ program_keyspace }}"
ml.exhaust.store.prefix="ml_reports"

Please define below variables

program_keyspace: "sunbird_programs"
ml.exhaust.store.prefix="ml_reports"

Cassandra Keyspace and Table for Program:-

https://github.com/shikshalokam/sunbird-utils/blob/release-5.2.0/sunbird-cassandra-migration/cassandra-migration/src/main/resources/db/migration/cassandra/sunbird_programs/V1.1_cassandra.cql

Flink Job Configurations for Lern:

Name of the Flink Job added

program-user-info

LR-285 - User detail flink job for ML-programs - setup/configuration details:

For this ticket, we have only done unit testing with the help of simulated events. Integration testing has not been done as the required workflows concerning this will only be enabled after Ed 6.0 release. As part of this ticket we have enabled new Flink jobs and they in no way impact any existing workflows Job name: program-user-info

The purpose of this job is to record the user's information when the user submits the program. Whenever a program is submitted, this job receives an event with the user's information as JSON data and then it parses and stores it as respective key-value pairs in Cassandra.

Keyspace name: sunbird_program

Schema of the Kafka Topic: Kafka Topic Name: {{envName}}.programuser.info Event Structure:-

{
      programId: {
        type : "ObjectId",
        required : true,
        index: true
      },
      programName: String,
      programExternalId: String,
      noOfResourcesStarted: {
        type:Number,
        index: true
        }
      userId: {
        type: String,
        index: true
      },
      requestForPIIConsent:true/false
      userProfile: Object,
      userRoleInformation: Object,
      appInformation: Object,
      createdAt: Date,
      updatedAt: Date,
      deleted:Boolean
}

Job Configurations:

kafka {
 input.topic = ${job.env}".programuser.info"
 groupId = ${job.env}"-programuser-group"
}
task {
 consumer.parallelism = 1
 downstream.parallelism = 1
 programUser{
  parallelism = 1
 }
}
ml-cassandra {
 keyspace = "sunbird_programs"
 table = "program_enrollment"
 port = "9042"
 host =
 }

Flink build Jenkins job name: /Build/job/Lern/job/FlinkJobs

Flink deploy Jenkins job name: /Deploy/job/<environment>/job/Lern/job/FlinkJobs/program-user-info

Jenkins job for building Cassandra: /Build/job/Core/job/Cassandra/

Jenkins job for deploying Cassandra: /Deploy/job/<environment>/job/Kubernetes/job/Cassandra

Data Security Policy setup

Configurations to be done by System admin:

  1. Setup default 'Data Security Policy' settings using tenant preference API.

curl --location --request PATCH '{{host}}/api/org/v2/preferences/update' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "orgId": "default",
        "key": "dataSecurityPolicy",
        "data": {
            "level": "PLAIN_DATASET",
            "dataEncrypted": "No",
            "comments": "Data is not encrypted",
            "job": {
                    "userinfo-exhaust": {
                        "level": "PASSWORD_PROTECTED_DATASET",
                        "dataEncrypted": "No",
                        "comments": "Password protected file."
                    },
                    "program-user-exhaust": {
                        "level": "PASSWORD_PROTECTED_DATASET",
                        "dataEncrypted": "No",
                        "comments": "Password protected file."
                    }
                },
            "securityLevels": {
                "PLAIN_DATASET": "Data is present in plain text/zip. Generally applicable to open datasets.",
                "PASSWORD_PROTECTED_DATASET": "Password protected zip file. Generally applicable to non PII data sets but can contain sensitive information which may not be considered open.",
                "TEXT_KEY_ENCRYPTED_DATASET": "Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open.",
                "PUBLIC_KEY_ENCRYPTED_DATASET": "Data encrypted via an org provided public/private key. Generally applicable to all PII data exhaust."
            }
        }
    }
}'

Details of Released Tag

Components
Build Jenkins Job
Build Tag
Deploy Jenkins Job
Deploy Tag
Comment

Kafka Setup

Deploy/Lern/KafkaSetup

Data pipeline

Build/Lern/FlinkJobs

Deploy/Lern/FlinkJobs

Add legacy-certificate-migrator into job list and deploy it.

Data Products

Build/Lern/LernDataProducts

Deploy/Lern/LernDataProducts

Batch Service

Build/Core/Lms

Deploy/Kubernetes/Lms

User&Org Service

Build/Core/Learner

Deploy/Kubernetes/Learner

Analytics

Deploy/Kubernetes/Analytics

Deploy with release-6.0.0 branch

Summary of the Changes

Details of the Changes:

LR-436 OldCertificateMigration spark data-product LR-437 LegacyCertificateMigrator Flink job LR-438 Sunbird RC changes for updating schema for issued date LR-330 Certificate template font url migration LR-395, LR-465 PII data security LR-451 Local setup of Data-pipeline - Ubuntu & Mac - Github and Microsite update LR-443 Local setup of UserOrg - Ubuntu & Mac - Github and Microsite update LR-445 Local setup of LMS - Ubuntu & Mac - Github and Microsite update LR-422 Point the channel create API to content-service instead of learning-service LR-519 Textbook APIs code cleanup from Course-Batch service LR-486 Microsite update with Certificate generation flow diagram LR-520 Group service - activity type should be case insensitive LR-556 Local setup of LMS - Ubuntu & Mac - Mock service setup LR-456 Local setup of Sunbird-utils - Ubuntu & Mac - Github and Microsite update

New APIs to onboard

- name: exhaustSubmitProxyAPI
  uris: "{{ course_service_prefix }}/v1/jobrequest/submit"
  upstream_url: "{{ lms_service_url }}/v1/jobrequest/submit"
  strip_uri: true
  plugins:
  - name: jwt
  - name: cors
  - "{{ statsd_pulgin }}"
  - name: acl
    config.whitelist:
    - courseAccess
  - name: rate-limiting
    config.policy: local
    config.hour: "{{ medium_rate_limit_per_hour }}"
    config.limit_by: credential
  - name: request-size-limiting
    config.allowed_payload_size: "{{ small_request_size_limit }}"
  - name: opa-checks
    config.required: false
    config.enabled: false

- name: exhaustListProxyAPI
  uris: "{{ course_service_prefix }}/v1/jobrequest/list"
  upstream_url: "{{ lms_service_url }}/v1/jobrequest/list"
  strip_uri: true
  plugins:
  - name: jwt
  - name: cors
  - "{{ statsd_pulgin }}"
  - name: acl
    config.whitelist:
    - courseAccess
  - name: rate-limiting
    config.policy: local
    config.hour: "{{ medium_rate_limit_per_hour }}"
    config.limit_by: credential
  - name: request-size-limiting
    config.allowed_payload_size: "{{ small_request_size_limit }}"
  - name: opa-checks
    config.required: false
    config.enabled: false
    
- name: orgAddEncryptionKey
  uris: "{{ org_service_prefix }}/v1/update/encryptionkey"
  upstream_url: "{{ learning_service_url }}/v1/org/update/encryptionkey"
  strip_uri: true
  plugins:
  - name: jwt
  - name: cors
  - "{{ statsd_pulgin }}"
  - name: acl
    config.whitelist:
    - orgSuperAdmin
  - name: rate-limiting
    config.policy: local
    config.hour: "{{ medium_rate_limit_per_hour }}"
    config.limit_by: credential
  - name: request-size-limiting
    config.allowed_payload_size: "{{ small_request_size_limit }}"
  - name: opa-checks
    config.required: false
    config.enabled: false    

Env Configurations (Needs to be done before service deployment):

The below environment variable needs to be configured in the 'sunbird-lms-service.env' file dev ops repo. Ref: https://github.com/project-sunbird/sunbird-devops/blob/release-5.3.0-lern/ansible/roles/stack-sunbird/templates/sunbird_lms-service.env

Variable Name
Values
Comments

exhaust_api_base_url

Obsrv exhaust API endpoint for batch service

exhaust_api_submit_endpoint

/request/submit

To submit job request from batch service

exhaust_api_list_endpoint

/request/list/

To list job request from batch service

sunbird_api_auth_token

"{{ core_vault_sunbird_api_auth_token }}"

Authentication token for APIs

content_read_url

/content/v3/read/

Exhaust Proxy API documentation

https://github.com/Sunbird-Lern/sunbird-course-service/blob/release-5.3.0/api-tests/Collection/Proxy%20Exhaust%20APIs.postman_collection.json

Data Security Policy setup

Configurations to be done by System admin:

  1. Execute CURL for providing link to download "Decryption Tool". Tool reference: https://github.com/Sunbird-Lern/sunbird-utils/blob/release-5.3.0/decryption-tool/decryption-tool.zip

Please upload the tool to your public cloud location or to your repository and provide the link to the same in below system setting variable value.

curl --location --request POST '{{host}}/api/data/v1/system/settings/set' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{api_key}}' \
--header 'x-authenticated-user-token: {{user_token}}' \
--data-raw '{
    "request": {
        "id": "decryptionToolLink",
        "field": "decryptionToolLink",
        "value": "{\"link\":\"<link to download decryption tool>\", \"Comments\": \"To use this tool, run the command with encrypted file and key to decrypt\"}"
    }
}'
  1. Setup default 'Data Security Policy' settings using tenant preference API.

curl --location --request POST '{{host}}/api/org/v2/preferences/create' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "orgId": "default",
        "key": "dataSecurityPolicy",
        "data": {
            "level": "PLAIN_DATASET",
            "dataEncrypted": "No",
            "comments": "Data is not encrypted",
            "job": {
                    "userinfo-exhaust": {
                        "level": "PASSWORD_PROTECTED_DATASET",
                        "dataEncrypted": "No",
                        "comments": "Password protected file."
                    }
                },
            "securityLevels": {
                "PLAIN_DATASET": "Data is present in plain text/zip. Generally applicable to open datasets.",
                "PASSWORD_PROTECTED_DATASET": "Password protected zip file. Generally applicable to non PII data sets but can contain sensitive information which may not be considered open.",
                "TEXT_KEY_ENCRYPTED_DATASET": "Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open.",
                "PUBLIC_KEY_ENCRYPTED_DATASET": "Data encrypted via an org provided public/private key. Generally applicable to all PII data exhaust."
            }
        }
    }
}'
  1. Setup default 'PII data security settings' using tenant preference API.

curl --location --request POST '{{host}}/api/org/v2/preferences/create' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "orgId": "default",
        "key": "userPrivateFields",
        "data": {
            "PIIFields": [
                "email",
                "phone",
                "userName",
                "prevUsedEmail",
                "prevUsedPhone",
                "recoveryEmail",
                "recoveryPhone"
            ]
        }
    }
}'

Configurations that can be done by Tenants:

  1. Use Tenant preference create API to create tenant specific 'Data Security Policy' settings similar to 'default' Data Security Policy settings but with tenant orgId.

Note: 
a. Tenant level security cannot be lower than 'default' Data Security Policy'.
b. Job Level security Policy in a Tenant specific configuration cannot be lower than Tenant Level configuration and cannot be lower than job level configuration in 'default' Data Security Policy'.
c. Below mapping shows the priority/grade of security policies 
"PLAIN_DATASET" < "PASSWORD_PROTECTED_DATASET" < "TEXT_KEY_ENCRYPTED_DATASET" < "PUBLIC_KEY_ENCRYPTED_DATASET"
  1. In order to use "PUBLIC_KEY_ENCRYPTED_DATASET" security configuration for an exhaust report, tenant admin should have uploaded public pem key file using below API.

curl --location --request PATCH '{{host}}/api/org/v1/update/encryptionkey' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{kong_api_token}}' \
--header 'x-authenticated-user-token: {{user_authentication_token}}' \
--form 'organisationId={{org_id}}' \
--form 'encryptionKey=@path_to_public_pem_file'

Steps to generate key pair for setting up Data Security policy configuration:

For Linux and Mac OS:

  1. To generate Private Key

openssl genrsa -out private.pem 4096
  1. To generate Public Key

openssl rsa -in private.pem -pubout -outform PEM -out public_key.pem

For Windows OS:

Please install GitBash: The Git installation package comes with SSH. Using Git Bash, which is the Git command line tool, you can generate SSH key pairs. Git Bash has an SSH client that enables you to connect to and interact with Triton containers on Windows.

To install Git:

  1. Download and initiate the Git installer.

  2. When prompted, accept the default components by clicking Next.

  3. Choose the default text editor. If you have Notepad++ installed, select Notepad++ and click Next.

  4. Select to Use Git from the Windows Command Prompt and click Next.

  5. Select to Use OpenSSL library and click Next.

  6. Select to Checkout Windows-style, commit Unix-style line endings and click Next.

  7. Select to Use MinTTY (The default terminal of mYSYS2) and click Next.

  8. Accept the default extra option configuration by clicking Install. When the installation completes, you may need to restart Windows.

Launching GitBash:

  1. press Start+R to launch the Run dialog.

  2. Type C:\Program Files\Git\bin\bash.exe and press Enter.

Generating Key pair:

  1. To generate Private Key

openssl genrsa -out private.pem 4096
  1. To generate Public Key

openssl rsa -in private.pem -pubout -outform PEM -out public_key.pem

Flink Job Configurations for Lern:

Name of the Flink Job added

legacy-certificate-migrator

Prerequired deployments for RC migration

LR-436 - Deploy Data-product

Data-product build Jenkins job: Build/Lern/LernDataProducts

Deploy Jenkins job: Deploy/{{env}}/Lern/LernDataProducts

LR-437 - Deploy legacy-certificate-migrator Flink job

Build Jenkins job: /Build/job/Lern/job/FlinkJobs

Deploy Jenkins job: /Deploy/job/<environment>/job/Lern/job/FlinkJobs

LR-438 - Update RC schema

Step 1 : Upload updated schema files. Deploy Jenkins job: Deploy/dev/Sunbird-RC/Upload_RC_Schema

Note: Since certificate signer service will cache the credential template. please make sure the credential template is updated in the respective path as per below file.

https://github.com/project-sunbird/sunbird-devops/blob/release-5.3.0-lern/utils/sunbird-RC/schema/credential_template.json

Step 2 : Deploy certificate signer service

Jenkins Job: Deploy/dev/Sunbird-RC/CertificateSign

Step to migrate old certificates to RC

Sunbird Lern BB is using Sunbird RC for generating & issuing e-credentials in its use cases (e.g.: course completion certificate) for all the latest completed courses (post March-2022). All the old certificates were custom generated and stored in Cassandra and cloud storage.

Once we migrate these certificates then we no longer need to store certificates in Cassandra and all the certificates will be using Sunbird RC going forward.

Reference Link: https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3117416449/LR-4+Design+of+migrating+existing+certificate+in+to+RC

Note: After migrating old certificates to RC, certificate verification of old certificates will become invalid. To support to old certificate verification, Sunbird ED building block is implementating in portal service in release 6.0. Kindly find the ticket in this link. So recommended to migrate the certificates after getting the old certification verification support as well.

Step 1

Create Kafka topic for only the purpose of this migration process

Topic name: {{env}}.legacy.certificate.migrate

Step 2

In the spark machine, update the old-certificate-migration-job model config in mount/data/analytics/scripts/lern-model-config.sh with correct values.

Sample model config:

{"search":{"type":"none"},"model":"org.sunbird.lms.audit.OldCertificateMigrationJob","modelParams":{"mode":"execute","store":"azure","sparkCassandraConnectionHost":"10.5.3.17", "cert_base_path": "https://dev.lern.sunbird.org", "cloud_storage_base_url": "https://sunbirddev.blob.core.windows.net", "cloud_store_base_path_placeholder": "CLOUD_BASE_PATH","content_cloud_storage_container": "sunbird-content-staging", "cloud_storage_cname_url": "https://obj.stage.sunbirded.org", "batchId": "01320961460024934435", "kafka_broker": "localhost:9092", "kafka_topic": "sunbirddevlern.legacy.certificate.migrate","output_file_path":"./reports/"},"parallelization":8,"appName":"OldCertificateMigrationJob"}

Note: migration job can be run single batch with "batchId": "01320961460024934435" and multiple batches with "batchId": "01320961460024934435,01220961460024934536" and for all batches with "batchId": "all" .

Step 3

Run the job with the below command in the spark machine.

/mount/data/analytics/scripts/lern-run-job.sh old-certificate-migration-job &

Note: logs can be found in below locations,

Joblog: /mount/data/analytics/scripts/logs/joblog.log

Execution log: /mount/data/analytics/logs/lern-data-products/{current_date}-job-execution.log

Note:

Verification steps can be found in the design page: https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3117416449/LR-4+Design+of+migrating+existing+certificate+in+to+RC#Verification-steps-for-the-certificate-migration-process

Steps to Font URL migration

All the templates are having dev URLs configured for Fonts in all the environments as per our observation. All these font URLs have to be migrated to the new cname URL

Note: Before font url migration, make sure all the font files are available at cname mapped account or cloud storage container. To verify, where the font files are available, open any svg template file in editor and check the font URL's host.

Please use java 11 for running the scripts

Step 1:

Download SVG file migrator and uploader jars by below command,

cd ~
mkdir svg_template_migration
cd svg_template_migration
wget "https://github.com/kumarks1122/sunbird-utils/raw/release-5.3.0-font-url-migration/svg_template_migration/template-migration/svg-migrator.jar"
wget "https://github.com/kumarks1122/sunbird-utils/raw/release-5.3.0-font-url-migration/svg_template_migration/template-upload/svg-uploader.jar"

Step 2:

Download the svg template files and update the font URLs in the template files.

java -jar svg-migrator.jar "{{ content search host }}" "0" "1000" "font_migration" "{{ Old URL }}" "{{ cname url }}"

#EXAMPLE
#java -jar svg-migrator.jar "dev.lern.sunbird.org" "0" "1000" "font_migration" "https://sunbirddev.blob.core.windows.net" "https://obj.diksha.gov.in"

Note: Before moving to next step, please verify atleast one svg file for whether the font URL got updated.

Step 3:

Upload the svg template files back to the cloud storage by below command.

java -jar svg-uploader.jar "{{ content search host }}" "0" "1000" "{{ storage key}}" "{{ storage secret }}" "{{svg file path}}" "{{storage type: (azure,..)}}" "{{ CSP endpoint (based on CSP it is optional) }}" "{{ region (based on CSP it is optional) }}"

#EXAMPLE
#java -jar svg-uploader.jar "dev.lern.sunbird.org" "0" "5" "sunbirddevbbpublic" "{{ secret }}" "/Users/{{username}}/svg_template_migration" "azure"

Configuration for making content read URL dynamic: https://project-sunbird.atlassian.net/browse/LR-579 Please define the below URL in the sunbird_lms-service.env file this will make the content read endpoint URL Configurable.

content_read_url=/content/v3/read/
PreviousRelease V 5.4.0NextRelease V 5.2.0

Last updated 11 months ago

Was this helpful?

QR Codes Image download Issue fix Bug:

{{ analytics_service_url | default('') }}

release-5.1.1_RC1
release-5.2.0_RC7
release-5.2.0_RC7
release-5.3.1_RC11
release-5.3.1_RC11
release-5.3.1_RC11
release-5.3.1_RC11
release-5.3.1_RC11
release-5.2.0_RC7
release-5.2.0_RC7
release-5.3.1_RC11
release-5.3.1_RC11
release-5.3.1_RC11
release-5.3.1_RC11
release-5.3.2_RC1
release-5.3.0-lern
release-5.3.1_RC1
release-5.3.0-lern
release-5.3.1_RC2
release-5.3.1_RC2
release-5.3.1_RC10
release-5.3.1_RC10
release-5.3.1_RC1
release-5.3.0-lern
https://project-sunbird.atlassian.net/browse/KN-889
release-5.3.0_RC5
release-5.3.0_RC5
release-5.3.0_RC6
release-5.3.0_RC6
release-5.3.0_RC1
release-5.3.0-lern
release-5.3.0_RC3
release-5.3.0_RC5
release-5.3.0_RC5
release-5.3.0_RC6
release-5.3.0_RC6
release-5.3.0_RC1
release-5.3.0-lern
release-5.3.0_RC2
release-5.3.0-lern
http://analytics-service:9000
Jenkins configuration