# Overview Sunbird Lern is primarily a set of microservices along with some reference UX widgets that offer capabilities to enable tracking of a user’s learning journey as well as enable interactions and collaboration among users. These capabilities can be leveraged by any adopter with requirements for measurement of learning progress, creation and management of cohorts/ batches of learners, as well as enabling collaboration capabilities such as Groups or Discussion forums. The Lern services are also employed for creation and management of user accounts and organizations within a Sunbird instance. A few examples of solutions that can be assembled with Lern include: * Integration of Sunbird to an existing system to allow valid users with accounts on the system to login via single sign-on to Sunbird * Bulk-create a set of user accounts for a set of participants in a workshop that uses an online platform/ Sunbird * Allow for generation of and issuing of certificates to participants of a course in a workshop that uses an online platform/ Sunbird * Assign rights (say content creation rights) to an identified set of users for say a content creation workshop * Assign a group of learners to a batch and allow them take a course within specified start-end dates. * Enable a discussion forum for a Batch of users taking a course or a Group of users so that participants can collaborate * Create a group of users, assign learning material, courses and quizzes to them, and track their progress or scores against Courses. * Send notifications to a set of users about new content / activities that have been added on the platform Sunbird Lern can be utilized to enable learning solutions that necessitate tracking of user progress and scores, issuance of certificates based on some criteria as well as to form defined Groups of users, or permit collaboration between them via forums. ### **Key Capabilities that Sunbird Lern can enable:** * User Account Creation & Administration Capabilities: New user account creation, user login/ authentication and user profile with passbook. Assigning and management of user roles. Management of location and other master data. Enable login via various methods including userID/ password as well as SSO via Google or similar systems * Launch Courses via Batches: Create and manage course batches, review user progress and performance on assessments, and issue rule-based certificate * Engagement & Collaboration: Engage users through groups and discussion forums, events, and notifications. * Notifications for users: Send notifications to users to inform them about programs deadlines or for system level activity such as generation of a certificate or creation of an account. \_\_ Sunbird Lern is leveraged in DIKSHA - Digital Infrastructure for Knowledge Sharing, the national school education platform of India. [**Contribute to Sunbird Lern**](https://github.com/orgs/Sunbird-Lern/discussions/15) ***Contributors:*** EkStep # Functional Capabilities Functional capabilities enabled by Sunbird Lean These functional capabilities are powered by different Sunbird Lern components - which can be configured by an adopter based on their specific needs. * **User account creation and management, user login as well as platform administration capabilities** - these are powered by the [User & Org Service](/use/developer-guide/user-and-org-service) component of Sunbird Lern * Enable users to create accounts to save their platform preferences, access relevant content based on their preferences etc. Users with accounts on the system who login also get access to richer platform features such as Courses and Learner passbook. * Platform administrators can be granted capabilities to manage user roles on the platform, as well as manage platform master data (eg. Location data, Framework values etc.) * Configure the platform to allow for user logins via various mechanisms including username/ password, Google login or single sign-on with other approved systems * **Enable batches of students to take courses on the platform**: Create and manage a course batch - this allows for a cohort of users to take the course within a specified time frame, review user course progress or assessment performance, issue rule based certificates for course completion, merit scores etc. The [Batch Service](/use/developer-guide/lms-service) component of the Lern Building block is used to power these capabilities within Sunbird Lern. * **Allow users on the platform to collaborate**: Sunbird Lern components can be configured to give users on the platform collaboration capabilities. These components include the Groups Service as well as Discussion Forums. The functionalities enabled by the Groups component includes the ability for users to create a Group on the platform, add/ remove users from the group, as well as assign activities (content, assessments etc) for the users in the group to undertake. The Group creator is also able to track how much of the activities the users have completed. A Group can also have a Dicussion Forum attached to it, which allows a user to start or contribute to discussions on relevant topics. Such capabilities allow for users to learn together as a cohort, as well as engage in useful collaborative activities to further their learning. The components used to enable collaboration include [Groups](/use/developer-guide/groups) as well as [Discussion Forum](/use/developer-guide/discussion-forum). * **Configure the platform to be able to send notifications to users on events of choosing** - these could be user driven or system driven events. This capability is enabled via the [Notification Service](/use/developer-guide/notification-service) component. Various components of Lern permit for different configurations, allowing the system to enable different workflows based on specific requirements. Learn more about the configurations that each component allows for, on the pages for each of the specific components. # Technical Architecture # Technical Architecture Diagram **Microservices:** LERN BB provides the following services: - User-Org Service, LMS (Batch) Service, Groups Service, Notification Service and Discussion Forum **Dependencies:** LERN BB Microservices has dependencies on other BB microservices as well as intra-BB Services based. Individual micro-service architecture will provide the exact picture of dependcies of respective micro-service. **Databases:** LERN BB use Cassandra database as the primary database across its micro-services; Redis for data caching mechanism; Elasticsearch for enabling data search capabilities; Druid and Cloud Storage for reports generation and storage; And, Postgres for KC integration and for storing reports process information. **Jobs:** We have flink jobs that reads events from Kafka for processing data in the background to enable multiple functionalities. **Reports:** The functional and business metrics are computed by fetching the data from various data sources using Spark and executed as Batch processing jobs.

Lern Technical Architecture

The diagram represents the components involved and their arrangement in **Lern**. * **Services** has build on play framework using akka actor. * ***Cassandra*** is a reference to Apache Cassandra, which is an open-source distributed NoSQL database. Cassandra is designed to handle large amounts of data across multiple servers while providing high availability and fault tolerance. * ***Elasticsearch*** is a distributed, open-source search and analytics engine. It is built on top of Apache Lucene, and it is designed to be scalable, fault-tolerant, and highly available. By leveraging ***Elasticsearch***, we can deliver an efficient and robust search experience to our users, ensuring that our application performs exceptionally well and remains flexible to adapt to future needs. * ***Redis*** is a popular open-source, in-memory data structure store often referred to as a "data structure server." It is designed for speed, efficiency, and versatility. ***Redis*** stores data in RAM providing extremely fast read and write operations. * By leveraging **Flink**, we enable real-time and batch data processing with low latency, high throughput, and fault tolerance. Lern uses Flink jobs for asynchronous processes. ### **Overall Architecture of Reports (Data Products):**

Data-Product Architecture

Reports (Data products) refer to products or services derived from data analysis or processing. They are typically created to provide valuable insights. Providing you with the list of data products available in the Sunbird platform below
{% embed url="" %} Lern Highlevel Architecture {% endembed %} # Tech Stack Complete Tech Stack used in Sunbird Lern Building Block
# Dependencies Dependencies of Sunbird Lern Building Block with Other Sunbird BBs Sunbird LERN uses other Sunbird Building Blocks to enable the important capabilites. The dependent building block details are listed below: ## Sunbird Knowlg Content-service in Sunbird Knowlg is used for creating, updating and validating the channel info during organisation creation. Also framework APIs in Sunbird Knowlg is used for validating the user framework selection. {% hint style="success" %} **Resolution:** Sunbird Knowlg : Content service provides capabilities such as creating content against channel and associating a framework to channel using channel configurations. Sunbird Lern - User\&Org service is user and organisation management system. It can be used independently to maintain user and organisation details and their association if some changes are made to organisation management apis to remove the content service dependency. {% endhint %} ## Sunbird RC *Credential Service* and *Credential Registry* in Sunbird RC is used by the batch service to generate certificates for the user. {% hint style="success" %} **Resolution:** Similar adopter service can be configured in Batch service to generate certificates instead of using Sunbird RC {% endhint %} ## Sunbird Ed Form APIs from Sunbird Ed is used to validate the profile information of the user. {% hint style="success" %} **Resolution:** User create and Update APIs in User& Org service will need to have Sunbird Ed dependency only if profileUserType and profileLocation need to be saved with user info. {% endhint %} ## Sunbird Telemetry Sunbird Telemetry is a specification to instrument all the key events. Using this specification reference applications & services will generate telemetry events. {% hint style="success" %} **Resolution:** Sunbird Lern generates various telemetry events for logging, audit, monitoring and tracing purpose. Currently the spec is tightly coupled with code base. {% endhint %} ## Sunbird Obsrv Telemetry service and data pipeline of Sunbird Obsrv is used to process and store the telemetry events and custom data products. {% hint style="success" %} **Resolution:** Any other service which can process the telemetry events generated by Sunbird Lern can be used instead of Sunbird Obsrv. {% endhint %} ## Sunbird inQuiry *Lern* uses Sunbird Inquiry to fetch meta data for QuestionSets to fetch the metadata of QuestionSet. # Product Roadmap Use this link to see the Jira Board that has the [Sunbird Lern roadmap](https://project-sunbird.atlassian.net/jira/software/c/projects/LR/boards/102/roadmap). The roadmap has the list of Epics that are to be taken up, as well as form the backlog for the Sunbird Lern team Sunbird Lern [ISSUE TRACKER](https://github.com/Sunbird-Lern/Community/issues) : This is the link to the set of issues/ submissions or requests that are being considered for development as part of the Sunbird Lern roadmap. You can upvote an issue if you find it relevant, or add a new issue to the list **Release-5.4.0** **(Planned release date - 07 Jul '23)** Release Plan: Design Discussions : 15 May '23 - 26 May '23 (2 weeks)\ Sprint 01 : 29 May '23 - 09 Jun '23 (2 weeks)\ Sprint 02 : 12 Jun '23 - 32 Jun '23 (2 weeks)\ Regression & Testing : 26 Jun '23 to 06 Jul (2 weeks)\ Production Release : 07 Jul '23 `Release List for Lern 5.4 on` [`Jira`](https://project-sunbird.atlassian.net/projects/LR/versions/10260/tab/release-report-all-issues) **Release-5.3.0** **(Planned release date - 26 May '23)** Release plan: Design Discussions : 16 Jan '23 - 27 Jan '23 (2 weeks)\ Sprint 01 : 17 Apr '23 - 28 Apr '23 (2 weeks)\ Sprint 02 : 01 May '23 - 12 May '23 (2 weeks)\ Regression & Testing : 15 May '23 to 25 May (2 weeks)\ Production Release : 26 May '23 `UPDATE:` `Actual release date for 5.3.0 : 27 May '23 (`[`Discussion Forum Update`](https://github.com/orgs/Sunbird-Lern/discussions/128)`)` `Release List for Lern 5.3 on` [`Jira`](https://project-sunbird.atlassian.net/projects/LR/versions/10233/tab/release-report-all-issues) **Release-5.2.0** **(Planned release date - 27 Mar '23)** [Click here](https://project-sunbird.atlassian.net/issues/?filter=12735) to see the list of issues planned for SB Lern Release 5.2 The following are the planned release dates of release 5.2: Design Discussions : 16 Jan '23 - 27 Jan '23 (2 weeks)\ Sprint 01 : 30 Jan '23 - 17 Feb '23 (3 weeks)\ Sprint 02 : 20 Feb '23 - 10 Mar '23 (3 weeks)\ Regression & Testing : 13 March '23 to '24 March (2 weeks)\ Production Release : 27 March '23\ Bug Fixes & Support : 28 March ' 23 to 31 March '23 (1 week) UPDATE: `Actual Release date for 5.2.0 : 06 April '23 (see Discussion Forum update` [`here`](https://github.com/orgs/Sunbird-Lern/discussions/59)`)`\ \ `Release list for Lern 5.2 on` [`Jira`](https://project-sunbird.atlassian.net/projects/LR/versions/10232/tab/release-report-all-issues) [**Release-5.1.0**](https://project-sunbird.atlassian.net/issues/?filter=12607) **(Planned release date - 13 Jan '23)** **Project: Making SB Lern Cloud Agnostic** **Task : Completion of Data migration (**[**LR-254**](https://project-sunbird.atlassian.net/browse/LR-4)**)** CSP data migration task for certificates in RC. **Project: Sunbird Lern integration with SB RC** **Task : Completion of Sunbird RC integration with Lern (**[**LR-4**](https://project-sunbird.atlassian.net/browse/LR-4)**,** [**LR-6**](https://project-sunbird.atlassian.net/browse/LR-6)**)** Integration of Sunbird RC with Sunbird Lern has been taken up over the last 2 releases in order to facilitate Registry driven credentials. A few pending backlog items in this project will be completed as part of Release 5.1.0, including migration of certificates issued so far to Sunbird RC. **Project: New Feature Development** **Task: Support for Optional material for Courses (**[**LR-1**](https://project-sunbird.atlassian.net/browse/LR-1)**)** All Course content is considered as mandatory for course completion as of today. There is a functional need called out for being able to add 'optional' material to courses - such that it does not contribute to course progress. This project will cover Lern changes that need to be done in order to enable Optional material for courses **Task : Course progress exhaust to capture number of attempts (**[**LR-127**](https://project-sunbird.atlassian.net/browse/LR-127)**)** The current Course progress exhaust is required to be enhanced to capture the number of attempts that a user has made against the assessment tagged to the course. **Project: Enabling ease of Adoption** **Task: Refactoring of SB Lern Batch Service (**[**LR-131**](https://project-sunbird.atlassian.net/browse/LR-131)**)** There are a few dependencies for the course service APIs and the DB layers that need to be resolved in order to remove the dependecies that Lern has on SB Obsrv and Knowlg. **Note :** The team was engaged with an interrupt - i.e. work on [CSP support](https://project-sunbird.atlassian.net/browse/LR-147) related items over the months of September and October 2022. The next release for SB Lern will be 5.1, for which Sprint 1 will commence on 31 Oct 2022. [**Release-5.0.0**](https://project-sunbird.atlassian.net/issues/?filter=12509) **(Planned release date - 19 Aug'22)** | **1.** [**Design and plan for decoupling Lern building block from other BB repos and creating its own set up**](https://project-sunbird.atlassian.net/browse/SB-30063) | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | The code base of some of the components are spread across different building blocks. This makes the installation and setup difficult. Due to this PR approval and merge takes time. Decoupling the code from other BB repos will help to maintain the repos and code most efficiently. For example, batch service code is spread across many repositories like Sunbird Knowlg, Sunbird Obsrv. Also sunbird-utils repo has the cassandra migration scripts from all components - this needs to be separated. | | **2.** [**Stabilising of components - increase code coverage and unit test cases**](https://project-sunbird.atlassian.net/browse/SB-30072) | | The code base needs to be further stabilised by bringing in some design changes and also by increasing code coverage and unit tests in components like batch service, notification service etc | | **3.** [**Deployment and Release processes**](https://project-sunbird.atlassian.net/browse/SB-30066) | | i. Build, Deploy and provisioning scripts : refactoring of the provisioning and deployment scripts for the BB | | ii. Be able to deploy existing microservices into a different namespace (SB Ed) | |

4. Making SB Lern Cloud agnostic (LR-113, LR-112, LR-111, LR-110, LR-109, LR-108, LR-107, LR-106, LR-105, LR-104, LR-103, LR-125, LR-128)

SB Lern currently has code that is specific to one CSP - this effort will ensure that such dependencies are removed, and SB Lern can function in a cloud agnostic fashion.

| #### [Sunbird Lern Backlog](https://project-sunbird.atlassian.net/issues/?filter=12361) This is the list of approved backlog items that can be picked up by the Sunbird Lern team as well as other contributors or adopters for development and submission as part of the Lern building block. # Overview Using the Sunbird Lern components: **Batch Service** The Batch Service APIs depend on the content-service APIs provided by the SB Knowlg building block (for content consumption details) as well as the SB Lern UserOrg APIs (for details of the users in the batch). There are also additional dependencies on SB Knowlg as well as the Druid Service in SB Obsrv \ **Groups** The Groups Service component has a dependency on the UserOrg component as well as the SB Knowlg building block Member details in a group are fetched using the User Org service, and addition of activities requires use of the content-service component from the SB Knowlg BB **Discussion Forum** The discussion forum component does not have any other cross dependencies, and can be installed independently **Notification Service** The Notification Service component can be installed independently, and used to send Email/SMS/FCM and feed/in-app notifications synchronously or asynchronously **User & Org Service** The UserOrg component can be installed independently. Note: The User update API has a cross building-block dependency Sunbird Ed. It uses the Sunbird Ed form API for user type as well as location type validation, for use cases specific to SB Ed. An adopter can choose to address this dependency on SB Ed by setting up the Form API configs as per their needs and workflows. {% hint style="info" %} For developer installation and setup of individual lern components please refer to [Sunbird Lern Developer Installation](https://lern.sunbird.org/use/developer-installation) documentation. {% endhint %} {% hint style="info" %} Please refer to the [Sunbird-Ed Deployment ](https://ed.sunbird.org/use/prerequisites-for-your-own-sunbird-ed-instance)for more details on deployment. {% endhint %} The deployment view diagram shown below explains how the Lern Building block components are deployed in SunbirdEd. ![](/files/kysNultwazjGesjLNTcL) # Release Notes In the subsequent pages you will find detailed documentation of these releases: {% content-ref url="/pages/zrgyk2XDr1Wt6ZRdD5vj" %} [Release V 4.7.0](/use/release-notes/release-v-4.7.0-live) {% endcontent-ref %} {% content-ref url="/pages/ffRVhDAlpabYHsTaqiaJ" %} [Release V 4.8.0](/use/release-notes/release-v-4.8.0-upcoming) {% endcontent-ref %} {% content-ref url="/pages/93znrEofS4z9egH2UxOB" %} [Release V 4.9.0](/use/release-notes/release-v-4.9.0) {% endcontent-ref %} {% content-ref url="/pages/aAoX406sevghooFncGXu" %} [Release V 4.10.0](/use/release-notes/release-v-4.10.0-latest) {% endcontent-ref %} {% content-ref url="/pages/jCp71axhPKq52Peu2cQ5" %} [Release V 5.0.0](/use/release-notes/release-v-5.0.0-latest) {% endcontent-ref %} {% content-ref url="/pages/SdG1QhXUV9kDGkxRJRgV" %} [Release V 5.0.1](/use/release-notes/release-v-5.0.1) {% endcontent-ref %} {% content-ref url="/pages/UNGTtsorUYz0aBOKBBWY" %} [Release V 5.1.0](/use/release-notes/release-v-5.1.0) {% endcontent-ref %} {% content-ref url="/pages/zyItu9QhySFe2gCn3bjM" %} [Release V 5.2.0](/use/release-notes/release-v-5.2.0) {% endcontent-ref %} # Release V 8.0.0 (Ongoing) ## Document Release Version | Project | Release Date | Version | | ------- | ------------ | ------- | | Lern | | V 8.0.0 | ## Overview This release contains software upgrade and ownership transfer functionality. ## New Features ### Ownership Transfer [LR-722](https://project-sunbird.atlassian.net/browse/LR-722) - Ownership Transfer API\ [LR-685](https://project-sunbird.atlassian.net/browse/LR-685) - Ownership Transfer Flink Job\ [LR-748](https://project-sunbird.atlassian.net/browse/LR-748) - Ownership Transfer delete user assets report ## Enhancements / Technical Tasks [LR-309](https://project-sunbird.atlassian.net/browse/LR-309) - The Keycloak version is upgraded with 21.1.2 from 7.0.1. , we are supporting the existing features.\ Migration activity details are mentioned [LR-738](https://project-sunbird.atlassian.net/browse/LR-738) - Scala upgrade from 2.11 to 2.12 for userorg, course, notification and group service. [LR-766](https://project-sunbird.atlassian.net/browse/LR-766) - Elasticsearch upgrade 6.8.22 to 7.17.13. Steps to upgrade Elasticsearch are available [here](https://project-sunbird.atlassian.net/wiki/spaces/SBDES/pages/3494182916/Elasticsearch+Version+Upgrade+6.8.22+to+7.17.13). ## Bug Fixes [LR-759](https://project-sunbird.atlassian.net/browse/LR-759) - After deleting a user, and when tried to login to the same user immediately, we are getting "Access denied" error ## Details of Released Tag *Upgrade Sunbird Lern from 7.0.0 to 8.0.0*
ComponentsBuild Jenkins JobBuild TagDeploy Jenkins JobDeploy TagComment
Elasticsearch ProvisionngNANAProvision/Core/ApplicationElasticSearchrelease-8.0.0
  • To upgrade the ElasticSearch from 6.8.22 to 7.17.21
Keycloak-21 ProvisioningBuild/Core/Keycloak21release-8.0.0_RC1Deploy/Kubernetes/Keycloak21release-8.0.0Migrating keycloak from version 7.0.1 to 21.1.2
OnboardAPIsNANADeploy/Kubernetes/OnboardAPIsrelease-8.0.0To onboard the ownership transfer API
UserOrg ServiceBuild/Core/UserOrgrelease-8.0.0_RC1Deploy/Kubernetes/UserOrgrelease-8.0.0
  • cloud_store_group_id: org.sunbird
  • cloud_store_artifact_id: cloud-store-sdk_2.12
  • cloud_store_version: 1.4.7
LMS ServiveBuild/Core/LMSrelease-8.0.0_RC1Deploy/Kubernetes/LMSrelease-8.0.0
  • cloud_store_group_id: org.sunbird
  • cloud_store_artifact_id: cloud-store-sdk_2.12
  • cloud_store_version: 1.4.7
Group ServiceBuild/Core/Groupsrelease-8.0.0_RC1Deploy/Kubernetes/Groupsrelease-8.0.0
Notification serviceBuild/Core/Notificationrelease-8.0.0_RC1Deploy/Kubernetes/Notificationrelease-8.0.0
Kafka SetupNANADeploy/Lern/KafkaSetuprelease-7.0.0_RC6
DataPipelineBuild/Lern/LernFlinkJobsrelease-8.0.0_RC1Deploy/Lern/LernFlinkJobsrelease-8.0.0_RC1

Define the core_vault_sunbird_keycloak_user_federation_provider_id in Lern inventory secret. Add user-deletion-cleanup and ml-user-delete into job list and deploy it.

  • cloud_store_group_id: org.sunbird
  • cloud_store_artifact_id: cloud-store-sdk_2.12
  • cloud_store_version: 1.4.6
Data ProductBuild/Lern/LernDataProductsrelease-8.0.0_RC1Deploy/Lern/LernDataProductsrelease-8.0.0_RC1
  • cloud_store_group_id: org.sunbird
  • cloud_store_artifact_id: cloud-store-sdk_2.12
  • cloud_store_version: 1.4.6
## Configurations * To know more about the configuration of delete user assets report visit [here](https://lern.sunbird.org/use/learn-more/delete-user-functionality) * To know more about the configuration of ownership transfer functionality visit [here](https://lern.sunbird.org/use/learn-more/asset-ownership-transfer) ## Release Notes: Dependent building blocks Sunbird-Knowlg: [Release notes](https://knowlg.sunbird.org/use/release-notes/release-6.1.0-latest) (V 6.1.0)\ Sunbird-Obsrv: [Release notes](https://obsrv.sunbird.org/previous-versions/sb-5.0-version/use/release-notes/release-v-5.1.3) (V 5.1.3)\ Sunbird-Ed: [Release notes](https://ed.sunbird.org/use/release/updating-sunbird-releases/5.2.0-to-6.0.0) (V 8.0.0)\ Sunbird-Inquiry: [Release notes](https://inquiry.sunbird.org/use/release-notes/inquiry-release-v5.7.0) (V 5.7.0)\ Sunbird-Telemetry: [Documentation](https://telemetry.sunbird.org/)\ Sunbird-RC: [Documentation](https://docs.sunbirdrc.dev/learn/readme) # Release V 7.0.0 ### Document Release Version | Project | Release Date | Version | | ------- | ------------ | ------- | | Lern | 30-Dec-2023 | V 7.0.0 | ### **Hot-fix: 5.3.1**** (24-04-2024)**
ComponentsBuild Jenkins JobBuild TagDeploy Jenkins JobDeploy TagComment
Group ServiceBuild/Core/Groupsrelease-7.0.0_RC2Deploy/Kubernetes/Groupsrelease-7.0.0LR-776
### Details of Released Tag
ComponentsBuild Jenkins JobBuild TagDeploy Jenkins JobDeploy TagComment
OnboardAPIsNANADeploy/Kubernetes/OnboardAPIsrelease-7.0.0To onboard the delete user API
Cassandra MigrationBuild/Core/Cassandrarelease-7.0.0_RC3Deploy/Kubernetes/Cassandrarelease-7.0.0

Select the sunbird in cassandra_keyspace_to_migrate while deploying

script_repo_branch_or_tag: release-7.0.0_RC3

ES mappingProvision/Core/ESMappingrelease-7.0.0Updates the es mapping to accept dynamic framework category to index user data
UserOrg ServiceBuild/Core/UserOrgrelease-7.0.0_RC5Deploy/Kubernetes/UserOrgrelease-7.0.0
LMS ServiveBuild/Core/LMSrelease-7.0.0_RC2Deploy/Kubernetes/LMSrelease-7.0.0
Group ServiceBuild/Core/Groupsrelease-7.0.0_RC1Deploy/Kubernetes/Groupsrelease-7.0.0
Notification serviceBuild/Core/Notificationrelease-7.0.0_RC2Deploy/Kubernetes/Notificationrelease-7.0.0
Kafka SetupNANADeploy/Lern/KafkaSetuprelease-7.0.0_RC6
DataPipelineBuild/Lern/LernFlinkJobsrelease-7.0.0_RC7Deploy/Lern/LernFlinkJobsrelease-7.0.0_RC7Define the core_vault_sunbird_keycloak_user_federation_provider_id in Lern inventory secret. Add user-deletion-cleanup and ml-user-delete into job list and deploy it.
Data ProductBuild/Lern/LernDataProductsrelease-7.0.0_RC7Deploy/Lern/LernDataProductsrelease-7.0.0_RC7
Discussions MiddlewareBuild/Core/DiscussionsMiddlewarerelease-7.0.0_RC1Deploy/Kubernetes/DiscussionsMWrelease-7.0.0Removed the unwanted logs
### [**Summary of the Changes**](https://project-sunbird.atlassian.net/issues/?filter=12863) **Details of the Changes:** [LR-676](https://project-sunbird.atlassian.net/browse/LR-676) - User PII Information and account deletion from LERN DBs [LR-699](https://project-sunbird.atlassian.net/browse/LR-699) - \[LERN] Making BMGS configurable [LR-687](https://project-sunbird.atlassian.net/browse/LR-687) - Removal of Adopter specific keywords from LERN repos [LR-637](https://project-sunbird.atlassian.net/browse/LR-637) - Postman API automation development for user-org micro-service. [LR-660](https://project-sunbird.atlassian.net/browse/LR-660) - Postman API automation development for LMS micro-service. [LR-662](https://project-sunbird.atlassian.net/browse/LR-662) - Postman API automation development for groups micro-service. [LR-664](https://project-sunbird.atlassian.net/browse/LR-664) - Postman API automation development for notification micro-service. [LR-756](https://project-sunbird.atlassian.net/browse/LR-756) - Adding a "status" column to Admin reports. ### Configurations To know more about the configuration of Delete user functionality visit [here](https://lern.sunbird.org/use/learn-more/delete-user-functionality) ### Release Notes: Dependent building blocks Sunbird-Knowlg: [Release notes](https://knowlg.sunbird.org/use/release-notes/release-5.7.0-latest) (V 5.7.0)\ Sunbird-Obsrv: [Release notes](https://obsrv.sunbird.org/previous-versions/sb-5.0-version/use/release-notes/release-v-5.1.3) (V 5.1.3)\ Sunbird-Ed: [Release notes](https://ed.sunbird.org/use/release/updating-sunbird-releases/5.2.0-to-6.0.0) (V 7.0.0)\ Sunbird-Inquiry: [Release notes](https://inquiry.sunbird.org/use/release-notes/inquiry-release-v5.7.0) (V 5.7.0)\ Sunbird-Telemetry: [Documentation](https://telemetry.sunbird.org/)\ Sunbird-RC: [Documentation](https://docs.sunbirdrc.dev/learn/readme) ### Steps to update user's cache in Redis As part of making framework categories configurable, framework category details in the user's cache are updated. To support the new data-product, the existing Redis user's data should be updated through the Usercache indexer job. Run the below job with the below params to update the same. Job : `Deploy/Lern/LernAnalyticsReplayJobs` Kindly refer the below image for params
In this release, a new column called **"status"** has been added to the user data frame (`userDF`) of [**UsercacheindexerJob**](https://github.com/Sunbird-Lern/data-products/blob/release-7.0.0/lern-data-products/src/main/scala/org/sunbird/userorg/job/report/UserCacheIndexerJob.scala) of dataproducts. inorder to cache the status column from the Cassandra DB to Redis. Also same has been added to usercacheupdater config to make sure it is fetched from Redis to the state admin report and user info report. This will help the admin understand whether the user is active/inactive or deleted.\ Select the `job_names_to_deploy` as **user cache-updater-v2 job** (**/Deploy/Lern/LernFlinkJobs**) and a user-cache-index job (**/Deploy/Lern/LernAnalyticsReplayJobs**) once so that the status column is fetched correctly. # Release V 5.4.0 ### Document Release Version | Project | Release Date | Version | | ------- | ------------ | ------- | | Lern | 11-Jul-2023 | V 5.4.0 | ### Details of Released Tag
ComponentsBuild Jenkins JobBuild TagDeploy Jenkins JobDeploy TagComment
ProxyBuild/Core/Proxy

Deploy/Kubernetes/nginx-private-ingress

/Deploy/Kubernetes/nginx-public-ingress

release-5.4.0-lern
OnboardAPIsDeploy/Kubernetes/OnboardAPIsrelease-5.4.0-lern
UserOrg ServiceBuild/Core/UserOrgrelease-5.4.0_RC1Deploy/Kubernetes/UserOrgrelease-5.4.0-lernLearner service is renamed as UserOrg now.
LMS ServiveBuild/Core/LMSrelease-5.4.0_RC1Deploy/Kubernetes/LMSrelease-5.4.0-lern
Group ServiceBuild/Core/Groupsrelease-5.4.0_RC1Deploy/Kubernetes/Groupsrelease-5.4.0-lern
Notification serviceBuild/Core/Notificationrelease-5.4.0_RC1Deploy/Kubernetes/Notificationrelease-5.4.0-lern
Analytics ServiceDeploy/Kubernetes/Analyticsrelease-5.4.0-lern
DataPipelineBuild/Lern/LernFlinkJobsrelease-5.4.0_RC1Deploy/Lern/LernFlinkJobsrelease-5.4.0_RC1
Data ProductBuild/Lern/LernDataProductsrelease-5.4.0_RC6Deploy/Lern/LernDataProductsrelease-5.4.0_RC6
KeycloakDeploy/Kubernetes/Keycloakrelease-5.4.0-lern
Knowledge-MWDeploy/Kubernetes/KnowledgeMWrelease-5.4.0-lern
### **Summary of the Changes** **Details of the Changes:** [LR-122](https://project-sunbird.atlassian.net/browse/LR-122) Lern repo and pod name correction to match the component name - Development\ [LR-588](https://project-sunbird.atlassian.net/browse/LR-588) PII Enhancement\ [LR-102](https://project-sunbird.atlassian.net/browse/LR-102) UserOrg: Ability for Lern to connect to shared instances of Cassandra, ES, Postgres, and Redis with lern specific keyspace and indexes\ [LR-511](https://project-sunbird.atlassian.net/browse/LR-511) Ability for Lern to connect to shared instances of Cassandra, ES, Postgres, and Redis with lern specific keyspace and indexes in BatchService \ [LR-512](https://project-sunbird.atlassian.net/browse/LR-512) Ability for Lern to connect to shared instances of Cassandra, and Redis with lern specific keyspace and indexes in GroupService \ [LR-513](https://project-sunbird.atlassian.net/browse/LR-513) Ability for Lern to connect to shared instances of Cassandra with lern specific keyspace and indexes in NotificationService \ [LR-394](https://project-sunbird.atlassian.net/browse/LR-394) \[Feed API] Update feed API with deleted feed id is giving 200k response and read is also giving feed details #### Github repositories name updates ``` old: https://github.com/Sunbird-Lern/sunbird-lms-service new: https://github.com/Sunbird-Lern/userorg-service old: https://github.com/Sunbird-Lern/sunbird-course-service new: https://github.com/Sunbird-Lern/lms-service old: https://github.com/Sunbird-Lern/sunbird-notification-service new: https://github.com/Sunbird-Lern/notification-service ``` ### Configurations Rename all the endpoints from **/learner** to **/userorg** Renamed the below variables.
oldnew
  • learner_replicas
  • learner_reservation_memory
  • learner_limit_memory
  • learner_reservation_cpu
  • learner_limit_cpu
  • learner_java_mem_limit
  • learner_replicacount
  • learner_repository
  • learner_cpu_req
  • learner_mem_req
  • learner_cpu_limit
  • learner_mem_limit
  • learner_maxsurge
  • learner_maxunavailable
  • learner_liveness_readiness
  • learner_envoy_cpu_req
  • learner_envoy_mem_req
  • learner_envoy_cpu_limit
  • learner_envoy_mem_limit
  • learner_opa_cpu_req
  • learner_opa_mem_req
  • learner_opa_cpu_limit
  • learner_opa_mem_limit
  • learner_initcontainer_cpu_req
  • learner_initcontainer_mem_req
  • learner_initcontainer_cpu_limit
  • learner_initcontainer_mem_limit
  • learner_autoscaling_enabled
  • learner_autoscaling_minReplicas
  • learner_autoscaling_maxReplicas
  • learner_autoscaling_targetCPUUtilizationPercentage
  • learner_autoscaling_targetMemoryUtilizationPercentage
  • learner_opa_enabled
  • learner_access_basepath
  • deploy_learner
  • learner_opa_decision_logs
  • userorg_replicas
  • userorg_reservation_memory
  • userorg_limit_memory
  • userorg_reservation_cpu
  • userorg_limit_cpu
  • userorg_java_mem_limit
  • userorg_replicacount
  • userorg_repository
  • userorg_cpu_req
  • userorg_mem_req
  • userorg_cpu_limit
  • userorg_mem_limit
  • userorg_maxsurge
  • userorg_maxunavailable
  • userorg_liveness_readiness
  • userorg_envoy_cpu_req
  • userorg_envoy_mem_req
  • userorg_envoy_cpu_limit
  • userorg_envoy_mem_limit
  • userorg_opa_cpu_req
  • userorg_opa_mem_req
  • userorg_opa_cpu_limit
  • userorg_opa_mem_limit
  • userorg_initcontainer_cpu_req
  • userorg_initcontainer_mem_req
  • userorg_initcontainer_cpu_limit
  • userorg_initcontainer_mem_limit
  • userorg_autoscaling_enabled
  • userorg_autoscaling_minReplicas
  • userorg_autoscaling_maxReplicas
  • userorg_autoscaling_targetCPUUtilizationPercentage
  • userorg_autoscaling_targetMemoryUtilizationPercentage
  • userorg_opa_enabled
  • userorg_access_basepath
  • deploy_user_org
  • userorg_opa_decision_logs
### Release Notes: Dependent building blocks Sunbird-Knowlg: [Release notes](https://knowlg.sunbird.org/use/release-notes/release-5.5.0-latest) (V 5.5.0)\ Sunbird-Obsrv: [Release notes](https://obsrv.sunbird.org/previous-versions/sb-5.0-version/use/release-notes/release-v-5.1.3) (V 5.1.3)\ Sunbird-Ed: [Release notes ](https://ed.sunbird.org/use/releases/release-notes/release-5.2.0)(V 5.2.0)\ Sunbird-Inquiry: [Release notes](https://inquiry.sunbird.org/use/release-notes/inquiry-release-v5.7.0) (V 5.7.0)\ Sunbird-Telemetry: [Documentation](https://telemetry.sunbird.org/)\ Sunbird-RC: [Documentation](https://docs.sunbirdrc.dev/learn/readme) # Release V 5.3.0 ### **Hot-fix: CSP**** (26-06-2024)** | Component | Build Job | Build Tag | Deploy Job | Deployment | Comment | | ---------------------------------------------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------ | -------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ETLJobs | Build/DataPipeline/ETLJobs | [release-5.1.1\_RC1](https://github.com/Sunbird-Ed/sunbird-data-products/tree/release-5.1.1_RC1) | Deploy/DataPipeline/ETLJobs | [release-5.2.0\_RC7](https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/tree/release-5.2.0_RC7) | | | ETLDruidContentIndexer | NA | NA | Deploy/DataPipeline/ETLDruidContentIndexer | [release-5.2.0\_RC7](https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/tree/release-5.2.0_RC7) | | | Data Products | Build/Lern/LernDataProducts | [release-5.3.1\_RC11](https://github.com/Sunbird-Lern/data-products/tree/release-5.3.1_RC11) | Deploy/Lern/LernDataProducts | [release-5.3.1\_RC11](https://github.com/Sunbird-Lern/data-products/tree/release-5.3.1_RC11) |

CSP related changes.

Note: While deploy select set the module value as lern-dataproducts,cronjobs

| | Ed data product in dock env | Build/Lern/LernDataProducts | [release-5.3.1\_RC11](https://github.com/Sunbird-Lern/data-products/tree/release-5.3.1_RC11) | Deploy/Dock/DataPipeline/EdDataProducts | [release-5.3.1\_RC11](https://github.com/Sunbird-Lern/data-products/tree/release-5.3.1_RC11) |

CSP related changes.

Note: While deploy select set the module value as dock-dataproducts

| | To run Ed related reports: **Live ETB QR Code-Content Linkage Status** | NA | NA | Deploy/Lern/LernAnalyticsReplayJobs | [release-5.3.1\_RC11](https://github.com/Sunbird-Lern/data-products/tree/release-5.3.1_RC11) |

Add the etb-metrics in job\_id list.
- job\_type: run-job
- job\_id: etb-metrics

| | To run Ed related reports: **Course Adoption Report v2** | NA | NA | Deploy/DataPipeline/Runreport | [release-5.2.0\_RC7](https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/tree/release-5.2.0_RC7) |

report\_id:

| | To run coKreat related report: Visitor's report | NA | NA | Deploy/DataPipeline/Runreport | [release-5.2.0\_RC7](https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/tree/release-5.2.0_RC7) | report\_id: vidyadaan\_visitor | | To run coKreat related report: Collection Level Content Gaps | NA | NA | Deploy/Dock/DataPipeline/AnalyticsReplayJobs | [release-5.3.1\_RC11](https://github.com/Sunbird-Lern/data-products/tree/release-5.3.1_RC11) | | | To run coKreat related report: Folder Level (first level) Content Gaps | NA | NA | Deploy/Dock/DataPipeline/AnalyticsReplayJobs | [release-5.3.1\_RC11](https://github.com/Sunbird-Lern/data-products/tree/release-5.3.1_RC11) | | | To run coKreat related report: Project level funnel report | NA | NA | Deploy/Dock/DataPipeline/AnalyticsReplayJobs | [release-5.3.1\_RC11](https://github.com/Sunbird-Lern/data-products/tree/release-5.3.1_RC11) | | | To run coKreat related report: Content Details Report | NA | NA | Deploy/Dock/DataPipeline/AnalyticsReplayJobs | [release-5.3.1\_RC11](https://github.com/Sunbird-Lern/data-products/tree/release-5.3.1_RC11) | | ### **Hot-fix: CSP**** (24-08-2023)** | Component | Build Job | Build Tag | Deploy Job | Deployment | Comment | | ----------------- | --------------------------- | -------------------------------------------------------------------------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Batch Service | Build/Core/Lms | [`release-5.3.2_RC1`](https://github.com/Sunbird-Lern/lms-service/tree/release-5.3.2_RC1) | Deploy/Kubernetes/Lms | [release-5.3.0-lern](https://github.com/project-sunbird/sunbird-devops/tree/release-5.3.0-lern) |

CSP related changes.

| | User\&Org Service | Build/Core/Learner | [release-5.3.1\_RC1](https://github.com/Sunbird-Lern/userorg-service/tree/release-5.3.1_RC1) | Deploy/Kubernetes/Learner | [release-5.3.0-lern](https://github.com/project-sunbird/sunbird-devops/tree/release-5.3.0-lern) |

CSP related changes.

| | Data pipeline | Build/Lern/FlinkJobs | [release-5.3.1\_RC2](https://github.com/Sunbird-Lern/data-pipeline/tree/release-5.3.1_RC2) | Deploy/Lern/FlinkJobs | [release-5.3.1\_RC2](https://github.com/Sunbird-Lern/data-pipeline/tree/release-5.3.1_RC2) |

CSP related changes.

| | Data Products | Build/Lern/LernDataProducts | [release-5.3.1\_RC10](https://github.com/Sunbird-Lern/data-products/tree/release-5.3.1_RC10) | Deploy/Lern/LernDataProducts | [release-5.3.1\_RC10](https://github.com/Sunbird-Lern/data-products/tree/release-5.3.1_RC10) |

CSP related changes.

| **Jenkins Configurations for csp support:** Configure the variables cloud\_store\_group\_id, cloud\_store\_artifact\_id and cloud\_store\_version with proper values in the Jenkins, it can configured in the global or to the individual service of build job.\ For lms, user-org, flinks-jobs, lerndataproducts build jobs configure like as we mentioned below.

Jenkins configuration

#### Configure the following values **Name - Default Value - Description** cloud\_store\_group\_id - ${cloud\_store\_group\_id} - Set the Cloud store sdk group id. e.g. org.sunbird\ cloud\_store\_artifact\_id - ${cloud\_store\_artifact\_id} - Set the Cloud store sdk artifact id. e.g. cloud-store-sdk cloud\_store\_version - ${cloud\_store\_version} - Set the Cloud store sdk version. e.g 1.4.6 #### Config changes in Lern common.yaml for data-products ``` cloud_storage_report_verfication_bucketname: "" # default value is "report-verification" dp_storage_endpoint_config: "" # default is "{{s3_storage_endpoint}}" ``` ### **Hot-fix: 5.3.1**** (05-07-2023)** | Component | Build Job | Build Tag | Deploy Job | Deployment | Comment | | ------------- | -------------- | ---------------------------------------------------------------------------------------- | --------------------- | ----------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Batch Service | Build/Core/Lms | [release-5.3.1\_RC1](https://github.com/Sunbird-Lern/lms-service/tree/release-5.3.1_RC1) | Deploy/Kubernetes/Lms | [release-5.3.0-lern](https://github.com/project-sunbird/sunbird-devops/tree/release-5.3.0-lern) |

QR Codes Image download Issue fix

Bug:

| ### Document Release Version | Project | Release Date | Version | | ------- | ------------ | ------- | | Lern | 27-May-2023 | V 5.3.0 | | Lern | 23-Jun-2023 | V 5.3.1 | ### Hot Fix :- ML PII Data Product (23-06-2023) ### Details of Released Tag
ComponentsBuild Jenkins JobBuild TagDeploy Jenkins JobDeploy TagComment
Kafka SetupDeploy/Lern/KafkaSetupverify if kafka topic = programuser.info is created or not
Data pipelineBuild/Lern/FlinkJobsrelease-5.3.0_RC5Deploy/Lern/FlinkJobsrelease-5.3.0_RC5Add program-user-info into job list and deploy it.
Data ProductsBuild/Lern/LernDataProductsrelease-5.3.0_RC6Deploy/Lern/LernDataProductsrelease-5.3.0_RC6Add program-user-exhaust into job list of Deploy/Lern/LernAnalyticsReplayJobs for running it.
Cassandra MigrationBuild/Core/Cassandrarelease-5.3.0_RC1Deploy/Kubernetes/Cassandrarelease-5.3.0-lernadd the sunbird_programs keyspace in Deploy Jenkins jobs
AnalyticsDeploy/Kubernetes/AnalyticsDeploy with release-6.0.0 branch
**Summary of the Changes** **Details of the Changes:** [LR-491](https://project-sunbird.atlassian.net/browse/LR-491) User detail (PII) report for ML programs - Data Product\ [LR-285](https://project-sunbird.atlassian.net/browse/LR-285) User detail (PII) report for ML programs - Flink Job #### Default values for config default config for [services](https://github.com/Sunbird-Lern/data-products/blob/release-5.3.0/ansible/roles/lern-data-products-deploy/defaults/main.yml#L35) ``` sunbird.program.report.keyspace="{{ program_keyspace }}" ml.exhaust.store.prefix="ml_reports" ``` Please define below variables ``` program_keyspace: "sunbird_programs" ml.exhaust.store.prefix="ml_reports" ``` #### Cassandra Keyspace and Table for Program:- #### ### Flink Job Configurations for Lern: | Name of the Flink Job added | | --------------------------- | | **program-user-info** |
LR-285 - User detail flink job for ML-programs - setup/configuration details: For this ticket, we have only done unit testing with the help of simulated events. Integration testing has not been done as the required workflows concerning this will only be enabled after Ed 6.0 release. As part of this ticket we have enabled new Flink jobs and they in no way impact any existing workflows\ \ **Job name: program-user-info** The purpose of this job is to record the user's information when the user submits the program. Whenever a program is submitted, this job receives an event with the user's information as JSON data and then it parses and stores it as respective key-value pairs in Cassandra. **Keyspace name**: sunbird\_program **Schema of the Kafka Topic:**\ **Kafka Topic Name**: `{{envName}}.programuser.info`\ **Event** **Structure:-** 
{
      programId: {
        type : "ObjectId",
        required : true,
        index: true
      },
      programName: String,
      programExternalId: String,
      noOfResourcesStarted: {
        type:Number,
        index: true
        }
      userId: {
        type: String,
        index: true
      },
      requestForPIIConsent:true/false
      userProfile: Object,
      userRoleInformation: Object,
      appInformation: Object,
      createdAt: Date,
      updatedAt: Date,
      deleted:Boolean
}
**Job Configurations:** 
kafka {
 input.topic = ${job.env}".programuser.info"
 groupId = ${job.env}"-programuser-group"
}
task {
 consumer.parallelism = 1
 downstream.parallelism = 1
 programUser{
  parallelism = 1
 }
}
ml-cassandra {
 keyspace = "sunbird_programs"
 table = "program_enrollment"
 port = "9042"
 host =
 }
Flink **build** Jenkins job name: **/Build/job/Lern/job/FlinkJobs** Flink **deploy** Jenkins job name: **/Deploy/job/\/job/Lern/job/FlinkJobs/program-user-info** Jenkins job for **building** Cassandra: **/Build/job/Core/job/Cassandra/** Jenkins job for **deploying** Cassandra: **/Deploy/job/\/job/Kubernetes/job/Cassandra**
### Data Security Policy setup **Configurations to be done by System admin:** 1. Setup **default** 'Data Security Policy' settings using tenant preference API. ``` curl --location --request PATCH '{{host}}/api/org/v2/preferences/update' \ --header 'x-authenticated-user-token: {{user_authentication_token}}' \ --header 'Authorization: Bearer {{kong_api_token}}' \ --header 'Content-Type: application/json' \ --data-raw '{ "request": { "orgId": "default", "key": "dataSecurityPolicy", "data": { "level": "PLAIN_DATASET", "dataEncrypted": "No", "comments": "Data is not encrypted", "job": { "userinfo-exhaust": { "level": "PASSWORD_PROTECTED_DATASET", "dataEncrypted": "No", "comments": "Password protected file." }, "program-user-exhaust": { "level": "PASSWORD_PROTECTED_DATASET", "dataEncrypted": "No", "comments": "Password protected file." } }, "securityLevels": { "PLAIN_DATASET": "Data is present in plain text/zip. Generally applicable to open datasets.", "PASSWORD_PROTECTED_DATASET": "Password protected zip file. Generally applicable to non PII data sets but can contain sensitive information which may not be considered open.", "TEXT_KEY_ENCRYPTED_DATASET": "Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open.", "PUBLIC_KEY_ENCRYPTED_DATASET": "Data encrypted via an org provided public/private key. Generally applicable to all PII data exhaust." } } } }' ``` ### Details of Released Tag
ComponentsBuild Jenkins JobBuild TagDeploy Jenkins JobDeploy TagComment
Kafka SetupDeploy/Lern/KafkaSetuprelease-5.3.0_RC3
Data pipelineBuild/Lern/FlinkJobsrelease-5.3.0_RC5Deploy/Lern/FlinkJobsrelease-5.3.0_RC5Add legacy-certificate-migrator into job list and deploy it.
Data ProductsBuild/Lern/LernDataProductsrelease-5.3.0_RC6Deploy/Lern/LernDataProductsrelease-5.3.0_RC6
Batch ServiceBuild/Core/Lmsrelease-5.3.0_RC1Deploy/Kubernetes/Lmsrelease-5.3.0-lern
User&Org ServiceBuild/Core/Learnerrelease-5.3.0_RC2Deploy/Kubernetes/Learnerrelease-5.3.0-lern
AnalyticsDeploy/Kubernetes/AnalyticsDeploy with release-6.0.0 branch
### **Summary of the Changes** **Details of the Changes:** [LR-436](https://project-sunbird.atlassian.net/browse/LR-436) OldCertificateMigration spark data-product\ [LR-437](https://project-sunbird.atlassian.net/browse/LR-437) LegacyCertificateMigrator Flink job\ [LR-438](https://project-sunbird.atlassian.net/browse/LR-438) Sunbird RC changes for updating schema for issued date\ [LR-330](https://project-sunbird.atlassian.net/browse/LR-330) Certificate template font url migration\ [LR-395](https://project-sunbird.atlassian.net/browse/LR-395), [LR-465](https://project-sunbird.atlassian.net/browse/LR-465) PII data security\ [LR-451](https://project-sunbird.atlassian.net/browse/LR-451) Local setup of Data-pipeline - Ubuntu & Mac - Github and Microsite update\ [LR-443](https://project-sunbird.atlassian.net/browse/LR-443) Local setup of UserOrg - Ubuntu & Mac - Github and Microsite update\ [LR-445](https://project-sunbird.atlassian.net/browse/LR-445) Local setup of LMS - Ubuntu & Mac - Github and Microsite update\ [LR-422](https://project-sunbird.atlassian.net/browse/LR-422) Point the channel create API to content-service instead of learning-service\ [LR-519](https://project-sunbird.atlassian.net/browse/LR-519) Textbook APIs code cleanup from Course-Batch service\ [LR-486](https://project-sunbird.atlassian.net/browse/LR-486) Microsite update with Certificate generation flow diagram\ [LR-520](https://project-sunbird.atlassian.net/browse/LR-520) Group service - activity type should be case insensitive\ [LR-556](https://project-sunbird.atlassian.net/browse/LR-556) Local setup of LMS - Ubuntu & Mac - **Mock service setup**\ [LR-456](https://project-sunbird.atlassian.net/browse/LR-456) Local setup of Sunbird-utils - Ubuntu & Mac - Github and Microsite update
### **New APIs to onboard** ``` - name: exhaustSubmitProxyAPI uris: "{{ course_service_prefix }}/v1/jobrequest/submit" upstream_url: "{{ lms_service_url }}/v1/jobrequest/submit" strip_uri: true plugins: - name: jwt - name: cors - "{{ statsd_pulgin }}" - name: acl config.whitelist: - courseAccess - name: rate-limiting config.policy: local config.hour: "{{ medium_rate_limit_per_hour }}" config.limit_by: credential - name: request-size-limiting config.allowed_payload_size: "{{ small_request_size_limit }}" - name: opa-checks config.required: false config.enabled: false - name: exhaustListProxyAPI uris: "{{ course_service_prefix }}/v1/jobrequest/list" upstream_url: "{{ lms_service_url }}/v1/jobrequest/list" strip_uri: true plugins: - name: jwt - name: cors - "{{ statsd_pulgin }}" - name: acl config.whitelist: - courseAccess - name: rate-limiting config.policy: local config.hour: "{{ medium_rate_limit_per_hour }}" config.limit_by: credential - name: request-size-limiting config.allowed_payload_size: "{{ small_request_size_limit }}" - name: opa-checks config.required: false config.enabled: false - name: orgAddEncryptionKey uris: "{{ org_service_prefix }}/v1/update/encryptionkey" upstream_url: "{{ learning_service_url }}/v1/org/update/encryptionkey" strip_uri: true plugins: - name: jwt - name: cors - "{{ statsd_pulgin }}" - name: acl config.whitelist: - orgSuperAdmin - name: rate-limiting config.policy: local config.hour: "{{ medium_rate_limit_per_hour }}" config.limit_by: credential - name: request-size-limiting config.allowed_payload_size: "{{ small_request_size_limit }}" - name: opa-checks config.required: false config.enabled: false ``` ### Env Configurations (Needs to be done before service deployment): The below environment variable needs to be configured in the 'sunbird-lms-service.env' file dev ops repo. Ref: | Variable Name | Values | Comments | | ------------------------------ | ----------------------------------------------------------------------------------------------------------- | -------------------------------------------- | | exhaust\_api\_base\_url | {{ analytics\_service\_url \| default('[http://analytics-service:9000](http://analytics-service:9000/)') }} | Obsrv exhaust API endpoint for batch service | | exhaust\_api\_submit\_endpoint | /request/submit | To submit job request from batch service | | exhaust\_api\_list\_endpoint | /request/list/ | To list job request from batch service | | sunbird\_api\_auth\_token | "{{ core\_vault\_sunbird\_api\_auth\_token }}" | Authentication token for APIs | | content\_read\_url | /content/v3/read/ | | ### Exhaust Proxy API documentation ### Data Security Policy setup **Configurations to be done by System admin:** 1. Execute CURL for providing link to download "Decryption Tool". Tool reference: {% hint style="info" %} Please upload the tool to your public cloud location or to your repository and provide the link to the same in below system setting variable value. {% endhint %} ``` curl --location --request POST '{{host}}/api/data/v1/system/settings/set' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer {{api_key}}' \ --header 'x-authenticated-user-token: {{user_token}}' \ --data-raw '{ "request": { "id": "decryptionToolLink", "field": "decryptionToolLink", "value": "{\"link\":\"\", \"Comments\": \"To use this tool, run the command with encrypted file and key to decrypt\"}" } }' ``` 2. Setup **default** 'Data Security Policy' settings using tenant preference API. ``` curl --location --request POST '{{host}}/api/org/v2/preferences/create' \ --header 'x-authenticated-user-token: {{user_authentication_token}}' \ --header 'Authorization: Bearer {{kong_api_token}}' \ --header 'Content-Type: application/json' \ --data-raw '{ "request": { "orgId": "default", "key": "dataSecurityPolicy", "data": { "level": "PLAIN_DATASET", "dataEncrypted": "No", "comments": "Data is not encrypted", "job": { "userinfo-exhaust": { "level": "PASSWORD_PROTECTED_DATASET", "dataEncrypted": "No", "comments": "Password protected file." } }, "securityLevels": { "PLAIN_DATASET": "Data is present in plain text/zip. Generally applicable to open datasets.", "PASSWORD_PROTECTED_DATASET": "Password protected zip file. Generally applicable to non PII data sets but can contain sensitive information which may not be considered open.", "TEXT_KEY_ENCRYPTED_DATASET": "Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open.", "PUBLIC_KEY_ENCRYPTED_DATASET": "Data encrypted via an org provided public/private key. Generally applicable to all PII data exhaust." } } } }' ``` 3. Setup **default '**PII data security settings' using tenant preference API. ``` curl --location --request POST '{{host}}/api/org/v2/preferences/create' \ --header 'x-authenticated-user-token: {{user_authentication_token}}' \ --header 'Authorization: Bearer {{kong_api_token}}' \ --header 'Content-Type: application/json' \ --data-raw '{ "request": { "orgId": "default", "key": "userPrivateFields", "data": { "PIIFields": [ "email", "phone", "userName", "prevUsedEmail", "prevUsedPhone", "recoveryEmail", "recoveryPhone" ] } } }' ``` **Configurations that can be done by Tenants:** 1. Use Tenant preference create API to create tenant specific 'Data Security Policy' settings similar to 'default' Data Security Policy settings but with tenant orgId. ``` Note: a. Tenant level security cannot be lower than 'default' Data Security Policy'. b. Job Level security Policy in a Tenant specific configuration cannot be lower than Tenant Level configuration and cannot be lower than job level configuration in 'default' Data Security Policy'. c. Below mapping shows the priority/grade of security policies "PLAIN_DATASET" < "PASSWORD_PROTECTED_DATASET" < "TEXT_KEY_ENCRYPTED_DATASET" < "PUBLIC_KEY_ENCRYPTED_DATASET" ``` 2. In order to use "PUBLIC\_KEY\_ENCRYPTED\_DATASET" security configuration for an exhaust report, tenant admin should have uploaded public pem key file using below API. ``` curl --location --request PATCH '{{host}}/api/org/v1/update/encryptionkey' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer {{kong_api_token}}' \ --header 'x-authenticated-user-token: {{user_authentication_token}}' \ --form 'organisationId={{org_id}}' \ --form 'encryptionKey=@path_to_public_pem_file' ``` ### Steps to generate key pair for setting up Data Security policy configuration: #### For Linux and Mac OS: 1. To generate Private Key ``` openssl genrsa -out private.pem 4096 ``` 2. To generate Public Key ``` openssl rsa -in private.pem -pubout -outform PEM -out public_key.pem ``` #### For Windows OS: Please install GitBash: The Git installation package comes with SSH. Using Git Bash, which is the Git command line tool, you can generate SSH key pairs. Git Bash has an SSH client that enables you to connect to and interact with Triton containers on Windows. **To install Git:** 1. Download and initiate the [Git installer](https://git-scm.com/download/win). 2. When prompted, accept the default components by clicking Next. 3. Choose the default text editor. If you have Notepad++ installed, select Notepad++ and click Next. 4. Select to Use Git from the Windows Command Prompt and click Next. 5. Select to Use OpenSSL library and click Next. 6. Select to Checkout Windows-style, commit Unix-style line endings and click Next. 7. Select to Use MinTTY (The default terminal of mYSYS2) and click Next. 8. Accept the default extra option configuration by clicking Install. When the installation completes, you may need to restart Windows. **Launching GitBash:** 1. press Start+R to launch the Run dialog. 2. Type C:\Program Files\Git\bin\bash.exe and press Enter. **Generating Key pair:** 1. To generate Private Key ``` openssl genrsa -out private.pem 4096 ``` 2. To generate Public Key ``` openssl rsa -in private.pem -pubout -outform PEM -out public_key.pem ``` ### ### Flink Job Configurations for Lern: | Name of the Flink Job added | | ------------------------------- | | **legacy-certificate-migrator** | ### Prerequired deployments for RC migration
LR-436 - Deploy Data-product Data-product build Jenkins job: **Build/Lern/LernDataProducts** Deploy Jenkins job: **Deploy/{{env}}/Lern/LernDataProducts**
LR-437 - Deploy legacy-certificate-migrator Flink job Build Jenkins job: **/Build/job/Lern/job/FlinkJobs** Deploy Jenkins job: **/Deploy/job/\/job/Lern/job/FlinkJobs**
LR-438 - Update RC schema **Step 1 : Upload updated schema files.**\ Deploy Jenkins job: **Deploy/dev/Sunbird-RC/Upload\_RC\_Schema** **Note**: Since certificate signer service will cache the credential template. please make sure the credential template is updated in the respective path as per below file. **Step 2 : Deploy certificate signer service** Jenkins Job: **Deploy/dev/Sunbird-RC/CertificateSign**
## Step to migrate old certificates to RC Sunbird Lern BB is using Sunbird RC for generating & issuing e-credentials in its use cases (e.g.: course completion certificate) for all the latest completed courses (post March-2022). All the old certificates were custom generated and stored in Cassandra and cloud storage. Once we migrate these certificates then we no longer need to store certificates in Cassandra and all the certificates will be using Sunbird RC going forward. Reference Link: **Note:** After migrating old certificates to RC, certificate verification of old certificates will become invalid. To support to old certificate verification, `Sunbird ED` building block is implementating in portal service in release 6.0. Kindly find the ticket in [this link](https://project-sunbird.atlassian.net/browse/ED-1594). So recommended to migrate the certificates after getting the old certification verification support as well. **Step 1** Create Kafka topic for only the purpose of this migration process **Topic name:** {{env}}.legacy.certificate.migrate **Step 2** In the spark machine, update the **`old-certificate-migration-job`** model config in **`mount/data/analytics/scripts/lern-model-config.sh`** with correct values. Sample model config: {% code overflow="wrap" %} ``` {"search":{"type":"none"},"model":"org.sunbird.lms.audit.OldCertificateMigrationJob","modelParams":{"mode":"execute","store":"azure","sparkCassandraConnectionHost":"10.5.3.17", "cert_base_path": "https://dev.lern.sunbird.org", "cloud_storage_base_url": "https://sunbirddev.blob.core.windows.net", "cloud_store_base_path_placeholder": "CLOUD_BASE_PATH","content_cloud_storage_container": "sunbird-content-staging", "cloud_storage_cname_url": "https://obj.stage.sunbirded.org", "batchId": "01320961460024934435", "kafka_broker": "localhost:9092", "kafka_topic": "sunbirddevlern.legacy.certificate.migrate","output_file_path":"./reports/"},"parallelization":8,"appName":"OldCertificateMigrationJob"} ``` {% endcode %} Note: migration job can be run single batch with `"batchId": "01320961460024934435"` and multiple batches with `"batchId": "01320961460024934435,01220961460024934536"` and for all batches with `"batchId": "all"` . **Step 3** Run the job with the below command in the spark machine. ``` /mount/data/analytics/scripts/lern-run-job.sh old-certificate-migration-job & ``` *Note:* logs can be found in below locations, Joblog: `/mount/data/analytics/scripts/logs/joblog.log` Execution log: `/mount/data/analytics/logs/lern-data-products/{current_date}-job-execution.log` **Note:** Verification steps can be found in the design page: ## Steps to Font URL migration All the templates are having dev URLs configured for Fonts in all the environments as per our observation. All these font URLs have to be migrated to the new cname URL **Note:** Before font url migration, make sure all the font files are available at cname mapped account or cloud storage container. To verify, where the font files are available, open any svg template file in editor and check the font URL's host. Please use java 11 for running the scripts #### Step 1: Download SVG file migrator and uploader jars by below command, ``` cd ~ mkdir svg_template_migration cd svg_template_migration wget "https://github.com/kumarks1122/sunbird-utils/raw/release-5.3.0-font-url-migration/svg_template_migration/template-migration/svg-migrator.jar" wget "https://github.com/kumarks1122/sunbird-utils/raw/release-5.3.0-font-url-migration/svg_template_migration/template-upload/svg-uploader.jar" ``` #### Step 2: Download the svg template files and update the font URLs in the template files. ``` java -jar svg-migrator.jar "{{ content search host }}" "0" "1000" "font_migration" "{{ Old URL }}" "{{ cname url }}" #EXAMPLE #java -jar svg-migrator.jar "dev.lern.sunbird.org" "0" "1000" "font_migration" "https://sunbirddev.blob.core.windows.net" "https://obj.diksha.gov.in" ``` ***Note**:* Before moving to next step, please verify atleast one svg file for whether the font URL got updated. #### Step 3: Upload the svg template files back to the cloud storage by below command. ``` java -jar svg-uploader.jar "{{ content search host }}" "0" "1000" "{{ storage key}}" "{{ storage secret }}" "{{svg file path}}" "{{storage type: (azure,..)}}" "{{ CSP endpoint (based on CSP it is optional) }}" "{{ region (based on CSP it is optional) }}" #EXAMPLE #java -jar svg-uploader.jar "dev.lern.sunbird.org" "0" "5" "sunbirddevbbpublic" "{{ secret }}" "/Users/{{username}}/svg_template_migration" "azure" ``` \ \ **Configuration for making content read URL dynamic:**\ [**https://project-sunbird.atlassian.net/browse/LR-579**](https://project-sunbird.atlassian.net/browse/LR-579)\ Please define the below URL in the sunbird\_lms-service.env file this will make the content read endpoint URL Configurable. ``` content_read_url=/content/v3/read/ ``` # Release V 5.2.0 ### Document Release Version | Project | Release Date | Version | | ------- | ------------ | ------- | | Lern | 27 MAR 23 | V 5.2.0 | ### Details of Released Tag
ComponentBuild Jenkins JobBuild TagsDeploy Jenkins JobDeploy TagsComment
Batch ServiceBuild/Core/Lms

release-5.2.0_RC2


Deploy/Kubernetes/Lmsrelease-5.2.0-lern
Data pipelineBuild/Lern/FlinkJobsrelease-5.2.0_RC1Deploy/Lern/FlinkJobsrelease-5.2.0_RC1deploy the user-cache-updater-v2 flink job only
User&Org ServiceBuild/Core/Learnerrelease-5.2.0_RC1 Deploy/Kubernetes/Learnerrelease-5.2.0-lern
Data ProductsBuild/Lern/LernDataProductsrelease-5.2.0_RC1Deploy/Lern/LernDataProductsrelease-5.2.0_RC1
Cassandra MigrationBuild/Core/Cassandrarelease-5.2.0_RC1Deploy/Kubernetes/Cassandrarelease-5.2.0-lern
SyncToolBuild/KnowledgePlatform/SyncToolrelease-5.4.0_RC1Deploy/KnowledgePlatform/Neo4jElasticSearchSyncTool

cmd: syncdialcodes

Sample params: 

--ids U7J3S8,R9Y6W5,Y3U3F1,D5C3D6,A7R6H3,J4F5V2,E1P7P2,Y5X5T7
SyncTool enhancement to be used by existing adopters for syncing "imageUrl" of DIAL codes to elastic search.
**Summary of the Changes** * **Refactoring of Dial code dependency**: An API was developed to fetch QR code image URLs and resolve relative paths from the DIAL service instead of the current connection to the Cassandra table. * **API automation using Postman for P0 APIs** * **Movement of UserCache and UserCacheIndexer in Data Pipeline to Lern** * **Test Automation for CSP** * **Cassandra migration and grouping cql scripts with respect to keyspaces** \ **Bug Fixes** - click [here](https://project-sunbird.atlassian.net/browse/LR-405?jql=created%20%3E%3D%202023-02-22%20AND%20created%20%3C%3D%202023-03-22%20AND%20project%20%3D%20LR%20AND%20issuetype%20%3D%20Bug%20AND%20status%20in%20\(%22Failed%20Validation%22%2C%20%22In%20Development%22%2C%20%22In%20Validation%22%2C%20Open%2C%20%22Selected%20for%20Contribution%22\)%20AND%20affectedVersion%20in%20\(5.2.0%2C%205.2.0.0\)%20AND%20labels%20%3D%20External_BB_Issue%20ORDER%20BY%20created%20DESC) to see the list of bugs fixed as a part of this release.\ \ **Details of the Changes:**\ [LR-301](https://project-sunbird.atlassian.net/browse/LR-301) API automation using Postman for P0 APIs\ [LR-302](https://project-sunbird.atlassian.net/browse/LR-302) Movement of UserCacheIndexer Data Product to Lern \ [LR-303](https://project-sunbird.atlassian.net/browse/LR-303) Movement of UserCache in Data Pipeline to Lern\ [LR-306](https://project-sunbird.atlassian.net/browse/LR-306) Test Automation for CSP \ [LR-322](https://project-sunbird.atlassian.net/browse/LR-322) API automation using Newman for P0 APIs\ [LR-325](https://project-sunbird.atlassian.net/browse/LR-325) BatchService: Refactoring of SB Lern Batch Service - DialCode Dependency \ [LR-101](https://project-sunbird.atlassian.net/browse/LR-101) Cassandra migration and grouping cql scripts with respect to keyspaces\ [LR-307](https://project-sunbird.atlassian.net/browse/LR-307) Setting up a complete testing env for Lern with all other BBs\ [LR-122](https://project-sunbird.atlassian.net/browse/LR-122) Lern repo and pod name correction to match the component name ### Env Configurations (Needs to be done before service deployment): The below environment variable needs to be configured in the dev ops repo in **'sunbird\_lms-service.env'** file. | Variable Name | Values | Comments | | ----------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------- | | sunbird\_dial\_service\_base\_url | [http://dial-service.{{env}}.svc.cluster.local:9000](http://dial-service.learn.svc.cluster.local:9000) | To store the dial service base path | | sunbird\_dial\_service\_search\_url | /dialcode/v3/search | To store the search url of the dial service | {% hint style="info" %} **Note:** **Only For the adopters who are migrating from the previous versions to 5.2.0, run 'syncdialcodes' command in "Neo4jElasticSearchSyncTool" to sync "imageUrl" of dialcodes to Elastic Search. DIAL codes list can be fetched using below content search CURL:** ```json curl --location --request POST '{{host}}/api/content/v1/search' \ --header 'Content-Type: application/json' \ --data-raw '{ "request": { "filters": { "primaryCategory": "Course" }, "exists": "dialcodes", "fields": ["dialcodes"], "limit": 10000 } }' ``` {% endhint %} {% hint style="warning" %} **Note: ****Only For the adopters who are migrating from the previous versions to 5.2.0, need to follow the following steps:** * Create the cassandra\_migration\_version and cassandra\_migration\_version\_counts tables in respective keyspaces by using the below queries. * Replace \ with the below keyspace names, sunbird\_groups sunbird\_notifications sunbird\_courses ```sql CREATE TABLE .cassandra_migration_version ( version text PRIMARY KEY, checksum int, description text, execution_time int, installed_by text, installed_on timestamp, installed_rank int, script text, success boolean, type text, version_rank int ) WITH bloom_filter_fp_chance = 0.01 AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} AND comment = '' AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'} AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND crc_check_chance = 1.0 AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99PERCENTILE'; CREATE TABLE .cassandra_migration_version_counts ( name text PRIMARY KEY, count counter ) WITH bloom_filter_fp_chance = 0.01 AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} AND comment = '' AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'} AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND crc_check_chance = 1.0 AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99PERCENTILE'; ``` * To export cassandra\_migration\_version table COPY, ```sql sunbird.cassandra_migration_version TO '/tmp/cassandra_migration_version.csv'; ``` * To import cassandra\_migration\_version table COPY ``` .cassandra_migration_version FROM '/tmp/cassandra_migration_version.csv'; ``` * To export cassandra\_migration\_version\_count table COPY ``` sunbird.cassandra_migration_version TO '/tmp/cassandra_migration_version_count.csv'; ``` * To import cassandra\_migration\_version\_count table COPY ``` .cassandra_migration_version_count FROM '/tmp/cassandra_migration_version_count.csv'; ``` {% endhint %} ### Flink Job Configurations for Lern: | Name of the Flink Job added | | --------------------------- | | **user-cache-updater-v2** |
LR-303 - Movement of UserCache in Data Pipeline to Lern - setup/configuration details Flink **build** Jenkins job name: **/Build/job/Lern/job/FlinkJobs** Flink **deploy** Jenkins job name: **/Deploy/job/\/job/Lern/job/FlinkJobs/user-cache-updater-v2**\
### **Data Product Configurations for Lern:** | DataProduct Name | | ---------------- | | UserCacheIndexer |
LR-302 Movement of UserCacheIndexer Data Product to Lern - setup/configuration details Please define the below configuration in Dataproducts (lern-data-products/src/main/resources/application.conf) for the UserCacheIndexerJob data product to work, ``` redis.host=__redis_host__ redis.port="6379" redis.connection.max=20 location.db.redis.key.expiry.seconds=3600 redis.connection.idle.max=20 redis.connection.idle.min=10 redis.connection.minEvictableIdleTimeSeconds=120 redis.connection.timeBetweenEvictionRunsSeconds=300 redis.max.pipeline.size="100000" #CassandraToRedis Config spark.cassandra.connection.host="localhost" cassandra.user.keyspace="sunbird" cassandra.user.table="user" redis.user.database.index="12" redis.user.input.index="4" redis.user.backup.dir="src/mount/data/analytics/content-snapshot/redisbackup" redis.scan.count="100000" redis.user.index.source.key="id" # this will be used as key for redis cassandra.read.timeoutMS="500000" cassandra.query.retry.count="100" cassandra.input.consistency.level="LOCAL_QUORUM" ```
# Release V 5.1.0 ### **Hot-fix: 5.1.1**** (03-04-2023)** | Component | Build Job | Build Tag | Deploy Job | Deployment | Comment | | ------------- | ---------------------------- | ------------------------------------------------------------------------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Batch Service | Build/job/Lern/job/FlinkJobs | [release-5.1.1\_RC1](https://github.com/Sunbird-Lern/data-pipeline/tree/release-5.1.1_RC1) | Deploy/job/Lern/job/FlinkJobs | [release-5.1.1\_RC1](https://github.com/Sunbird-Lern/data-pipeline/tree/release-5.1.1_RC1) |

relational-cache-updater, activity-aggregate-updater jobs need to be deployed

Bug:

| ### Document Release Version | Project | Release Date | Version | | ------- | ------------ | ------- | | Lern | 31 Jan 23 | V 5.1.0 | ### Details of Released Tag | Components | Jenkins Job | Deploy Tags (Devops) | Build Tags (Github Repo Tags) | Github Repository | Comments | | ----------------- | ------------------------------------ | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------- | | Batch Service | Build/Core/Lms | release-5.1.0 |

sunbird-course-service : release-5.1.0\_RC2


| | | | Batch Service | Build/job/Lern/job/FlinkJobs | release-5.1.0 | data-pipeline : [release-5.1.0\_RC1](https://github.com/Sunbird-Lern/data-pipeline/tree/release-5.1.0_RC1) | | relational-cache-updater, activity-aggregate-updater jobs need to be deployed | | User\&Org Service | Build/Core/Learner | release-5.1.0 | sunbird-lms-service : [release-5.1.0\_RC2](https://github.com/Sunbird-Lern/sunbird-lms-service/releases/tag/release-5.1.0_RC2) | | | | Data Products | Build/job/Lern/job/LernDataProducts/ | release-5.1.0 | data-products : [release-5.1.0\_RC1](https://github.com/Sunbird-Lern/data-products/releases/tag/release-5.1.0_RC1) | | | ### **Summary of the Changes** * Support for optional material in a course * Migration of existing certificate in to RC * Making Identity manager optional for setup #### Affected Areas: * Flink Jobs - RelationCacheUpdate, ActivityAggregateUpdater for removing optional contents from course completion calculation * Identity manager made optional for local setup in UserOrg and Batch Service * Exhaust Reports - Progress Exhaust ### Details of the Changes [LR-1 ](https://project-sunbird.atlassian.net/browse/LR-1)Backend :: Support for optional material in a course - Consumption and reporting related changes [LR-4 ](https://project-sunbird.atlassian.net/browse/LR-4)Design on Migration of existing certificate in to RC [LR-241 ](https://project-sunbird.atlassian.net/browse/LR-241)UserOrg: Identity manager should be optional for setup [LR-242](https://project-sunbird.atlassian.net/browse/LR-242) UserOrg:Decoupling the external dependencies so that installation is easier like configuring the form api validation to Sunbird Ed [LR-251 ](https://project-sunbird.atlassian.net/browse/LR-251)BatchService : Identity manager should be optional for setup [LR-131](https://project-sunbird.atlassian.net/browse/LR-131) BatchService: Refactoring of SB Lern Batch Service # Release V 5.0.1 ### Document Release Version | Project | Release Date | Version | | ------- | ------------ | ------- | | Lern | 29 Nov 22 | V 5.0.1 | ### Details of Released Tag | Components | Jenkins Job | Deploy Tags (Devops) | Build Tags (Github Repo Tags) | Github Repository | Comments | | ----------------- | ------------------------------------ | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | Batch Service | Build/Core/Lms | release-5.1.0\_RC1 |

sunbird-course-service : release-5.0.1\_RC3


| | | | Batch Service | Build/Core/Cert | release-5.1.0\_RC1 | cert-service : [release-5.0.1\_RC2](https://github.com/Sunbird-Lern/cert-service/releases/tag/release-5.0.1_RC2) | | | | Batch Service | Build/job/Lern/job/FlinkJobs | release-5.0.1\_RC5 |

data-pipeline : release-5.0.1\_RC5


| | Collection-cert-pre-processor, Collection-certificate-generator jobs need to be deployed | | User\&Org Service | Build/Core/Learner | release-5.1.0\_RC1 | sunbird-lms-service : [\*\*\*\* ](https://github.com/Sunbird-Lern/sunbird-lms-service/releases/tag/release-5.0.0_RC1)[release-5.0.1\_RC2](https://github.com/Sunbird-Lern/sunbird-lms-service/releases/tag/release-5.0.1_RC2) | | | | Data Products | Build/job/Lern/job/LernDataProducts/ | release-5.0.1\_RC4 |

data-products : release-5.0.1\_RC4

| | | ### **Summary of the Changes** * Making SB Lern Cloud agnostic #### Affected Areas: * Textbook upload and download * QRCode list download from workspace * Old certificate download * New certificate generation and download * Exhaust Reports - UserInfo, Progress and Reponse Exhausts * OrgAdmin Reports - Org Consent Report and Geo Reports ### Details of the Changes [LR-262 ](https://project-sunbird.atlassian.net/browse/LR-262)OCI and Open stack support analysis [LR-263 ](https://project-sunbird.atlassian.net/browse/LR-263)CSP integration testing in Non-Ed env [LR-254 ](https://project-sunbird.atlassian.net/browse/LR-254)BatchService: CSP Data migration for certificate obj data in RC ### Env Configurations (Needs to be done before service deployment): The below environment variable needs to be configured in the devops repo. | Variable Name | Values | Comments | | -------------------------- | ------------------------------------------ | --------------------------------- | | cloud\_storage\_base\_url | | To store the CSP base path | | cloud\_storage\_cname\_url | | To store the cname url of the CSP |
Ansible Changes need to be updated in Common.yml, secrets.yml, hosts.yml ``` sunbird-devops-private/ansible/inventory/{{env}}/KnowledgePlatform hosts: ## Lern dataproducts [learning] [raw-broker] [report-cassandra] [raw-coordinator] [redis] [raw-overlord] [lp-cassandra] common: ## Lern dataproducts dp_vault_artifacts_container: db_admin_password: db_password: postgres: db_url: #"{{ groups['postgres'][0] }}" db_username: #analytics db_name: db_password: db_table_name: db_port: 5432 db_admin_user: db_admin_password: data_exhaust_webhook_url: data_exhaust_Channel: data_exhaust_name: user_port: 6379 secrets: ## Lern dataproducts dp_vault_data_exhaust_token: dp_vault_pgdb_admin_password: dp_vault_pgdb_password: dp_vault_druid_postgress_pass: core_vault_sunbird_api_auth_token: core_vault_sunbird_encryption_key: ### This variable added for admin user reports which is bein used to encrypt and decrypt data in cassandra. ```
### Data Migrations: (Run these scripts after service deployment) * **This script is to update the variable based relative url for cloud resources to course\_batch(cassandra) and job\_request(postgres) database tables.** Verification Steps after migration: Check few records in course\_batch table to see whether the certificate template cloud url is changed to variable format and also check in job\_request table to see whether the report url is in relative path format. * **Update blob URL or CNAME URL in ES ad RC DB** To change actual URL to CNAME URL or In case of opting new CSP provider, to change to new blob URL execute below scripts : RC PostgreSQL DB table V\_TrainingCertificate data migration: ES Migrations(course-batch index, trainingcertificate index)[ : ](https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3231449089/CSP+Changes+for+Course+Batch+and+RC)[https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3231449089/CSP+Changes+for+Course+Batch+and+](https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3231449089/CSP+Changes+for+Course+Batch+and+RC)RC Verification Steps after migration: Check ES course\_batch index and TrainingCertificate index to see whether the certificate template cloud url is changed to new csp blob url or cname url. Also check in postgres V\_TrainingCertificate table to see whether the certificate template url is changed to new csp blob url or cname url. * **Update system settings values for sunbird** Run the system-setting for setting the sunbird id value and the read the api ``` curl --location --request POST 'https://staging.sunbirded.org/api/data/v1/system/settings/set ' \ --header 'Authorization: Bearer {{authorization-key}}' \ --header 'Content-Type: application/json' \ --data-raw '{ "request" : { { "id": "sunbird", "field": "sunbird", "value": "{\\\"latestVersion\\\":\\\"v1\\\",\\\"v1\\\":{\\\"url\\\":\\\"https:\/\/obj.stage.sunbirded.org\/portal\/terms-and-conditions-v1.html\\\"}}" } } }' ``` # Release V 5.0.0 ### Document Release Version
ProjectRelease DateVersion
Lern18 Aug 22V 5.0.0
### Details of Released Tag
ComponentsTags
Batch Service

sunbird-course-service : release-5.0.0_RC6

cert-service : release-5.0.0_RC4

certficate-registry : release-5.0.0_RC1

data-pipeline : release-5.0.0_RC6

User&Org Servicesunbird-lms-service : release-5.0.0_RC5
Group Servicegroups-service : release-5.0.0_RC3
Discussion Forum Service

discussions-middleware : release-5.0.0_RC2

sunbird-nodebb : release-5.0.0_RC1

Notification Servicesunbird-notification-service : release-5.0.0_RC6
Data Productsdata-products : release-5.0.0_RC4
### **Summary of the Changes** * Migrating component repositories in to Lern organisation in GitHub as a first step in making the installation and setup easy for adopters and contributors * Increasing code coverage and unit test cases of all the components in Lern as part of stabilising the components * Refactoring of the provisioning and deployment scripts of Lern BB * Making SB Lern Cloud agnostic ### **Details of the Changes** {% tabs %} {% tab title="User\&Org Service" %} | JIRA ID | Descriptions | | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [SB-30076](https://project-sunbird.atlassian.net/browse/SB-30076) | UserOrg - increase code coverage and unit test cases | | [SB-30067](https://project-sunbird.atlassian.net/browse/SB-30067) | UserOrg - Deployment and Release processes | | [SB-29842](https://project-sunbird.atlassian.net/browse/SB-29842) | UserOrg Dataproducts Migration to Lern | | [SB-29826](https://project-sunbird.atlassian.net/browse/SB-29826) | Sunbird-apimanager-util migration to Lern | | [SB-29825](https://project-sunbird.atlassian.net/browse/SB-29825) | Sunbird-auth migration to Lern | | [SB-29824](https://project-sunbird.atlassian.net/browse/SB-29824) | Sunbird-utils - DB Migration to Lern | | [SB-29823](https://project-sunbird.atlassian.net/browse/SB-29823) | UserOrg Migration to Lern | | [SB-29813](https://project-sunbird.atlassian.net/browse/SB-29813) | OrgSearch to allow partial search and fuzzy Search | | [LR-103](https://project-sunbird.atlassian.net/browse/LR-103) | Making SB Lern Cloud agnostic : Code changes to generalise CSP support in UserOrg | | [LR-124](https://project-sunbird.atlassian.net/browse/LR-124) | Cassandra related changes for supporting multiple Data Centers | | [LR-232](https://project-sunbird.atlassian.net/browse/LR-232) | CSP changes for cert, learner & course service | | [LR-128](https://project-sunbird.atlassian.net/browse/LR-128) | [Data Migration related to CSP changes : Report url in Job\_request table needs to be updated](https://project-sunbird.atlassian.net/browse/LR-128) | | [LR-125](https://project-sunbird.atlassian.net/browse/LR-125) | [Data Migration related to CSP changes : Template url in DB Certificate Objects needs to be updated - batch tables](https://project-sunbird.atlassian.net/browse/LR-125) | | [LR-113](https://project-sunbird.atlassian.net/browse/LR-113) | [Data Migration related to CSP changes : Existing reports need to be migrated](https://project-sunbird.atlassian.net/browse/LR-113) | | [LR-112](https://project-sunbird.atlassian.net/browse/LR-112) | [Data Migration related to CSP changes : Credential template, context files to be stored in to new CSP](https://project-sunbird.atlassian.net/browse/LR-112) | | [LR-111](https://project-sunbird.atlassian.net/browse/LR-111) | [Data Migration related to CSP changes : Old Certificates in Azure needs to be migrated](https://project-sunbird.atlassian.net/browse/LR-111) | | [LR-110](https://project-sunbird.atlassian.net/browse/LR-110) | [Data Migration related to CSP changes : Template url in DB Certificate Objects needs to be updated - RC tables and ES](https://project-sunbird.atlassian.net/browse/LR-110) | | [LR-109](https://project-sunbird.atlassian.net/browse/LR-109) | [Data Migration related to CSP changes:Exisisting Certificate templates needs to be migrated.](https://project-sunbird.atlassian.net/browse/LR-109) | Configurations: ``` Sunbird-lms-service: isMultiDCEnabled={{cassandra_isMultiDCEnabled}} sunbird_cassandra_consistency_level={{sunbird_cassandra_consistency_level}} sunbird_user_cert_kafka_topic={{kafka_topic_lms_user_account}} sunbird_cloud_service_provider={{cloud_service_provider}} sunbird_account_name={{sunbird_public_storage_account_name}} sunbird_account_key={{sunbird_public_storage_account_key}} ``` {% endtab %} {% tab title="Batch Service" %}
JIRA IDDescription
SB-30075Batch service - increase code coverage and unit test cases
SB-29843Batch service Dataproducts Migration to Lern
SB-30068Batch service - Deployment and Release processes
SB-29841assessment aggregate updater Migration to Lern
SB-29840activity aggregate updater Migration to Lern
SB-29839course-service migration to Lern
SB-29837enrolment-reconciliation Migration to Lern
SB-29836relation cache updater Migration to Lern
SB-29835colletion-certificate-generator Migration to Lern
SB-29834collection-cert-pre-processor Migration to Lern
SB-29833certificate processor Migration to Lern
SB-29832certificate-registry migration to Lern
SB-29831cert-service migration to Lern
LR-104Making SB Lern Cloud agnostic : Code changes to generalise CSP support in BatchService
LR-105Making SB Lern Cloud agnostic : Code changes to generalise CSP support in datapipeline
LR-106Making SB Lern Cloud agnostic : Code changes to generalise CSP support in data products
LR-108Signed URL Generation for old certificates - cert-service
LR-107RC deploy helm chart changes to upload Credential template, context files
Configurations: ``` https://github.com/project-sunbird/sunbird-devops/blob/learn-bb/ansible/roles/stack-sunbird/templates/ Sunbird-course-service: isMultiDCEnabled={{cassandra_isMultiDCEnabled}} # Add proper cloud service provider (azure,aws,gcloud) sunbird_cloud_service_provider={{cloud_service_provider}} sunbird_account_name={{sunbird_public_storage_account_name}} sunbird_account_key={{sunbird_public_storage_account_key}} #deleted the below variable sunbird_content_azure_storage_container=sunbird-content-dev # Added below variable for supporting multiple cloud service providers # Provide corresponding cloud service provider(azure,aws,gcloud) container name here sunbird_content_cloud_storage_container=sunbird-content-dev ``` 
Cert-service:
CONTAINER_NAME={{cert_service_container_name}}
# Add proper cloud service provider (azure,aws,gcloud)
CLOUD_STORAGE_TYPE={{cloud_service_provider}}
PRIVATE_CLOUD_STORAGE_SECRET={{sunbird_private_storage_account_key}}
PRIVATE_CLOUD_STORAGE_KEY={{sunbird_private_storage_account_name}}
PUBLIC_CLOUD_STORAGE_KEY={{sunbird_public_storage_account_name}}
PUBLIC_CLOUD_STORAGE_SECRET={{sunbird_public_storage_account_key}}
``` Certificate-registry: isMultiDCEnabled={{cassandra_isMultiDCEnabled}} ``` ``` Data-pipeline: isMultiDCEnabled={{cassandra_isMultiDCEnabled}} # Add proper cloud service provider (azure,aws,gcloud) cloud_storage_type :azure/aws/gcloud ``` ``` data-products: cloud_storage_type :azure/aws/gcloud ``` {% endtab %} {% tab title="Discussion Forum" %} | JIRA ID | Descriptions | | ----------------------------------------------------------------- | ----------------------------------------------- | | [SB-30070](https://project-sunbird.atlassian.net/browse/SB-30070) | DF - Deployment and Release processes | | [SB-29820](https://project-sunbird.atlassian.net/browse/SB-29820) | DF Migration to Lern and deployment setup | | [SB-30073](https://project-sunbird.atlassian.net/browse/SB-30073) | DF - increase code coverage and unit test cases | | {% endtab %} | | {% tab title="Group Service" %} | JIRA ID | Description | | ----------------------------------------------------------------- | ---------------------------------------------------------- | | [SB-30074](https://project-sunbird.atlassian.net/browse/SB-30074) | Group service - increase code coverage and unit test cases | | [SB-30069](https://project-sunbird.atlassian.net/browse/SB-30069) | Group service - Deployment and Release processes | | [SB-29819](https://project-sunbird.atlassian.net/browse/SB-29819) | Group-service Migration to Lern and deployment setup | Configurations: ``` groups-service: File Path: cassandra-utils/src/main/resources/cassandra.config.properties Changes: isMultiDCEnabled=false File Path: sb-utils/src/main/resources/cassandra.config.properties Changes: isMultiDCEnabled=false ``` {% endtab %} {% tab title="Notification Service" %} | JIRA ID | Description | | ----------------------------------------------------------------- | ------------------------------------------------------------------------- | | [SB-30077](https://project-sunbird.atlassian.net/browse/SB-30077) | Sunbird-notification-service - increase code coverage and unit test cases | | [SB-29828](https://project-sunbird.atlassian.net/browse/SB-29828) | Sunbird-notification-jobs migration to Lern | | [SB-29827](https://project-sunbird.atlassian.net/browse/SB-29827) | Sunbird-notification-service migration to Lern | | [SB-30071](https://project-sunbird.atlassian.net/browse/SB-30071) | Sunbird-notification-service - Deployment and Release processes | Configurations: ``` Sunbird-notification-service: File path: sb-utils/src/main/resources/cassandra.config.properties Changes: isMultiDCEnabled=false ``` {% endtab %} {% endtabs %} Detailed Information is present in the [JIRA](https://project-sunbird.atlassian.net/issues/?filter=12509) list. **Configurations:** 1. Nodebb upstream branch: v1.18.6 2. Jenkins build, deploy and upload related changes for Flink jobs are present in below link: {% embed url="" %} 3\. There is a new variable added in devops repo to configure the bb name for kafka topics of flink jobs. This variable should be appended after the env name. {% embed url="" %} ``` List of Lern Flink jobs: collection-cert-pre-processor collection-certificate-generator activity-aggregate-updater relation-cache-updater merge-user-courses assessment-aggregator enrolment-reconciliation notification-job ``` ``` env variable changes are listed below: kubernetes/ansible/roles/flink-jobs-deploy/defaults/main.yml ### Merge User Courses Job related Vars merge_user_courses_consumer_parallelism: 1 merge_user_courses_parallelism: 1 merge_user_courses_course_batch_parallelism: 1 merge_user_courses_course_date_format: "yyyy-MM-dd HH:mm:ss:SSSZ" ### Notification Job related Vars notification_job_consumer_parallelism: 1 notification_job_parallelism: 1 ### assessment-aggregator related vars assessaggregator_parallelism: 1 assessaggregator_consumer_parallelism: 1 assessaggregator_downstream_parallelism: 1 assessaggregator_scoreaggregator_parallelism: 1 middleware_cassandra_courses_keyspace: sunbird_courses middleware_cassandra_assessment_aggregator_table: assessment_aggregator middleware_cassandra_assessment_question_type : question middleware_cassandra_user_enrolments_table: user_enrolments middleware_cassandra_user_activity_agg_table: user_activity_agg content_read_api_host: "http://dev.sunbirded.org" content_read_api_endpoint: "/api/content/v1/read/" merge-user-courses: job_class_name: 'org.sunbird.job.merge.user.courses.task.MergeUserCoursesStreamTask' replica: 1 jobmanager_memory: 1024m taskmanager_memory: 1024m taskslots: 1 cpu_requests: 0.3 assessment-aggregator: job_class_name: 'org.sunbird.dp.assessment.task.AssessmentAggregatorStreamTask' replica: 1 jobmanager_memory: 1024m taskmanager_memory: 1024m taskmanager_process_memory: 1700m jobmanager_process_memory: 1600m taskslots: 1 cpu_requests: 0.3 scale_enabled: false notification-job: job_class_name: 'org.sunbird.job.notification.task.NotificationStreamTask' replica: 1 jobmanager_memory: 1024m taskmanager_memory: 1024m taskslots: 1 cpu_requests: 0.3 Stop the existing Samza jobs - (merge-user-courses and notification-job) Stop the assessment-aggregator job from sunbird-data-pipeline and remove it from the corresponding Jenkins job as well All these 3 jobs will be running from LERN repo now ``` 4\. Jenkins build, deploy and upload related changes for data products are in below link: {% embed url="" %} ``` LERN data-products list exhaust / progressexhaustjob exhaust / responseexhaust exhaust / userinfoexhaust job / course consumption job / course enrolment job / stateadminreport job / stateadmingeoreport job / collectionsummaryjobv2 audit / collection reconcilation audit / coursebatch status updater updater / Cassandramigrator ``` 5\. Jenkins build, deploy and upload related changes for microservices are in below link: {% embed url="" %} **Devops config changes:** {% code overflow="wrap" %} ``` File Path: ansible/inventory/env/group_vars/all.yml ### Release-5.0.0 cloud service provider changes for supporting multiple providers ### ### cloud_service_provider value should be either (azure, aws, gcloud) as per cloud sdk dependency ### cloud_service_provider: "azure" ## modified consistency levels from quorum to local_quorum for multiple data centers support File Path: ansible/roles/stack-sunbird/templates/sunbird_cert-registry-service.env Changes: sunbird_cassandra_consistency_level=local_quorum File Path: ansible/roles/stack-sunbird/templates/sunbird_groups-service.env Changes: sunbird_cassandra_consistency_level=local_quorum File Path: ansible/roles/stack-sunbird/templates/sunbird_learner-service.env Changes: sunbird_cassandra_consistency_level=local_quorum sunbird_user_cert_kafka_topic={{env_name}}{{bb}}.lms.user.account.merge File Path: ansible/roles/stack-sunbird/templates/sunbird_lms-service.env Changes: sunbird_cassandra_consistency_level=local_quorum File Path: ansible/roles/stack-sunbird/templates/sunbird_notification-service.env Changes: sunbird_cassandra_consistency_level=local_quorum sunbird_notification_kafka_topic={{env_name}}{{bb}}.lms.notification File Path: kubernetes/helm_charts/sunbird-RC/registry/schemas/TrainingCertificate.json Changes: "credentialTemplate": "https://{{upstream_url}}/schema/credential_template.json" File Path: utils/sunbird-RC/schema/credential_template.json Changes: "https://{{upstream_url}}/schema/v1_context.json", "https://{{upstream_url}}/schema/sunbird_context.json" ``` {% endcode %} Design Documentation: {% embed url="" %} Sunbird-Lern Code, Flink Jobs and Data Products {% endembed %} {% embed url="" %} Lern BB repositories {% endembed %} {% embed url="" %} Sunbird-Lern existing jenkins jobs {% endembed %} # Release V 4.10.0 ### Document Release Version
ProjectRelease DateVersion
Lern10 June 22 (tentative)V 4.10.0
### Details of Released Tag | Component | Tag | | ------------------- | -------------------------------------------- | | Cassandra migration | sunbird-utils : release-4.10.0\_RC1 | | User\&Org Service | sunbird-lms-service : release-4.10.0\_RC1 | | Discussion Forum | discussions-middleware : release-4.10.0\_RC1 | ### **Summary of the Changes** * DF Moderation changes * OrgSearch to allow partial search and fuzzy Search ### **Details of the Changes** {% tabs %} {% tab title="User\&Org Service" %} | JIRA ID | Descriptions | | ----------------------------------------------------------------- | -------------------------------------------------- | | [SB-27866](https://project-sunbird.atlassian.net/browse/SB-27866) | User accounts without Phone or Email Issue Fix | | [SB-29813](https://project-sunbird.atlassian.net/browse/SB-29813) | OrgSearch to allow partial search and fuzzy Search | | {% endtab %} | | {% tab title="Discussion Forum" %} | JIRA ID | Descriptions | | ----------------------------------------------------------------- | -------------------------------------- | | [SB-29794](https://project-sunbird.atlassian.net/browse/SB-29794) | Discussion forum >> Moderation changes | **New Environment variables** ``` enable_audit_event={{ enable_audit_event | default(true) }} moderation_flag={{ moderation_flag | default(false) }} ``` {% endtab %} {% endtabs %} Detailed Information is present in the [JIRA](https://project-sunbird.atlassian.net/issues/?filter=12500) list. ### Manual Configurations | Manual Tasks | Details | Comments | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ---------------------- | | Reindex Org ES Index in UserOrg Service | | New index name - orgv3 | # Release V 4.9.0 ### Document Release Version | Project | Release Date | Version | | ------- | ------------ | ------- | | Lern | 18 May 2022 | V 4.9.0 | ### Details of Released Tag: | Component | Tag | | ----------------- | --------------------------------------------------------------------------------------------------------------- | | Batch Service | [**release-4.9.0\_RC4**](https://github.com/sunbird-lern/sunbird-course-service/releases/tag/release-4.9.0_RC4) | | User\&Org Service | [**release-4.9.0\_RC1**](https://github.com/sunbird-lern/sunbird-lms-service/releases/tag/release-4.9.0_RC1) | ### **Summary of the Changes** * Enhancements for User and Org Data required for iGOT * Enhancements in Course Service * Enhanced Org Type Management * Extended User Profile ### **Details of the Changes** {% tabs %} {% tab title="Batch Service" %} | JIRA ID | Descriptions | | ----------------------------------------------------------------- | ----------------------------------------------- | | [SB-29551](https://project-sunbird.atlassian.net/browse/SB-29551) | Enhancements in Course Service | | [SB-29362](https://project-sunbird.atlassian.net/browse/SB-29362) | Storing RC context JSON in azure for deployment | | {% endtab %} | | {% tab title="User\&Org Service" %} | JIRA ID | Descriptions | | ----------------------------------------------------------------- | ---------------------------------------------------- | | [SB-29553](https://project-sunbird.atlassian.net/browse/SB-29553) | Enhancements for User and Org Data required for iGOT | | [SB-29490](https://project-sunbird.atlassian.net/browse/SB-29490) | New feature - Enhanced Org Type Management | | [SB-29489](https://project-sunbird.atlassian.net/browse/SB-29489) | New feature - Extended User Profile | | {% endtab %} | | | {% endtabs %} | | Detailed Information is present in the [JIRA](https://project-sunbird.atlassian.net/issues/?filter=12456) list. **Manual Configuration** |

Add soft link to Core/keys directory
in private repo for Sunbird-RC directory

| ln -s ../Core/keys/ keys | | --------------------------------------------------------------------------------------- | ------------------------ | # Release V 4.8.0 ### Document Release Version | Project | Release Date | Version | | ------- | ------------- | ------- | | Lern | 28 April 2022 | V 4.8.0 | ### Details of Released Tag: | Component | Tag | | -------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Batch Service | [**release-4.8.0\_RC2**](https://github.com/project-sunbird/sunbird-course-service/releases/tag/release-4.8.0_RC2) | | CertRegistry | release-4.8.5\_RC7 | | kp flink | release-4.8.5\_RC4 | | Sunbird-Rc/CertificateApi | | | Sunbird-Rc/CertificateSign | | | Sunbird-Rc/Registry | | ### **Summary of the Changes** * Certificate Registry Integration with RC * Update Lern BB Microsite * Build and Deployment automation of Sunbird RC * Contribute to RC on QR code generation with a specific type * Certificate Generator Job Integration with RC to re-issue the certificate ### **Details of the Changes** {% tabs %} {% tab title="Batch Service" %} | JIRA ID | Descriptions | | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | [SB-29268](https://project-sunbird.atlassian.net/browse/SB-29268) | v1 RC create certificate API request is passed with valid parameters. | | [SB-29262](https://project-sunbird.atlassian.net/browse/SB-29262) | \[Certificate key API] Certificate Key search API with Invalid osid | | [SB-29126](https://project-sunbird.atlassian.net/browse/SB-29126) | Contribute to RC on QR code genration with specific type | | [SB-29124](https://project-sunbird.atlassian.net/browse/SB-29124) | Script to migrate templates in new format to support certificate generation using RC | | [SB-28980](https://project-sunbird.atlassian.net/browse/SB-28980) | Added validation to Certificate SignatoryList attributes | | [SB-28969](https://project-sunbird.atlassian.net/browse/SB-28969) | Enable flag for validate Certificate SignatoryList | | [SB-28746](https://project-sunbird.atlassian.net/browse/SB-28746) | Configuring and setting up sunbird-RC instance | | [SB-28733](https://project-sunbird.atlassian.net/browse/SB-28733) | Certificate Generator Job Integration with RC to re-issue certificate | | [SB-28732](https://project-sunbird.atlassian.net/browse/SB-28732) | Certificate Generator Job Integration with RC to issue certificate and store certificate type, id and issue date in passbook | | [SB-29068](https://project-sunbird.atlassian.net/browse/SB-29068) | Certificate Registry Integration with RC for download and search certificate | | [SB-29118](https://project-sunbird.atlassian.net/browse/SB-29118) | API to store and fetch public key from RC | | {% endtab %} | | {% tab title="Notification Service" %} | JIRA ID | Descriptions | | ----------------------------------------------------------------- | ------------------------------------------------ | | [SB-29311](https://project-sunbird.atlassian.net/browse/SB-29311) | Redirection on onclick of notification bell icon | | {% endtab %} | | | {% endtabs %} | | Detailed Information is present in the [JIRA](https://project-sunbird.atlassian.net/issues/?filter=12417) list. **Environment Changes:** | Variable Names | Env | value | | ----------------------------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | enable\_suppress\_exception | kp flink | | | enable\_rc\_certificate | kp flink | | | CERTIFICATE\_PRIVATE\_KEY | CertificateSign |

add cert private key variable in secrets.yml

0

| | collection\_certificate\_generator\_enable\_rc\_certificate | sunbird-learning-platform | | **Manual Configurations:** |

Template migration for sunbirdRc integration, please follow the wiki for the same

| | | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |

Run curl command from registry service pod
and share response id with dev team

|

curl --location --request POST '' \
--header 'Content-Type: application/json' \
--data-raw '{
"value": "",
"alg": "RSA"
}'

| | Create sunbird-RC inventory folder(soft link to core common.yml, host, secrets.yml | | | Create database in postgress kong (CREATE DATABASE registry;) | | | Upload\_RC\_Schema | | # Release V 4.7.0 ### Document Release Version | Project | Release Date | Version | | ------- | ------------- | ------- | | Lern | 01 March 2022 | V 4.7.0 | ### Details of Released Tag: | Component | Tag | | ----------------- | --------------------------------------------------------------------------------------------------------------- | | User\&Org Service | [**release-4.9.0\_RC4**](https://github.com/sunbird-lern/sunbird-course-service/releases/tag/release-4.9.0_RC4) | | Discussion Forum | [**release-4.7.0\_RC4**](https://github.com/Sunbird-Lern/discussions-middleware/releases/tag/release-4.7.0_RC4) | ### **Summary of the Changes** In 4.7.0, the following changes were implemented by the Lern BB, * Fixed course enrolment list API issue, while fetching more than 1000 enrolments * Sorted enrolment API list response as per last access date * Added log4j vulnerability fix * Standardized Error code implementation in userOrg service * Extensive updates to the Lern microsite documentation () ### **Details of the Changes** {% tabs %} {% tab title="UserOrg Service" %} | JIRA ID | Descriptions | | ----------------------------------------------------------------- | ---------------------------------------------- | | [SB-28201](https://project-sunbird.atlassian.net/browse/SB-28201) | Error Code implementation | | [SB-27875](https://project-sunbird.atlassian.net/browse/SB-27875) | Consent report data validation and corrections | | [SB-28039](https://project-sunbird.atlassian.net/browse/SB-28039) | Log4j vulnerability Fix | | [SB-28560](https://project-sunbird.atlassian.net/browse/SB-28560) | Deprecated FLAG\_REVIEWER role | | {% endtab %} | | {% tab title="Batch Service" %} | JIRA ID | Descriptions | | ----------------------------------------------------------------- | ---------------------------------------------------------------------------- | | [SB-28497](https://project-sunbird.atlassian.net/browse/SB-28497) | Fixed enrolment API sorting issue | | [SB-28560](https://project-sunbird.atlassian.net/browse/SB-28560) | ES-client upgrade to 6.8.22. Error code updation for UserOrgService API call | | [SB-28064](https://project-sunbird.atlassian.net/browse/SB-28064) | Log4j vulnerability Fix | | {% endtab %} | | {% tab title="Discussion Forum" %} | JIRA ID | Description | | ----------------------------------------------------------------- | ----------------------------------- | | [SB-28567](https://project-sunbird.atlassian.net/browse/SB-28567) | DF audit events configuration check | | [SB-28743](https://project-sunbird.atlassian.net/browse/SB-28743) |

pdata version update

| | {% endtab %} | | | {% endtabs %} | | Detailed Information is present in the [JIRA](https://project-sunbird.atlassian.net/issues/?filter=12362) list. # Developer Guide Sunbird Lern enables a host of capabilities that have been utilised by various adopters to create program specific workflows for their users. For instance, [Shikshalokam](https://shikshalokam.org) has created an leadership platform to train leaders in education - using componets from Sunbird Lern such as the User & Org service as well as the Batch Service. The flexibility of being able to choose the components that are required for the adopter, and the ability to configure them at will (instead of having to build ground up) is what makes it easier to deploy solutions using the Sunbird suite. The following table lists out the various components that make up the Sunbird Lern building block, with a brief description of what each module pertains to. This can be used to get an understanding of what capabilities this specific building block can be used to enable. Further descriptions of each component, the configurations available within etc. are detailed in the pages specific to the component.
User & Org ServiceAPIs that enable user and org management operations (including C-R-U-D) for users and organisations on the platform, as well as several other capabilities including Consent management as well as OTP services. See the page for the 'User Org service' component for further details.https://github.com/sunbird-lern/sunbird-lms-service
Batch ServiceThe Batch Service comprises APIs that permit for creation of cohorts of users in the context of content collections, allowing for Batch management capabilities (tracking progress of users within a batch, assigning of mentors to batches as well as start and end dates for batches, attaching credentials etc.)https://github.com/sunbird-lern/sunbird-course-service
Group ServiceAPIs that enable Groups related functionalities which include create/ deactivate groups, add/remove members, add/remove learning assets and enable/disable forums.https://github.com/sunbird-lern/groups-service
Discussion ForumGroup of APIs that enable embedding of discussion forums in any context within Sunbird. Also see Discussion Forum UX tool.https://github.com/sunbird-lern/discussions-middleware
Notification ServiceThis module powers the creation and sending of notifications to sets of users - this is a capability that can be leveraged by other workflows to communicate to their end users.https://github.com/sunbird-lern/sunbird-notification-service
# USER & ORG SERVICE This component consists of various services which support user authentication, API management and token generation, Samza jobs for asynchronous notification along with user and organisation management. ![User Org service - Features](/files/hspnXdUZSDD1tkbTKwA6) ### User & Org Service: This service provides a set of APIs to manage the user, organisation, and location information. 1. **Location** - It is mainly the geographical location of the organisation or user. Currently, it supports - state, district, block, and cluster location types. 2. **Organisation** - Supports tenant or non-tenant organisations. Tenant organisation has to have a channel and slug which is unique. Non-tenant organisations should have a channel which is having a tenant mapped to it. There should be a default organisation called ‘custodian’ created during setup. 3. **User** - The user can be an LUA(Logged in User) or MUA(Managed User) user. LUA should have an email or phone number, which can be used for login. MUA will not have email and phone, their profile can be only used after logging in with LUA credentials. Each user should be mapped to the custodian or other tenant orgs. ### Terms: #### **Slug** * Slug is a text and its a set of characters, usually 2 to 4 in length, is used in a URL * Slug and channel are abbreviations of organisation name, where a slug is a URL compatible, while the channel is not. * Most often the channel and slug are the same, but it is not guaranteed. **Ex:** slug : channel1003 #### Channel * Channel is a text and is Usually, a two-letter string that denotes a unique hashtagid registered in the global index. * Channel is an entity in content-service(KP) against which framework, BGMS, and contents are mapped. * It has a name, id/code, and description. * Channel id/code can be the Organisation id or any dummy digits. * It is independent of Organisation, but for the portal or app to use it, the channel id needs to be mapped to the organisation id. * Channel abbreviations like '**tn'** and '**ka**' are not used in content-service(KP) * Channel is a neo4j object, there is no tabular structure. * The channel is ‘exactly’ the same for the rootOrg-subOrg combination. **Ex:** channel: channel1003 #### Tenant Organisation * It is an entity in UserOrg service against which all users are mapped. * Main properties are id, name, organisationtype, orglocation, isTenant, slug, channel, externalid. * Slug and channel are abbreviation of organisation name, where slug is URL compatible, while channel is not. * A default tenant organisation needs to be created in sunbird, which is termed as custodian org, with a unique slug and channel. * While creating org, assigning a channel is mandatory. If tenant org needs to be created, then the channel should be registered in content-service, also it should be unique. Organisationid is passed to channel API to register it as a channel in content service. * If the channel is already existing in content service then use the channel id/code as the organisation id. * Sunbird Ed searches the tenant organisation using slug and get the organisation id. This is passed in channel/v1/read to get the channel configurations and details from content-service. * Tenant org and channel both are the same for the Sunbird-Ed, only that different properties of the channel like content, framework, BGMS are handled by content-service and user, location, user details are handled by userOrg service. #### Non-Tenant Organisation * While creating a non-tenant organisation, mention an existing tenant's channel. This channel's org id is searched before creating a new organisation. * If it is not a tenant org, then the channel should be internally mapped to a tenantOrg by specifying the tenant org channel. * Non-tenant organisation - channel relation is not used in the Sunbird-Ed #### Org External Id * Org External Id is a unique id of the organisation. #### Organisation Type * It will save the int values corresponding to the organisation types like board/school/contentOrg. * The types will be represented with bit positions * bit 0 isBoard * bit 1 isSchool * bit 2 canCreateContent * bit 3 isBoard So the value in digit for board - 5, school - 2. #### RootOrg Vs Tenant Org * Most often the rootOrg and the tenant org are the same for representing the board, but it is not guaranteed. #### Tenant Association * There is only one tenant association possible for a user. #### Non-Tenant Association * There can only be one active non-tenant association as of now for a user. * When a new association is added, the old one is made inactive. **Adopters:** Diksha **Contributors**: EkStep **Last Release Date**: April 28, 2022 **Version:** 4.9.0 # Features/Core capabilities ### **Key Features:** | Feature | Description | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | User Management (CRUD) | Creation of LUA and MUA users and updating user profile, fetching user information, block and unblock user, associating user with organisation etc are implemented via this feature | | Organisation Management (CRUD) | This feature provides CRUD APIs for maintaining organisation details. | | Location Management (CRUD) | Provides CRUD APIs for managing location master data. | | Consent Management | This helps to capture user consent to share the PII in organisation level and course collection level. | | OTP Services | Helps to generate OTP and send notification either through email or phone. Also OTP validation, expiry, rate limit are managed using this feature. | | Notes Management | User can capture notes when content is being played. This service provides the CRUD APIs for the same. | | Tenant Configurations | This service can be used to maintain tenant organisation level configurations. | | Backend Services | Admins can upload master data in to location, organisation. This service allows to create users and update user information. | |


System Settings

| This service is used to configure system / application settings like default organisation, various T\&C etc | # Architecture ## UserOrg Service Architecture The below diagram represents the components involved and their arrangement in **UserOrg** service.

UserOrg Architecture

### Flow Diagram:

UserOrg Flow Diagram

* User service includes multiple operations to creating new users and search the user. * Keyclock for user authentication. * Enabled Notification feature and messaging like Email and SMS on any operations performed. * Updation of user details using jobs. * OTP Creations on user merge action. * Admin Util will do token verification. * Content service is for to register the channel and framework validation. * Userorg service will send an audit event to `{env}.telemetry.raw` topic. This event will be processed by pipeline-preprocessor(SB-Obsrv) and valid events will pushed to the audit topic to cache the user data by `user-cache-updater-v2` flink job. {% embed url="" %} User-Org Highlevel Architecture {% endembed %} # Code Flow The below diagram represents the code flow of **UserOrg** service.

code flow

### Repository #### UserOrg Service {% embed url="" %} UserOrg Service {% endembed %} UserOrg Service APIs {% embed url="" %} UserOrg Service APIs {% endembed %} #### Flink Jobs {% embed url="" %} UserOrg data-popeline {% endembed %} {% embed url="" %} UserOrg Flink Jobs {% endembed %} #### Reports {% embed url="" %} UserOrg data-product {% endembed %} {% embed url="" %} UserOrg Reports {% endembed %} # Developer Installation ### Getting Started Guide This guide helps you to install [User\&Org Service](https://github.com/sunbird-lern/sunbird-lms-service) on developer machine. It includes instructions for installing the Cassandra, Elasticsearch, Keycloak server in standalone mode, along with [admin-util ](https://github.com/sunbird-lern/sunbird-apimanager-util)service for managing users and organisations. # System Requirements Ensure that your laptop or desktop has the following minimum system requirements: * Operating System: Windows 7 and above, or 4.2 Mac OS X 10.0 and above/Linux * RAM: >4 GB * CPU: 4 cores, >2 GHz # Tech Stack | Technology | Version | | ------------- | ------- | | Java | 11 | | Keycloak | 7.0.1 | | Elasticsearch | 6.8.11 | | Cassandra | 3.11.8 | # Installation Guide ### UserOrg Service Developer installation: {% embed url="" %} ### UserOrg Service Setup: To make the User/Org service completely working, some pre-required configuration setup is mandatory. Follow the steps given in the link below to complete the setup : {% embed url="" %} # Keycloak Local setup Local setup documentation of keycloak and integration with user-org service {% embed url="" %} # Additional Installation Dependencies: ### Authentication: This service is used to validate a user while authenticating using keycloak. Keycloak uses this internally to fetch user information from the User & Org Service. **GitHub Repository:** [GitHub - sunbird-lern/sunbird-auth: Repository for sunbird authentication service](https://github.com/sunbird-lern/sunbird-auth) **Adopters:** Diksha **Contributors**: EkStep **Last Release Date**: **Version :** 3.8.0 ### API manager util: Wrapper for Kong admin util. This wrapper exposes APIs which can be used to register kong consumers and credentials. This service should not be exposed to the internet. **GitHub Repository:** [GitHub - sunbird-lern/sunbird-apimanager-util: Wrapper for Kong admin util](https://github.com/sunbird-lern/sunbird-apimanager-util) **Adopters:** Diksha **Contributors**: EkStep **Last Release Date**: Oct 21 2021 **Version :** 4.3.0 ### Background Jobs: This repository is for event driven jobs user in Sunbird User & Org Service **GitHub Repository:** [GitHub - project-sunbird/sunbird-lms-jobs: Repository for background jobs in Sunbird LMS](https://github.com/Sunbird-Lern/data-pipeline/tree/master/lms-jobs) **Adopters:** Diksha **Contributors**: EkStep **Last Release Date**: **Version :** 3.7.0 # Source Code **Source Code:** UserOrg Service {% embed url="" %} Authentication - Keycloak SPI {% embed url="" %} API Manager Util {% embed url="" %} Report Jobs and other adhoc scripts: Geo report, User-consent report (before release-5.0.0) [StateAdminGeoReportJob.scala](https://github.com/Sunbird-Ed/sunbird-data-products/blob/master/data-products/src/main/scala/org/sunbird/analytics/job/report/StateAdminGeoReportJob.scala), [StateAdminReportJob.scala](https://github.com/Sunbird-Ed/sunbird-data-products/blob/master/data-products/src/main/scala/org/sunbird/analytics/job/report/StateAdminReportJob.scala) {% embed url="" %} Report Jobs and other adhoc scripts: (after release-5.0.0) {% embed url="" %} Environment and API Configurations {% embed url="" %} Database setup: {% embed url="" %} ES Mappings: {% embed url="" %} # Installation Configuration ### Telemetry Config ``` #Telemetry producer related info telemetry_pdata_id=local.sunbird.learning.service telemetry_pdata_pid=learning-service telemetry_pdata_ver=4.7.0 ``` ### KeyClock Config ``` sunbird_keycloak_required_action_link_expiration_seconds=155520000 ``` ### SSO Config: ``` sso.url= sso.realm=sunbird sso.connection.pool.size=20 sso.enabled=true sunbird_sso_client_id= sunbird_sso_username= sunbird_sso_password= sunbird_sso_url= sunbird_sso_realm= ``` ### Elastic Search Config: ``` es.cluster.name=test es.host.name=localhost es.host.port=9300 #elastic search top n result count for telemetry searchTopN=5 ``` ### Cassandra Config ``` coreConnectionsPerHostForLocal=4 coreConnectionsPerHostForRemote=2 maxConnectionsPerHostForLocal=10 maxConnectionsPerHostForRemote=4 maxRequestsPerConnection=32768 heartbeatIntervalSeconds=60 poolTimeoutMillis=0 queryLoggerConstantThreshold=300 ``` ### Kafka Config ``` kafka_urls=localhost:9092 kafka_linger_ms=5 ``` ### Notification Config ``` notification_service_base_url= notification_service_v2_send_url=/private/v2/notification/send notification_service_v1_update_url=/private/v1/notification/feed/update notification_service_v1_read_url=/private/v1/notification/feed/read notification_service_v1_delete_url=/private/v1/notification/feed/delete #NIC nic_sms_gateway_provider_base_url=https://smsgw.sms.gov.in/failsafe/HttpLink ``` ### Message Config ``` sunbird.msg.91.country=91 sunbird.msg.91.sender=TesSun sunbird.msg.91.auth= sunbird.msg.91.method=POST sunbird.msg.91.route=4 sunbird.msg.91.baseurl=http://api.msg91.com/ sunbird.msg.91.get.url=api/sendhttp.php? sunbird.msg.91.post.url=api/v2/sendsms ``` ### Mail Template Config ``` orgName=Diksha onboarding_mail_subject=Welcome to {0} onboarding_welcome_message=Welcome to {0} onboarding_welcome_mail_body=Please ensure that you change your password according to instructions when you log in for the first time. mail_note=Note: This is an automatic alert email. Replies to this mail box will not be monitored. If you are not the intended recipient of this message, or need to communicate with the team, write to ``` ### Mail Config ``` sunbird_mail_server_host= sunbird_mail_server_port= sunbird_mail_server_username= sunbird_mail_server_password= sunbird_mail_server_from_email=support@open-sunbird.org sunbird_account_name= sunbird_account_key= sunbird_encryption_key=SunBird sunbird_encryption=ON ``` ### SMS Config ``` nic_sms_gateway_provider_base_url=https://smsgw.sms.gov.in/failsafe/HttpLink sms_gateway_provider=91SMS ``` ### Other Config ``` sunbird_installation=sunbird sunbird_analytics_api_base_url=https://dev.ekstep.in/api/data/v3 ekstep_api_base_url=https://dev.ekstep.in/api sunbird_allowed_login=You can use your cellphone number to login sunbird_web_url=https://dev.sunbirded.org sunbird_framework_read_api=/v1/framework/read fcm.url=https://fcm.googleapis.com/fcm/send #put the default evn logo url here or System Env variable with #same key. code will first search from EVN then here. sunbird_env_logo_url=http://via.placeholder.com/100x50 system_settings_properties=phoneUnique,emailUnique sunbird_default_welcome_sms=Welcome to DIKSHA. sunbird_url_shortner_base_url=https://api-ssl.bitly.com/v3/shorten?access_token= sunbird_url_shortner_access_token= ekstep.channel.reg.api.url=/channel/v3/create ekstep.channel.list.api.url=/channel/v3/list ekstep.channel.update.api.url=/channel/v3/update sunbird_valid_location_types=state,district,block,cluster,school; sunbird_default_channel= sunbird_url_shortner_enable=false sunbird_analytics_blob_account_name= sunbird_analytics_blob_account_key= sunbird_state_img_url=https://sunbirddev.blob.core.windows.net/orgemailtemplate/img/File-0128212938260643843.png sunbird_diksha_img_url=https://sunbirddev.blob.core.windows.net/orgemailtemplate/img/File-0128212989820190722.png sunbird_cert_completion_img_url=https://sunbirddev.blob.core.windows.net/orgemailtemplate/img/File-0128212919987568641.png sunbird_reset_pass_msg=Your have requested to reset password. Click on the link to set a password: {0} sunbird_reset_pass_mail_subject=Reset Password sunbird_subdomain_keycloak_base_url=https://merge.dev.sunbirded.org/auth/kafka_linger_ms=5 sunbird_user_upload_error_visualization_threshold=20001 migrate_user_template=You can now access your {0} state teacher account using {1}. Please log out and login once again to see updated details. sunbird_account_merge_subject=Account merged successfully sunbird_pass_regex=(?=.*[0-9])(?=.*[a-z])(?=.*[A-Z])(?=.*[!\"#$%&'()*+,-./:;<=>?@\\[\\]^_`{|}~])(?=\\S+$).{8,} sunbird_user_create_sync_type=ES sunbird_user_create_sync_topic=local.user.events sigterm_stop_delay=40 limit_managed_user_creation=true adminutil_base_url = http://adminutil:4000/ adminutil_sign_endpoint = v1/sign/payload self_declared_mandatory_fields = Diksha UUID,Status,State provided ext. ID,Channel,Persona self_declared_optional_fields = School Name,School UDISE ID,Email ID,Phone number,Error Type enable_captcha=true consent_expiry_in_days=100 feed_limit=30 learner_in_memory_cache_ttl=14400 user_index_alias=user_alias defaultMonthDate = -12-31 org_index_alias=org_alias stacktrace_char_length=2500 channel_registration_disabled=false ``` # Data Models * **Cassandra** - Cassandra is the primary data store for UserOrg. * **ElasticSearch** - ElasticSearch is used as secondary data store. * **Redis** - Redis is used for caching API response. # Cassandra Cassandra Migration in [sunbird-utils](https://github.com/sunbird-lern/sunbird-utils) needs to be run before user-org service setup to create necessary tables required in sunbird keyspace. ### **Cassandra** Database \[keyspace : sunbird] {% embed url="" %} # Data Dictionary List of tables in Cassandra database used in User-Org service ### Table List: 1. ### **user** 2. ### user\_lookup 3. ### organisation 4. ### usr\_external\_identity 5. ### user\_organisation 6. ### org\_external\_identity 7. ### system\_settings 8. ### role 9. ### role\_group 10. ### url\_action 11. ### user\_roles 12. ### bulk\_upload\_process 13. ### bulk\_upload\_process\_task 14. ### tenant\_preference\_v2 15. ### cassandra\_migration\_version 16. ### cassandra\_migration\_version\_counts 17. ### user\_consent 18. ### user\_declarations 19. ### email\_template 20. ### otp 21. ### rate\_limit 22. ### page\_management 23. ### page\_section 24. ### location 25. ### user\_notes # User ### **unbird.user (PRIMARY KEY: id)** Table used for storing user profile details
Column NameData TypeDescriptionSample Value
idtextUUID9b774c71-6034-4de7-aa38-5382fc673b14
alltncacceptedmap<text, text>

Terms and Conditions that the user has accepted.

1) Org Admin TNC if the user has ORG_ADMIN role

2) Report Viewer role TNC if the user has REPORT_VIEWER role

3)Groups tnc - if the user has groups.

"allTncAccepted": { "reportViewerTnc": { "tncAcceptedOn": "2023-01-02 05:25:21:586+0000", "version": "4.0.0" }, "orgAdminTnc": { "tncAcceptedOn": "2020-11-26 08:17:59:547+0000", "version": "3.5.0" }, "groupsTnc": { "tncAcceptedOn": "2020-12-03 09:14:47:437+0000", "version": "3.5.0" } }
channeltextTenant Organisation channel valueTN, AP
countrycodetextCountry code of the user+91
createdbytextuuid of the created user, null in case user has signed up by himself8c774c71-6034-4de7-aa38-5382fc673b14
createddatetextDate on which the user was created2020-09-28 15:47:15:919+0000
dobtextDate of Birth of the user. Only year is provided by the user. Month and date(12-31) is appended to it by the system1987-12-31
emailtextEmail id of the user in encrypted formattestdoc@yopmail.com
emailverifiedbooleanEmail is verified or not using OTP. This flag is not used anymoretrue or false
firstnametextFirst name of the user
flagsvalueintValue will by 4 , if the user is uploaded by the tenant or registered through tenant login page. ( Earlier there were different values updated as per the user email verified, phone verified and tenant verified. But currently only tenant verification is stored.)4
frameworkmap<text, frozen<list<text>>>User chosen framework{ "board": ["State (Tamil Nadu)"], "gradeLevel": ["Class 1"], "id": ["tn_k-12_5"], "medium": ["English"], "subject": ["Mathematics"] }
isdeletedboolean

User is soft deleted or not. Scenarios:

1) Merge one user to another, then the first one gets deleted.

2) User Block will update the is_deleted flag to true and status to 0 (inactive)

true or false
lastnametextLast Name of the user
locationidslist<text>Not Used
loginidtextNot Used
managedbytextLogged in user/Parent uuid of the managed user. If managedby column has value( parent uuid), that means this row of record is child user's and email and phone number will be blank for this user.7c784c71-9034-4de7-aa38-5382fc673b14
maskedemailtextMasked email id value te*****@yopmail.com
maskedphonetextMasked phone number value98******09
phonetextPhone number of the user in encrypted format
phoneverifiedbooleanEmail is verified or not using OTP. This flag is not used anymore
prevusedemailtextPreviously used email id in encrypted format
prevusedphonetextPreviously used phone number in encrypted format
profilelocationtextUsed to store location data of the user from release-3.9.0. This stores the location types and code from sunbird.location table. This is validated against the configuration of location types in properties file "sunbird_valid_location_types" and also to ED form api configuration "profileconfig_v2"

"profileLocation": [ { "code": "32", "type": "state" }, { "code": "3210", "type": "district" },

{"code": "321001", "type": "block"} ,{"code": "32100123", "type": "cluster"]

profileusertypetextNot Used from release-4.4.0.This field was in use from release-3.9.0 to 4.4.0.{ "type": "administrator", "subType":"hm" }
profileusertypestextUsed to store user type (role) and subusertype (subrole) from release-4.4.0. This is validated against the ED form api configuration "profileconfig_v2""profileUserTypes": [ { "type": "administrator", "subType":"hm" }, { "type": "administrator", "subType":"deo" }],
recoveryemailtextRecovery email of the user in encrypted form
recoveryphonetextRecovery phone number of the user in encrypted form
roleslist<text>Not used
rootorgidtextTenant Org id of the user0126796199493140480
statusintUser is active or not. '1' means active, '0' is inactive1
tncacceptedontimestampterms and conditions accepted time in long format1687933998587
tncacceptedversiontextVersion of the terms and conditions. If any change in terms and conditions it is added with a new version and link is updated in system configuration.v13
updatedbytextUUID of the user who updated the profile. 7c784c71-9034-4de7-aa38-5382fc673b14
updateddatetextDate in which this table record is updated last.2023-06-28 06:34:02:020+0000
useridtextUUID of the user, same as the id column value9b774c71-6034-4de7-aa38-5382fc673b14
usernametextUsername of the user, this can be given via create api or if not given its automatically created by appending firstname and random text.
usersubtypetextNot used
usertypetextNot used
### sunbird.user\_lookup \[PRIMARY KEY (type, value)] Table used for supporting user lookup based on email/phone/username.
Column NameData TypeDescriptionSample Data
typetextfield/column typeusername/email/phone
valuetextencrypted value corresponding to the field of the user in 'userid' column0D8+4tIadSIevBPI5TRVpz01bytYGtT59Fm6X1hbXTxLMh873Ibbq1Orv9HTxpM3orvEZRVVOy3E\n56xU3yrF/qtCQr8582HbLJKeRdHyl+sKLJ7enl6IgnHREqS+/HVeT6a+wzaAmCWueMEdPmZuRg==
useridtextUUID of the user51b39c1d-a021-4ce8-9487-4dcf843eb925
### sunbird.usr\_external\_identity \[PRIMARY KEY (provider, idtype, externalid) Table used for storing user information from the third party system (SSO integrated systems)
Column NameData TypeDescriptionSample Data
providertextOrganisation of the user0132818330295992324
idtypetextType of the IDUDAI
externalidtextID of the user in the third party systemckc971
createdbytextUUID of the admin who onboarded the user1405f334-ee59-42fc-befb-51986221881e
createdontimestampTimestamp at which user record was created on2021-05-18 07:35:22.646000+0000
lastupdatedbytextNot Used
lastupdatedontimestampNot Used
originalexternalidtextNot Used
originalidtypetextNot Used
originalprovidertextNot Used
useridtextUUID of the user in sunbird systemee27b0bc-be5c-427f-9cc4-a87079ef3dd7
### sunbird.user\_organisation \[PRIMARY KEY (userid, organisationid)]
Column NameData TypeDescriptionSample Data
useridtextUUID of the useradc314fa-cd62-4d8c-8b06-5c30091d0009
organisationidtextRootOrg Identifier to which user is associated with01276175508980531215
addedbytextNot Used
addedbynametextNot Used
approvaldatetextDate on which self declared user record was approved on2019-03-27 13:07:14:732+0000
approvedbytextNot Used
associationtypeintStores information about how user to organsiation association was established1/2/4
SSO -> 1
Self Declaation -> 2
System Upload -> 4
hashtagidtexthashtagid of the RootOrg to which user is associated with01276175508980531215
idtext
isapprovedbooleanUsed to indicate whether user record was approved for association with organisationTrue
isdeletedbooleanUsed to indicate whether user record is deletedFalse
isrejectedbooleanUsed to indicate whether user record was rejected for association with organisationFalse
orgjoindatetextTimestamp at which user was associated with organisation2019-03-27 13:07:14:732+0000
orgleftdatetextTimestamp at which user was disassociated with organisation2019-06-18 11:24:03:263+0000
positiontextNot used
roleslist<text>Roles associated with the user['CONTENT_CREATOR', 'COURSE_MENTOR']
updatedbytextOrg Admin user UUID who updated the user information59177b28-14a1-41e3-a5f9-87bff98443b3
updateddatetextTimestamp at which user record was updated on2019-04-10 10:32:09:999+0000
# Organisation ### sunbird.organisation (**PRIMARY KEY: id)** Table used for storing Tenants/Organisations and Sub-Organisations
Column NameData TypeDescriptionSample Value
idtextId of the organisation0126796199493140480
channeltextAbbreviation of the Tenant Organisation nameTN , AP, CBSE etc if Tenant Org Name is Tamil Nadu, Andra Pradesh, Central Board of Secondary Education etc
contactdetailtextNot Used
createdbytextUUID of the created user7c784c71-9034-4de7-aa38-5382fc673b14
createddatetextDate on which Organisation detail as added to the table
descriptiontextDescription on the Organisation
emailtextEmail Id f the Organisation
externalidtextUnique identifier or registered Organisation Code which is used for reference externally3453456 or something unique to the org
hashtagidtextSame as organisation id0126796199493140480
isrootorgbooleanNot Used
isssoenabledbooleanIf the login from tenant site is enabled to sunbird system.True or False
istenantbooleanTo indicate whether Organisation is a Tenant ORg or Sub- Org under the Tenant OrgTrue or False
keysmap<text, frozen<list<text>>>Public key shared by the Organisation
locationidslist<text>Not used
organisationtypeintType of the Organisation2, 5 etc
orglocationtextLocation details of the organisation. This is validated against the configuration of location types in properties file "sunbird_valid_location_types"

[ { "code": "32", "type": "state" }, { "code": "3210", "type": "district" },

{"code": "321001", "type": "block"} ,{"code": "32100123", "type": "cluster"]

orgnametextName of the OrganisationKids of India
providertextSame as channel.
rootorgidtextnull in case of Tenant Orgs and Tenant Org id incase of sub-orgs.null or 0126796199493140480
slugtextSlug and channel are abbrevations of organisation name, where slug is URL compatable, while channel is not.tn, ap etc
statusintActive or not. '1' means active, '0' is inactive1 or 0
updatedbytextUUID of the user who updated the organisation detail.
updateddatetextDate in which this table record is updated last.
### sunbird.org\_external\_identity \[PRIMARY KEY: (provider, externalid)] Table used to store details of the organisation as stored in the third party system (SSO integrated system)
Column NameData TypeDescriptionSample Data
providertextTenant Organisation to which the sub-organisation belongs toekstep
externalidtextExternal Id of the organisation as maintained in the third party system28110100101
orgidtextIdentifier of the organisation maintained in the Sunbird0127257746440601604
# System Settings ### sunbird.system\_settings (**PRIMARY KEY: id)** Table used for storing system level configuration values used by LERN and other sunbird BBs.
Column NameData TypeDescriptionSample Value
idtextUnique code for identifying the settings or configurationcustodianOrgId, tncConfig etc
fieldtextAs of now same as id itselfcustodianOrgId, tncConfig etc
valuetextValue of the configuration. This can be a string or a json string

0126796199493140480 or

"{"latestVersion":"v13","v13":{"url":"https://obj.stage.sunbirded.org/privacy-policy/terms-of-use.html"}}"

# Role ### sunbird.role \[PRIMARY KEY: id] Table used to store Roles definition.
Column NameData TypeDescriptionSample Data
idtextRole IdentifierCONTENT_REVIEWER
nametextRole NameContent Reviewer
rolegroupidlist<text>Role Group used for API access management['CONTENT_CURATION']
statusintstatus of the role1 - Valid
### sunbird.role\_group \[PRIMARY KEY: id] Table used to store role group information
Column NameData TypeDescriptionSample Data
idtextRole Group identifierORG_MANAGEMENT
nametextRole Group NameOrg Management
url_action_idslist<text>URL action Identifiers (Refer to sunbird.url_action)['createOrg', 'updateOrg', 'removeOrg', 'createUser', 'updateUser']
### sunbird.url\_action \[PRIMARY KEY: id] Table used to define url action to API endpoint mapping
Column NameData TypeDescriptionSample Data
idtextURL action IdentifierupdateOrg
nametextURL action NameupdateOrg
urllist<text>API endpoint list['/v1/organisation/update']
### sunbird.user\_roles \[PRIMARY KEY (userid, role)] Tole used to store user to role mapping
Column NameData TypeDescriptionSample Data
useridtextUUID of the User5e48c2ba-cd3a-4d85-9d3b-28dc329d7dd9
roletextRole IdentifierBOOK_REVIEWER
createdbytextRole added By08631a74-4b94-4cf7-a818-831135248a4a
createddatetextRecord creation timestamp2021-08-11 08:53:49:662+0000
scopetextRole scope - organisation to which role is defined for the user[{"organisationId":"01334203864941363283"}]
updatedbytextRole Updated By
updateddatetextLast record Updated timestamp
# Bulk Upload Process ### sunbird.bulk\_upload\_process (**PRIMARY KEY: id)** Table used for storing bulk upload process information.
Column NameData TypeDescriptionSample Data
idtextProcess Identifier013164532565106688101
createdbytextBulk upload triggered byec8deeb2-4ded-4fa2-ac48-023ad8298d92
createdontimestampBulk upload triggered on2020-12-03 12:24:00.303000+0000
datatextBulk uploaded file data9C9dOj89ssd0Rae5fSjSVmXmNdMSm7OpbtcYhFsc9WMwI92NvcdeNvC+IEgmUg0rdejQd642yHeV\na6E=
failureresulttextRecords that could not be processed[]
lastupdatedontimestamprecord last updated on2020-12-03 12:24:00.303000+0000
objecttypetextrecords' object type: location/user/organisation/MigrationUser/SELF_DECLARED_USERSELF_DECLARED_USER
organisationidtextorganisation Identifier to which bulk upload has been triggered for01269878797503692810
processendtimetext
processstarttimetextStart time at which record processing got triggered2020-12-03 12:24:00.303000+0000
retrycountintNumber of times record processing was tried2
statusintStatus of the bulk upload process3
storagedetailstextCloud Storage location of the downloadable CSV with process result information
DfCr74W+cCyXsW0KiI9+0fE/qnW1SYZLJeyMEoWC7pV9wu2rSCdgiQ2JBt2B/0nQswU6sFZZ8+dX\noxDWPWJ5FXSKc2v9cXPWEGnscNhsEFrqOIUemwlJe5mcIjhM2Opel2940M1Vslf/cZMCI3xwpSUq\nUCYFRMDmK6lNotysYPISemcCMnZEk+qhOT8vhvpeKo1Xv/0/jBfbKI03JhZruX703HqiOZOcR072\nkYFoPRDaxoOFc+rXLm+hHg6LOXypFVTNhs9/3Sa80SViXQCn4bnc5FlOjy26iVIxqdYi6bCEfQqd\n6EAFIFGwzFRTS/giFGHVEadH0V0aMXTQbYUJ3Q==
successresulttextRecords that were processed successfully
[{"firstName":"Partner Content Creator","phone":null,"roles":["CONTENT_CREATOR"],"userType":"TEACHER","userId":"fd3cbea8-8608-43ce-ba7f-4fb587a10b45","email":"T+CY41939vm5qKAAcTLaxxK+trKhBNYnDDP/VRRvhac0gVteBTNegJgigEyqBs0qGcAkB/m3MS5f\n7RPeVRTnKnT88bYbiAGo78uwUmwN4GcDevB7iVni9p2Sbmsd5V9h/tJR3ZPAAFSTyJ9yFc8WHcvy\nG0t0HiJjD+xEIZgJ6RQ=","orgId":"0128925770740285447","organisationId":"0128925770740285447","emailVerified":true,"createdBy":"30f05f74-2ee8-4ed0-b20a-d1f880268a74","rootOrgId":"0128925770740285447","channel":"2.5.0","phoneVerified":false,"profileVisibility":{},"isDeleted":false,"createdDate":"2020-05-04 07:28:34:162+0000","status":1,"userName":"5Q14OzLBlLI7HHuph8OqS1N9fKpmXHYm+y4p8QljjhRY1J0CQtV30E8Ym7r8KzpXw5EzAHe3Vj+Z\nzy4i+VpmXXlF/Duv7I/HjdoA+q/U2hTPImOINHWbZp0d3J4XvFK2x7BDdjyMwJA4M25P8OEXyfJn\npkKp6ASi2X8EnVJq+wE=","loginId":"5Q14OzLBlLI7HHuph8OqS1N9fKpmXHYm+y4p8QljjhRY1J0CQtV30E8Ym7r8KzpXw5EzAHe3Vj+Z\nzy4i+VpmXcJii4KPr2fcgMs2OGMaLfKN0JRcaxV4Dhk2DKbBCAprp/fgHO9L04/v2xepR3Lj95ZX\nR3WFhFjJrrpMKskBxnM=","externalIds":null,"id":"fd3cbea8-8608-43ce-ba7f-4fb587a10b45","orgName":"Preprod2.5.0","operation":"create"},{"firstName":"Partner Book Creator","phone":null,"roles":["BOOK_CREATOR"],"userType":"TEACHER","userId":"cf37aa84-c441-4222-8158-3dbca16f3837","email":"T+CY41939vm5qKAAcTLaxxK+trKhBNYnDDP/VRRvhac0gVteBTNegJgigEyqBs0qGcAkB/m3MS5f\n7RPeVRTnKnT88bYbiAGo78uwUmwN4GcDevB7iVni9p2Sbmsd5V9h/tJR3ZPAAFSTyJ9yFc8WHcvy\nG0t0HiJjD+xEIZgJ6RQ=","orgId":"0128925770740285447","organisationId":"0128925770740285447","emailVerified":true,"createdBy":"30f05f74-2ee8-4ed0-b20a-d1f880268a74","rootOrgId":"0128925770740285447","channel":"2.5.0","phoneVerified":false,"profileVisibility":{},"isDeleted":false,"createdDate":"2020-05-04 07:28:34:237+0000","status":1,"userName":"TvHDg/MI2JpJFXgjp3YIG4LMGVGHK+jOhtjZhSRlLdN0QYfVVdKS0fH1182ZYFR3wH0ApdRZW2Uu\ncmm1xMNH7hR1NfeTnnYNoC5Lx7PvqjoLgACzaXPkpdqojfIVFRCFT6a+wzaAmCWueMEdPmZuRg==","loginId":"TvHDg/MI2JpJFXgjp3YIGxHeQiq+pucBuZCTotHSROCYKG6HLWzgt45LtXLalf8VOa8Jkm7i4ZoB\nDHKKfmpitKHKqYPck44ON4KMwv0OHXcC0RETYKK/WhokPxDLwYjBXXEKEarPqozxAXE+a5dMmjUl\n1FlHleh2UkyWuT7bVWQ=","externalIds":null,"id":"cf37aa84-c441-4222-8158-3dbca16f3837","orgName":"Preprod2.5.0","operation":"create"}]
taskcountint8
telemetrycontextmap<text, text>Used to store request context information in case of MigrationUser/SELF_DECLARED_USER type bulk uploads
{'actorId': '013164536804098048102', 'actorType': 'System', 'channel': '0126796199493140480', 'env': 'SelfDeclaredUserUpload', 'method': 'POST', 'requestType': 'API_CALL', 'telemetry_pdata_id': 'staging.sunbird.learning.service', 'telemetry_pdata_pid': 'learner-service', 'telemetry_pdata_ver': '3.5.0', 'url': '/v2/bulk/user/upload', 'x-request-id': '12b90c4428da29db327c8b17f3258c1c'}
uploadedbytextsame as createdbyec8deeb2-4ded-4fa2-ac48-023ad8298d92
uploadeddatetextsame as createdon2020-12-03 12:24:00.303000+0000
### sunbird.bulk\_upload\_process\_task (PRIMARY KEY: processid, sequenceid)
Column NameData TypeDescriptionSample Data
processidtextProcess Identifier0134312772714250242
sequenceidintRecord sequence2
createdontimestampRecord created on2021-12-15 09:11:59.078000+0000
datatextRecord details{"code":"998","parentCode":"ak0015","name":"d02","parentId":"1efe2f58-b1a3-432f-820e-9a8b9d843237","id":"259278c4-7c69-428e-85cb-0f2df8d167c4","type":"district"}
failureresulttextReason for record processing failure{"code":"999","parentCode":"ak0015","name":"d01","parentId":"65359881-ace2-44e9-8212-c74c637fea9b","id":"ff47b704-9856-4712-ad88-0625853f0e26","type":"state","operation":"update","err_msg":"Update of type is not allowed."}
iterationidintNumber of iterations1
lastupdatedontimestampRecord last updated on2021-12-15 09:11:59.322000+0000
statusint

Status of record processing

NEW(0),
IN_PROGRESS(1),
INTERRUPT(2),
COMPLETED(3),
FAILED(9)
0/1/2/3/9
successresulttextrecord details when processed successfully{"code":"998","parentCode":"ak0015","name":"d02","parentId":"1efe2f58-b1a3-432f-820e-9a8b9d843237","id":"259278c4-7c69-428e-85cb-0f2df8d167c4","type":"district","operation":"update"}
# Tenant Preference ### dataSecurityPolicysunbird.tenant\_preference\_v2 \[PRIMARY KEY (orgid, key)] Table used to store Tenant preference information
Column NameData TypeDescriptionSample Data
orgidtextorganisation identifier whose preference details are storeddefault
keytextpreference key iddataSecurityPolicy/certList/selfdeclarations/userPrivateFields ....
createdbytextRecord created Byfbe926ac-a395-40e4-a65b-9b4f711d7642
createdontimestampRecord created On2023-04-26 09:37:31.315000+0000
datatextPreference details
{"level":"PLAIN_DATASET","dataEncrypted":"No","comments":"Data is not encrypted","job":{"progress-exhaust":{"level":"PUBLIC_KEY_ENCRYPTED_DATASET","dataEncrypted":"No","comments":"Password protected file."},"response-exhaust":{"level":"TEXT_KEY_ENCRYPTED_DATASET","dataEncrypted":"No","comments":"Password protected file."},"userinfo-exhaust":{"level":"PASSWORD_PROTECTED_DATASET","dataEncrypted":"Yes","comments":"Password protected file."},"program-user-exhaust":{"level":"PLAIN_DATASET","dataEncrypted":"No","comments":"Data is not encrypted"}},"securityLevels":{"PLAIN_DATASET":"Data is present in plain text/zip. Generally applicable to open datasets.","PASSWORD_PROTECTED_DATASET":"Password protected zip file. Generally applicable to non PII data sets but can contain sensitive information which may not be considered open.","TEXT_KEY_ENCRYPTED_DATASET":"Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open.","PUBLIC_KEY_ENCRYPTED_DATASET":"Data encrypted via an org provided public/private key. Generally applicable to all PII data exhaust."}}
updatedbytextRecord updated Byfbe926ac-a395-40e4-a65b-9b4f711d7642
updatedontimestampRecord updated On2023-06-27 08:44:54.019000+0000
# Cassandra Migration Version ### sunbird.cassandra\_migration\_version (**PRIMARY KEY: version)** Table used to capture cassandra CQL script version status during LERN release deployment
Column NameData TypeDescriptionSample Data
versiontextCurrent CQL script version deployed1.113
checksumint-1988945306
descriptiontextNot Usedcassandra
execution_timeintTime taken in milliseconds to execute the cql script2323
installed_bytextNot Used
installed_ontimestampTimestamp on which the cql script was executed2020-12-18 11:04:46.095000+0000
installed_rankintThe order in which this migration was applied amongst all others. (For out of order detection)226
scripttextCQL script nameV1.113_cassandra.cql
successbooleanIndicator to mention whether CQL script was run successfully or notTrue
typetextThe type of migration (CQL, JAVA_DRIVER, ...)CQL
version_rankintThe position of this version amongst all others. (For easy order by sorting)112
### sunbird.cassandra\_migration\_version\_counts (PRIMARY KEY: name) Table used to store information about the last run CQL migration script data
Column NameData TypeDescriptionSample Data
nametextField Nameinstalled_rank
countcounterField Value257
# User Consent ### sunbird.user\_consent \[PRIMARY KEY (user\_id, consumer\_id, object\_id)] Table used to store user consent to share PII
Column NameData TypeDescriptionSample Data
user_idtextUUID of the user0dd78c26-4226-4cd5-b883-23e7dc277047
consumer_idtextorganisation Identifier01269878797503692810
object_idtextobject Identifier on which consent is provided on01269878797503692810 OR do_2130448790797926401216
categorieslist<text>
consent_datatext
consumer_typetextORGANISATION
created_ontimestampRecord created on2021-12-06 11:03:09.059000+0000
expirytimestampRecord expires on2022-03-16 11:03:09.057000+0000
idtextconsent identifierusr-consent:ed9d0289-8d5b-4d39-89d2-1a1b5ce69828:01269878797503692810:01269878797503692810
last_updated_ontimestamprecord recently updated on2021-12-06 11:03:09.059000+0000
object_typetextObject type on which user has provided consent to share PII Organisation/Collection
statustextConsent statusACTIVE/REVOKED
### sunbird.user\_declarations \[PRIMARY KEY (userid, orgid, persona)] Table used to store user declared fields details
Column NameData TypeDescriptionSample Data
useridtextUUID of the usered9d0289-8d5b-4d39-89d2-1a1b5ce69828
orgidtextOrganisation Identifier to which user belongs to01269878797503692810
personatextType of the teacher: default/teacherteacher
createdbytextNot used
createdontimestamprecord created on2020-12-07 12:06:59.467000+0000
errortypetextDeclared record validation error typeERROR-DISTRICT, ERROR-PHONE, ERROR-EMAIL, ERROR-SCHOOL ORG NAME, ERROR-SCHOOL ORG ID, ERROR-ID, ERROR-NAME, ERROR-STATE
statustextStatus of the declaration:
SUBMITTED, VALIDATED, REJECTED, ERROR		SUBMITTED
updatedbytextNot used
updatedontimestampNot used
userinfomap<text, text>User declared information{'declared-email': '', 'declared-ext-id': '987654321', 'declared-phone': '', 'declared-school-name': 'PUPS, REDDIYARPATTI', 'declared-school-udise-code': '33291500301'}
# Email Template ### sunbird.email\_template (PRIMARY KEY: name) Table used to store email notification templates
Column NameData TypeDescriptionSample Data
nametextname of the templatepublishContent
createdbytextNot Used
createdontimestampNot Used
lastupdatedbytextNot Used
lastupdatedontimestampNot Used
templatetextemail template
<!doctype html><html> <head> <meta name="viewport" content="width=device-width"> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> <title></title> <style>/* ------------------------------------- INLINED WITH htmlemail.io/inline ------------------------------------- */ /* ------------------------------------- RESPONSIVE AND MOBILE FRIENDLY STYLES ------------------------------------- */ @media only screen and (max-width: 620px){table[class=body] h1{font-size: 28px !important; margin-bottom: 10px !important;}table[class=body] p, table[class=body] ul, table[class=body] ol, table[class=body] td, table[class=body] span, table[class=body] a{font-size: 16px !important;}table[class=body] .wrapper, table[class=body] .article{padding: 10px !important;}table[class=body] .content{padding: 0 !important;}table[class=body] .container{padding: 0 !important; width: 100% !important;}table[class=body] .main{border-left-width: 0 !important; border-radius: 0 !important; border-right-width: 0 !important;}table[class=body] .btn table{width: 100% !important;}table[class=body] .btn a{width: 100% !important;}table[class=body] .img-responsive{height: auto !important; max-width: 100% !important; width: auto !important;}}/* ------------------------------------- PRESERVE THESE STYLES IN THE HEAD ------------------------------------- */ @media all{.ExternalClass{width: 100%;}.ExternalClass, .ExternalClass p, .ExternalClass span, .ExternalClass font, .ExternalClass td, .ExternalClass div{line-height: 100%;}.apple-link a{color: inherit !important; font-family: inherit !important; font-size: inherit !important; font-weight: inherit !important; line-height: inherit !important; text-decoration: none !important;}.btn-primary table td:hover{background-color: #34495e !important;}.btn-primary a:hover{background-color: #34495e !important; border-color: #34495e !important;}}</style> </head> <body class="" style="background-color: #f6f6f6; font-family: sans-serif; -webkit-font-smoothing: antialiased; font-size: 14px; line-height: 1.4; margin: 0; padding: 0; -ms-text-size-adjust: 100%; -webkit-text-size-adjust: 100%;"> <table border="0" cellpadding="0" cellspacing="0" class="body" style="border-collapse: separate; mso-table-lspace: 0pt; mso-table-rspace: 0pt; width: 100%; background-color: #f6f6f6;"> <tr> <td style="font-family: sans-serif; font-size: 14px; vertical-align: top;">&nbsp;</td><td class="container" style="font-family: sans-serif; font-size: 14px; vertical-align: top; display: block; Margin: 0 auto; max-width: 580px; padding: 10px; width: 580px;"> <div class="content" style="box-sizing: border-box; display: block; Margin: 0 auto; max-width: 580px; padding: 10px;"> <span class="preheader" style="color: transparent; display: none; height: 0; max-height: 0; max-width: 0; opacity: 0; overflow: hidden; mso-hide: all; visibility: hidden; width: 0;"></span> <table class="main" style="border-collapse: separate; mso-table-lspace: 0pt; mso-table-rspace: 0pt; width: 100%; background: #ffffff; border-radius: 3px;"> <tr> <td class="wrapper" style="font-family: sans-serif; font-size: 14px; vertical-align: top; box-sizing: border-box; padding: 20px;"> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: separate; mso-table-lspace: 0pt; mso-table-rspace: 0pt; width: 100%;"> <tr> <tr> <td> #if ($orgImageUrl) <p> <img src="$orgImageUrl" alt="logo" align="right" width="180" height="100"> </p>#end </td></tr><td style="font-family: sans-serif; font-size: 14px; vertical-align: top;"> #if ($name) <p style="font-family: sans-serif; font-size: 14px; font-weight: normal; margin: 0; Margin-bottom: 15px;">Hi $name,</p>#end <p style="font-family: sans-serif; font-size: 14px; font-weight: normal; margin: 0; Margin-bottom: 15px;">$body</p><table border="0" cellpadding="0" cellspacing="0" class="btn btn-primary" style="border-collapse: separate; mso-table-lspace: 0pt; mso-table-rspace: 0pt; width: 100%; box-sizing: border-box;"> <tbody> <tr> <td align="left" style="font-family: sans-serif; font-size: 14px; vertical-align: top; padding-bottom: 15px;"> <table border="0" cellpadding="0" cellspacing="0" style="border-collapse: separate; mso-table-lspace: 0pt; mso-table-rspace: 0pt; width: auto;"> <tbody> #if ($actionUrl) <tr> <td style="font-family: sans-serif; font-size: 14px; vertical-align: top; background-color: #3498db; border-radius: 5px; text-align: center;"> <a href="$actionUrl" target="_blank" style="display: inline-block; color: #ffffff; background-color: #3498db; border: solid 1px #3498db; border-radius: 5px; box-sizing: border-box; cursor: pointer; text-decoration: none; font-size: 14px; font-weight: bold; margin: 0; padding: 12px 25px; text-transform: capitalize; border-color: #3498db;">#if ($actionName) <span>$actionName</span> #end </a> </td></tr>#end </tbody> </table> </td></tr></tbody> </table> <p style="font-family: sans-serif; font-size: 14px; font-weight: normal; margin: 0; Margin-bottom: 0;">Regards,</p><p style="font-family: sans-serif; font-size: 14px; font-weight: normal; margin: 0; Margin-bottom: 0;">Team #if ($orgName) <span> $orgName</span> #end </p><p> Note: This is an automatic alert email. Replies to this mail box will not be monitored. If you are not the intended recipient of this message, or need to communicate with the team, write to $fromEmail. </p></td></tr></table> </td></tr></table> </div></td><td style="font-family: sans-serif; font-size: 14px; vertical-align: top;">&nbsp;</td></tr></table> </body></html>
# OTP ### sunbird.otp \[PRIMARY KEY: (type, key)] Table used to store OTP information
Column NameData TypeDescriptionSample Data
typetextOTP Communication modeEMAIL/PHONE
keytextEmail/Phone of the user98******10
attemptedcountintNumber of times OTP mismatch was identified 2
createdontimestamprecord created on timestamp
otptextOTP254789
### sunbird.rate\_limit \[PRIMARY KEY: (key, unit)] Table used to store rate limit information (number of requests per hour/day)
Column NameData TypeDescriptionSample Data
keytextEmail/Phone of the usertestaugustuser@yopmail.com
unittextDAY/HOURDAY
countint1
rateintNumber of notifications that can be sent as per limit20
# Page Management (LMS Service) ### sunbird.page\_management \[PRIMARY KEY: id] Table used to store configurable page information
Column NameData TypeDescriptionSample Data
idtextPage Identifier01228382478150860822
appmaptextPage configuration with section information for mobile application
[{"id":"01228382278062080019","index":1,"group":1},{"id":"01228382243946496017","index":1,"group":2}]
created_datetimestampNot used
createdbytextUUID of the user who created the page490ed1d2-c69d-4cf0-a50c-f37e658c128d
createddatetextRecord creation date2017-07-08 05:03:29:679+0000
nametextName of the pageCourse
organisationidtextorganisation Identifier for which the page is configured for
portalmaptextPage configuration with section information for web portal application
[{"id":"01228382278062080019","index":1,"group":1},{"id":"01228382243946496017","index":1,"group":2}]
updated_datetimestampNot used
updatedbytextNot used
updateddatetextNot used
### sunbird.page\_section \[PRIMARY KEY: id] Table used to store page section details
Column NameData TypeDescriptionSample Data
idtextPage Section Identifier0131191751772733441
alttextNot used
created_datetimestampNot used
createdbytextUUID of user who created the section9bb884fc-8a56-4727-9522-25a7d5b8ea06
createddatetextRecord creation date2020-09-30 10:28:07:042+0000
datasourcetextNot used
descriptiontextNot used
displaytextSection display title in different languages{"name":{"en":"Latest Courses"}}
dynamicfilterstextNot used
imgurltextNot used
nametextSection TitleLatest Courses
searchquerytextQuery to display the contents as part of section{"request":{"filters":{"contentType":["Course"],"objectType":["Content"],"status":["Live"]},"sort_by":{"lastPublishedOn":"desc"},"limit":10}}
sectiondatatypetextData type of the SectionContentBrowser/Content
statusintStatus of the record1
updated_datetimestampNot used
updatedbytextRecord updated By9bb884fc-8a56-4727-9522-25a7d5b8ea06
updateddatetextRecord last updated on2020-09-30 10:46:11:529+0000
# Location ### Sunbird.location (**PRIMARY KEY: id)** Table is used for storing locations.
Column NameData TypeDescriptionSample Value
idtextUUID of the location0c0391ba-610b-4796-8645-338d047b1e28
codetextUnique code with which location is identified externally or outside sunbird system4021
nametextName of the location
parentidtextParent location id in sunbird system91d9baae-14f1-477a-955c-f91bd9037f0b
typetextType of the location. This is validated against the configuration of location types in properties file "sunbird_valid_location_types" . This can be changedstate, district, block, cluster
# User Notes ### sunbird.user\_notes \[PRIMARY KEY: id]
Column NameData TypeDescriptionSample Data
idtextIdentifier0127589083239956481071
contentidtextcontent identifier on which user notes is recordeddo_21273722793293414414343
courseidtextCourse Id in which the content is presentdo_21273766432272384014646
createdbytextuser UUID who created the notes54f4b468-a13b-4207-bbb1-b61d5889b2d3
createddatetexttimestamp at which user notes was created2019-05-10 10:17:50:399+0000
isdeletedbooleanIf user notes was deleted or notFalse
notetextnotes dataNeed to revise the content
tagslist<text>
titletextTitle of the notesRevisit content
updatedbytextuser UUID who updated the record54f4b468-a13b-4207-bbb1-b61d5889b2d3
updateddatetexttimestamp at which user notes was updated2019-05-10 10:23:58:131+0000
useridtextuser UUID who created the notes54f4b468-a13b-4207-bbb1-b61d5889b2d3
# Deprecated ### Sunbird.action\_group (**PRIMARY KEY: id)**
Column NameData TypeDescription
idtext
actionidtext
groupnametext
### sunbird.user\_action\_role \[PRIMARY KEY: id]
Column NameData TypeDescription
idtext
roleidtext
actiongroupidlist<text>
### sunbird.user\_skills \[PRIMARY KEY: id]
Column NameData TypeDescription
idtext
createdbytext
createdontimestamp
endorsementcountint
endorserslistfrozen<list<frozen<map<text, text>>>>
lastupdatedbytext
lastupdatedontimestamp
skillnametext
skillnametolowercasetext
useridtext
### sunbird.user\_org \[PRIMARY KEY: id]
Column NameData TypeDescription
idtext
addedbytext
addedbynametext
approvaldatetext
approvedbytext
hashtagidtext
isapprovedboolean
isdeletedboolean
isrejectedboolean
organisationidtext
orgjoindatetext
orgleftdatetext
positiontext
roleslist<text>
updatedbytext
updateddatetext
useridtext
### sunbird.user\_cert \[PRIMARY KEY: id]
Column NameData TypeDescription
idtext
accesscodetext
createdattimestamp
isdeletedboolean
oldidtext
storemap<text, text>
updatedattimestamp
useridtext
### sunbird.user\_feed \[PRIMARY KEY: id]
Column NameData TypeDescription
idtext
categorytext
createdbytext
createdontimestamp
datatext
expireontimestamp
priorityint
statustext
updatedbytext
updatedontimestamp
useridtext
### sunbird.tenant\_preference \[PRIMARY KEY: id]
Column NameData TypeDescription
idtext
datatext
keytext
orgidtext
roletext
tenantnametext
### Sunbird.address (**PRIMARY KEY: id)**
Column NameData TypeDescription
idtext
addressline1text
addressline2text
addtypetext
citytext
countrytext
createdbytext
createddatetext
isdeletedboolean
statetext
updatedbytext
updateddatetext
useridtext
zipcodetext
### sunbird.cert\_registry (PRIMARY KEY: id)
Column NameData TypeDescription
idtext
accesscodetext
createdattimestamp
createdbytext
datatext
isrevokedboolean
jsonurltext
pdfurltext
reasontext
recipienttext
relatedtext
updatedattimestamp
updatedbytext
### sunbird.config\_path\_audit (PRIMARY KEY: ((id, cloud\_store\_type), created\_date))
Column NameData TypeDescription
idtext
cloud_store_typetext
created_datebigint
cloud_store_accounttext
cloud_store_containertext
cloud_store_pathtext
versionbigint
### sunbird.geo\_location (PRIMARY KEY: id)
Column NameData TypeDescription
idtext
createdbytext
createddatetext
locationtext
rootorgidtext
topictext
typetext
updatedbytext
updateddatetext
usercountint
usercountttltext
### sunbird.master\_action (PRIMARY KEY: id)
Column NameData TypeDescription
idtext
nametext
### sunbird.media\_type
Column NameData TypeDescription
idtext
nametext
### sunbird.report\_tracking \[PRIMARY KEY: id]
Column NameData TypeDescription
idtext
createddatetext
datatext
emailtext
fileurltext
firstnametext
formattext
periodtext
resourceidtext
resourcenametext
statusint
trycountint
typetext
updateddatetext
uploadeddatetext
useridtext
### sunbird.shadow\_user \[PRIMARY KEY (channel, userextid)]
Column NameData TypeDescription
channel
userextid
addedby
attemptedcountint
claimedontimestamp
claimstatusint
createdontimestamp
email
name
orgextid
phone
processid
updatedontimestamp
userid
useridslist<text>
userstatusint
# Elastic Search Elastic Search indexes used by UserOrg : * [userv3](https://github.com/project-sunbird/sunbird-devops/blob/release-5.3.0-lern/ansible/roles/es-mapping/files/indices/userv3.json) - This index is used to store user data. * [userfeed](https://github.com/project-sunbird/sunbird-devops/blob/release-5.3.0-lern/ansible/roles/es-mapping/files/indices/userfeed.json) - This index is used to store userfeed data. * [usernotes](https://github.com/project-sunbird/sunbird-devops/blob/release-5.3.0-lern/ansible/roles/es-mapping/files/indices/usernotes.json) - This index is used to store usernotes data. * [orgv3](https://github.com/project-sunbird/sunbird-devops/blob/release-5.3.0-lern/ansible/roles/es-mapping/files/indices/orgv3.json) - This index is used to store org data. * [location](https://github.com/project-sunbird/sunbird-devops/blob/release-5.3.0-lern/ansible/roles/es-mapping/files/indices/location.json) - This index is used to store location data. **Note:** userv3 index is mapped to user\_alias and orgv3 index mapped to org\_alias. #### Elastic Search Indices and Mappings Setup {% embed url="" %} # Redis Redis is used for caching the user details metadata. Redis key is \ of the user. (dbIndex: 12) Redis data is updated by user cache updater flink job and also user cache indexer data product. {% embed url="" %} Flink Job {% endembed %} {% embed url="" %} Spark Job {% endembed %} #### Sample Data: ``` HGETALL user:08631a74-4b94-4cf7-a818-831135248a4a 1) "usersubtype" 2) "" 3) "board" 4) "State (Tamil Nadu)" 5) "subject" 6) "[]" 7) "email" 8) "mqfHvB25AM9v+F86DWgPjE9tDNkvu6n6u1pqKAS9BqT9U4TkpKJju5BgSFzLp2VlxqPrgzDT5J\nhlKFh45G+E77Q53w5xYOrsbUoABF4Vci58JJHbN4Wzb2fV8/5Fl+T5ZUuUNdJcoZQazqPM4rCkTZ\nilMv5S3xBPGCQxJxIp8=" 9) "phone" 10) "lRpq1LqQ69CQ1SzAWRLuyEA7F1VH53KfAkxcXvCR22QKUkb2TdKftlbD1GltiFLBvHCrxqsOsl\n3eoQuhF0P+C2h4oA83Y+obFB6eagd2T5iGFTQ4RsFjBUMhUsdDrBT6a+wzaAmCWueMEdPmZuRg==" 11) "framework" 12) "tn_k-12_5" 13) "language" 14) "" 15) "userlogintype" 16) "teacher" 17) "district" 18) "CHITTOOR" 19) "orgname" 20) "Tamil Nadu" 21) "state" 22) "Andhra Pradesh" 23) "usersignintype" 24) "Validated" 25) "grade" 26) "[\"Class 1\",\"Class 2\",\"Class 10\",\"Class 11\",\"Class 12\"]" 27) "lastname" 28) "" 29) "rootorgid" 30) "01269878797503692810" 31) "profileusertypes" 32) "\\[{\"type\":\"teacher\"}\\]" 33) "medium" 34) "[\"Tamil\"]" 35) "firstname" 36) "content reviewer" 37) "userid" 38) "08631a74-4b94-4cf7-a818-831135248a4a" 39) "usertype" 40) "teacher" ``` # APIs The **UserOrg microservice** provides services to enable and manage the lifecycle, administration, and permissions of users; registries of users, organisations, and geographic locations; to send user notifications through multiple channels such as email, SMS, and OTPs. The APIs are grouped as: | Feature | API Documentation | | ------------------------------ | ---------------------------------------------------------------------------------------------------- | | User Management (CRUD) | | | Organisation Management (CRUD) | | | Location Management (CRUD) | | | Consent Management | | | OTP Services | | | Data Sync APIs | | | Notes Management | | | Tenant Configurations | | | Bulk Upload | | |


System Settings

| | | Notification | | # User Management {% openapi src="/files/rvSBKNmW4XVKEmD7kC1o" path="/user/v1/create" method="post" %} [userManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2FPstUZ35tRgtwMdyjZoFz%2Fopenapi%20\(4\).yaml?alt=media\&token=e40f18fe-3270-42d1-8d4d-8db8421a2aa1) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v2/create" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v3/create" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/sso/create" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/signup" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v2/signup" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v4/create" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/managed/create" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/search" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v2/search" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v3/search" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/update" method="patch" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v2/update" method="patch" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v3/update" method="patch" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/read/{userId}" method="get" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v2/read/{userId}" method="get" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v3/read/{userId}" method="get" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v4/read/{userId}" method="get" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v5/read/{userId}" method="get" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/get/{idType}/{id}" method="get" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v2/exists/{idType}/{id}" method="get" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/exists/{idType}/{id}" method="get" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/block" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/unblock" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/profile/read" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/managed/{userId}" method="get" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/declarations" method="patch" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/migrate" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/account/merge" method="patch" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/role/assign" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v2/role/assign" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} {% openapi src="/files/u3hJEydd8nAqcWM9PqJG" path="/user/v1/tnc/accept" method="post" %} [userService.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-8b19b2aa113f014b2fb9fa4449f95f8ff670b888%2FuserService.yaml?alt=media) {% endopenapi %} # User Deletion API API for deleted user ### Overview The User Deletion API allows authorized users to delete a user from the system. This API requires the user ID of the user to be deleted and is accessible via an HTTP POST request. ### Delete a user `POST` /api/user/v1/delete ### **Headers** | Name | Value | | ------------------------------------------------------------ | ------------------------- | | Content-Type\* | `application/json` | | Authorization\* | `Bearer ` | | X-Authenticated-User-token\* | `` | ### Body {% tabs %} {% tab title="Example" %} ```json { "request": { "userId": "user id to be deleted" } } ``` {% endtab %} {% tab title="Schema" %} | Name | Type | Description | | ---------------------------------------- | ------ | ---------------------------------- | | userId\* | string | User id of the user to be deleted | | {% endtab %} | | | | {% endtabs %} | | | ### **Response** {% tabs %} {% tab title="200" %} ```json { "id": "api.user.delete", "ver": "v1", "ts": "2024-05-20 10:08:56:812+0000", "params": { "resmsgid": "e413d019f31bcb2d3736b7054f1fd6e2", "msgid": "e413d019f31bcb2d3736b7054f1fd6e2", "err": null, "status": "SUCCESS", "errmsg": null }, "responseCode": "OK", "result": { "response": "SUCCESS" } } ``` {% endtab %} {% tab title="400" %} ```json { "id": "api.user.delete", "ver": "v1", "ts": "2024-05-20 10:09:45:837+0000", "params": { "resmsgid": "4abb4c0bdd99feea7a7070eeade15aaa", "msgid": "4abb4c0bdd99feea7a7070eeade15aaa", "err": "UOS_USRDLT0008", "status": "FAILED", "errmsg": "User is already deleted." }, "responseCode": "CLIENT_ERROR", "result": {} } ``` {% endtab %} {% tab title="403" %} ```json You do not have permission to perform this operation ``` {% endtab %} {% endtabs %} # Ownership Transfer API API for ownership transfer ### Overview The Ownership Transfer API allows authorized users to transfer ownership of roles and assets from one user to another within an organization. ### Delete a new user `POST` /api/user/v1/ownership/transfer ### **Headers** | Name | Value | | ------------------------------------------------------------ | ------------------------- | | Content-Type\* | `application/json` | | Authorization\* | `Bearer ` | | X-Authenticated-User-token\* | `` | ### Body {% tabs %} {% tab title="Example" %} ```json { "request": { "context": "User Deletion", "organisationId": "organisation-id-of-deleted-user", "actionBy": { "userId": "org-admin-user-id" }, "fromUser": { "userId": "deleted-user-id", "roles": [ { "role": "role-1", "scope": [ { "organisationId": "organisation-id-of-role-1" } ] }, { "role": "role-2", "scope": [ { "organisationId": "organisation-id-of-role-2" } ] } ] }, "toUser": { "userId": "new-user-id", "roles": [ { "role": "role-1", "scope": [ { "organisationId": "organisation-id-of-role-1" } ] }, { "role": "role-2", "scope": [ { "organisationId": "organisation-id-of-role-2" } ] } ] }, "objects": [{ "objectType": "asset-object-type", "identifier": "asset-identifier", "primaryCategory": "asset-category", "name": "asset-name" }] } } ``` {% endtab %} {% tab title="Schema" %} #### Parameters
ParameterTypeRequiredDescription
contextstringYesThe context of the transfer, typically "User Deletion".
organisationId*stringYesThe ID of the organization associated with the deleted user.
actionBy.userId*stringYesThe ID of the organization admin performing the transfer.
fromUser.userId*stringYesThe ID of the deleted user.
fromUser.roles*arrayYesThe roles to be transferred.
fromUser.roles[].rolestringYesThe role name.
fromUser.roles[].scopearrayYesThe scope of the role.
fromUser.roles[].scope[].organisationIdstringYesThe ID of the organization associated with the role.
toUser.userId*stringYesThe ID of the user receiving the ownership transfer.
toUser.roles*arrayYesThe roles being transferred.
toUser.roles[].rolestringYesThe role name.
toUser.roles[].scopearrayYesThe scope of the role.
toUser.roles[].scope[].organisationIdstringYesThe ID of the organization associated with the role.
objectsarrayYesThe assets to be transferred.
objects[].objectTypestringYesThe type of the asset.
objects[].identifierstringYesThe identifier of the asset.
objects[].primaryCategorystringYesThe category of the asset.
objects[].namestringYesThe name of the asset.
{% endtab %} {% endtabs %} ### **Response** **Response** {% tabs %} {% tab title="200" %} ```json { "id": "api.user.ownership.transfer", "ver": "v1", "ts": "2024-05-20 10:36:04:695+0000", "params": { "resmsgid": "b681b51992d3833bd500323585629b33", "msgid": "b681b51992d3833bd500323585629b33", "err": null, "status": "SUCCESS", "errmsg": null }, "responseCode": "OK", "result": { "status": "Ownership transfer process is submitted successfully!" } } ``` {% endtab %} {% tab title="400" %} ```json { "id": "api.user.ownership.transfer", "ver": "v1", "ts": "2024-05-20 10:36:21:087+0000", "params": { "resmsgid": "92e62ce028dc5912e88edd8ef43c9a3b", "msgid": "92e62ce028dc5912e88edd8ef43c9a3b", "err": "UOS_UOWNTRANS0028", "status": "FAILED", "errmsg": "Organization ID is mandatory in the request." }, "responseCode": "CLIENT_ERROR", "result": {} } ``` {% endtab %} {% tab title="401" %} ```json { "id": "api.user.ownership.transfer", "ver": "v1", "ts": "2024-05-20 10:37:12:052+0000", "params": { "resmsgid": "0f5d442e3e19232cf5176cb2b8ee63a9", "msgid": "0f5d442e3e19232cf5176cb2b8ee63a9", "err": "UOS_0070", "status": "FAILED", "errmsg": "You are not authorized." }, "responseCode": "UNAUTHORIZED", "result": {} } ``` {% endtab %} {% endtabs %} # Organisation Management {% openapi src="/files/r3o08FRkuoo655wIQWpk" path="/org/v1/create" method="post" %} [OrganisationManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-a07bf893965dbedd63f735032474958c00b8bca9%2FOrganisationManagement.yaml?alt=media) {% endopenapi %} {% openapi src="/files/r3o08FRkuoo655wIQWpk" path="/org/v1/read" method="post" %} [OrganisationManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-a07bf893965dbedd63f735032474958c00b8bca9%2FOrganisationManagement.yaml?alt=media) {% endopenapi %} {% openapi src="/files/r3o08FRkuoo655wIQWpk" path="/org/v1/search" method="post" %} [OrganisationManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-a07bf893965dbedd63f735032474958c00b8bca9%2FOrganisationManagement.yaml?alt=media) {% endopenapi %} {% openapi src="/files/r3o08FRkuoo655wIQWpk" path="/org/v1/update" method="patch" %} [OrganisationManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-a07bf893965dbedd63f735032474958c00b8bca9%2FOrganisationManagement.yaml?alt=media) {% endopenapi %} {% openapi src="/files/r3o08FRkuoo655wIQWpk" path="/org/v1/status/update" method="patch" %} [OrganisationManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-a07bf893965dbedd63f735032474958c00b8bca9%2FOrganisationManagement.yaml?alt=media) {% endopenapi %} {% openapi src="/files/r3o08FRkuoo655wIQWpk" path="/org/v1/assign/key" method="post" %} [OrganisationManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-a07bf893965dbedd63f735032474958c00b8bca9%2FOrganisationManagement.yaml?alt=media) {% endopenapi %} # Location Management {% openapi src="/files/HJozu5gmRhb0u1x8oEft" path="/data/v1/location/create" method="post" %} [LocationManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-21ee3fbfc148f7405fd6e1a5894c5bc919220a2f%2FLocationManagement.yaml?alt=media) {% endopenapi %} {% openapi src="/files/HJozu5gmRhb0u1x8oEft" path="/data/v1/location/search" method="post" %} [LocationManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-21ee3fbfc148f7405fd6e1a5894c5bc919220a2f%2FLocationManagement.yaml?alt=media) {% endopenapi %} {% openapi src="/files/HJozu5gmRhb0u1x8oEft" path="/data/v1/location/update" method="patch" %} [LocationManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-21ee3fbfc148f7405fd6e1a5894c5bc919220a2f%2FLocationManagement.yaml?alt=media) {% endopenapi %} {% openapi src="/files/HJozu5gmRhb0u1x8oEft" path="/data/v1/location/delete/{locationId}" method="delete" %} [LocationManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-21ee3fbfc148f7405fd6e1a5894c5bc919220a2f%2FLocationManagement.yaml?alt=media) {% endopenapi %} # Consent Management {% openapi src="/files/mlHcC4s8l5PqQeMkfLgG" path="/user/v1/consent/update" method="post" %} [ConsentManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-cfa35d4719320afd380d6db30a706a6b1df5b14c%2FConsentManagement.yaml?alt=media) {% endopenapi %} {% openapi src="/files/mlHcC4s8l5PqQeMkfLgG" path="/user/v1/consent/read" method="post" %} [ConsentManagement.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-cfa35d4719320afd380d6db30a706a6b1df5b14c%2FConsentManagement.yaml?alt=media) {% endopenapi %} # OTP Services {% openapi src="/files/DIOpHRXPKwBNb0X34CBV" path="/otp/v1/generate" method="post" %} [OTP.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-b2d70501acbc7dbe2a59fad0751e20f8f7bed269%2FOTP.yaml?alt=media) {% endopenapi %} {% openapi src="/files/DIOpHRXPKwBNb0X34CBV" path="/otp/v1/verify" method="post" %} [OTP.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-b2d70501acbc7dbe2a59fad0751e20f8f7bed269%2FOTP.yaml?alt=media) {% endopenapi %} # Tenant Configurations {% openapi src="/files/OnSAVQxBzqirsn7dbZ8a" path="/org/v2/preferences/read" method="post" %} [TenantConfigurations.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-de068ac6d0b60d414b7dbbe78c1834c2c2e2bab8%2FTenantConfigurations.yaml?alt=media) {% endopenapi %} {% openapi src="/files/OnSAVQxBzqirsn7dbZ8a" path="/org/v2/preferences/create" method="post" %} [TenantConfigurations.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-de068ac6d0b60d414b7dbbe78c1834c2c2e2bab8%2FTenantConfigurations.yaml?alt=media) {% endopenapi %} {% openapi src="/files/OnSAVQxBzqirsn7dbZ8a" path="/org/v2/preferences/update" method="patch" %} [TenantConfigurations.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-de068ac6d0b60d414b7dbbe78c1834c2c2e2bab8%2FTenantConfigurations.yaml?alt=media) {% endopenapi %} # Bulk Upload {% openapi src="/files/8MoCaTLgdoOcaO1ogNSs" path="/user/v1/upload" method="post" %} [BulkUploadServices.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-56f3ca8a2fb2b0d6eb577b79bf3244cb83604fd0%2FBulkUploadServices.yaml?alt=media) {% endopenapi %} {% openapi src="/files/8MoCaTLgdoOcaO1ogNSs" path="/data/v1/upload/status/{processId}" method="get" %} [BulkUploadServices.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-56f3ca8a2fb2b0d6eb577b79bf3244cb83604fd0%2FBulkUploadServices.yaml?alt=media) {% endopenapi %} {% openapi src="/files/8MoCaTLgdoOcaO1ogNSs" path="/data/v1/bulk/location/upload" method="post" %} [BulkUploadServices.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-56f3ca8a2fb2b0d6eb577b79bf3244cb83604fd0%2FBulkUploadServices.yaml?alt=media) {% endopenapi %} {% openapi src="/files/8MoCaTLgdoOcaO1ogNSs" path="/org/v1/upload" method="post" %} [BulkUploadServices.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-56f3ca8a2fb2b0d6eb577b79bf3244cb83604fd0%2FBulkUploadServices.yaml?alt=media) {% endopenapi %} ## Sample file 1. Users Bulk Upload - {% file src="/files/TgBhRZPNT015zb5YEmxO" %} 2. Location Bulk Upload - {% file src="/files/CIdmhOvxSDjtJgRmH91h" %} 3. Organizations Bulk Upload - {% file src="/files/g3oEwkdjsdjdeb6I8Mtb" %} # System Settings {% openapi src="/files/nYXHvgOKwZGiaSQ7NCJD" path="/data/v1/system/settings/set" method="post" %} [SystemSettings.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-c1c7a9efc1c81a3f0a9f17b900b6c97f1a5eb5a3%2FSystemSettings.yaml?alt=media) {% endopenapi %} {% openapi src="/files/nYXHvgOKwZGiaSQ7NCJD" path="/data/v1/system/settings/get/{configName}" method="get" %} [SystemSettings.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-c1c7a9efc1c81a3f0a9f17b900b6c97f1a5eb5a3%2FSystemSettings.yaml?alt=media) {% endopenapi %} {% openapi src="/files/nYXHvgOKwZGiaSQ7NCJD" path="/data/v1/system/settings/list" method="get" %} [SystemSettings.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2Fgit-blob-c1c7a9efc1c81a3f0a9f17b900b6c97f1a5eb5a3%2FSystemSettings.yaml?alt=media) {% endopenapi %} # API Management Service The API management microservice provides services to create, manage and validate API tokens to register mobile and desktop apps, and to issue and refresh API tokens of registered mobile and desktop app devices. {% openapi src="/files/2vDT7Cm33TGgWaFz3eYc" path="/api-manager/v1/consumer/{consumer}/credential/register" method="post" %} [kongcredentialregisterapiv1.yml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2F9n7oehjOa54zxG4loWrH%2Fkongcredentialregisterapiv1.yml?alt=media\&token=3b331af7-a058-4be6-87d9-f909a9bd1576) {% endopenapi %} {% openapi src="/files/sE8bsWGzFv3uqvnJf8Ru" path="/api-manager/v2/consumer/{consumer}/credential/register" method="post" %} [kongcredentialregisterapiv2.yml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2FOGUqErUbIlGe8861LpHr%2Fkongcredentialregisterapiv2.yml?alt=media\&token=52144657-7ccb-4bd3-99cf-8bec4904d95d) {% endopenapi %} {% openapi src="/files/Qkm147vP854D7wfNZ7zz" path="/auth/v1/refresh/token" method="post" %} [refreshtokenapi.yml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2FkI3Q1tivcIOVbVK3VVH3%2Frefreshtokenapi.yml?alt=media\&token=e6df1b59-7864-4017-a784-a0a5363bade4) {% endopenapi %} {% openapi src="/files/7xiKYQjzdLDoMCFbif8v" path="/api-manager/v2/consumer/desktop\_device/credential/register" method="post" %} [deviceregistry.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2F5k5ppjP5l2ScwdmUSsJi%2Fdeviceregistry.yaml?alt=media\&token=459a467a-d2dc-4235-abb3-2e197088b2b2) {% endopenapi %} # Data Sync Data synchronization API(s) establish consistency among data from a source to a target data storage. {% openapi src="/files/0EAga22JYPnKP8oj3Eow" path="/data/v1/index/sync" method="post" %} [datasyncapi.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2F41wo0HSKhb4vpYoxIEe2%2Fdatasyncapi.yaml?alt=media\&token=62197a55-c0f7-41df-b1ff-90c5af389aed) {% endopenapi %} # Notification APIs {% openapi src="/files/DcJN0wML9shau5u56tud" path="/user/v1/notification/email " method="post" %} [notificationapi.yaml](https://308610995-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4ZKyfmmhMWpPkD6iYvKF%2Fuploads%2FsqWKsASTf86QsTngY2cw%2Fnotificationapi.yaml?alt=media\&token=aa7f0221-1325-472e-a5f8-4b6763474c99) {% endopenapi %} # Flink Jobs UserOrg service uses User cache updater flink job to de-normalise the user data and stores it in Redis. ### Reference * # User Cache Updater 'user-cache-updater-2.0' job is used to generate the user-metadata information which is complied by fetching information from various Cassandra tables and are stored into the Redis cache. This user-metadata information is used by few exhaust reports. #### Additional Reading: , #### Design guide: **Configuration variables:**
VariableDefault ValuePurpose
kafka.input.topic{{env}}.telemetry.auditKafka topic from which messages/events are read to be processed.
kafka.groupId{{env}}-user-cache-updater-groupKafka input topic group Id
redis-meta.database.userstore.id12Redis index to which user metadata is to be written to for caching
redis-meta.database.key.expiry.seconds3600Redis cache expiry in seconds
user-read.api.url"/learner/private/user/v1/read"API Endpoint for fetching User profile details
regd.user.producer.pidlearner-serviceused to specify service providing user microservice
user.self.signin.types["google","self"]used to specify self sign-in modes available in application
user.validated.types["sso"]used to specify sign-in modes where user validation is signed from third party system
user.self.signin.key"Self-Signed-In"
user.valid.key"Validated"
user.read.url.fields"locations,organisations"used to specify the user metadata properties that are to be cached to Redis
user.read.api.error["CLIENT_ERROR"]
**Sample event:** ```json { "eid": "AUDIT", "ets": 1573121861118, "ver": "3.0", "mid": "1573121861118.40f9136b-1cc3-458d-a04a-4459606df", "actor": { "id": "5609876543234567890987654345678", "type": "Request" }, "context": { "channel": "01285019302823526477", "pdata": { "id": "dev.sunbird.portal", "pid": "learner-service", "ver": "2.5.0" }, "env": "User", "did": "user-3", "cdata": [ { "id": "google", "type": "SignupType" } ], "rollup": { "l1": "01285019302823526477" } }, "object": { "id": "user-1", "type": "user" }, "edata": { "state": "Update", "props": [ "recoveryEmail", "recoveryPhone", "userId", "id", "externalIds", "updatedDate", "updatedBy" ] }, "syncts": 1573121861125, "@timestamp": "2019-11-07T10:17:41.125Z", "flags": { "tv_processed": true, "dd_processed": true }, "type": "events", "ts": "2019-11-07T10:17:41.118+0000" } ``` **Source code:** {% embed url="" %} # User Deletion Cleanup Flink Job The **`UserDeletionCleanupFunction`** job handles the cleanup and deletion of user-related data from various storage systems when users are deleted. It ensures that all related information is removed or masked appropriately in multiple data sources, including Cassandra and Keycloak. Here's what the job does in simpler terms: * **Identify User Deletion Events**: * The job listens for events indicating that a user is to be deleted. * It retrieves user information from a user organization service to validate the deletion event. * **Keycloak User Removal**: * The job connects to Keycloak to remove the user credentials. * If removal fails, it masks sensitive information (e.g., email, phone) to ensure user data is not exposed. * **Database Operations**: * The job connects to Cassandra and updates the user lookup table to remove entries related to the user (like email, phone, external ID, and username). * It updates the main user table in Cassandra to reflect the deletion status by clearing or masking personal information. * It also removes user entries from the external identity table in Cassandra. * The job updates the organization table to mark the user as deleted and record the date of deletion. * **Audit Events and Logging**: * The job generates audit events to track the cleanup process for telemetry and auditing purposes. * It logs detailed information during the deletion process for troubleshooting and error handling. In summary, this job ensures that when a user is deleted, all related data is appropriately removed or masked across multiple storage systems, maintaining data privacy and consistency. It plays a crucial role in the cleanup process, ensuring that user data is properly handled upon deletion. ### **Configuration Variables:** | Variable | Default value | purpose | | ------------------------------------------------- | ------------------------------------------------- | --------------------------------------------------------------- | | kafka.input.topic | {{env}}.delete.user | Kafka topic from which messages/events are read to be processed | | kafka.groupId | {{env}}-delete-user-group | Kafka input topic group Id | | user.keyspace | sunbird | Cassandra keyspace name | | user.lookup.table | user\_lookup | Cassandra table used to store user lookup data | | user.table | user | Cassandra table used to store user details | | user.externalIdentity.table | usr\_external\_identity | Cassandra table used to store user extrenal identity details | | user.org.table | user\_organisation | Cassandra table used to store organisation details | | service.lms.basePath | | lms base url | | service.userorg.basePath | | User-Org service URL | | sunbird\_keycloak\_user\_federation\_provider\_id | sunbird\_keycloak\_user\_federation\_provider\_id | fedaration provider id for key cloak. | | user\_read\_api | | API route for fetching user profile details | | batch\_search\_api | | API route for fetching batch details | | user.ownership.transfer.parallelism | 1 | Degree of parallelism for the user ownership | ### **Sample event:** ```json { "eid": "BE_JOB_REQUEST", "ets": 1619527882745, "mid": "LP.1619527882745.32dc378a-430f-49f6-83b5-bd73b767ad36", "actor": { "id": "delete-user", "type": "System" }, "context": { "pdata": { "id": "org.sunbird.platform", "ver": "1.0" } }, "object": { "id": "", "type": "User" }, "edata": { "organisationId": "" "userId": "", "suggested_users": [ { "role": "ORG_ADMIN", "users": [""] }, { "role": "CONTENT_CREATOR", "users": [""] }, { "role": "COURSE_MENTOR", "users": [""] } ], "action": "delete-user", "iteration": 1 } } ``` # Ownership Transfer Flink Job The **`UserOwnershipTransferFunction`** job facilitates the transfer of course batch ownership from one user to another. It achieves this by performing database and search updates to ensure consistent data. The job interacts with several data sources, including Cassandra and Elasticsearch, to locate the relevant information and update it accordingly. Here's what the job does in simpler terms: * **Identify Ownership Transfers**: * This job takes an event indicating an ownership transfer between users. * It processes the event to determine the source user (`fromUserId`) and the target user (`toUserId`). * **Database Operations**: * It connects to Cassandra and retrieves information about course batches created by the source user and where they act as a mentor. * The job creates Cassandra update queries to change the ownership from the source user to the target user in the appropriate fields. * It executes these updates in batches to ensure efficient processing. * **Search Operations**: * It interacts with Elasticsearch to find the corresponding course batch documents. * The job updates the documents to reflect the new ownership, replacing the source user with the target user in the appropriate fields. * **Metrics and Logging**: * The job maintains metrics to track the number of processed events, successful operations, and database updates. * It logs detailed information for auditing and error handling. In summary, this job facilitates the smooth transfer of course ownership, updating relevant data in multiple locations to maintain consistency across the system. It plays a crucial role in ensuring that the information in both the database and search systems accurately reflects the ownership transfer. ### **Configuration variables:** | Variable | Default value | purpose | | ----------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | kafka.input.topic | {{env}}.user.ownership.transfer | Kafka topic from which messages/events are read to be processed | | kafka.groupId | {{env}}-user-ownership-transfer-group | Kafka input topic group Id | | lms-cassandra.keyspace | sunbird\_courses | Cassandra keyspace name | | lms-cassandra.course\_batch.table | course\_batch | Cassandra table used to store batch details of a collection. Batch status, start date , end date , batch enrolment end date, enrolment type (open/invite-only), certificate templates etc are stored in this table. | | threshold.batch.write.size | 10 | Property used to specify batch size of the database update queries while updating a specific cassandra table in batch format | | service.lms.basePath | | lms base url | | service.userorg.basePath | | User-Org service URL | | user\_read\_api | /private/user/v1/read/ | API route for fetching user profile details | | batch\_search\_api | /v1/course/batch/search | API route for fetching batch details | | user.ownership.transfer.parallelism | 1 | Degree of parallelism for the user ownership | ### **Sample event:** ```json { "eid": "BE_JOB_REQUEST", "ets": 1712750750956, "mid": "LP.1712750750956.07a0a24d-37ef-462c-a614-b76ad2a6a6ac", "actor": { "type": "System", "id": "ownership-transfer" }, "context": { "pdata": { "ver": "1.0", "id": "org.sunbird.platform" } }, "object": { "type": "user", "id": "72d8cd69-2469-4234-82e7-6b849e0a28d9" }, "edata": { "organisationId": "01394517023437619214_1111", "actionBy": { "userId": "ad8c3adf-2447-4559-af15-f6d1057a0b8a", "userName": "gtest-user-007" }, "context": "User Deletion", "action": "ownership-transfer", "fromUserProfile": { "userId": "72d8cd69-2469-4234-82e7-6b849e0a28d9", "userName": "gtest-user-005", "channel": "", "organisationId": "", "roles": [ "BOOK_CREATOR", "CONTENT_CREATOR" ] }, "iteration": 1, "assetInformation": { "name": "TestContent", "identifier": "do_123", "primaryCategory": "Practice Question Set", "objectType": "QuestionSet" }, "toUserProfile": { "userId": "4c009ce1-b069-4d27-879b-605c55ff4ef9", "userName": "gtest-user-006", "firstName": "G-Test", "lastName": "User-006", "roles": [ "BOOK_CREATOR", "CONTENT_CREATOR" ] } } } ``` ### **Source code:** {% embed url="" %} # Reports There are two reports which comes under ‘MANAGE’ page in Sunbird Ed for 'ORG\_ADMIN' role. There reports are fetched from azure blob and details are displayed. Also allows to download in .csv format. UserOrg service jobs include: * State Admin Geo Report * State Admin Report or User Consent Report * User Cache Indexer Job #### Configuration and setup: These can be configured as cron jobs. \#Ansible: sunbirdstaging-admin-user-reports-3AMIST\ 30 21 \* \* \* /mount/data/analytics/scripts/run-job.sh admin-user-reports\ \#Ansible: sunbirdstaging-admin-user-reports-2PMIST\ 30 8 \* \* \* /mount/data/analytics/scripts/run-job.sh admin-user-reports\ \#Ansible: sunbirdstaging-admin-geo-reports-3PMIST\ 30 9 \* \* \* /mount/data/analytics/scripts/run-job.sh admin-geo-reports\ \#Ansible: sunbirdstaging-admin-geo-reports-4AMIST\ 30 22 \* \* \* /mount/data/analytics/scripts/run-job.sh admin-geo-reports Also can be run explicitly by using below jenkins job : Deploy/DataPipeline/AnalyticsReplayJobs set Job\_type : run-job set job\_id : admin-user-reports → for running User-Consent report admin-geo-reports → for running Geo-report # Standard Exhaust These reports are not generated on demand; rather, they are produced as a daily job, generating CSV output without any user requests.
The standard exhausts run daily as a job in the scheduled intervals and produce the report without any request from the organization administrator, course administrator, or report administrator # State Admin Geo report ### **1. Geo-report:** * geo-summary shows the number of schools, districts and blocks in a particular org. * geo-summary-district shows the data with respect to district. StateAdminGeoReportJob generates three folders in azure blob : geo-detail, geo-summary, geo-summary-district **geo-detail** contains csv reports with channel name and geo-summary, geo-summary-district contains json reports with channel name. \ **geo-summary** shows the number of schools, districts and blocks in a particular org. Example: \[{"index":1,"districtName":"CHITTOOR","blocks":1,"schools":1}]. \ **geo-summary-district** provides the data with respect to district. Example: \[{"index":1,"districtName":"ARIYALUR","blocks":6,"schools":796},{"index":2,"districtName":"CHENNAI","blocks":10,"schools":1472},]\
\ **Data provider:** **Cassandra** 1. organisation 2. location 3. user # State Admin Report The consent Report gives the user info organization-wise as per the consent given by the user. StateAdminReportJob generates a folder declared\_user\_detail which contains the user-declared consent and user details. It is generated in CSV format with respect to the organization name. It also provides organisation-wise consent details.
**Data Provider:** 1. user\_declaration 2. user\_consent 3. organization 4. location 5. user **Consent report CSV content:**
Column LabelColumn TypeData TypeDescription
NameStaticStringName of the user
User UUIDStaticStringThe system generated unique user ID
StateStaticStringUser declared state for self signed up users. If the user is a org validated user then the state as passed from org SSO or derived from Sub-Org ID.
DistrictStaticStringUser declared district for self signed up users. If the user is a org validated user then the district as passed from org SSO or derived from Sub-Org ID.
BlockStaticStringUser declared block for self signed up users. If the user is a org validated user then the block as passed from org SSO or derived from Sub-Org ID.
ClusterStaticStringIf the user is a org validated user then the cluster as passed from org SSO or derived from Sub-Org ID.
User declared cluster for self signed up users.
Sub-org NameStaticStringIf user is org validated then the sub-org name mapped to this user. If user is self-declared then the user declared org/sub-org name.
Sub-org IDStaticStringIf user is org validated then the sub-org ID is mapped to this user. If the user is self-declared then the user declared org/sub-org ID.
Organization provided unique IDStaticStringSelf declared users this is their declared ID. For organization validated users this is their organization ID
User PhoneStaticStringUser declared unmasked mobile number from consent declaration
User Email IDStaticStringUser declared unmasked email ID from consent declaration
User TypeStaticStringUser type from user profile
User-Sub TypeStaticStringUser sub type from user profile
Root Org of userStaticStringRoot/Tenant Org the user belongs to
# Other Jobs # User Cache Indexer Job The User Cache Indexer job reads user data from Cassandra, performs transformations and joins, and stores the processed data in Redis for caching and indexing purposes.
**Data provider:** \ **Cassandra** 1. user 2. location **User Redis**\ \ For more information please visit,
{% content-ref url="/pages/dTF51Av1HH1juddPGIhc" %} [UserCacheUpdaterFlinkJob](/use/developer-guide/user-and-org-service/caching-and-denormalising-user-metadata/usercacheupdaterflinkjob) {% endcontent-ref %} {% content-ref url="/pages/nbxtQQJ8sFfHbFtGL5xE" %} [ETLUserCacheUpdaterJob](/use/developer-guide/user-and-org-service/caching-and-denormalising-user-metadata/etlusercacheupdaterjob) {% endcontent-ref %} # Delete Users Assets Report ### The "Deleted Users Assets Report" provides an overview of assets (content and course assets) created by users who have been deleted from the system. The report involves the following steps: 1. **Fetch Deleted Users:** Retrieve a list of users who have been marked as deleted from the database. 2. **Extract User IDs:** Extract the unique identifiers for these deleted users. 3. **Fetch Content Assets:** Query an external API to fetch assets (such as documents, media, etc.) created by these deleted users. 4. **Fetch Course Assets:** Query an external API to fetch course-related assets (like course batches) associated with these deleted users. 5. **Fetch ML Assets from API or MongoDB**: The job queries an external API (ML\_ASSET\_SEARCH\_URL) to fetch ML assets (solutions and programs) created by the deleted users. Alternatively, if the API call fails, the job directly queries MongoDB to retrieve the required ML asset data. This involves connecting to the MongoDB instance and executing aggregate queries to filter and retrieve the relevant ML assets based on user IDs. 6. **Process DataFrames:** Integrate the dataframes for deleted users and their assets, and perform necessary processing, such as filtering and deduplication. 7. **Save Report to Blob Storage:** Save the final processed report to a cloud-based blob storage, typically in CSV format. This report is useful for auditing, tracking, and removing assets that are no longer needed, ensuring system cleanliness and compliance with data retention policies.
\ **Data provider:** * Cassandra - User table * Course Batch API * Content Search API **Delete Asset report CSV content:** | Column Label | Column Type | Data Type | Description | | --------------- | ----------- | --------- | ------------------------------------------------------------------------------------------- | | userId | Static | String | User Id of the deleted User | | username | Static | String | Decrypted user name of the deleted user | | roles | Static | String | Roles of the deleted user | | assetIdentifier | Static | String | Identifier of the asset / course batch / ML program / solution | | assetName | Static | String | Name of the asset / course batch / ML program / solution | | assetStatus | Static | String | Status of the asset / course batch / ML program / solution i.e live/draft | | objectType | Static | String | Object type of the asset/course batch / ML program / solution i.e QuestionSet/content/batch | \ **Sample data:** ``` userId username roles assetIdentifier assetName assetStatus objectType cc3220fe-ace3-4716-a3a4-4ed799740947 TestUser1 ORG_ADMIN do_11395143971887513612 Test SRK QuestionSet Live QuestionSet ed8f22b1-1c33-4ca5-b8fd-572844ef3a7a Demouser CONTENT_CREATOR do_11395139703507353614 QuestionSet-FT-2014 Draft QuestionSet ed8f22b1-1c33-4ca5-b8fd-572844ef3a7a DeleteuserTest BOOK_CREATOR do_11395217540852940811 sameple course Live Content ``` # Data Products Developer Installation Getting Started Guide Data-products is a collection of scala scripts which are used to generate reports, updating data in the redis and migration of data using spark. This guide helps you to install [Data-products](https://github.com/Sunbird-Lern/data-products) on developer machine. {% embed url="" %} # System Requirements Ensure that your laptop or desktop has the following minimum system requirements: * Operating System: Windows 7 and above, or 4.2 Mac OS X 10.0 and above/Linux * RAM: >4 GB * CPU: 4 cores, >2 GHz # Tech Stack | Technology | Version | | ------------- | ------- | | Java | 11 | | Scala | 2.12 | | Spark | 3.1.3 | | Elasticsearch | 6.8.11 | | Cassandra | 3.11.8 | | Postgres | | | Redis | | | Druid | | # Installation Dependencies ### Analytics framework: Analytics job driver and analytics framework is used to trigger the job in job manager. **GitHub Repository:** [Sunbird-Obsrv/sunbird-analytics-core](https://github.com/Sunbird-Obsrv/sunbird-analytics-core) **Adopters:** Diksha **Contributors**: EkStep **Jenkins Job:** /Build/DataPipeline/AnalyticsCore /Deploy/DataPipeline/AnalyticsCore It will generate `analytics-framework-2.0.jar` in `/mount/data/analytics/models-2.0/` ### Core Data-products: Batch-models module is used from this library handling the execution of job **GitHub Repository:** [Sunbird-Obsrv/sunbird-core-dataproducts](https://github.com/Sunbird-Obsrv/sunbird-core-dataproducts) **Adopters:** Diksha **Contributors**: EkStep **Jenkins Job:** /Build/DataPipeline/CoreDataProducts /Deploy/DataPipeline/CoreDataProducts It will generate `batch-models-2.0.jar` in `/mount/data/analytics/models-2.0/` # Local installation of data-products ## Setup of data-products in local machine Each data-product is an independent spark job used for generating reports and data migrations, so each has different sets of data provider dependencies. So the data-products can be tested locally with the test cases. Kindly find the local setup guide from the below link. {% embed url="" %} README.md {% endembed %} # Server setup Guide This guide helps to deploy in data-products in server. ## Setup and execution of data-products in the server Each data-product is an independent spark job that runs in a spark-submit mode for generating reports and data migrations. So it requires, all the data sources and dependency libraries to be present before executing data-product. ### Building data-product Job Path: Build/Lern/LernDataProducts ### **Deploying data-product** Job Path: Deploy/{{env}}/Lern/LernDataProducts Params: * **`module`** - this parameter used to deploy respective process of data-products deployment 1. `lern-dataproducts` - to deploy data-products 2. `lern-dataproducts-spark-cluster` - to deploy data-products in spark cluster (such as hd insight cluster) 3. `cronjobs` - to update cronjobs in server * `remote` - to which spark server to deploy the above module ### **Cron jobs** Data-product is running in demon mode which is getting triggered based on schedule by using cronjobs. 
## Cron job list for LERN data-products

#Ansible: sunbird-job-manager
30 2 * * * /mount/data/analytics/scripts/start-jobmanager.sh
#Ansible: sunbird-course-batch-status-updater
*/60 * * * * /mount/data/analytics/scripts/lern-run-job.sh course-batch-status-updater
#Ansible: sunbird-admin-user-reports-2PMIST
30 8 * * * /mount/data/analytics/scripts/lern-run-job.sh admin-user-reports
#Ansible: sunbird-admin-user-reports-3AMIST
30 21 * * * /mount/data/analytics/scripts/lern-run-job.sh admin-user-reports
#Ansible: sunbird-admin-geo-reports-3PMIST
30 9 * * * /mount/data/analytics/scripts/lern-run-job.sh admin-geo-reports
#Ansible: sunbird-admin-geo-reports-4AMIST
30 22 * * * /mount/data/analytics/scripts/lern-run-job.sh admin-geo-reports
#Ansible: sunbird-progress-exhaust
0 08 * * * /mount/data/analytics/scripts/lern-run-job.sh progress-exhaust
#Ansible: sunbird-response-exhaust
0 09 * * * /mount/data/analytics/scripts/lern-run-job.sh response-exhaust
#Ansible: sunbird-cassandra-migration
15 19 * * * /mount/data/analytics/scripts/lern-run-job.sh cassandra-migration
#Ansible: sunbird-userinfo-exhaust
0 10 * * * /mount/data/analytics/scripts/lern-run-job.sh userinfo-exhaust
#Ansible: sunbird-collection-summary
30 09 * * * /mount/data/analytics/scripts/lern-run-job.sh collection-summary-report
### Provisioning Postgres DB for exhaust job execution In data-products, exhaust jobs are using job\_request table from postgres DB for maintaining the exhaust job requests. Job Path: Provision/{{env}}/DataPipeline/PostgresDbUpdate ### Running a data-product through Jenkins Data-product is running in server using cronjobs. For development and testing purpose, below Jenkins job can be used to trigger the job with respective job id. Job Path: Deploy/{{env}}/Lern/LernAnalyticsReplayJobs Params: * job\_type - `run-job` * job\_id - specific job id such as (admin-user-reports, progress-exhaust) * batch\_identifier - specific batch id * start\_date - Data consumption start date. not required for LERN data-products * end\_date - Data consumption end date. not required for LERN data-products * private\_branch - specific private branch * branch\_or\_tag - public branch ### Running a data-product using shell command The data-product can be executed with the following shell command in the server ``` /mount/data/analytics/scripts/lern-run-job.sh { job-id } #Example: /mount/data/analytics/scripts/lern-run-job.sh admin-user-reports ``` # Installation Configuration # Data-product creation guide Each data-product is an independent spark job that runs in a spark-submit mode for generating reports and data migrations. So, even for a new data-product, we will have to add a new scala script with some class base classes extended. ### Data-product execution flow

Data-product tech level architecture

### Exhaust job execution flow
As mentioned in Data-product execution overflow, all the data-products are under the `JobExecutor` from sunbird-core-dataproducts. Thus, before creating a data-product, dependency libraries need to be setup. Required Baseclasses a new Data-product: * `IJob` It is an abstract class in from sunbird-core-dataproducts which used to represent script as data-product job to the job manager * `BaseReportsJob` It has the spark utility functions such as creating a spark session for a data-product. ## Configurations Data-product can be executed with following two levels of cofiguration. ### `application level config` This config is provided from the `application.conf` file which is common for all the data-products and will not be modified frequently. Github Path for the template which is used to create the `.conf` file: {% embed url="" %} ### `lern-model-config` Each job is collecting data from different data-providers and has various types of inputs. So, model config is implemented to serve data-product level configuration Github Path for the template which is used to create the model-config file: {% embed url="" %} ### Script changes for creating a new data-product Since the data-product is a batch processing scripts, in server data-products are triggered using shell scripts. So whenever the data-product is implemented we need to add the job id and model-config in the below shell script templates. `lern-run-job` : job-id which will be used as identifier and respective data-product classpath will be added in this scripts. {% embed url="" %} `lern-model-config` : Respective job id job config has to be added in the this script {% embed url="" %} # Troubleshooting a data-product ## Logs for Data-products In the server, data-product generates two types of logs when running a job. ### 1. Execution log The spark script code execution log can be found in the below location of the server. ``` /mount/data/analytics/logs/lern-data-products/{{date}}-job-execution.log ``` ### 2. Joblog Also, from the job we are having logs for identifying the status of the job. This type of log configurable to put either in log file or kafka topic based on below config. ``` log.appender.kafka.enable="{{ boolean }}" log.appender.kafka.broker_host="{{ host }}:9092" log.appender.kafka.topic="{{ env }}.druid.events.log" ``` If `log.appender.kafka.enable` is `false`, the log goes into the below file. ``` /mount/data/analytics/scripts/logs/joblog.log ``` # Logs, Telemetry Events Telemetry is a specification to instrument all the key events. Using this specification reference applications & services will generate telemetry events. For more info refer Sunbird-Telemetry documentation [here](https://telemetry.sunbird.org/). Also to know how telemetry is processed refer to Telemetry Processing Documentation in . Logs also use telemetry specification and request tracing/ tracking is accomplished by passing the request context into the logs and telemetry. This helps in troubleshooting issues. Request context has below details: ``` uid //user id from user authentication token in header (x-authenticated-user-token) did //device id from request header (x-device-id) sid //session id from request header (x-session-id) reqId //request id from request header (x-request-id) ``` This request id also set to API response as msgid. This helps in tracking once request that invokes multiple APIs and flows.
### Configuration Telemetry has request context information and it is set during the telemetry logging. Below are the properties set by the application to add the producer information into the telemetry data: \#Telemetry producer related info ``` telemetry_pdata_id=local.sunbird.learning.service telemetry_pdata_pid=learning-service telemetry_pdata_ver=5.1.0 ``` Telemetry is logged and also send to SB-Obsrv data-pipeline using logback in user-org service. logback.xml configuration has a kafka appender which will send the telemetry generated by the code to a kafka topic. ``` ${ENV_NAME}.telemetry.raw ``` Logback.xml configuration: 
   <appender name="kafka-appender" class="com.github.danielwegener.logback.kafka.KafkaAppender">
    <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
      <pattern>%msg</pattern>
    </encoder>

    <topic>${ENV_NAME}.telemetry.raw</topic>
    <!-- ensure that every message sent by the executing host is partitioned to the same partition strategy -->
    <keyingStrategy class="com.github.danielwegener.logback.kafka.keying.NoKeyKeyingStrategy" />
    <!-- block the logging application thread if the kafka appender cannot keep up with sending the log messages -->
    <deliveryStrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy" />

    <!-- each <producerConfig> translates to regular kafka-client config (format: key=value) -->
    <!-- producer configs are documented here: https://kafka.apache.org/documentation.html#newproducerconfigs -->
    <!-- bootstrap.servers is the only mandatory producerConfig -->
    <producerConfig>bootstrap.servers=${SUNBIRD_KAFKA_URL}</producerConfig>
    <!-- don't wait for a broker to ack the reception of a batch.  -->
    <producerConfig>acks=0</producerConfig>
    <!-- wait up to 1000ms and collect log messages before sending them as a batch -->
    <producerConfig>linger.ms=15000</producerConfig>
    <!-- even if the producer buffer runs full, do not block the application but start to drop messages -->
    <producerConfig>max.block.ms=0</producerConfig>
    <!-- define a client-id that you use to identify yourself against the kafka broker -->
    <producerConfig>client.id=${HOSTNAME}-${CONTEXT_NAME}-logback-relaxed</producerConfig>

    <!-- there is no fallback <appender-ref>. If this appender cannot deliver, it will drop its messages. -->

  </appender>
  <logger name="TelemetryEventLogger" level="INFO">
    <appender-ref ref="kafka-appender" />
  </logger>
### List of Events
Audit Event ``` { "eid":"AUDIT", "ets":1649247985143, "ver":"3.0", "mid":"d808691c-e253-43a6-a8a0-aa03bc67b6ce", "actor":{ "id":"50792198-c6d7-4964-8d4c-da6891ceed0a", "type":"User" }, "context":{ "channel":"0126796199493140480", "pdata":{ "id":"staging.sunbird.learning.service", "pid":"learner-service", "ver":"4.7.0" }, "env":"User", "cdata":[ { "id":"d808691c-e253-43a6-a8a0-aa03bc67b6ce", "type":"Request" } ], "rollup":{ "l1":"0126796199493140480" } }, "object":{ "type":"User" }, "edata":{ "state":"Update", "props":[ "identifier", "tncAcceptedOn", "id", "tncAcceptedVersion" ] } } ```
Search Event ``` { "eid":"SEARCH", "ets":1649247379860, "ver":"3.0", "mid":"4e0e50ed-92c0-5c80-ff29-b3a194a1911f", "actor":{ "id":"86fe48dd-72d5-4f27-a9b4-c55580878ec4", "type":"User" }, "context":{ "channel":"0126796199493140480", "pdata":{ "id":"staging.dock.portal", "pid":"learner-service", "ver":"4.7.0" }, "env":"User", "did":"487975adbe74ea73faea476eab1ebb31", "cdata":[ { "id":"4e0e50ed-92c0-5c80-ff29-b3a194a1911f", "type":"Request" } ], "rollup":{ "l1":"0126796199493140480" } }, "edata":{ "size":1, "query":"", "filters":{ "id":[ "0126796199493140480" ] }, "sort":{ }, "type":"Org_alias", "topn":[ { "id":"0126796199493140480" } ] } } ```
Error Event ``` { "eid":"ERROR", "ets":1649248112302, "ver":"3.0", "mid":"31eab671-1395-4135-8723-15ffa5d349cb", "actor":{ "id":"internal", "type":"Consumer" }, "context":{ "channel":"0126796199493140480", "pdata":{ "id":"staging.sunbird.learning.service", "pid":"learner-service", "ver":"4.7.0" }, "env":"Organisation", "cdata":[ { "id":"31eab671-1395-4135-8723-15ffa5d349cb", "type":"Request" } ], "rollup":{ } }, "edata":{ "err":"UOS_ORGSER0017", "stacktrace":"Invalid value null for parameter hashTagId. Please provide a valid value. org.sunbird.validator.BaseRequestValidator.lambda$validateListValues$6(BaseRequestValidator.java:291)java.base/java.util.ArrayList.forEach(ArrayList.java:1541)org.sunbird.validator.BaseRequestValidator.validateListValues", "errtype":"api_access", "requestid":"31eab671-1395-4135-8723-15ffa5d349cb" } } ```
Log Event ``` { "eid":"LOG", "actor":{ "id":"internal", "type":"Consumer" }, "edata":{ "level":"info", "type":"Api_access", "message":"", "params":[ { "method":"POST" }, { "url":"/v1/org/search" }, { "duration":0 }, { "status":"OK" } ] }, "ver":"3.0", "syncts":1649247488365, "@timestamp":"2022-04-06T12:18:08.365Z", "ets":1649247476273, "context":{ "channel":"0126796199493140480", "pdata":{ "id":"staging.sunbird.learning.service", "pid":"learner-service", "ver":"4.7.0" }, "env":"Organisation", "cdata":[ { "id":"9c158007-345b-44d7-a128-6c36f7a42cfb", "type":"Request" } ], "rollup":{ } }, "flags":{ "pp_validation_processed":true }, "mid":"9c158007-345b-44d7-a128-6c36f7a42cfb", "type":"events" } ```
### Sample Telemetry Data
Audit Event Data ``` { "eid": "AUDIT", "ets": 1566563420660, "ver": "3.0", "mid": "1566563420660.f46c14d1-8c9a-417f-82b8-f125ba32b828", "actor": { "id": "internal", "type": "Consumer" }, "context": { "channel": "0128220189818880000", "pdata": { "id": "staging.diksha.learning.service", // Producer ID. "ver": "1.15", // Version of the App "pid": "learner-service"// Optional. In case the component is distributed, then which instance of that component }, "env": "User", "cdata": [ { "id": "4e2afe8e-fb44-4788-9f49-0ef61c5c808b", "type": "User" }, { "id": "11166", "type": "Certificate" } ], "rollup": { "l1": "0128220189818880000" } }, "object": { "id": "11166", "type": "Certificate" }, "edata": { "state": "Create", // defines the state i.e: Mergecert, Mergeuser "props": [ "certId", // certificate Id "userId" // user Id ] } } ```
# Configuration # Functional Configurations {% tabs %} {% tab title="Functional Config" %} ``` sunbird_username_num_digits=4 download_link_expiry_timeout=300 sunbird_default_country_code=+91 sunbird_encryption_key=SunBird sunbird_encryption=ON sunbird_otp_allowed_attempt=2 ``` ### Bulk Upload Config ``` #size of bulk upload data is 1001 including header in csv file sunbird_user_bulk_upload_size=1001 bulk_upload_org_data_size=300 # Bulk upload file max size in MB file_upload_max_size=10 ``` ### Batch Service Config ``` # Batch size for cassandra batch operation cassandra_write_batch_size=100 ``` ### OTP Config ``` sunbird_otp_expiration=1800 sunbird_otp_length=6 sunbird_otp_hour_rate_limit=5 sunbird_otp_day_rate_limit=20 ``` ### Other Config ``` managed_user_limit=30 sunbird_api_request_lower_case_fields=source,externalId,userName,provider,loginId,email,prevUsedEmail sunbird_rate_limit_enabled=true sunbird_health_check_enable=true sunbird_sync_read_wait_time=1500 sunbird_gzip_size_threshold=262144 sunbird_fuzzy_search_threshold=0.5 ekstep_authorization= ``` {% endtab %} {% tab title="UserOrg System Config" %} | JSON Key | Key | Default Value | | ------------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------ | | ENV\_NAME | ENV\_NAME | sunbirdstaging | |


PORTAL\_SERVICE\_PORT

| PORTAL\_SERVICE\_PORT | | |

SUNBIRD\_KAFKA\_URL

| sunbird\_KAFKA\_URL | 11.3.2.16:9092,11.3.2.15:9092,11.3.2.14:9092 | |


| accesstoken.publickey.basepath | /keys/ | |


| api\_actor\_provider | local (off) | |


| background\_actor\_provider | local (remote) | |


EKSTEP\_BASE\_URL

| ekstep\_api\_base\_url | | |


EKSTEP\_AUTHORIZATION

| ekstep\_authorization | eg: public\_key | |


| feed\_limit | 30 | |


FORM\_API\_ENDPOINT

| form\_api\_endpoint | /plugin/v1/form/read | |


GOOGLE\_CAPTCHA\_PRIVATE\_KEY

| google\_captcha\_mobile\_private\_key | eg: mbl\_private\_key | |

GOOGLE\_CAPTCHA\_PRIVATE\_KEY

| google\_captcha\_private\_key | eg: private\_key | |

SUNBIRD\_KAFKA\_URL

| kafka\_urls | 11.3.2.16:9092,11.3.2.15:9092,11.3.2.14:9092 | | | learner\_in\_memory\_cache\_ttl | 600 | | NOTIFICATION\_SERVICE\_BASE\_URL | notification\_service\_base\_url | | |


| org\_index\_alias | org\_alias | |


| quartz\_shadow\_user\_migration\_timer | 0 0/10 \* 1/1 \* ? \* | | ACCOUNT\_KEY | sunbird\_account\_key | eg: key | | ACCOUNT\_NAME | sunbird\_account\_name | sunbirdstagingpublic | |

ANALYTICS\_API\_BASE\_URL

| sunbird\_analytics\_api\_base\_url | | | ANALYTICS\_ACCOUNT\_KEY | sunbird\_analytics\_blob\_account\_key | eg: key | | ANALYTICS\_ACCOUNT\_NAME | sunbird\_analytics\_blob\_account\_name | sunbirdstagingprivate | | SUNBIRD\_API\_BASE\_URL | sunbird\_api\_base\_url | | | SUNBIRD\_AUTHORIZATION | sunbird\_authorization | eg: key | | | sunbird\_badger\_baseurl | | |


SUNBIRD\_CASSANDRA\_CONSISTENCY\_LEVEL

| sunbird\_cassandra\_consistency\_level | quorum | | SUNBIRD\_CASSANDRA\_IP | sunbird\_cassandra\_host | 11.3.2.4,11.3.2.5,11.3.2.6 | |


| sunbird\_cassandra\_password | password | |


| sunbird\_cassandra\_port | 9042, 9042, 9042 | |


| sunbird\_cassandra\_username | cassandra | |


| sunbird\_default\_channel | ntp | | SUNBIRD\_EMAIL\_MAX\_RECEPIENT\_LIMIT | sunbird\_email\_max\_recipients\_limit | 100 | | ENCRYPTION\_KEY | sunbird\_encryption\_key | eg: key | |


| sunbird\_encryption\_mode | local | | SUNBIRD\_ENV\_LOGO\_URL | sunbird\_env\_logo\_url | | | SUNBIRD\_ES\_IP | sunbird\_es\_host | 11.3.2.811.3.2.8,11.3.2.9,11.3.2.10 | | SUNBIRD\_ES\_PORT | sunbird\_es\_port | 9300,9300,93009300 | | SUNBIRD\_FUZZY\_SEARCH\_THRESHOLD | sunbird\_fuzzy\_search\_threshold | 0.5 | | SUNBIRD\_HEALTH\_CHECK\_ENABLE | sunbird\_health\_check\_enable | false | | SUNBIRD\_INSTALLATION | sunbird\_installation | sunbird\_Stage | | SUNBIRD\_INSTALLATION\_DISPLAY\_NAME | sunbird\_installation\_display\_name\_for\_sms | DIKSHA | |


| sunbird\_installation\_email | | | SUNBIRD\_KEYCLOAK\_LINK\_EXPIRATION\_TIME | sunbird\_keycloak\_required\_action\_link\_expiration\_seconds | 2592000 | | SUNBIRD\_KEYCLOAK\_USER\_FEDERATION\_PROVIDER\_ID | sunbird\_keycloak\_user\_federation\_provider\_id | eg: id | | EMAIL\_SERVER\_FROM | sunbird\_mail\_server\_from\_email | | | EMAIL\_SERVER\_HOST | sunbird\_mail\_server\_host | smtp.sendgrid.net | | EMAIL\_SERVER\_PASSWORD | sunbird\_mail\_server\_password | eg: password | | EMAIL\_SERVER\_PORT | sunbird\_mail\_server\_port | 587 | | EMAIL\_SERVER\_USERNAME | sunbird\_mail\_server\_username | apikey | |


| sunbird\_msg\_91\_auth | eg: key | |


| sunbird\_msg\_sender | DKSAPP | |


| sunbird\_mw\_system\_host | learner-service | |


| sunbird\_mw\_system\_port | 8088 | | SUNBIRD\_OTP\_ALLOWED\_ATTEMPT | sunbird\_otp\_allowed\_attempt | 2 | | SUNBIRD\_OTP\_EXPIRATION | sunbird\_otp\_expiration | 1800 | | SUNBIRD\_OTP\_LENGTH | sunbird\_otp\_length | 6 | |


| sunbird\_pg\_db | quartz | |


| sunbird\_pg\_host | staging-pg11.postgres.database.azure.com | |


| sunbird\_pg\_password | eg: password | |


| sunbird\_pg\_port | 5432 | |


| sunbird\_pg\_user | sunbirdstaging\@staging-pg11 | |


| sunbird\_quartz\_mode | cluster | |


| sunbird\_remote\_bg\_req\_router\_path | akka.tcp\://SunbirdMWSystem\@actor-service:8088/user/BackgroundRequestRouter | |


| sunbird\_remote\_req\_router\_path | akka.tcp\://SunbirdMWSystem\@actor-service:8088/user/RequestRouter | |


| sunbird\_reset\_pass\_msg | You have requested to reset password. Click on the link to set a password: {0} | | SUNBIRD\_SSO\_CLIENT\_ID | sunbird\_sso\_client\_id | lms | | SUNBIRD\_SSO\_CLIENT\_SECRET | sunbird\_sso\_client\_secret | eg: key | | SUNBIRD\_SSO\_LB\_IP | sunbird\_sso\_lb\_ip | | | SUNBIRD\_SSO\_PASSWORD | sunbird\_sso\_password | eg: password | | SSO\_PUBLIC\_KEY | sunbird\_sso\_publickey | eg: key | | SSO\_REALM | sunbird\_sso\_realm | sunbird | | SSO\_URL | sunbird\_sso\_url | | | SSO\_USERNAME | sunbird\_sso\_username | sunbird-staging-new-admin | | SUNBIRD\_SUBDOMAIN\_KEYCLOAK\_BASE\_URL | sunbird\_subdomain\_keycloak\_base\_url | | | | sunbird\_url\_shortner\_access\_token | eg: token | | SUNBIRD\_URL\_SHORTNER\_ENABLE | sunbird\_url\_shortner\_enable | True | | BULK\_UPLOAD\_USER\_DATA\_SIZE | sunbird\_user\_bulk\_upload\_size | 1001 | | SUNBIRD\_USER\_CERT\_KAFKA\_TOPIC | sunbird\_user\_cert\_kafka\_topic | sunbirdstaging.lms.user.account.merge | | SUNBIRD\_WEB\_URL | sunbird\_web\_url | | | PDATA\_ID | telemetry\_pdata\_id | staging.sunbird.learning.service | | PDATA\_PID | telemetry\_pdata\_pid | learner-service | |


| user\_index\_alias | user\_alias | | {% endtab %} | | | {% tab title="Admin Utils System Config" %} | Key | Default Value | | -------------------------------------------- | ------------------------------------------------------------------------ | | ACCESS\_TOKEN\_VALIDITY | 86400 | | AM\_ADMIN\_API\_ACCESS\_BASEPATH | /keys/ | | AM\_ADMIN\_API\_ACCESS\_KEYCOUNT | 10 | | AM\_ADMIN\_API\_ACCESS\_KEYPREFIX | accessv1\_key | | AM\_ADMIN\_API\_ACCESS\_KEYSTART | 1 | | AM\_ADMIN\_API\_DESKTOP\_DEVICE\_BASEPATH | /keys/ | | AM\_ADMIN\_API\_DESKTOP\_DEVICE\_KEYCOUNT | 10 | | AM\_ADMIN\_API\_DESKTOP\_DEVICE\_KEYPREFIX | desktop\_devicev2\_key | | AM\_ADMIN\_API\_DESKTOP\_DEVICE\_KEYSTART | 1 | | AM\_ADMIN\_API\_ENDPOINT | | | AM\_ADMIN\_API\_KEYS | mobile\_device,desktop\_device,portal\_anonymous,portal\_loggedin,access | | AM\_ADMIN\_API\_MOBILE\_DEVICE\_BASEPATH | /keys/ | | AM\_ADMIN\_API\_MOBILE\_DEVICE\_KEYCOUNT | 10 | | AM\_ADMIN\_API\_MOBILE\_DEVICE\_KEYPREFIX | mobile\_devicev2\_key | | AM\_ADMIN\_API\_MOBILE\_DEVICE\_KEYSTART | 11 | | AM\_ADMIN\_API\_PORTAL\_ANONYMOUS\_BASEPATH | /keys/ | | AM\_ADMIN\_API\_PORTAL\_ANONYMOUS\_KEYCOUNT | 10 | | AM\_ADMIN\_API\_PORTAL\_ANONYMOUS\_KEYPREFIX | portal\_anonymous\_key | | AM\_ADMIN\_API\_PORTAL\_ANONYMOUS\_KEYSTART | 1 | | AM\_ADMIN\_API\_PORTAL\_LOGGEDIN\_BASEPATH | /keys/ | | AM\_ADMIN\_API\_PORTAL\_LOGGEDIN\_KEYCOUNT | 10 | | AM\_ADMIN\_API\_PORTAL\_LOGGEDIN\_KEYPREFIX | portal\_loggedin\_key | | AM\_ADMIN\_API\_PORTAL\_LOGGEDIN\_KEYSTART | 1 | | DEFAULT\_CONSUMER\_GROUP | contentUser | | EMBED\_ROLE | true | | ENDPOINTS\_HEALTH\_ID | apihealth | | ENDPOINTS\_HEALTH\_SENSITIVE | false | | ENDPOINTS\_METRICS\_ID | metrics | | ENDPOINTS\_METRICS\_SENSITIVE | false | | JAVA\_OPTS | -Xms256m -Xmx256m | | LEARNER\_API\_AUTH\_KEY | eg: key | | LEARNER\_BASE\_API\_URL | | | REFRESH\_TOKEN\_DOMAIN | | | REFRESH\_TOKEN\_KID | eg: token | | REFRESH\_TOKEN\_LOG\_OLDER\_THAN | 30 | | REFRESH\_TOKEN\_OFFLINE\_VALIDITY | 15552000 | | REFRESH\_TOKEN\_PRELOAD | true | | REFRESH\_TOKEN\_PUBLIC\_BASEPATH | /keys/ | | REFRESH\_TOKEN\_PUBLIC\_KEYPREFIX | refresh\_token\_public\_key | | REFRESH\_TOKEN\_SECRET\_KEY | eg:key | | SERVER\_PORT | 4000 | | SPRING\_PROFILES\_ACTIVE | production | | {% endtab %} | | | {% endtabs %} | | # System Settings This page explains how a System Administrator can configure some settings in the system for different purposes. #### Configurable Parameters
S NO.PARAMETERDESCRIPTIONEXAMPLE
1custodianOrgChannelset default channel into system. The self sign-up or Google sign up user are under this channelsunbird
2custodianRootOrgIdset org id of custodianOrgChannel created earlier
3contentComingSoonMsgmessage for the rootOrgs whose content is being created or they don’t have content yet
4courseFrameworkIdframework ID for course creation, this framework needs to be created firstTPD
5tncConfigterms and condition page
6consumptionFaqspublic page url for consumption FAQ
### Read Values that are Already Set Use the following curl command to check the value that is already set for a particular parameter. Replace the value in the {key} with the required ID. The key refers to the ID to be configured using the cURL command. ``` curl -X GET \ /data/v1/system/settings/get/{key} \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' ``` \ Configure Custodian Channel Configure the Sunbird LMS custodian channel ID using the following cURL command. ``` curl -X POST \ /data/v1/system/settings/set \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'X-Authenticated-User-Token: ' \ -d '{ "request": { "id": "custodianOrgChannel", "field": "custodianOrgChannel", "value": "" } }' ``` ### Configure Custodian Org ID Configure the Sunbird LMS custodian Org ID using the following cURL command. Use the Org ID for the custodian channel set earlier. The custodian channel ID and the custodian Org ID work as a pair. ``` curl -X POST \ /data/v1/system/settings/set \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'X-Authenticated-User-Token: ' \ -d '{ "request": { "id": "custodianRootOrgId", "field": "custodianRootOrgId", "value": "" } }' ``` ### Configure the Content Coming Soon Page Configure the message on the Content Coming Soon page using the following cURL command. This message is used by those root organizations that do not have any content, as on date. ``` curl -X POST \ /data/v1/system/settings/set \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'X-Authenticated-User-Token: ' \ -d '{ "request": { "id": "contentComingSoonMsg", "field": "contentComingSoonMsg", "value": "[{\"rootOrgId\":\"{RootOrgId}\",\"value\":\"Org specific coming soon message\",\"translations\":\"{\\\"en\\\":\\\"Coming soon message\\\"}\"}\"}]" } }' ``` ### Configure Course Framework ID Configure the Sunbird LMS Course Framework ID using the following cURL command. Create the same framework in the Knowledge Platform sub system. ``` curl -X POST \ /data/v1/system/settings/set \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'X-Authenticated-User-Token: ' \ -d '{ "request": { "id": "courseFrameworkId", "field": "courseFrameworkId", "value": "" } }' ``` ### Configure the Terms and Conditions page Configure the Sunbird LMS Terms and Conditions page configuration using the following cURL command. ``` curl -X POST \ /data/v1/system/settings/set \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'X-Authenticated-User-Token: ' \ -d '{ "request": { "id": "tncConfig", "field": "tncConfig", "value": "{"latestVersion":"v1","v1":{"url":"{public url for config html page}"}}" } }' ``` ### Configure FAQs for User Consumption Confifure the Sunbird LMS FAQs for users using the following cURL command. ``` curl -X POST \ /data/v1/system/settings/set \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -H 'X-Authenticated-User-Token: ' \ -d '{ "request": { "id": "consumptionFaqs", "field": "consumptionFaqs", "value": "{consumption faq public html page url}" } }' ```
# Email Configuration In sunbird, SendGrid is used for sending email notifications. Below are the confgurations that need to be updated for sending email. These configurations are common for user-org service, notification service and notification data-pipeline flink job. ``` sunbird_mail_server_host = sunbird_mail_server_port = sunbird_mail_server_username = sunbird_mail_server_password = sunbird_mail_server_from_email = support@open-sunbird.org sendgrid_connection_reset_interval //default is 60000L ``` #### Adding Email Template to Cassandra DB Userorg service stores email templates in the table **email\_template** within the **sunbird** keyspace. The following command allows you to view email templates currently available in Cassandra DB: `SELECT * from sunbird.email_template;` Command to add an email template to Cassandra DB using CQL shell. Ensure that the template name is unique so that it doesnot override the existing template information in Cassandra DB: ``` INSERT INTO sunbird.email_template(name, template) VALUES('myEmailTemplate', '
 
#if ($orgImageUrl)

logo

#end
#if ($name)

Hi $name,

#end

$body

') ``` # SMS Configuration Create an account on [msg91](https://msg91.com/in) and login to get the [authKey](https://msg91.com/help/MSG91/where-can-i-find-my-authentication-key). Register the sender ID. Register the SMS template in the DLT portal and get a template ID. Add the approved template to the msg91 portal along with the template ID. * [Step-by-step process to configure SMS](https://msg91.com/help/MSG91/step-by-step-process-to-configure-sms) * [Where can I find my authentication key?](https://msg91.com/help/MSG91/where-can-i-find-my-authentication-key) * [How to Add, Update, or Delete a sender ID (header) on MSG91?](https://msg91.com/help/MSG91/how-to-add-sender-id-in-msg91) * [How to add or delete an SMS template?](https://msg91.com/help/MSG91/how-to-add-or-delete-an-sms-template) #### Use the below msg91 curl to verify sending the SMS: ``` curl --location --request POST 'https://api.msg91.com/api/v2/sendsms' \ --header 'accept: application/json' \ --header 'authkey: ' \ --header 'content-type: application/json' \ --data-raw '{ "sender": "", "route": "4", "country": "91", "unicode": 1, "sms": [ { "message": "", "to": [ "" ] } ], "DLT_TE_ID": "