📙
Sunbird Lern
AskDot
  • LEARN
    • Overview
    • Functional Capabilities
    • Technical Architecture
      • Technical Architecture Diagram
      • Tech Stack
    • Dependencies
    • Product Roadmap
  • USE
    • Overview
    • Release Notes
      • Release V 8.0.0 (Ongoing)
      • Release V 7.0.0
      • Release V 5.4.0
      • Release V 5.3.0
      • Release V 5.2.0
      • Release V 5.1.0
      • Release V 5.0.1
      • Release V 5.0.0
      • Release V 4.10.0
      • Release V 4.9.0
      • Release V 4.8.0
      • Release V 4.7.0
    • Developer Guide
      • USER & ORG SERVICE
        • Features/Core capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Keycloak Local setup
          • Additional Installation Dependencies:
          • Source Code
          • Installation Configuration
        • Data Models
          • Cassandra
            • Data Dictionary
              • User
              • Organisation
              • System Settings
              • Role
              • Bulk Upload Process
              • Tenant Preference
              • Cassandra Migration Version
              • User Consent
              • Email Template
              • OTP
              • Page Management (LMS Service)
              • Location
              • User Notes
              • Deprecated
          • Elastic Search
          • Redis
        • APIs
          • User Management
            • User Deletion API
            • Ownership Transfer API
          • Organisation Management
          • Location Management
          • Consent Management
          • OTP Services
          • Tenant Configurations
          • Bulk Upload
          • System Settings
          • API Management Service
          • Data Sync
          • Notification APIs
        • Flink Jobs
          • User Cache Updater
          • User Deletion Cleanup Flink Job
          • Ownership Transfer Flink Job
        • Reports
          • Standard Exhaust
            • State Admin Geo report
            • State Admin Report
          • Other Jobs
            • User Cache Indexer Job
            • Delete Users Assets Report
          • Data Products Developer Installation
            • System Requirements
            • Tech Stack
            • Installation Dependencies
            • Local installation of data-products
            • Server setup Guide
            • Installation Configuration
            • Data-product creation guide
            • Troubleshooting a data-product
        • Logs, Telemetry Events
        • Configuration
          • Functional Configurations
          • System Settings
          • Email Configuration
          • SMS Configuration
          • OTP based SMS Configuration
        • Roles
          • System Roles
          • User Roles
          • RBAC
        • Dependencies
        • Keycloak on Sunbird
        • How to Guide
          • Create Organization
          • Create User
          • Add new user type & location
        • Single Sign-on with Sunbird
        • Caching and Denormalising User Metadata
          • ETLUserCacheUpdaterJob
          • UserCacheUpdaterFlinkJob
        • Unit Tests and Code Coverage
        • FAQs
      • LMS(BATCH) SERVICE
        • Features/Core Capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Source Code
        • Data Models
          • Cassandra
            • Data Dictionary
          • ElasticSearch
          • Redis
        • APIs
        • Flink Jobs
          • Merge User Courses
          • Relation Cache Updater
          • Activity Aggregate Updater
          • Assessment Aggregator
          • Enrolment Reconciliation
          • Collection Certificate Pre-Processor
          • Collection Certificate Generator
        • Reports
          • On-Demand Exhaust
            • Response Exhaust
            • User Info Exhaust
            • Progress Exhaust
          • Other Jobs
            • Collection Summary Job
            • Course Batch Status Updater Job
            • Cassandra Migrator Job
            • Score Metric Migration Job
            • Assessment Score Correction Job
            • Collection Reconciliation Job
            • Course Enrollment Job
            • Course Consumption Job
        • Logs, Telemetry Events
        • Configuration
          • System Configuration
          • Functional Configuration
        • Dependencies
        • Certificate Registry and Credentialing using Sunbird-RC
          • Configuring PublicKey in Sunbird-RC
          • Server Setup guide for Sunbird-RC
          • API Guide For Sunbird-RC
        • Certificate Flow
          • Certificates Creation and Configuration
        • FAQs
      • GROUPS
        • Features/Core Capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Source Code
          • Installation Configuration
        • Data Models
          • Cassandra
            • Data Dictionary
          • Redis
        • APIs
          • Create Group
        • Logs, Telemetry Events
          • UI Telemetry Events
          • Service Telemetry Events
        • Configuration
          • Functional Configurations
        • Design References
        • Dependencies
        • Unit Tests and Code Coverage
      • NOTIFICATION SERVICE
        • Features/Core Capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Source Code
          • Installation Configuration
        • Data Models
          • Data Dictionary
        • APIs
        • Flink Jobs
        • Logs, Telemetry Events
        • Configuration
          • Functional Configurations
        • Dependencies
        • Unit Tests and Code Coverage
      • DISCUSSION FORUM
        • Features/Core capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
            • Nodebb setup
            • Discussion Middleware Setup
            • Discussion Middleware
            • Discussion forum integration with any application
            • Discussion-UI setup along with demo application.
            • Sunbird-lern portal for DF
          • Source Code
        • APIs
          • Category APIs
          • User APIs
          • Post APIs
          • Topic APIs
          • Forum APIs
        • Context Schema
        • Configurations
          • Nodebb Admin panel settings
          • Discussion MW & Nodebb System Config
        • Telemetry Events
        • Dependencies
      • ML SERVICE
        • DATA PIPELINE (Flink Jobs)
          • Program User Info
            • Component Diagram
            • Data Model
            • Folder Structure
          • Ml User Delete
            • Component Diagram
            • Data Model
            • Folder Structure
        • DATA PRODUCTS
          • Program Exhaust
            • Component Diagram
            • Folder Structure
    • Server Installation
    • Dependency setup
    • Deprecation
      • Release-5.4.0
    • Jenkins Jobs
    • Release Calendar
    • Learn More
      • Tech References
      • Telemetry Processing
      • Data Dictionary
        • Sample Data
      • Delete User Functionality
      • Asset Ownership Transfer
    • FAQs
  • Engage
    • Discuss
    • Contribute to Sunbird Lern
    • Raise an Issue
    • Contribution Guidelines
Powered by GitBook
On this page
  • User Interaction Diagram
  • CSV File Structure
  • CSV Columns Contents
  • Consent Fields
  • Sample CSV Data

Was this helpful?

Edit on GitHub
  1. USE
  2. Developer Guide
  3. ML SERVICE
  4. DATA PRODUCTS
  5. Program Exhaust

Component Diagram

PreviousProgram ExhaustNextFolder Structure

Last updated 1 year ago

Was this helpful?

Program Exhaust is a service that generates CSV file containing user details for those who has joined a program.

  1. Database Layer:

  • PostgreSQL Database (job_request): This is where job requests are stored. These requests include information about which users have requested reports.

  • Cassandra Database (user_consent): This is where user consent data is stored. It contains information about whether users have agreed to share their PII data.

  • Cassandra Database (program_enrollment): This is where program enrollment data is stored when a user joins new program.

  • Redis Cache (user): Contains cached user data that can be quickly retrieved. Data provider

Database
Table

PostgreSQL

job_request

Redis Cache

user

Cassandra

program_enrolment, user_consent

  1. Data Processing Layer: Apache Spark is used to perform transformations, sort columns, eliminate duplicates, and replace unknown values with null. This process enhances data quality, organizes data logically before storing to CSV.

User Interaction Diagram

This interaction diagram details the complete process of requesting and generating reports. The user can request a user info exhaust report through SunbirdEd from the program dashboard. Using exhaust APIs, this will map the request to SunbirdLern Which internally calls SunbirdObsrv service. userInfoExhaust data-product will be triggered by a scheduled cron task, which will query cassandra and redis to get data and process it using Spark to transform data and generate the report. The user receives the same report once it has been created.

CSV File Structure

Format

Nomenclature

Example

Security Levels

CSV zip (Password protected)

program-user-exhaust/<request_id>_<UpdatedDate>.zip

9CD107F48B5AF0D163F8AE8410829674_20230622.zip

L3 - Data encrypted with a user provided encryption key. Generally applicable to non PII data but can contain sensitive information which may not be considered open

CSV Columns Contents

Column Label

Column Type

Data Type

Description

User UUID

Static

String

The system generated unique user ID

User Name

Static

String

Name of the user(Firstname and Lastname)

Mobile Number

Static

String

User declared unmasked mobile number

Email ID

Static

String

User declared unmasked email ID

Consent Provided

Static

String

Yes/No. Flag to denote whether user has consented to the data sharing.

Consent Provided Date

Static

Date

Date when the user has consented to share the data

Program Name

Static

String

Program title

Program ID

Static

String

Unique Program Identifier.

State

Static

String

User declared state for self signed up users. If the user is a org validated user then the state as passed from org SSO or derived from sub-org ID.

District

Static

String

User declared district for self signed up users. If the user is a org validated user then the district as passed from org SSO or derived from sub-org ID.

Block

Static

String

Block name mapped to the user’s org/sub-org id

Cluster

Static

String

Cluster name mapped to the user’s org/sub-org id

School Id

Static

String

User declared school id for self signed up users. If the user is a org validated user then the school id as passed from org SSO or derived from sub-org ID

School Name

Static

String

School name mapped to the user’s org/sub-org id

User Type

Static

String

Type of the user

User Sub Type

Static

String

Sub Type of the user

Org Name

Static

String

Name of user org - Custodian for self signed up users and respective org name for org validated users

Consent Fields

Following are the fields/columns that will be available in the CSV file only when the user consented for the data sharing.

User Name
Mobile number
Email ID

On user consent

On user consent

On user consent

Sample CSV Data

User UUID
User Name
Mobile number
Email ID
Consent Provided
Consent Provided Date
Program Name
Program ID
State
District
Block
Cluster
School Id
School Name
Usertype
Usersubtype

ef3733df-41e7-4bf1-b67c-091fd0762ee

opa5

7272272777

pao@test.com

false

01/01/23

'Prerak Head Teacher of the Block 19-20'

PGM-Prerak-Head-Teacher-of-the-Block-19-20-Feb2021

Uttar Pradesh

AGRA

ACHHNERA

ZPHS AGALI

9150206302

JHS NAGLA SADLE COMPOSITE

administrator

hm,dikshapreprodcustodian