📙
Sunbird Lern
AskDot
  • LEARN
    • Overview
    • Functional Capabilities
    • Technical Architecture
      • Technical Architecture Diagram
      • Tech Stack
    • Dependencies
    • Product Roadmap
  • USE
    • Overview
    • Release Notes
      • Release V 8.0.0 (Ongoing)
      • Release V 7.0.0
      • Release V 5.4.0
      • Release V 5.3.0
      • Release V 5.2.0
      • Release V 5.1.0
      • Release V 5.0.1
      • Release V 5.0.0
      • Release V 4.10.0
      • Release V 4.9.0
      • Release V 4.8.0
      • Release V 4.7.0
    • Developer Guide
      • USER & ORG SERVICE
        • Features/Core capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Keycloak Local setup
          • Additional Installation Dependencies:
          • Source Code
          • Installation Configuration
        • Data Models
          • Cassandra
            • Data Dictionary
              • User
              • Organisation
              • System Settings
              • Role
              • Bulk Upload Process
              • Tenant Preference
              • Cassandra Migration Version
              • User Consent
              • Email Template
              • OTP
              • Page Management (LMS Service)
              • Location
              • User Notes
              • Deprecated
          • Elastic Search
          • Redis
        • APIs
          • User Management
            • User Deletion API
            • Ownership Transfer API
          • Organisation Management
          • Location Management
          • Consent Management
          • OTP Services
          • Tenant Configurations
          • Bulk Upload
          • System Settings
          • API Management Service
          • Data Sync
          • Notification APIs
        • Flink Jobs
          • User Cache Updater
          • User Deletion Cleanup Flink Job
          • Ownership Transfer Flink Job
        • Reports
          • Standard Exhaust
            • State Admin Geo report
            • State Admin Report
          • Other Jobs
            • User Cache Indexer Job
            • Delete Users Assets Report
          • Data Products Developer Installation
            • System Requirements
            • Tech Stack
            • Installation Dependencies
            • Local installation of data-products
            • Server setup Guide
            • Installation Configuration
            • Data-product creation guide
            • Troubleshooting a data-product
        • Logs, Telemetry Events
        • Configuration
          • Functional Configurations
          • System Settings
          • Email Configuration
          • SMS Configuration
          • OTP based SMS Configuration
        • Roles
          • System Roles
          • User Roles
          • RBAC
        • Dependencies
        • Keycloak on Sunbird
        • How to Guide
          • Create Organization
          • Create User
          • Add new user type & location
        • Single Sign-on with Sunbird
        • Caching and Denormalising User Metadata
          • ETLUserCacheUpdaterJob
          • UserCacheUpdaterFlinkJob
        • Unit Tests and Code Coverage
        • FAQs
      • LMS(BATCH) SERVICE
        • Features/Core Capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Source Code
        • Data Models
          • Cassandra
            • Data Dictionary
          • ElasticSearch
          • Redis
        • APIs
        • Flink Jobs
          • Merge User Courses
          • Relation Cache Updater
          • Activity Aggregate Updater
          • Assessment Aggregator
          • Enrolment Reconciliation
          • Collection Certificate Pre-Processor
          • Collection Certificate Generator
        • Reports
          • On-Demand Exhaust
            • Response Exhaust
            • User Info Exhaust
            • Progress Exhaust
          • Other Jobs
            • Collection Summary Job
            • Course Batch Status Updater Job
            • Cassandra Migrator Job
            • Score Metric Migration Job
            • Assessment Score Correction Job
            • Collection Reconciliation Job
            • Course Enrollment Job
            • Course Consumption Job
        • Logs, Telemetry Events
        • Configuration
          • System Configuration
          • Functional Configuration
        • Dependencies
        • Certificate Registry and Credentialing using Sunbird-RC
          • Configuring PublicKey in Sunbird-RC
          • Server Setup guide for Sunbird-RC
          • API Guide For Sunbird-RC
        • Certificate Flow
          • Certificates Creation and Configuration
        • FAQs
      • GROUPS
        • Features/Core Capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Source Code
          • Installation Configuration
        • Data Models
          • Cassandra
            • Data Dictionary
          • Redis
        • APIs
          • Create Group
        • Logs, Telemetry Events
          • UI Telemetry Events
          • Service Telemetry Events
        • Configuration
          • Functional Configurations
        • Design References
        • Dependencies
        • Unit Tests and Code Coverage
      • NOTIFICATION SERVICE
        • Features/Core Capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
          • Source Code
          • Installation Configuration
        • Data Models
          • Data Dictionary
        • APIs
        • Flink Jobs
        • Logs, Telemetry Events
        • Configuration
          • Functional Configurations
        • Dependencies
        • Unit Tests and Code Coverage
      • DISCUSSION FORUM
        • Features/Core capabilities
        • Architecture
          • Code Flow
        • Developer Installation
          • System Requirements
          • Tech Stack
          • Installation Guide
            • Nodebb setup
            • Discussion Middleware Setup
            • Discussion Middleware
            • Discussion forum integration with any application
            • Discussion-UI setup along with demo application.
            • Sunbird-lern portal for DF
          • Source Code
        • APIs
          • Category APIs
          • User APIs
          • Post APIs
          • Topic APIs
          • Forum APIs
        • Context Schema
        • Configurations
          • Nodebb Admin panel settings
          • Discussion MW & Nodebb System Config
        • Telemetry Events
        • Dependencies
      • ML SERVICE
        • DATA PIPELINE (Flink Jobs)
          • Program User Info
            • Component Diagram
            • Data Model
            • Folder Structure
          • Ml User Delete
            • Component Diagram
            • Data Model
            • Folder Structure
        • DATA PRODUCTS
          • Program Exhaust
            • Component Diagram
            • Folder Structure
    • Server Installation
    • Dependency setup
    • Deprecation
      • Release-5.4.0
    • Jenkins Jobs
    • Release Calendar
    • Learn More
      • Tech References
      • Telemetry Processing
      • Data Dictionary
        • Sample Data
      • Delete User Functionality
      • Asset Ownership Transfer
    • FAQs
  • Engage
    • Discuss
    • Contribute to Sunbird Lern
    • Raise an Issue
    • Contribution Guidelines
Powered by GitBook
On this page

Was this helpful?

Edit on GitHub

Last updated 1 year ago

Was this helpful?

The three exhaust reports depend on the user metadata information which is generated from the user-cache-updater flink job by fetching information from different core Cassandra tables and are stored in the Redis cache. From Release-3.7.0 the design for the flink job has been changed and now the information will be fetched from the User Read API.

JIRA Link:

Reference Wiki Links:

  1. User Table Changes:

  2. Org Table Changes:

Implementation Design:

User Read API Request curl --location --request GET 'http://{Private LB}/learner/private/user/v1/read/:uuid?fields=locations' User Read API Response

Queries:

  1. Should we fetch extra information eg webpage, createdBy, createdFor, etc? Fetch only if required for denorm job

  2. Should we fetch externalid from now or not? Fetch only if required for denorm job

  3. Should we directly fetch username or have a logic of firstname+lastname? firstname+lastname should be the username for reports

Design for the fields to be fetched:

Field Name
Type
Logic to fetch
Description

User-ID

String

userId

It indicates user unique Identifier

Mobile Number

String

encPhone

User phone number in an encrypted format

Email ID

String

encEmail

User mail id in an encrypted format

First Name

String

firstName

User first name

Last Name

String

lastName

User Last Name

Rootorgid

String

rootOrgId

User root org id (can be used to differentiate between custodian and state user)

Board

String

framework.board

User’s board Assumption: It is single valued

Medium

List[String]

framework.medium

User medium

Subject

List[String]

framework.subject

User subjects

Language

List[String]

language

User Language

Grade

List[String]

framework.gradeLevel

User grades

framework

String

framework.id

User’s framework id Assumption: It is single valued

UserType

String

profileUserType.type

User Type

UserSubType

String

profileUserType.subType

User’s Sub Type

Orgname

String

rootOrg.orgName

User’s Org Name It is fetched from rootOrg Map.

School Name

String

Fetch {name} where userLocation[*].type = 'school'

User’s School Name. It is fetched from the userLocation Map

School UDISE Code

String

Fetch {code} where userLocation[*].type = 'school'

User’s School UDISE Code. It is fetched from the userLocation Map

State Name

String

Fetch {name} where userLocation[*].type = 'state'

User’s State Name. It is fetched from the userLocation Map

District Name

String

Fetch {name} where userLocation[*].type = 'district'

User’s District Name. It is fetched from the userLocation Map

Block Name

String

Fetch {name} where userLocation[*].type = 'block'

User’s Block Name. It is fetched from the userLocation Map

Cluster Name

String

Fetch {name} where userLocation[*].type = 'cluster'

User’s Cluster Name. It is fetched from the userLocation Map

{
  "id": ".private.user.v1.read.123456",
  "ver": "private",
  "ts": "2021-03-09 11:33:42:061+0530",
  "params": {
    "resmsgid": null,
    "msgid": "06ff91d6-043e-4714-b002-6f8b90a04723",
    "err": null,
    "status": "success",
    "errmsg": null
  },
  "responseCode": "OK",
  "result": {
    "response": {
      "webPages": [],
      "maskedPhone": null,
      "subject": [],
      "channel": "root2",
      "language": [],
      "updatedDate": null,
      "password": null,
      "managedBy": null,
      "flagsValue": 2,
      "id": "a962a4ff-b5b5-46ad-a9fa-f54edf1bcccb",
      "recoveryEmail": "",
      "identifier": "a962a4ff-b5b5-46ad-a9fa-f54edf1bcccb",
      "thumbnail": null,
      "updatedBy": null,
      "accesscode": null,
      "locationIds": [
        "8db3345c-1bfc-4276-aef1-7ea0f3183211",
        "d087424e-18cb-49b0-865c-98f265c73ed3",
        "13087424e-18cb-49b0-865c-98f265c73ed3",
        "43087424e-18cb-49b0-865c-98f265c73ed3"
      ],
      "externalIds": [
        {
          "idType": "root2",
          "provider": "root2",
          "id": "123456"
        }
      ],
      "registryId": null,
      "rootOrgId": "0127738024883077121",
      "prevUsedEmail": "",
      "firstName": "Creator",
      "tncAcceptedOn": null,
      "allTncAccepted": {},
      "phone": "",
      "dob": null,
      "grade": [],
      "currentLoginTime": null,
      "userType": null,
      "status": 1,
      "lastName": "test",
      "tncLatestVersion": "v1",
      "gender": null,
      "roles": [
        "PUBLIC"
      ],
      "prevUsedPhone": "",
      "stateValidated": false,
      "isDeleted": false,
      "organisations": [
        {
          "organisationId": "0127738024883077121",
          "updatedBy": null,
          "addedByName": null,
          "addedBy": null,
          "roles": [
            "PUBLIC"
          ],
          "approvedBy": null,
          "updatedDate": null,
          "userId": "a962a4ff-b5b5-46ad-a9fa-f54edf1bcccb",
          "approvaldate": null,
          "isDeleted": false,
          "hashTagId": "0127738024883077121",
          "isRejected": null,
          "id": "0132322953313320960",
          "position": null,
          "isApproved": null,
          "orgjoindate": "2021-03-09 11:31:31:930+0530",
          "orgLeftDate": null
        }
      ],
      "profileUserType": {
        "usertype": "administrator",
        "usersubtype": "deo"
      },
      "userLocations": [
        {
          "code": "21",
          "name": "Odisha",
          "id": "8db3345c-1bfc-4276-aef1-7ea0f3183211",
          "type": "state",
          "parentId": null
        },
        {
          "code": "2112",
          "name": "CUTTACK",
          "id": "d087424e-18cb-49b0-865c-98f265c73ed3",
          "type": "district",
          "parentId": "8db3345c-1bfc-4276-aef1-7ea0f3183211"
        },
        {
          "code": "211",
          "name": "BLOCK1",
          "id": "13087424e-18cb-49b0-865c-98f265c73ed3",
          "type": "block",
          "parentId": "8db3345c-1bfc-4276-aef1-7ea0f3183211"
        },
        {
          "code": "211",
          "name": "CLUSTER1",
          "id": "43087424e-18cb-49b0-865c-98f265c73ed3",
          "type": "cluster",
          "parentId": "8db3345c-1bfc-4276-aef1-7ea0f3183211"
        },
        {
          "name": "DPS, MATHURA",
          "id": "63087424e-18cb-49b0-865c-98f265c73ed3",
          "type": "school",
          "externalid": "3183211"
        }
      ],
      "countryCode": null,
      "tncLatestVersionUrl": "https://dev-sunbird-temp.azureedge.net/portal/terms-and-conditions-v1.html",
      "maskedEmail": "am******@yopmail.com",
      "tempPassword": null,
      "email": "am******@yopmail.com",
      "rootOrg": {
        "dateTime": null,
        "preferredLanguage": "English",
        "keys": {},
        "approvedBy": null,
        "channel": "root2",
        "description": "Root Org2",
        "updatedDate": null,
        "addressId": null,
        "orgType": null,
        "provider": null,
        "locationId": null,
        "orgCode": null,
        "theme": null,
        "id": "0127738024883077121",
        "communityId": null,
        "isApproved": null,
        "email": null,
        "slug": "root2",
        "isSSOEnabled": null,
        "thumbnail": null,
        "orgName": "Root Org2",
        "updatedBy": null,
        "locationIds": [],
        "externalId": null,
        "isRootOrg": true,
        "rootOrgId": "0127738024883077121",
        "approvedDate": null,
        "imgUrl": null,
        "homeUrl": null,
        "orgTypeId": null,
        "isDefault": null,
        "createdDate": "2019-05-31 16:41:29:485+0530",
        "createdBy": null,
        "parentOrgId": null,
        "hashTagId": "0127738024883077121",
        "noOfMembers": null,
        "status": 1
      },
      "phoneVerified": false,
      "profileSummary": null,
      "recoveryPhone": "",
      "avatar": null,
      "userName": "creatortest_72yz",
      "userId": "a962a4ff-b5b5-46ad-a9fa-f54edf1bcccb",
      "userSubType": null,
      "promptTnC": true,
      "emailVerified": true,
      "lastLoginTime": null,
      "createdDate": "2021-03-09 11:31:23:189+0530",
      "framework": {
        "board": [
          "IGOT-Health"
        ],
        "gradeLevel": [
          "Volunteers"
        ],
        "id": [
          "igot_health"
        ],
        "medium": [
          "English"
        ],
        "subject": [
          "IRCS"
        ]
      },
      "createdBy": null,
      "location": null,
      "tncAcceptedVersion": null
    }
  }
}
  1. USE
  2. Developer Guide
  3. USER & ORG SERVICE
  4. Caching and Denormalising User Metadata

UserCacheUpdaterFlinkJob

PreviousETLUserCacheUpdaterJobNextUnit Tests and Code Coverage
https://project-sunbird.atlassian.net/browse/SB-21691
https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/2110881881
https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/2260074547
usercacheupdaterflinkdesign