Delete User Functionality

Overview

The user deletion requirement in Lern has been originated from the below requirement.

PRD: [PRD] Delete Account functionality

BE Design Lern - [Design] Delete Account Functionality

FE Design Lern - [Design] [Front-end]Delete User Functionality

What is changing?

The user can request for deletion of their account in Sunbird, this means two primary actions to happen.

1. The user's Personal Identifiable Information (PII) needs to be removed

2. The assets created by this user (such as questions, question sets, content, etc.) need to be transferred to an identified user.

Changes on Lern:

1. A user deletion API which produces a kafka event on <env>.delete.user topic.

2. For more details on the user deletion cleanup flink job, please visit

Release Tags:

Components	Build Jenkins Job	Build Tag	Deploy Jenkins Job	Deploy Tag	Comment
OnboardAPIs	NA	NA	Deploy/Kubernetes/OnboardAPIs	release-7.0.0	To onboard the delete user API
Cassandra Migration	Build/Core/Cassandra	release-7.0.0_RC3	Deploy/Kubernetes/Cassandra	release-7.0.0	Select the sunbird in cassandra_keyspace_to_migrate while deploying script_repo_branch_or_tag: release-7.0.0_RC3
UserOrg Service	Build/Core/UserOrg	release-7.0.0_RC5	Deploy/Kubernetes/UserOrg	release-7.0.0
Kafka Setup	NA	NA	Deploy/Lern/KafkaSetup	release-7.0.0_RC6
DataPipeline	Build/Lern/LernFlinkJobs	release-7.0.0_RC6	Deploy/Lern/LernFlinkJobs	release-7.0.0_RC6	Define the core_vault_sunbird_keycloak_user_federation_provider_id in Lern inventory secret. Add user-deletion-cleanup and ml-user-delete into job list and deploy it.
Data Product	Build/Lern/LernDataProducts	release-7.0.0_RC5	Deploy/Lern/LernDataProducts	release-7.0.0_RC5

Adoption of the feature Through Flink Job

User delete Flink job

This flink job was introduced as part of the 7.0.0 release, if any adopter wants to use this feature before the release have to configure and use this flink job.

1. Added below partition related settings and replication_factor related settings in ansible/roles/setup-lern-kafka/defaults/main.yml file of data pipeline repository.

   - name: delete.user
    retention_time: 172800000
    replication_factor: 1
    
   - name: delete.user
    num_of_partitions: 2
    replication_factor: 1

2. Added below user delete flinkjob related configuration in kubernetes/helm_charts/datapipeline_jobs/values.j2 file of data pipeline repository.

user-deletion-cleanup:
  user-deletion-cleanup: |+
    include file("/data/flink/conf/base-config.conf")
    kafka {
      input.topic = ${job.env}".delete.user"
      groupId = ${job.env}"-delete-user-group"
    }
    task {
      user.deletion.cleanup.parallelism = {{ user_deletion_cleanup_job_parallelism }}
    }

    service {
        lms {
            basePath = "{{ lms_service_base_url }}"
        }
        userorg {
            basePath = "{{ userorg_service_base_url }}"
        }
    }

    sunbird_keycloak_user_federation_provider_id={{core_vault_sunbird_keycloak_user_federation_provider_id}}
    user_read_api = "/user/v5/read/"
    batch_search_api = "/course/v1/batch/list"

    user {
        keyspace = "sunbird"
        lookup.table = "user_lookup"
        table = "user"
        externalIdentity.table = "usr_external_identity"
        org.table = "user_organisation"
    }
    

  flink-conf: |+
    jobmanager.memory.flink.size: {{ flink_job_names['user-deletion-cleanup'].jobmanager_memory }}
    taskmanager.memory.flink.size: {{ flink_job_names['user-deletion-cleanup'].taskmanager_memory }}
    taskmanager.numberOfTaskSlots: {{ flink_job_names['user-deletion-cleanup'].taskslots }}
    parallelism.default: 1
    jobmanager.execution.failover-strategy: region
    taskmanager.memory.network.fraction: 0.1

3. Added below user delete flinkjob related configuration in kubernetes/ansible/roles/flink-jobs-deploy/defaults/main.yml of data pipeline repository

user_deletion_cleanup_job_consumer_parallelism: 1
user_deletion_cleanup_job_parallelism: 1
  
  user-deletion-cleanup:
    job_class_name: 'org.sunbird.job.deletioncleanup.task.UserDeletionCleanupStreamTask'
    replica: 1
    jobmanager_memory: 1024m
    taskmanager_memory: 1024m
    taskslots: 1
    cpu_requests: 0.3

Ownership transfer delete user assets report:

This report was introduced as part of the 8.0.0 release, if any adopter wants to use this feature before the release have to use the above flink job

1. Added below ownership transfer delete user assets report related configuration in ansible/roles/lern-data-products-deploy/templates/lern-model-config.j2 of data products repository.

if [ ! -z "$3" ]; then specificUserId=$3; configuredUserId=$3; fi #used as specfic user id in usercacheindxerjob,used as configured user id in deleteusersassetsreportjob
if [ ! -z "$5" ]; then refreshUserData=$5; configuredChannel=$5;fi #used as refreshUserData flag in usercacheindxerjob,used as configured channel flag in deleteusersassetsreportjob


"delete-users-assets-report-job")
echo '{"search":{"type":"none"},"model":"org.sunbird.userorg.job.report.DeletedUsersAssetsReportJob","modelParams":{"store":"{{dp_object_store_type}}","storageKeyConfig":"storage.key.config","storageSecretConfig":"storage.secret.config","storageContainer":"{{reports_container}}","storageEndpoint":"{{dp_storage_endpoint_config}}","configuredUserId":'$configuredUserId',"configuredOrganisationId":[],"configuredChannel":'$configuredChannel', "sparkCassandraConnectionHost":"{{ core_cassandra_host }}"}}'
;;

2. Added below ownership transfer delete user assets report related configuration in ansible/roles/lern-data-products-deploy/templates/lern-run-job.j2 of data products repository

"delete-users-assets-report-job") echo 'org.sunbird.userorg.job.report.DeletedUsersAssetsReportJob'
;;

3. Added below ownership transfer delete user assets report related configuration in lern-data-products/src/main/resources/application.conf of data products repository.

content.search.url="http://10.246.3.3/search/v3/search" 
course.batch.search.url="http://10.246.3.3/lms/v1/course/batch/search"
sunbird_instance_name="Sunbird" 
delete.user.cloud.objectKey="reports/"

To learn more about Delete user Assets report visit here Jenkins Job Details For the deployment of the above flink job:

delete-users-assets-report-job

Data products jenkins build job name: /Build/job/Lern/job/LernDataProducts/

Data products jenkins deploy job name: /Deploy/job/<environment>/job/Lern/job/LernDataProducts/LernAnalyticsReplayJobs/delete-users-assets-report-job

Adoption of the feature Through API

This API was introduced as part of the 7.0.0 release and can be used directly from this release. Before to this release adopters have to use either the flink job or extend this logic

Delete user API

1. Added the below configuration in the user org service application.conf file of user org service

      "/user_deletion_background_job_actor"
      {
        router = smallest-mailbox-pool
        nr-of-instances = 5
        dispatcher = brr-usr-dispatcher
      }
    "/user_deletion_background_job_actor/*"
       {
         dispatcher = akka.actor.brr-usr-dispatcher
       }

2. Added the below property in external resource.properties related to user delete kafka topic in user org service

user-deletion-roles=public
user-deletion-broadcast-topic={{env_name}}.delete.user

3. Added the below configuration in ansible/roles/kong-api/defaults/main.yml for owuser delete API in sunbird devops repository.

- name: deleteUser
  uris: "{{ user_service_prefix }}/v1/delete"
  upstream_url: "{{ userorg_service_url }}/v1/user/delete"
  strip_uri: true
  plugins:
  - name: jwt
  - name: cors
  - "{{ statsd_pulgin }}"
  - name: acl
    config.whitelist:
    - userUpdate
  - name: rate-limiting
    config.policy: local
    config.hour: "{{ medium_rate_limit_per_hour }}"
    config.limit_by: credential
  - name: request-size-limiting
    config.allowed_payload_size: "{{ medium_request_size_limit }}"
  - name: opa-checks
    config.required: true
    config.enabled: true

4. Added the user delete topic to the userorgservice.env file in Sunbird DevOps repository.

user-deletion-roles=public
user-deletion-broadcast-topic={{env_name}}.delete.user