Folder Structure

Program User Info Service folder structure is designed to organize the different modules and files that constitute the program user info flink job. It follows a modular approach, facilitating easy management and development of the service.

The structure is as follows

.  
│
└── ml-jobs
    ├── pom.xml
    └── program-user-info
        ├── pom.xml
        └── src
            ├── main
            │   ├── resources
            │   │   └── program-user-info.conf
            │   └── scala
            │       └── org
            │           └── sunbird
            │               └── dp
            │                   └── userinfo
            │                       ├── domain
            │                       │   └── Event.scala
            │                       ├── functions
            │                       │   └── ProgramUserInfoFunction.scala
            │                       ├── task
            │                       │   ├── ProgramUserInfoConfig.scala
            │                       │   └── ProgramUserInfoStreamTask.scala
            │                       └── util
            │                           └── Commons.scala
            └── test
                ├── resources
                │   ├── test.conf
                │   └── test.cql
                └── scala
                    └── org
                        └── sunbird
                            └── dp
                                ├── fixture
                                │   └── EventFixture.scala
                                └── spec
                                    └── ProgramUserInfoTaskTestSpec.scala
       
    

ml-jobs

This is the main directory that contains one Flink job called program-user-info.

program-user-info

This folder contains all the required directory for actual implementation of logics.

resources [main]

This folder contains application related configurations in a file called program-user-info.conf

domain

This directory contains a file called Even.scala which is used to declare methods to get specific keys out of JSON.

functions

This folder contains the file ProgramUserInfoFunction.scala which helps in processing and flattening the JSON into one-level key-value pairs. This file also contains the logic to store the flattened data into Cassandra database.

task

This folder contains a file called ProgramUserInfoStreamTask.scala which is an entry point for this service. and another file called ProgramUserInfoConfig.scala which holds the variable constants.

resources [test]

This folder contains test related configurations values in a file called test.conf. Also contains test.cql file which holds the Cassandra query to create keyspace for the testing purpose.

fixture

This directory contains a hard coded kafka events for the testing purpose.

spec

In the spec folder we have the main test case file called ProgramUserInfoTaskTestSpec.scala that is used to test our stream processing task.

Source code

data-pipeline/ml-jobs/program-user-info at release-5.3.0 · Sunbird-Lern/data-pipelineGitHub