Nobill_framework
Overview:
We have 3 main source system:
1)Nobill-*In this data is in form of file .Its is our local server
([Link]) and for nobill source for [Link] the data here is
read from S3 and then sent to landing area where it performs
task and then stored back in [Link] perform Dynamic jobs in
Nobill Source system .
*File structure of Nobill:It has 6 header ,10lines of data and
the footer as record count or row [Link] footer record_count
is 11 then it means that there are 11 records present.
*Here in Nobill file the header’s are the file name it self like
example file name is
Domain.name_Table.version_period.start_period.end_Sequen
[Link] and these are considered as the 6 headers.
Files like transaction history has 6 header and 1 footer ,files
like snapshot has 5 header and 1 footer
Note: *The files in Nobill are in form of header and footer.
*In MFS files are in column schema.
*Previously ftp used to act as a mediator but now the
data is fetched directly from S3 to the nobill .
2)Database(it can be different database like mariadb,mssql
etc).
3)Google Big Query-Applications like UAT,frindi,SA
Nobill_framework
We have 2 zones Raw and transient [Link], from the
source the data is gone to raw zone directly as it [Link] the
transformations is made in transient zone .
From transient Zone its transferred to cleansed zone in
columnar form where we apply schema compliance check,
data quality rules, structure of file,merging of files like take
10 files as 1 file . And then the data is transfered to curated
zone where we apply data modelling steps but its sometimes
difficult to perform data modelling in talend so we can can use
spyspark/pycharm,and then the final data is sent to consumer
zone where we apply data Visualization [Link]
curated to consumer it is 1-1 [Link] transient zone to
cleanse ,cleanse to curated and curated to consumer zone its
1-1 mapping.
Nobill source system has 18 records like
event_record,sms_records [Link] we consider the object
name as the file name itself.
Master_info_list ->Has Columns like id,
object_name,source_system,active_flag as Yes/No,
source_path etc, this source path contains the path to process
the job to go in the curated zone.
In this file directly the source path is not given ,1st connect to
database and run the SQL query to run that in context then it
is loaded in context,then that context is given as a path.
Vm_job_details-Whenever we set the job to run
automatically and it fails to run then that details is stored in
job_details.It has the records like weather job is successfully
running,job updates,rejected records etc.
Nobill_framework
Stage_Id-It consists of the zone(raw/transient) and the load
type(incremental/full_load) like example if stage_id=12 it
means its raw zone an its full load
Raw cleansed curated consumer
11 21 31 41
12 22 32 42
Context_parameter-We sometimes set the contect variables
in the context table when we need to use it multiple times in
the job and sometimes define variables directly in the job
when its useful only to that particular [Link] can also take
java_component to change context value and then load
context -here we need to hardcore the values.
Ftp_count-This table is only for the files and not for the
[Link] names has
object_name,period_start,period_end.We do the source_count
on record count because if the count of files in database and
source_count of ftp_count is same the its taken as
[Link] also has details of rejected records.
JOB -tjob_vm_ksa_nobill_call_record 0.1
1)In this Job we perform 1)Downloading files from S3 to
landing area in Node1
2)Checking if the file is duplicate
3)If file is duplicate then move the file in duplicate folder and
then delete file from landing and S3 path.
4)If the file is new then check for schema and file structure
validation.
Nobill_framework
1ST operation is read and load the context from context table to
the job
Main_audit_vertica-It has the database connection which we
have put in [Link] we have the file which has the
database_login,password,etc
So we made the file with db connection details and put in
local so that talend understands which db to [Link] do
context load with key-value pair which has the password as
encrypted.
We build the joblet so that when we want to use the particular
job multiple times so set it into joblet.
We decrypt the data so that the database reads the password so
that it connects to the database.
Vm_context_parameter-It loads as file with context value
which are neded to run a particular job.
Then we load the context_value from the master_info_list
table because the context values are present in the
master_info_list table.
Then we perform downloading files from S3 to landing
file directory-S3 folder has files so it will check and give
empty space.
Schema_compilence_check-It checks for column datatype
and size .
Read_file_content-To check weather all rows and columns
are present ,if not then it will throw an error.
Nobill_framework
Next,we are performing 1)validating header and footer of file
and loading its value
If header and footer is incorrect then auditing error and end
the job
Suppose if the file has 6 header data and footer and if one also
header has null value then it wont process further .Also for 6
header 10 lines of data and 1 footer which is 17 lines of data
the to consider the header we have set global integer as 17-1
where the last line record that is footer as record count is
considered.
Batch_process-To back track the [Link] set the batch_date
to the records in source so that when we open the file in
batch_process it will give us the exact location from where the
record is coming [Link] runs 10 files at ones and not more
than 10 files.
Next we performed 1)loading files from landing area->raw
and transient in S3.
2)performing auditing
3)deleting the required files.
We use the source_file_size(tSystem2)-It’s a command
prompt which checks for the size .This runs on the linux
server.
Command given to check the size of the file is “du-
h”+context.file_name.
Nobill_framework
On local server with the help of command prompt that
directory opens where there is the files and command prompt
opens and automatically the command triggers.
Tjob_vm_start_audit:
Job_details entry moved to history as status running of job
for further reference and deleted from the job detail table.
The job is upated with the unique_id as constraint like
unique_id=”context.stage_id+context.source_system+context.
object_name”.