Load Manager and Data Transformation Manager (DTM) Process

All you need to know about folders in Infromatica
December 22, 2016
Informatica – Difference between Connected Lookup & Unconnected Lookup
December 22, 2016
Show all

Load Manager and Data Transformation Manager (DTM) Process

Load Manager and DTM Process

Load Manager and DTM Process

Whenever the workflow is triggered, the PowerCenter Integration service uses the Load Manager process and the Data Transformation Manager Process (DTM) to run the workflow and carry out below tasks for the workflow.

Tasks performed by Load Manager:

  • Locks the workflow and reads workflow properties.
  • Reads the parameter file and expands workflow variables.
  • Creates the workflow log file.
  • Runs workflow tasks.
  • Distributes sessions to worker servers.
  • Starts the DTM to run sessions.
  • Runs sessions from master servers.
  • Sends post-session email if the DTM terminates abnormally.

After the load manager performs validations for the session, it creates the DTM process. The DTM process is the second process associated with the session run. The primary purpose of the DTM process is to create and manage threads that carry out the session tasks.

When a session is run, the DTM process reads source data and passes it to the transformations thread for processing. When the PowerCenter Server runs a session, the DTM performs the following tasks:

  • Fetches session and mapping metadata from the repository.
  • Creates and expands session variables.
  • Creates the session log file.
  • Validates session code pages if data code page validation is enabled. Checks query conversions if data code page validation is disabled.
  • Verifies connection object permissions.
  • Runs pre-session shell commands.
  • Runs pre-session stored procedures and SQL.
  • Creates and runs mapping, reader, writer, and transformation threads to extract, transform, and load data.
  • Runs post-session stored procedures and SQL.
  • Runs post-session shell commands.
  • Sends post-session email.

The DTM allocates process memory for the session and divide it into buffers. This is also known as buffer memory. It creates the main thread, which is called the master thread. The master thread creates and manages all other threads.

If the session is partitioned, the DTM creates a set of threads for each partition to allow concurrent processing. When Informatica server writes messages to the session log it includes thread type and thread ID. Following are the types of threads that DTM creates:

Master Thread: Main thread of the DTM process. Creates and manages all other threads.

Mapping Thread: One Thread to Each Session. Fetches Session and Mapping Information.

Pre and Post Session Thread: One Thread each to Perform Pre and Post Session Operations.

Reader Thread: One Thread for Each Partition for Each Source Pipeline.

Writer Thread: One Thread for Each Partition if target exist in the source pipeline write to the target.

Transformation Thread: One or More Transformation Thread for Each Partition.

Sid
Sid
Business Intelligence Consultant and Trainer with 13+ years of extensive work experience on various client engagements. Delivered many large data warehousing projects and trained numerous professionals on business intelligence technologies. Extensively worked on all facets of data warehousing including requirement gathering, gap analysis, database design, data integration, data modeling, enterprise reporting, data analytics, data quality, data visualization, OLAP. Has worked on broad range of business verticals and hold exceptional expertise on various ETL tools like Informatica Powercenter, SSIS, ODI and IDQ, Data Virtualization, DVO, MDM.

Leave a Reply

Your email address will not be published. Required fields are marked *

Share
+1
Tweet
Pin
Share