FAQ:SSIS:Service Speed

From Melissa Data Wiki
Jump to navigation Jump to search

← SSIS:Data Quality Components

Melissa’s SSIS Enterprise level components can process hundreds of thousands of records per hour. SSIS is available in 2 processing modes, Cloud and On Premise, which will affect throughput speeds.

Major Factors

Major factors impacting SSIS processing speed include:

  • Number of threads or processes
    • SSIS Cloud Processing components can handle multiple requests, increasing throughput speeds.
    • We recommend up to 10 threads for our web services; if you require significantly more speed, contact your sales representative to discuss available options.
    • We recommend creating multiple instances of our on-premise components to increase speed, utilizing a Balanced Data Distributor component to send data to each instance.
  • Type of input method provided
    • Using Melissa Address Key (MAK) as input will process faster since MAK is a globally unique identifier
    • Using parsed data elements (such as AddressLine1, City, and State) instead of unstructured data such as Free Form inputs will result in faster processing speeds.
  • Quality of input data
    • Data of poor quality that requires additional input, corrections and standardization will reduce processing speed
  • Number of output columns
    • Requesting additional output columns will reduce processing speeds as more data is requested from our services
  • Premium Mode Processing
    • Phone and Email services offer Premium mode processing, which will reduce processing speeds due to performing live requests that provide higher accuracy data
  • Type of SSIS input source utilized
    • SSIS can use a variety of input sources available via SSIS Toolbox. We recommend OLE DB or Flat file source for faster processing
  • Data Flow Buffer Size - DefaultBufferMaxRows and DefaultBufferSize Properties
    • The buffer size controls the amount of rows transferred by the data flow. Specifications for buffer size vary due to environment and input source speed. Typically, a slow source input speed will require a smaller buffer size
  • Adding Options and Actions
    • Including additional options or actions from our services will reduce processing speeds as additional data is requested or additional logic is utilized
  • Utilizing SQL Server Agent/Command Prompt to run projects
    • Processing speed can increase by utilizing SQL Server Agent or by running the project via Command Prompt. Process quicker without the overhead of Visual Studio

On Premise Components

Speeds will vary due to hardware differences; higher speeds may be achieved with multiple instances of on-premise components.

SSIS Component Throughput (records / hour)
Contact Verify – Name, Address, Phone, Geo, Email (Express) 500,000 – 600,000
Generalized Cleanser 9,000,000 – 12,000,000
MatchUp (Basic Match Code) 3,000,000 – 5,000,000
Profiler 800,000 – 1,250,000

Cloud Processing Components

SSIS Component Throughput (records / hour)
Contact Verify – Name, Address, Phone, Geo, Email (Express) 90,000 – 160,000
Contact Verify – Name, Address, Phone, Geo, Email (Premium) 50,000 – 70,000
Personator (Check, Verify) 200,000 – 220,000
Personator (Check, Verify, Move) 170,000 – 190,000
Personator (Check, Verify, Move, Append – Always) 140,000 – 160,000
SmartMover NCOA 1,500,000 – 2,000,000
SmartMover CCOA 450,000 – 490,000
BusinessCoder 220,000 – 270,000
Property 200,000 – 260,000
Global Verify – Name, Address, Phone (Express), Email (Express) 110,000 – 150,000
Global Verify – Name, Address, Phone (Premium), Email (Premium) 60,000 – 85,000
IPLocator 5,000,000 – 6,000,000
Personator World – Check – Name, Address, Phone (Express), Email (Express) 80,000 – 100,000