Showing posts with label SSIS. Show all posts
Showing posts with label SSIS. Show all posts

Wednesday, 13 May 2015

Find SSIS package details scheduled in jobs - SQL Server by serverku

I would like to share the script to find out the scheduled jobs which call SSIS packages in SQL Server.
USE MSDB
GO
SELECT
sj.job_id as JobId,
sj.name as JobName,
sjs.step_name as StepName,
sjs.Command as Command
FROM sysjobs sj
INNER JOIN sysjobsteps sjs
ON(sj.job_id = sjs.job_id)
WHERE sjs.subsystem = 'SSIS'
GO
The script return job name and the SSIS package full path as a command like "/FILE " here SSIS package full path" /CHECKPOINTING OFF /REPORTING E"

Tuesday, 12 May 2015

Download all files through FTP in SQL Server by serverku

We have completed the reading of earlier methods to receive single file and receive file dynamically through the FTP in SSIS. Hope you like those posts. Previously we have downloaded a file which has specific extension with format of current date. The same thing and criteria will be repeated again, but one thing is different and it's not downloading a single file and not the specific current date format. It's a download of multiple files which are having an extension with current year like you can see in below shot.

1. Here you can see the files from FTP, among them we will download files meet with the criteria. You can see target files.


2. Drag and drop FTP Task as shown on screen


3. Choose new FTP connection option for to connect FTP and apply FTP configuration as shown below.


4. Here we want to download files, so we need to go ahead with 'Receive' operation.


5. Configure Local Folder for file to be downloaded.


6. Create one variable for the Remote Path for the FTP and assign the value of the FTP directory from where the file will be received.


7. Here we haven't given static value of Remote Path and file name, but the qualified name will be created dynamically. So apply expression for the Remote Path option like following.


8. Finally, execute the created SSIS package and you will see the files received in the local folder. You can see the target files received.


What is the next? Stay tuned for the next posts.

Monday, 11 May 2015

Download multiple files through the FTP in SQL Server - SSIS by serverku

We have completed the reading of earlier methods to receive single file and receive file dynamically through the FTP in SSIS. Hope you like those posts. Previously we have downloaded a file which has specific extension with format of current date.

The same thing and criteria will be repeated again, but one thing is different and it's not downloading a single file and not the specific current date format. It's a download of multiple files which are having an extension with current year like you can see in below shot.

1. Here you can see the files from FTP, among them we will download files meet with the criteria. You can see target files.


2. Drag and drop FTP Task as shown on screen.


3. Choose new FTP connection option for to connect FTP and apply FTP configuration as shown below.


4. Here we want to download files, so we need to go ahead with 'Receive' operation.


5. Configure Local Folder for file to be downloaded.


6. Create one variable for the Remote Path for the FTP and assign the value of the FTP directory from where the file will be received.


7. Here we haven't given static value of Remote Path and file name, but the qualified name will be created dynamically. So apply expression for the Remote Path option like following.


8. Finally, execute the created ssis package and you will see the files received in the local folder. You can see the target files received.


What is the next? Stay tuned for the next posts.

Download file having specific extension in SQL Server - SSIS by serverku

Earlier we have seen some of the methods to download single file from FTP using tsql scripts and SSIS. So it's the same with single file, but a little bit different as we will download the file which have specific extension. Like we will have to receive the files which have extension of specific format of current date.

1. Here you can see the files from FTP, among them we will download one file. You can see target file.


2. Drag and drop FTP Task as shown in screen.


3. Choose new FTP connection option for to connect FTP and apply FTP configuration as shown below.


4. Here we want to download files, so we need to go ahead with 'Receive' operation.


5. Configure Local Folder for file to be downloaded.


6. Create one variable for the Remote Path for the FTP and assign the value of the FTP directory from where the file will be received.


7. Here we haven't given static value of Remote Path and file name, but the qualified name will be created dynamically. So apply expression for the Remote Path option like following.



8. Finally, execute the created ssis package and you will see the file received in the local folder. You can see the target file received in local directory.

Just it. Please share your ideas and comments here. Hope you enjoyed this post. Stay tuned for more posts.

Saturday, 9 May 2015

Download single file from FTP in SQL Server - SSIS by serverku

Here I am sharing the same thing, but it will be using SSIS instead of tsql script. Here I just want to show you a demo to download single file from an FTP. So let us move to the current post and follow the steps for same,

1. Here you can see the files from FTP, among them we will download one file.


2. Drag and drop FTP Task as shown in screen.


3. Choose new FTP connection option for to connect FTP and apply FTP configuration as shown below.


4. Here we want to download files, so we need to go ahead with 'Receive' operation.


5. Configure Local Folder for file to be downloaded.


6. From Remote Path option, select one of the file from FTP which is going to be received in local folder which we have configured in the previous step.


7. Finally, execute the created ssis package and you will see the file received in the local folder.


Hope you enjoyed this post. Stay tuned for more posts. Stay tuned for more!

Tuesday, 5 May 2015

SSIS package to get windows scheduled task status by serverku

We talked lots about SQL Server, Now moving to post this article of windows combining with SQL. You all know we can schedule ssis packages with scheduled jobs in SQL Server and with windows task scheduler too. We can easily get information and statistics of scheduled jobs which I ported just recently. We should have the same statistics for the windows task scheduled like a scheduled job from SQL Server and we should have an alert for its failures.

In this article i am creating one ssis package to achieve the same thing which we discussed above. So let us start and first creating prerequisite objects needed in ssis package.

1.  Creating staging table where we will import all windows scheduled tasks statistics.
CREATE TABLE [dbo].[WindowsTasksStatus]
(
[seq] [int] IDENTITY(1,1) NOT NULL,
[HostName] [varchar](200) NULL,
[TaskName] [varchar](1000) NULL,
[NextRunTime] [varchar](100) NULL,
[Status] [varchar](100) NULL,
[LastRunTime] [varchar](100) NULL,
[LastResult] [varchar](100) NULL,
[Schedule] [varchar](4000) NULL,
[TaskToRun] [varchar](100) NULL
)
2.  Create a batch file named WindowsTaskStatus.bat with following code. Pass server name in place of <HostName> and provide a path where to export csv file.
SCHTASKS /Query /S HostName /FO csv /V > E:\WindowsTaskStatus\hhax4_WindowsTaskStatus.csv
After creating all prerequisites, create a ssis package as per image and the all steps following,


Let me elaborate each task used in above ssis package.

Step 1 :  Execute SQL Task
In the first step, we will truncate the table on every execution of this package, so this table have following code with connection of SQL server database where we imports staging data.
Truncate table dbo.WindowsTasksStatus
Step 2 : Execute Process Task
In this step, we will call batch files created above.


Step 3 : For Each Loop Container
We will each files created in the second step and pass file name in variable, We have used For Each Loop Container because we can schedule other servers in the same batch file to export windows task status.




Step 4 : Data Flow Task
This task will pickup file passed from variable and import file data (Flat File Source) into a table (OLE DB Destination) which we created earlier. We also provide expression of connection string as FileName through property of the connection created for Flat File Source.


Step 5 : File System Task
After processing the files, it will be deleted by this task.


Step 6 : Execute SQL Task
Now we have staging table and also data imported from csv files for windows tasks statistics. So this final step to get the status of windows task's status. Here we will fetch the data from table for those windows tasks which have a last run status is not successful with last failed date time, next run time and excluding tasks which are currently executing. Following is the code used in this step which will fire an alert through an email.
DECLARE @Subject VARCHAR (1000) 
DECLARE @Body VARCHAR(max)
DECLARE @hpart VARCHAR(400)
DECLARE @MailProfile VARCHAR(50)

-- Use your mail profile here
SET @MailProfile = '<Mail Profile>'

CREATE TABLE #failedtasks
(
hostname VARCHAR(100),
taskname VARCHAR(100),
lastruntime VARCHAR(100),
nextruntime VARCHAR(100)
)

SET @Subject = 'Failed windows tasks alert'
-- Preparing column header
SET @hpart= '<tr><td>HostName</td><td>TaskName</td><td>LastRunTime</td><td>NextRunTime</td></tr>'

; WITH ctetasks
AS (SELECT hostname,
taskname,
lastruntime,
Cast (Substring(lastruntime, Charindex(',', lastruntime)+1, Len(
lastruntime))
+ ' '
+ Substring(lastruntime, 0, Charindex(',', lastruntime))
AS
DATETIME) AS
LastRunDateTime,
nextruntime
FROM dbo.windowstasksstatus
WHERE lastresult <> 0
-- Excuding task with disabled status
AND nextruntime <> 'Disabled'
AND lastruntime <> 'Disabled'
AND lastruntime <> 'N/A'
AND nextruntime <> 'N/A'
AND hostname <> 'HostName'
-- Excuding task which are running
AND status <> 'Running'
)

-- Importing above result in temp table
INSERT INTO #failedtasks
SELECT hostname,
taskname,
lastruntime,
nextruntime
FROM ctetasks
WHERE Cast(lastrundatetime AS DATETIME) >= Dateadd(dd, 0, Datediff(dd, 0,
Getdate()))

-- creating a mail body from temp table data
SET @Body = Cast ( ( SELECT td = hostname, '', td = taskname, '', td =
lastruntime, '', td =
nextruntime FROM #failedtasks FOR xml path('tr'), type ) AS VARCHAR(
max) ) + N''
SET @Body = '<table border="1">' + @hpart + @Body
+ '</table>' + '<br />'

-- Firing an email
IF ( @Body IS NOT NULL )
BEGIN
EXEC msdb.dbo.Sp_send_dbmail
@from_address = 'Windows Tasks Alert <alert@hhaexchange.com>',
@recipients = 'prajapatipareshm@gmail.com',
@subject = @subject,
@body = @Body,
@profile_name = @MailProfile,
@body_format = 'HTML';
END

-- Dropping temp table
DROP TABLE #failedtasks
You may be using this or any other way to get windows scheduled tasks statistics, Please share your valuable inputs here.

Script to get transactional replication errors for all subscriptions - SQL Server by serverku

Earlier post had same title just with a little bit difference of giving subscribers and all subscribers, so sharing a script which you can watch transactional replication error for all subscribers and get an alert at the time of error raise. Before going ahead with this post I would like you to read my earlier post and share your thoughts there.
So diverting on this post and again sharing script to get the same information, but it for all subscribers, you can also use the parameter of subscriber in where condition to apply filters.
Use distribution
GO
SELECT ma.publisher_db,
ma.publication,
ma.subscriber_db,
msre.time,
msre.error_text
FROM msrepl_errors msre
INNER JOIN msdistribution_history msh
ON ( msre.id = msh.error_id )
INNER JOIN msdistribution_agents ma
ON ( ma.id = msh.agent_id )
ORDER BY msre.time DESC
This is just a script which I want to present here. Now how can watch and keep attention when such any error occurs for transactional replication error in last 5 minutes and get an alert for the same with following script which can be scheduled every 5 minutes or whatever frequency as per filter applied of error time in query and run against a distribution database on distributor server,
USE distribution 
GO

DECLARE @body VARCHAR(max),
@subject VARCHAR(100),
@Publication VARCHAR(50),
@SubscriberDB VARCHAR(50),
@ErrorText VATCHAR(max)

SELECT @Publication = ma.publication,
@SubscriberDB = ma.subscriber_db,
@ErrorText = msre.error_text
FROM msrepl_errors msre
INNER JOIN msdistribution_history msh
ON ( msre.id = msh.error_id )
INNER JOIN msdistribution_agents ma
ON ( ma.id = msh.agent_id )
WHERE msre.time >= Dateadd(minute, -5, Getdate())
-- Capturing for last 5 minutes
ORDER BY msre.time DESC

SET @subject = 'Error captured for ' + @SubscriberDB
+ ' subscriber database of ' + @Publication
+ ' publication'
SET @body = @ErrorText

-- Sending an email
IF ( @subject IS NOT NULL )
BEGIN
EXEC msdb.dbo.Sp_send_dbmail
@recipients = 'prajapatipareshm@gmail.com',
@subject = @subject,
@body = @body,
@profile_name = '<Profile name>',
@body_format = 'HTML';
END

Have you scheduled any such script to get an alert and watch for any transaction replication error? Your ideas and thoughts are most welcome.

Saturday, 2 May 2015

Failed to acquire connection. Connection may not be configured correctly or you may not have the right permissions on this connection - SQL Server SSIS by serverku

This week, I resolved one issue which I received continuously which is nothing but exactly displayed in the post title.  The issue occurred  when I ran ssis package with batch file or scheduled job and same error message raised.

Source: Start Execute SQL Task
   Description: Failed to acquire connection "<Connection Name>". Connection may not be configured correctly or you may not have the right permissions on this connection.


As per title, it seems an issue with connection of server configured in ssis package. I thought same because this type of error comes when server is not connected or it does not have proper and rights to connect. First, I checked all server connections and tried to resolve it, but don't succeed.

Please note I received this error only while running ssis package with scheduled job or batch file, But it was working fine while running ssis package directly. It means it is not an issue in ssis package, but it is with batch files or schedule job. After trying I visited one conversion which says it may be issued with 32 bit/64 bit mode. And here I configured size package with schedule job and batch file with servers having operating system 64 bit.

After changing ssis package with schedule jobs and batch file with 32 bits, it works fine for me. Let me share,

With Batch file,
I changed path of dtexec.exe from <drive>:\Program Files\Microsoft SQL Server\100\DTS\Binn to <drive>:\Program Files(x86)\Microsoft SQL Server\100\DTS\Binn like following batch file content,
"C:\Program Files (x86)\Microsoft SQL Server\100\dts\Binn\dtexec.exe" /FILE  "C:\SSIS\SSISTest\bin\testing.dtsx" /MAXCONCURRENT " -1 " /CHECKPOINTING OFF  /REPORTING EWCDI 

With Scheduled job,
I am sharing one screen shot of schedule job step configuration of ssis package,


The day after this changed, I always see this package succeed. I am leaving one discussion open here “why it was failing with a 64 bit mode run?

Tuesday, 28 April 2015

Apply discrepancies at destination using SSIS - tablediff Utility in SQL Server by serverku

The last time we saw tablediff Utility basis and tablediff utility for multiple tables using SSIS. As you know using sssis package it generated discrepancies log files for database changes which we have to apply on destination servers\databases to make them seem. Now this post is extended for the same in which we will take care to automatically apply the discrepancies for each file generated at destination.
I am writing here for the next portion of previous post using ssis package for multiple tables and changes to be generated. Let create remaining part and add to existing package.


In the previous post we have visited first three steps and here I added last three steps (4 to 6). For the steps (1 to 3) please visit this post. Let’s continue with the remaining steps,

Step 4 : For each loop container
Retrieving each log files for the process,


Here we have taken one more variable to capture full log file path in variable named “Filename” and the following process will do the same,


Step 5 : Execute SQL Task
Taking destination database connection and using file connection,


To apply multiple file changes to destination, taken expression for FileConnection,


Step 6 : File System Task
Moving files to other location after process, so does not repeat next time,


After completing and running all the steps we will have the destination database to same as source , let us run first three steps (1 to 3) after changes apply and verify if found any other discrepancies,


You can see after this run, we have a message by tablediff utility is “Source table and destination table are identical” and it won't generate any discrepancies log files more. I want to know you are using, these steps to apply changes to destination using ssis? Waiting for your comments!

SQL Server tablediff utility for multiple tables using SSIS by serverku

As we saw the last post for the basic concept of tablediff utility. We learned one example, using static table, now I would like to go it with SISS package and also using more tables comparison. So, lets start it with some demo objects and created as follows. Here I am creating two tables with different databases and same SQL server instance.
USE SourceDB
GO

CREATE TABLE dbo.SourceObj1
(
id int PRIMARY KEY ,
name varchar(10),
CreatedDate DATETIME DEFAULT GETDATE()
)

CREATE TABLE dbo.SourceObj2
(
id int PRIMARY KEY ,
name varchar(10),
CreatedDate DATETIME DEFAULT GETDATE()
)

INSERT dbo.SourceObj1
(
id,
name
)
SELECT 1,'test1'
UNION ALL
SELECT 2,'test2'
GO

INSERT dbo.SourceObj2
(
id,
name
)
SELECT 1,'test3'
UNION ALL
SELECT 2,'test4'
GO


USE DestDB
GO

CREATE TABLE dbo.DestObj1
(
id int PRIMARY KEY ,
name varchar(10),
CreatedDate DATETIME DEFAULT GETDATE()
)

CREATE TABLE dbo.DestObj2
(
id int PRIMARY KEY ,
name varchar(10),
CreatedDate DATETIME DEFAULT GETDATE()
)

INSERT dbo.DestObj1
(
id,
name
)
SELECT 1,'test1'
UNION ALL
SELECT 3,'test3'
GO

INSERT dbo.DestObj2
(
id,
name
)
SELECT 1,'test3'
UNION ALL
SELECT 3,'test4'
GO


SELECT
'Sourcedb.dbo.SourceObj1' as ObjectName,
*
FROM Sourcedb.dbo.SourceObj1
SELECT
'Destdb.dbo.DestObj1' as Objectname,
*
FROM Destdb.dbo.DestObj1
SELECT
'Sourcedb.dbo.SourceObj2' as ObjectName,
*
FROM Sourcedb.dbo.SourceObj2
SELECT
'Destdb.dbo.DestObj2' as ObjectName,
*
FROM Destdb.dbo.DestObj2
GO
Let see the same data inserted,




Now I will create a ssis package with passing dynamic table name and other require details. But before that I need to populate one table with same details which we need require as argument to be passed in a batch file,
USE Maintenance
GO

CREATE TABLE DatasyncDetails
(
Id int identity(1,1),
SourceDatabase varchar(50),
DestDatabase varchar(50),
SourceSchema varchar(20),
DestSchema varchar(50),
SourceTable varchar(50),
DestTable varchar(20),
)


INSERT INTO DatasyncDetails
(
SourceDatabase,
DestDatabase,
SourceSchema,
DestSchema,
SourceTable,
DestTable
)

SELECT
'SourceDB',
'DestDB',
'DBO',
'DBO',
'SourceObj1',
'DestObj1'

UNION ALL

SELECT
'SourceDB',
'DestDB',
'DBO',
'DBO',
'SourceObj2',
'DestObj2'

SELECT
SourceDatabase,
SourceSchema,
SourceTable,
DestDatabase,
DestSchema,
DestTable
FROM Maintenance.dbo.DatasyncDetails
GO
Let's check the data populated which we have to process for discrepancies,


All required demo objects created to prepare for ssis package,


and the variables used ,


I have presented two whole snaps with details of whole flow, Now we will the all the steps one by one.

Step 1 : Execute SQL Task



Step 2 : For Each Loop Container



Step 3 :  Execute Process Task

Here we will create a batch file and call this task where arguments pass from parameter we mapped in an earlier task. You can see the content of batch file and the argument used inside,
"C:\Program Files\Microsoft SQL Server\90\COM\tablediff.exe" 
–sourceserver [PARESH\MSSQLSERVER2012]
-sourcedatabase [%1]
-sourceschema [%2]
-sourcetable [%3]
-sourceuser [dba]
-sourcepassword [dba@1234]
-destinationserver [PARESH\MSSQLSERVER2012]
-destinationdatabase [%4]
-destinationschema [%5]
-destinationtable [%6]
-destinationuser [dba]
-destinationpassword [dba@1234]
-et Difference
-f C:\DiffOutput\%7
Now I will use this batch file in execute process task and the use the arguments passed by For Each Loop container.






Expression of Arguments :
@[User::SourceDB]  +" "+  @[User::SourceSchema] + " " +  @[User::SourceTable] +" "+  @[User::DestDB]  +" "+  @[User::DestSchema] + " " +  @[User::DestTable] +" "+ @[User::SourceTable]
Finally done with all the steps and will have to run the package and will review the resulted SQL script log for discrepancies. So let's run it and review the out files.
Running…


Output files,
-- Host: PARESH\MSSQLSERVER2012
-- Database: [DestDB]
-- Table: [DBO].[DestObj1]
UPDATE [DBO].[DestObj1] SET [CreatedDate]='2012-10-06 11:39:10.620' WHERE [id] = 1
INSERT INTO [DBO].[DestObj1] ([CreatedDate],[id],[name]) VALUES ('2012-10-06 11:39:10.620',2,'test2')
DELETE FROM [DBO].[DestObj1] WHERE [id] = 3

-- Host: PARESH\MSSQLSERVER2012
-- Database: [DestDB]
-- Table: [DBO].[DestObj2]
UPDATE [DBO].[DestObj2] SET [CreatedDate]='2012-10-06 11:39:10.627' WHERE [id] = 1
INSERT INTO [DBO].[DestObj2] ([CreatedDate],[id],[name]) VALUES ('2012-10-06 11:39:10.627',2,'test4')
DELETE FROM [DBO].[DestObj2] WHERE [id] = 3
We can use more tables to find differences between them as we did. Here you can use server name, username & password as an argument and make it fully dynamic, but if those SQL servers can be connect from there. In the next post I will add some additional task to apply differences automatically at destination servers\databases. Hope you will like and share it.