Showing posts with label Administration. Show all posts
Showing posts with label Administration. Show all posts

Tuesday, 5 May 2015

Script to get failed scheduled jobs in SQL Server by serverku

As we were talking about to get start time and end time of schedule jobs a week before, We are going to use same information, but it will be with the scheduled jobs having last status is filed for any steps. So I request to go through that post. Also visit post to get running jobs and the way to get it. The DBA needs to know the scheduled job status on routine duty.

Some of the jobs having schedule occurrence one time and some of them have recursive occurrence. Need to list out those jobs having last status is Fail. It may be possible some jobs have multiple steps and among them some fail, so those jobs should be coming into the list. And have to exclude running jobs, even if are failed earlier.

Here I am sharing one script which will alert us with those jobs which failed today and have following criteria,
  1. List out failed jobs for today
  2. List out jobs which have a last status as fail
  3. Need Job Name, Start Time and End Time too
  4. Excluding running jobs
USE msdb
GO

-- Creating a temp table
CREATE TABLE #RunningJobs
(
job_id UNIQUEIDENTIFIER NOT NULL,
last_run_date INT NOT NULL,
last_run_time INT NOT NULL,
next_run_date INT NOT NULL,
next_run_time INT NOT NULL,
next_run_schedule_id INT NOT NULL,
requested_to_run INT NOT NULL,
request_source INT NOT NULL,
request_source_id sysname COLLATE database_default NULL,
running INT NOT NULL,
current_step INT NOT NULL,
current_retry_attempt INT NOT NULL,
job_state INT NOT NULL
)

-- Insert the running jobs in this temp table
INSERT INTO #RunningJobs
EXECUTE master.dbo.xp_sqlagent_enum_jobs 1,'sa'

SELECT
SERVERPROPERTY ('servername') as ServerName,
res.name as JobName,
-- converting run_date and run_time to proper date time format for Start Time
CAST(
CONVERT(CHAR(8), run_date, 112) + ' ' +
STUFF(STUFF(RIGHT('000000' + CONVERT(VARCHAR(8), run_time), 6), 5, 0, ':'), 3, 0, ':')
AS DateTime) AS StartedAt,
-- converting run_date,run_time and run_duration to proper date time format for Finish Time
(CONVERT (VARCHAR,
(DATEADD(ss,
(CASE LEN(run_duration)
WHEN 1
THEN run_duration
WHEN 2
THEN run_duration
WHEN 3
THEN (CAST(Left(right(run_duration,3),1) as int)*60)
+ (right(run_duration,2))
WHEN 4
THEN (CAST(Left(right(run_duration,4),2) AS int)*60)
+ (right(run_duration,2))
WHEN 5
THEN (CAST(Left(right(run_duration,5),1) AS int)*3600)
+ (CAST(Left(right(run_duration,4),2) AS int)*60)
+ right(run_duration,2)
WHEN 6
THEN (CAST(Left(right(run_duration,6),2) AS int)*3600)
+ (CAST(Left(right(run_duration,4),2) AS int)*60)
+ right(run_duration,2)
END ),
CAST(
CONVERT(CHAR(8), run_date, 112) + ' ' +
STUFF(STUFF(RIGHT('000000' + CONVERT(VARCHAR(8), run_time), 6), 5, 0, ':'), 3, 0, ':')
AS DateTime)))
,120)) AS FailedAt
FROM (SELECT Row_number()
OVER(
partition BY sj.name
ORDER BY run_time DESC) AS rnk,
sj.job_id,
sj.name,
run_date,
run_time,
run_duration,
run_status
FROM msdb.dbo.sysjobhistory sjh WITH (nolock)
INNER JOIN msdb.dbo.sysjobs sj WITH (nolock)
ON sjh.job_id = sj.job_id
WHERE sjh.step_id <> 0
-- considering for today only
AND run_date = CONVERT(VARCHAR(8), Getdate(), 112)
) res
WHERE
-- Failed status
res.run_status = 0
-- Lastly failed
AND res.rnk = 1
-- excluding running jobs
AND NOT EXISTS (SELECT rj.job_id
FROM #runningjobs rj
WHERE rj.job_id = res.job_id
AND rj.running = 1)
-- Droping table
DROP TABLE #RunningJobs
We can also set an alert for those failed jobs in email. I would like you to share your thoughts and the way you are using to get failed job details.

Script to get transactional replication errors for all subscriptions - SQL Server by serverku

Earlier post had same title just with a little bit difference of giving subscribers and all subscribers, so sharing a script which you can watch transactional replication error for all subscribers and get an alert at the time of error raise. Before going ahead with this post I would like you to read my earlier post and share your thoughts there.
So diverting on this post and again sharing script to get the same information, but it for all subscribers, you can also use the parameter of subscriber in where condition to apply filters.
Use distribution
GO
SELECT ma.publisher_db,
ma.publication,
ma.subscriber_db,
msre.time,
msre.error_text
FROM msrepl_errors msre
INNER JOIN msdistribution_history msh
ON ( msre.id = msh.error_id )
INNER JOIN msdistribution_agents ma
ON ( ma.id = msh.agent_id )
ORDER BY msre.time DESC
This is just a script which I want to present here. Now how can watch and keep attention when such any error occurs for transactional replication error in last 5 minutes and get an alert for the same with following script which can be scheduled every 5 minutes or whatever frequency as per filter applied of error time in query and run against a distribution database on distributor server,
USE distribution 
GO

DECLARE @body VARCHAR(max),
@subject VARCHAR(100),
@Publication VARCHAR(50),
@SubscriberDB VARCHAR(50),
@ErrorText VATCHAR(max)

SELECT @Publication = ma.publication,
@SubscriberDB = ma.subscriber_db,
@ErrorText = msre.error_text
FROM msrepl_errors msre
INNER JOIN msdistribution_history msh
ON ( msre.id = msh.error_id )
INNER JOIN msdistribution_agents ma
ON ( ma.id = msh.agent_id )
WHERE msre.time >= Dateadd(minute, -5, Getdate())
-- Capturing for last 5 minutes
ORDER BY msre.time DESC

SET @subject = 'Error captured for ' + @SubscriberDB
+ ' subscriber database of ' + @Publication
+ ' publication'
SET @body = @ErrorText

-- Sending an email
IF ( @subject IS NOT NULL )
BEGIN
EXEC msdb.dbo.Sp_send_dbmail
@recipients = 'prajapatipareshm@gmail.com',
@subject = @subject,
@body = @body,
@profile_name = '<Profile name>',
@body_format = 'HTML';
END

Have you scheduled any such script to get an alert and watch for any transaction replication error? Your ideas and thoughts are most welcome.

Monday, 4 May 2015

Configuration Replication failure and retry alert - SQL Server by serverku

As I wrote about replication latency from publisher to distributor and distributor to subscriber for replication and an alert for replication latency, same as we need to have an alert in case of replication failure. I would like to review some of my earlier posts about replication before moving to this post which you may like,
  1. Skip Log Reader error in SQL Server replication - How to
  2. Review of some replication issues and workaround - SQL Server
  3. Could not find the Distributor or the distribution database for the local server-Error while posts a tracer token in Replication
Let the discussion to move ahead and check as how can we set up an alert for in case of replication failure. So going with steps,

1. Create a job which having static code to fire an email as follows and do not need to schedule it,
DECLARE @Subject varchar(50), @body varchar(200)
SET @Subject = 'Replication Alert (Agent Failure)'
SET @body = 'Replication Alert (Agent Failure).<BR />Check the status of agent from replication monitor.'

EXEC msdb.dbo.sp_send_dbmail
@recipients = 'prajapatipareshm@gmail.com',
@subject = @Subject,
@body = @body,
@profile_name = '<Profile Name>',
@body_format = 'HTML';
2. Go to Replication monitor from Replication tab under SQL Server instance.


3. Select the Publication for which you want to receive an alert for failure and move to Warning tab as follows,


4. Go Configure alert and select “Replication : agent failure” as mentioned in below image,


5. For agent property, click on configure button  –> Response tab and select the job which we created earlier in step 1 in Execute job check box,


This is the just steps to configure an alert for replication agent failure. We can also add a notification alert to job operators also, so whenever the replication agent fails, this configuration runs the job and it will fire an email. We can set up the same configuration for replication agent retry too.

Script to get replication latency and alert in SQL Server by serverku

Recently I had posted for one issue I faced for replication trace token, Could not find the Distributor or the distribution database for the local server. Hope you read and liked it. Today I am writing for replication latency and its alert and I am using one script to post a tracer token which I used same in an earlier post where you can see the script there and details for same. So move on the topic here as how can we get replication latency information and alert for same. Putting in steps,

1. Post tracer tokens : Schedule a script to post a tracer tokens frequently, says 5 minutes, which posts a trace token into the transaction log at the Publisher and begins the process of tracking latency statistics and run this script at publisher,
USE publisherdb 
GO

DECLARE @out_tracer_token_id INT

EXEC sys.Sp_posttracertoken
@publication = N'<Publication Name>', -- Put Publication name here
@tracer_token_id=@out_tracer_token_id out

SELECT @out_tracer_token_id
 2. Script : After scheduling above scripts every 5 minutes, which will put latency history in MStracer_history system table in distribution database, following script use to get details for the latency from publisher to distributor and distributor to subscriber which belongs to distribution database,
USE distribution 
GO

SELECT publisher_db AS
[PublisherDB]
,
publication
AS [Publication],
name AS
[Subscriber],
subscriber_db AS
[SubscriberDB],
RIGHT('0' + Cast([timetosubscriber]/3600 AS VARCHAR(3)), 2)
+ ':'
+ RIGHT('0' + Cast(([timetosubscriber] % 3600) / 60 AS VARCHAR(2)), 2)
+ ':'
+ RIGHT('0' + Cast(([timetosubscriber] % 60) AS VARCHAR(2)), 2) AS
[DistToSubLatency],
RIGHT('0' + Cast([timetopublisher]/3600 AS VARCHAR(3)), 2)
+ ':'
+ RIGHT('0' + Cast(([timetopublisher] % 3600) / 60 AS VARCHAR(2)), 2)
+ ':'
+ RIGHT('0' + Cast(([timetopublisher] % 60) AS VARCHAR(2)), 2) AS
[PubToDistLatency],
RIGHT('0' + Cast([totaltime]/3600 AS VARCHAR(3)), 2)
+ ':'
+ RIGHT('0' + Cast(([totaltime] % 3600) / 60 AS VARCHAR(2)), 2)
+ ':'
+ RIGHT('0' + Cast(([totaltime] % 60) AS VARCHAR(2)), 2) AS
[TotalLatency]
FROM (SELECT DISTINCT msda.publisher_db,
syss.name,
msda.subscriber_db,
msda.publication,
publisher_commit,
distributor_commit,
Datediff(ss, publisher_commit, distributor_commit) AS
[TimeToPublisher],
subscriber_commit,
Datediff(ss, distributor_commit, subscriber_commit) AS
[TimeToSubscriber],
Datediff(ss, publisher_commit, distributor_commit)
+ Datediff(ss, distributor_commit, subscriber_commit) AS
[TotalTime]
FROM mstracer_history msth
INNER JOIN msdistribution_agents msda
ON msth.agent_id = msda.id
INNER JOIN sys.servers syss
ON msda.subscriber_id = syss.server_id
INNER JOIN mstracer_tokens
ON msth.parent_tracer_id = mstracer_tokens.tracer_id
WHERE subscriber_commit > Dateadd(mi, -5, Getdate())) res
ORDER BY [totaltime] DESC
3. Alert : As we have seen a script to get the latency for replication, herewith I am sharing a script to get alerts for highest latency, In script logic is implies to send an alert only when a publisher to distributor or distributor to subscriber latency exceed 1 minute. You can schedule this script to every 5 minutes as we use condition to scan latency history inserted in the last 5 minutes,
USE distribution 
GO

DECLARE @subject SYSNAME
DECLARE @body VARCHAR(4000)
DECLARE @SubscriberTime VARCHAR(20)
DECLARE @PublisherTime VARCHAR(20)
DECLARE @TotalTime VARCHAR(20)
DECLARE @PublisherDB VARCHAR(20)
DECLARE @Publication VARCHAR(20)
DECLARE @Subscriber VARCHAR(20)
DECLARE @SubscriberDB VARCHAR(20)

SELECT @PublisherDB = publisher_db,
@Publication = publication,
@Subscriber = name,
@SubscriberDB = subscriber_db,
@SubscriberTime = RIGHT('0' + Cast([timetosubscriber]/3600 AS VARCHAR(3))
, 2)
+ ':'
+ RIGHT('0' + Cast(([timetosubscriber] % 3600) / 60 AS
VARCHAR
(2)), 2)
+ ':'
+ RIGHT('0' + Cast(([timetosubscriber] % 60) AS VARCHAR
(2)), 2
),
@PublisherTime = RIGHT('0' + Cast([timetopublisher]/3600 AS VARCHAR(3)),
2)
+ ':'
+ RIGHT('0' + Cast(([timetopublisher] % 3600) / 60 AS
VARCHAR(2
)), 2)
+ ':'
+ RIGHT('0' + Cast(([timetopublisher] % 60) AS VARCHAR(2
)), 2),
@TotalTime = RIGHT('0' + Cast([totaltime]/3600 AS VARCHAR(3)), 2)
+ ':'
+ RIGHT('0' + Cast(([totaltime] % 3600) / 60 AS VARCHAR(2)),
2)
+ ':'
+ RIGHT('0' + Cast(([totaltime] % 60) AS VARCHAR(2)), 2)
FROM (SELECT DISTINCT msda.publisher_db,
syss.name,
msda.subscriber_db,
msda.publication,
publisher_commit,
distributor_commit,
Datediff(ss, publisher_commit, distributor_commit) AS
[TimeToPublisher],
subscriber_commit,
Datediff(ss, distributor_commit, subscriber_commit) AS
[TimeToSubscriber],
Datediff(ss, publisher_commit, distributor_commit)
+ Datediff(ss, distributor_commit, subscriber_commit) AS
[TotalTime]
FROM mstracer_history msth
INNER JOIN msdistribution_agents msda
ON msth.agent_id = msda.id
INNER JOIN sys.servers syss
ON msda.subscriber_id = syss.server_id
INNER JOIN mstracer_tokens
ON msth.parent_tracer_id = mstracer_tokens.tracer_id
WHERE subscriber_commit > Dateadd(mi, -5, Getdate())) res
WHERE ( [timetopublisher] > 60
OR [timetosubscriber] > 60 )
-- Fetch if publisher to distributor or distributor to subscriber latency greater than 60 seconds
ORDER BY [totaltime] DESC

SET @subject='Replication Latency Alert'
SET @body=
'Replication latency exceeded the highest acceptable replication delay'
SET @body=@body
+ 'Publisher to Distributor: ' + @PublisherTime
+ 'Distributor to Subscriber: '+ @SubscriberTime
+ 'Total Delay: '+ @TotalTime
+ 'Publication: '+ @Publication
+ 'Subscriber: ' + @Subscriber

IF ( @body IS NOT NULL )
BEGIN
EXEC msdb.dbo.Sp_send_dbmail
@recipients = 'prajapatipareshm@gmail.com',
@subject = @subject,
@body = @body,
@profile_name = '<Profile name>',
@body_format = 'HTML';
END
This is the details I wanted to present here to catch up replication latency and hope you may like it.

The row was not found at the Subscriber when applying the replicated command - Alternate workaround - SQL Server Replication by serverku

Before a week ago, I shared one post related to this title. Please read first workaround for same. I hope you liked it. Here I would like to share another way which may drive towards an alternative solution to get it resolved.

Alternate workaround :  tablediff utility
In first workaround in the last post, we used some script to get missing rows or the rows where we had an issue and applied at subscriber to complete it. But here we have another method to get missing rows and can apply at the destination. Before moving this method, I would like to read the following post related to same.
  1. The row was not found at the Subscriber when applying the replicated command-Replication error in SQL Server
  2. SQL Server tablediff Utility – Introduction
  3. SQL Server tablediff utility for multiple tables using SSIS
  4. Apply discrepancies at destination using SSIS - tablediff Utility in SQL Server
Did you read all posts? Ok, now we can go ahead. Actually, you know the workaround after reading above posts, Even let me share too. Yes, we can do it with tablediff utility. With following above links below is the script in context to the same server and following databases\tables again with transactional replication,
  • Primary database : Test
  • Secondary database : Test1
  • Replicated table : dbo.sample1
"C:\Program Files\Microsoft SQL Server\90\COM\tablediff.exe" 
-sourceserver [DemoServer]
-sourcedatabase [test]
-sourceschema [dbo]
-sourcetable [sample1]
-sourceuser [sa]
-sourcepassword [test@1234]
-destinationserver [DemoServer]
-destinationdatabase [test1]
-destinationschema [dbo]
-destinationtable [sample1]
-destinationuser [sa]
-destinationpassword [test@1234]
-et Difference
-f C:\DiffOutput
Make sure above statement must be in single line statement.

After running above script which will compare two tables data which have an issue (we have already script to know tables having an issue from first workaround) from source database\table and destination\table and we have generated missing rows script SQL file at C:\DiffOutput and file content as follows,
-- Host: [DemoServer]
-- Database: [test1]
-- Table: [dbo].[sample1]
INSERT INTO [test1].[dbo].[sample1] ([id],[name]) VALUES (2,'test2')
This will also generate delete script for the rows which exists at the destination but does not exist at source, insert script for missing rows from source to destination and update script if whole rows are different if it have, but in our case we have only insert script. Please take a note this is another alternate solution and tablediff utility may degrade performance for large tables or tables having so many numbers of rows. But this may help you to achieve your happy solution if you do not aware first workaround.

Delete database backup history in SQL Server by serverku

As I have written the article to clean up the old database backup for the maintenance activities. The same thing i am doing here as a maintenance activity, but not for the clean up of database backups, It is for database backups history cleanup from MSDB database. Please read my earlier posts related database backup, you may like them.
  1. Database Backup Statistics and History
  2. Archive old database backup files using forfiles.exe and FOR /F
  3. Archive old database backup files using TSQL Script
It is required to clean up the database backup history periodically if no longer needed. Because if the msdb database has more history then it will take the time to load it or sometime it will fail to load because of time out. The script to clean up the database backups, history and a demo are as follows, First we will look for the history of all the database backups done by this SQL instance.
USE msdb 
GO

SELECT bs.database_name,
bs.backup_start_date,
CASE bs.type
WHEN 'D' THEN 'Full Backup'
WHEN 'I' THEN 'Diff Backup'
WHEN 'L' THEN 'Log Backup'
END AS BackupType,
bmf.physical_device_name
FROM msdb..backupset bs
INNER JOIN msdb..backupmediafamily bmf
ON ( bs.media_set_id = bmf.media_set_id )
ORDER BY bs.backup_start_date DESC

We have two methods to delete the database backup history.

#1. Based on DateTime By passing the date in stored procedure , SP will delete all the database history older then specified date time. Let us run below query and get the output.
DECLARE @BackupOlderdate DATETIME
SET @BackupOlderdate = '2011-05-17 14:22:13.000'
EXEC sp_delete_backuphistory @BackupOlderdate
GO

#2. Based on Database
By passing the Database name in stored procedure, SP will delete all the database history for specified database only. Let us run below query and get the output.
DECLARE @DBname sysname
SET @DBname = 'trialmaster'
EXEC sp_delete_database_backuphistory @DBname
GO

Please comment here if you know any other methods to delete it.

Sunday, 3 May 2015

Find running scheduled jobs in SQL Server - Alternate way by serverku

A few days before, I just posted one article to check running scheduled jobs and I used master. dbo. xp_sqlagent_enum_jobs system objects and joined with dbo.sysjobs object from the msdb database for the same. Please read that post before moving down. This was one way to get the list of running jobs and now here I am sharing another alternate way to find same details. For this I am using dbo. sp_help_job objects from msdb database and @execution_status parameter to get the details. Let me share a script here,
USE msdb
GO

–- 1- Executing
EXEC dbo.sp_help_job @execution_status = 1;
And the result looks,


@execution_status parameter status values are following for which you list out the jobs with that passed status,

0 - Returns only those jobs that are not idle or suspended.
1 - Executing.
2 - Waiting for thread.
3 - Between retries.
4 - Idle.
5 - Suspended.
7 - Performing completion actions.

You may be using this script to get job status. Please put your comments if any other alternate way you have other than these.

There is insufficient system memory in resource pool 'default' to run this query - Error in SQL Server by serverku

Backup and restore database activity is common and routine task for all DBA when need to move a database at another place. I did the same thing while needed to move a database from one server to another server. But suddenly I encountered one issue while restoring database to the destination server and it was about to memory. Earlier I posted one tip in just learned of the same memory issue, but it was for replicaion. Please visit and read this link before going ahead this post. You can see that error “There is insufficient system memory to run this query” .

Now looking here too for the error while restoring database is the same, but slightly with different statement as follows,

“Msg 3013, Level 16, State 1, Line 1
RESTORE DATABASE is terminating abnormally.
Msg 701, Level 17, State 17, Line 1
There is insufficient system memory in resource pool 'default' to run this query.
”

An error seems the same with different operation, but the solution is the same.

DBCC FREESESSIONCACHE
DBCC FREEPROCCACHE
DBCC DROPCLEANBUFFERS
DBCC FREESYSTEMCACHE('ALL')

This is quick solution, but I am not recommending to apply this solution because this may create an issue as it will flush all caches and may affect on performance. So it is better to find cause of memory consumption and resolve it or apply optimized way like we can increase RAM, see if any idle sessions or session with high memory usage. In an earlier post of such error with replication, received some comments which can be helpful to us. I would like you put your valuable inputs here.

When exactly scheduled jobs started and completed - SQL Server by serverku

Some days ago, I worked to get failed job detail and an alert for the same. I used some system tables from the msdb database to fetch the details for the same. And msdb.dbo.sysjobhistory is one of the table among them, which used in the script. But this table doesn't contain data which shows the date and time when the job exactly started and completed. There are some fields which help us to get the same details,
  • run_date
  • run_time
  • run_duration
Let us run the query and check data from table,
USE msdb
GO

SELECT TOP 3 job_id,
run_date,
run_time,
run_duration
FROM msdb.dbo.sysjobhistory WITH (nolock)
ORDER BY run_date DESC,
run_time DESC

To get these details in shape and more readable format as we will use run_date, run_time and run_duration and convert them to date time format to achieve Start and Finish datetime. We will also use one more system table dbo.sysjobs for the job name and following is the script and output too,
USE msdb 
GO

SELECT TOP 3
sj.job_id AS JobId,
sj.name AS JobName,
run_date AS RunDate,
run_time AS Runtime,
run_duration AS RunDuration,
CAST(
CONVERT(CHAR(8), run_date, 112) + ' ' +
STUFF(STUFF(RIGHT('000000' + CONVERT(VARCHAR(8), run_time), 6), 5, 0, ':'), 3, 0, ':')
AS DateTime) AS StartedAt,
(CONVERT (VARCHAR,
(DATEADD(ss,
(CASE LEN(run_duration)
WHEN 1
THEN run_duration
WHEN 2
THEN run_duration
WHEN 3
THEN (CAST(Left(right(run_duration,3),1) as int)*60)
+ (right(run_duration,2))
WHEN 4
THEN (CAST(Left(right(run_duration,4),2) AS int)*60)
+ (right(run_duration,2))
WHEN 5
THEN (CAST(Left(right(run_duration,5),1) AS int)*3600)
+ (CAST(Left(right(run_duration,4),2) AS int)*60)
+ right(run_duration,2)
WHEN 6
THEN (CAST(Left(right(run_duration,6),2) AS int)*3600)
+ (CAST(Left(right(run_duration,4),2) AS int)*60)
+ right(run_duration,2)
END ),
CAST(
CONVERT(CHAR(8), run_date, 112) + ' ' +
STUFF(STUFF(RIGHT('000000' + CONVERT(VARCHAR(8), run_time), 6), 5, 0, ':'), 3, 0, ':')
AS DateTime)))
,120)) AS CompletedAt
FROM msdb.dbo.sysjobhistory sjh WITH (nolock)
INNER JOIN msdb.dbo.sysjobs sj WITH (nolock)
ON sjh.job_id = sj.job_id
WHERE sjh.step_id = 0
ORDER BY run_date DESC,
run_time DESC
(Click on image to enlarge)


Here we have all the details which we need. Hope this help you. Stay tuned for more posts.