Showing posts with label Cross Join. Show all posts
Showing posts with label Cross Join. Show all posts

Thursday, 9 July 2015

DEBUG Stored Procedures Using PRINT or RAISERROR in TSQL

DEBUG Stored Procedures Using PRINT or RAISERROR in TSQL

By Strictly Software 

Usually when I am having to write stored procedures with many sections within a transaction I set a @DEBUG BIT variable at the top of the procedure to help with debug.

Then I add checks to see if it is enabled and at various points in the procedure, usually at the start and end plus before and after each block of code I would output some debug telling me what is going on.

This really helps me when I have problems with the procedure and need to know where any bug is happening.

An example of a basic stored procedure with transactions so I can rollback the code if something goes wrong, plus the use of a TRY CATCH so that I can log error details to a table is shown below.

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO

CREATE PROCEDURE [dbo].[usp_net_clean_up]
	@SitePK int
AS
BEGIN
	
	SET NOCOUNT ON;

	DECLARE @ROWS INT,			
		@DEBUG BIT

	SELECT @DEBUG = 1

	IF @DEBUG = 1
	  PRINT 'IN usp_net_clean_up - SitePK: ' + CAST(@SitePK as varchar)

	BEGIN TRAN

	BEGIN TRY

        UPDATE	NIGHTLY_JOBS
	SET	Pending = 0
	WHERE	SitePK = @SitePK

	-- capture data errors
	SELECT @ROWS = @@ROWCOUNT

	IF @ROWS = 0 OR @ERROR != 0
	  BEGIN
		IF @DEBUG = 1
		  PRINT 'No rows updated'

		GOTO HANDLE_ERROR
	  END

	UPDATE	LOCK_FILES
	SET	Locked = 0
	WHERE	Locked = 1
		AND SitePK = @SitePK
		AND DATEDIFF(Day,Stamp,GETDATE())=0

	SELECT @ROWS = @@ROWCOUNT

	IF @ROWS = 0 OR @ERROR != 0
	  BEGIN
		IF @DEBUG = 1
		  PRINT 'No rows updated'

		GOTO HANDLE_ERROR
	  END

	END TRY
	BEGIN CATCH
		
		IF @DEBUG = 1
		  BEGIN
			PRINT 'IN CATCH ERROR'
			PRINT ERROR_MESSAGE()
			PRINT ERROR_LINE()
		  END

		 -- all ERROR functions will be available inside this proc
		  EXEC dbo.usp_sql_log_error @SitePK

		  -- rollback after anything you want to do such as logging the error 
		  -- to a table as that will get rolled back as well if you don't!
		  IF @@TRANCOUNT > 0
		    ROLLBACK TRAN

		  GOTO HANDLE_ERROR
	END CATCH

	IF @DEBUG = 1
	  PRINT 'End of proc - no errors'

	IF @@TRANCOUNT > 0
	  COMMIT TRAN

	GOTO EXIT_PROC


	HANDLE_ERROR:
	  IF @DEBUG = 1
	    PRINT 'IN HANDLE ERROR'

	  RETURN 1 -- I use 1 for success despite SQL recommendations!

	EXIT_PROC:
	  IF @DEBUG = 1
	    PRINT 'IN EXIT_PROC'

	  RETURN 0 -- failure
END

However another way to output debug messages is with the RAISERROR function and the use of placeholders for values, a bit like the sprint function in PHP.

To be honest I only used the function previously to raise custom errors but you easily use it for debugging as well.

This is an example of a super fast way to insert 100,000 rows into a table (using the TEMPDB), and using RAISERROR to output debug messages related to the time it takes plus some examples of the placeholder types which are listed at the bottom of the page.

SET NOCOUNT ON
SET DATEFORMAT YMD

DECLARE @StartStamp VARCHAR(20),
	@StopStamp VARCHAR(20),
	@ROWS INT
SELECT @StartStamp = CONVERT(varchar, GETDATE(),13)

RAISERROR('Start insert at %s',0,1,@StartStamp) WITH NOWAIT;

-- ensure our table doesn't already exist
IF OBJECT_ID('tempdb.dbo.random_data','U') IS NOT NULL
  DROP TABLE tempdb.dbo.random_data;

RAISERROR('Start insert of data',0,1)
-- super fast insert of 100,000 rows into a table
SELECT TOP (100000)
        RowNo   = ISNULL(CAST( ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS INT),0),        
        RandomID = NEWID() 
INTO	tempdb.dbo.random_data
FROM	master.sys.all_columns ac1
CROSS JOIN master.sys.all_columns ac2
CROSS JOIN master.sys.all_columns ac3

SELECT @ROWS = @@ROWCOUNT, @StopStamp = CONVERT(varchar, GETDATE(),13)

RAISERROR('There are %d rows in the table. Error Code was %u. Insert completed at %s',0,1,@ROWS,@@ERROR,@StopStamp) WITH NOWAIT;
-- output results
SELECT * FROM tempdb.dbo.random_data
-- drop our table
DROP TABLE tempdb.dbo.random_data;
-- ensure it is dropped
IF OBJECT_ID('tempdb.dbo.random_data','U') IS NULL
  RAISERROR('Dropped table %s',0,1,'tempdb.dbo.random_data')
ELSE
  RAISERROR('Could not drop table %s',0,1,'tempdb.dbo.random_data')

When I run this code the debug in the messages tab of SQL Query Analyser are:

09 Jul 2015 11:37:18
Start insert at 09 Jul 2015 11:37:18
Start insert of data
There are 100000 rows in the table. Error Code was 0. Insert completed at 09 Jul 2015 11:37:18
Dropped table tempdb.dbo.random_data

As you can see you can enter multiple parameters into the RAISERROR function.

The only thing you must remember is that you cannot use functions such as GETDATE(), CONVERT() or CAST() as substitution values. They must all be literals which is why I am converting the time stamps into strings first.

I hope you also notice how fast the insert of 100,000 rows all with unique values in the 2nd column is.

It takes less than one second!

This method is much faster than any WHILE loop or CURSOR method to insert records and you should add it to your box of tricks for future use.

Some people use number tables for insert jobs like this but there is no need when you have system tables with thousands of rows within them.

The trick is to use two CROSS JOINS to to the system tables master.sys.all_columns to create the necessary rows for the insert.

CROSS JOINS are hardly used by many people but they are very useful when you need to do insert jobs like this quickly.

Note how each row is ordered sequentially from 1 to 100,000 and the 2nd column has a unique GUID inside it. This is created from the NEWID() function.

So these are just two methods for adding debug to your TSQL code and it is up to you which method you find the most useful for the job at hand.

RAISERROR function parameter substitution values

%d or %i = signed integer

%o = unsigned octal

%p = pointer

%s = string

%u = unsigned integer

%x or %X = unsigned hexadecimal

By Strictly Software 

© 2015 Strictly Software

Sunday, 7 September 2008

When would you use a CROSS JOIN

Cross Joins??

Like most people I would imagine when I first heard about Cross Joins in SQL many years ago I remember thinking to myself when would I ever find myself
in a situation developing a website where I'd make use of them. They are one of those features that are infrequently used in day to day web development
but I over time I have come across a few situations where they have been very useful and solved some specific problems. I'm sure there are many
more that can be mentioned but I will talk about 2 specific situations where I have made use of them recently.


1. Many to Many Save

On nearly all of my sites I have one or more pages that output the results of some sort of search that the user has made on a previous page.
On the results page the user can select one or more of these records to view in more detail through selecting them with a checkbox or highlighting
the row with a double click etc. On some sites however I also have a folder option that allows the user to save those selected records into a new or
existing folder/group for later use. For example taking a jobboard as an example the user could select 3 candidates "Mr Smith, Mr Jones and Mr Robins"
from the results and then select 2 folders to save these candidates to "Top Candidates, Tuesdays Selections".
In the old days before I fully understood the flexibility and power of SQL I would do something like the following

(pseudo code)

For Each Candidate in Selected Candidates
For Each Folder in Selected Folders

'* Execute SQL either in a proc or inline
INSERT INTO CANDIDATE_GROUPS
(Folder, Candidate)
VALUES
(Folder, Candidate)

Next Folder
Next Candidate


Which for my example would involve 5 loop iterations, 5 separate calls to the database and then 5 separate insert statements. Obviously if you were
selecting more candidates and more folders this would increase the number of loops.

What always do nowadays is collect the two strings of delimited values and then call a stored procedure once passing the strings as parameters.
Then making use of my SPLIT user defined function I simply use a cross join to populate the table.

usp_asp_save_candidate_groups
@Candidates varchar(1000),
@Groups varchar(1000)
AS

SET NOCOUNT ON

INSERT INTO CANDIDATE_GROUPS
(Folder, Candidate)
SELECT a.[VALUE],b.[VALUE]
FROM dbo.udf_SPLIT(@Groups,',') as a,
dbo.udf_SPLIT(@Candidates,',') as b


This way I have reduced the application code to the bare minimum and we only have one INSERT statement. The overhead is now the CROSS JOIN and the SPLIT functions that convert a string of delimited values to a TABLE variable however this is always going to outperform nested loop inserts by a mile. Even though the SPLIT function makes use of multiple INSERTS to populate the TABLE variable that's then selected from the CROSS JOIN code outperforms the nested loops by factors of 20+ from tests I have run. Plus you are reducing network traffic and simplifying your application code.


2. Matrix Table

On a recent jobboard website I had to come up with a matrix table so that the user had access to the number of live jobs in the system for every possible category combination. You may have seen the sort of thing I mean where you have a filter form that lists job related categories with
the number of jobs in brackets to the side of them e.g

Job Type
-Full Time (50)
-Part Time (23)

Sector
-Management (4556)
-IT (3434)
-Sales (2456)

The page had to load fast so I didn't want to calculate the counts on the fly and there were 10 category types with over 150 categories that could be saved against a job in any possible combination. This meant that that as the user narrowed down his or her job search the counts had to reflect the options already selected. The first idea was to create a matrix table that would hold a row for each possible combination and a column for the number of jobs.

JobSector JobType Industry Region Location Role LiveJobs
12 8762 562 992 NULL NULL 345
12 8762 562 993 NULL NULL 321
12 8762 562 994 NULL NULL 78
12 8762 562 995 NULL NULL 963
12 8762 562 996 NULL NULL 13

This would have allowed me to do a simple statement such as

SELECT LiveJobs FROM MATRIX WHERE JobSector=12 AND Region=992

to get the number of live jobs per category combination.


Finance job in Afghanistan anyone?

So I knocked up some SQL to test how quick it would take to generate this matrix and clicked the run button. However it soon came apparent that this wasn't going to work as well as expected. After an hour or so the transaction log had filled the disk up and we had only populated 30 million or so rows out of some stupidly high number running into the billions.
I quickly decided this wasn't the way to go. Apart from the problems with generating this matrix table quickly most of the possible combinations were never going to actually ever have
a job count greater than 0. I doubt many UK based jobboards have ever posted jobs in the Private equity and venture capital sector based in Afghanistan so it seemed a pointless overhead trying to generate category combinations that would never be accessed.

So after a re-think I decided to rewrite the matrix using CROSS JOINs to populate the table so that instead of holding all possible category combinations it would only hold category combinations that were actually saved against the job. If a job was saved with only one category
for each possible category type then there would only be one row in the matrix table for that job. If however one category type had 3 options selected and all the rest one then there were be 3 rows and so on. This meant we would only ever hold data for categories being used and the table would be as small or as large at it needed to be.

A cut down version of the finished SQL that creates the matrix table is below. I have a working table #SITE_JOB_CATEGORY_VALUES that holds each job and each saved category ID and from the first SELECT that outputs the job ID it CROSS JOINS each possible derived table
to get all the combinations of saved category values against that job.


INSERT INTO SITE_JOB_CATEGORY_MATRIX_A
(JobPK,JobSector,JobType,Industry)
SELECT sjcv.JobFK,a.JobSector,b.JobType,c.Industry
FROM #SITE_JOB_CATEGORY_VALUES as sjcv,
(
SELECT a.JobFK, CategoryFK as JobSector
FROM (SELECT @JobPK as JobFK) as a
LEFT JOIN #SITE_JOB_CATEGORY_VALUES as b
ON b.JobFk = a.JobFK
AND CategoryTypeFK = 2248
) as a ,(
SELECT a.JobFK, CategoryFK as JobType
FROM (SELECT @JobPK as JobFK) as a
LEFT JOIN #SITE_JOB_CATEGORY_VALUES as b
ON b.JobFk = a.JobFK
AND CategoryTypeFK = 2249
) as b ,(
SELECT a.JobFK, CategoryFK as Industry
FROM (SELECT @JobPK as JobFK) as a
LEFT JOIN #SITE_JOB_CATEGORY_VALUES as b
ON b.JobFk = a.JobFK
AND CategoryTypeFK = 2190
) as c
GROUP BY sjcv.JobFK,a.JobSector,b.JobType,c.Industry


The reason that each derived table selects first from a constant and then LEFT JOINS to the main category table is that each derived table needs to return a value even if its a NULL otherwise the other derived tables wouldn't join to it and I'd lose the row for that job.

Depending on how many categories were selected under each category type (as some options allow multiple select) one job could have 1 or multiple rows with all the variations possible.

For example this job has one category value per category type apart from Region and Location that have 2 each which means there is a total of 4 rows to be outputted into the matrix.

JobPK JobSector JobType Industry Region Location Role
3434 12 8762 562 992 3543 78
3434 12 8762 562 993 3543 78
3434 12 8762 562 992 3544 78
3434 12 8762 562 993 3544 78

To get the job count you would just do a DISTINCT on the JobPK and the WHERE contains the columns and values that the user is filtering by.

The application references this table through the use of a SYNONYM as behind the scenes there will be 2 of these matrix tables. The current one and then every 15 minutes an MS Agent job runs that creates the new matrix and once done it points the SYNONYM to it and drops the existing table.

Using this approach I can rebuild the matrix table every 15 minutes in a manner of seconds. I have a clustered covering index on all the columns ordered by the most selective categories to the least and I can output the job count per category and run database searches using the users selections in lightening speed. The only downside is that there is a 15 minute delay between a job being posted on the site and it being searchable but that seems to be an acceptable limit to the posters.

So those are just two uses of a CROSS JOIN I have used lately and I am sure there are many more useful implementations. If you have any more please lets hear about them.