I did promise to announce the winners of the SQL Test competition I ran recently on this blog.

I would like to thank Carol D for her excellent write up on her experiences. Carol gained a lot from trying her first code kata but especially the discipline of writing tests first for small pieces of functionality at a time.  Steve also made  another good contribution and describes the new experience of using TDD to write SQL.  The other winners were Alex, Arthur and Manoj.  Thanks to all the contributors for making it such an interesting discussion.

In Part 11, we delved into mocking stored procedures and explored how to populate output parameters or add a row to a table when faking a procedure. In this post, we will look at views, including writing tests against the views themselves and also how to mock a view that another object under test depends upon.

As we’re going to focus our attention on views, let’s assume that we have the following schema in place – most of which is completely unrelated to the views we’re going to create. But that’s a lot like like the real world where the tSQLt Framework’s ability to mock the relevant objects means we can safely ignore the rest of the schema. You can download the DDL for this complete strucure here.

Testing a Simple View

There are two scenarios we are interested in, the first is where we are writing tests to validate the behaviour of the view itself, the second is where we are writing tests for an object that that references a view. The first case is pretty straightforward and for the second case, it is not so well known that tSQLt allows us to mock a view in the same way that we can fake a table. So we will start with testing views then move on to how to fake a view.

So given the schema above, let’s create a view called OrderItemView. Among other things we want this view to include a Value for each line item (i.e. Price x Quantity). So the very simplest test we can write looks like this:

CREATE PROCEDURE [ViewTests].[test_OrderItemView_calculates_correct_value]
AS
BEGIN
    --! Assemble
    EXEC tSQLt.FakeTable @TableName = 'dbo.OrderItem' ;

    INSERT dbo.OrderItem (Quantity, Price) VALUES (5, 1.54) ;

    DECLARE @Expected float; SET @Expected = 7.7;

    --! Act
    DECLARE @Actual float; SELECT @Actual = [Value] FROM dbo.OrderItemView

    --! Assert
    EXEC tSQLt.AssertEquals @Expected, @Actual
END

In this first test, we fake the OrderItem table to isolate the dependencies and add one row to the mocke version of the table. You’ll notice that we only need to populate the Quantity and Price columns for this test. Although on the real table most of the remaining columns are defined as NOT NULL, all columns on a faked table allow NULL so we can ignore them if they’re not relevant to the test in hand. This can not only save time writing this test, it also means we don’t have to go back and refactor this test in future when other non-null columns are added to the OrderItem table. When we first run this test, as expected we get this error:

[ViewTests].[test_OrderItemView_calculates_correct_value] failed: Invalid object name 'dbo.OrderItemView'.{test_OrderItemView_calculates_correct_value,13}

+----------------------+
|Test Execution Summary|
+----------------------+

|No|Test Case Name                                             |Result|
+--+-----------------------------------------------------------+------+
|1 |[ViewTests].[test_OrderItemView_calculates_correct_value]|Error |
-----------------------------------------------------------------------------
Msg 50000, Level 16, State 10, Line 1
Test Case Summary: 1 test case(s) executed, 0 succeeded, 0 failed, 1 errored.
-----------------------------------------------------------------------------

So what is the minimum code we can write to pass this test?

CREATE VIEW [dbo].[OrderItemView]
AS
SELECT
    Price * Quantity AS [Value]
FROM
    dbo.OrderItem AS oi

Ok, so as views go this one is of pretty limited use but we do have our first passing test and we would expect this test to continue to pass as we add more functionality. Let’s skip ahead to the next interesting feature. Our view now looks like this, returning all the columns from OrderItem including DispatchDate which, as you would expect, allows NULL. However, the application that uses this data needs the dispatch date to be some fixed point in the future as it doesn’t handle NULL date/times very well. Arguably, this isn’t the best way to address this issue but for the sake of this example, our view needs to ensure that DispatchDate is never NULL.

CREATE VIEW [dbo].[OrderItemView]
AS
SELECT
      ItemId
    , OrderId
    , ProductId
    , Quantity
    , Price
    , DispatchDate
    , Price * Quantity AS [Value]
FROM
    dbo.OrderItem AS oi

So, to test this we want to consider what happens when there is or is not a value for DispatchDate. Again, because we’re using FakeTable, we only need to populate the columns of interest and we just need one row with a DispatchDate and one row without. The easiest way to write this test is to put the results we expected to see into a table called #expected and select the columns of interest from the view into an #actual table then use tSQLt.AssertEqualsTable to compare the two.

CREATE PROCEDURE [ViewTests].[test_OrderItemView_supplies_default_DispatchDate]
AS
BEGIN
    --! Assemble
    EXEC tSQLt.FakeTable @TableName = 'dbo.OrderItem' ;

    --! Use a fixed date and time to avoid issues with ms differences causing
    --! this test to fail intermittently
    DECLARE @DispatchDate datetime; SET @DispatchDate = DATEADD(day, -1, GETDATE()) ;

    INSERT dbo.OrderItem (ItemId, DispatchDate)
          SELECT 1,@DispatchDate
    UNION SELECT 2, NULL

    CREATE TABLE #expected (ItemId int NULL, DispatchDate datetime NULL) ;

    INSERT #expected (ItemId, DispatchDate)
          SELECT 1, @DispatchDate
    UNION SELECT 2, '22220222 22:22:22'

    --! Act
    SELECT
          ItemId
        , DispatchDate
    INTO
        #actual
    FROM dbo.OrderItemView

    --! Assert
    EXEC tSQLt.AssertEqualsTable #expected, #actual ;
END

As we are practicing test-first development, this test will fail until we write just enough code to pass it.

[ViewTests].[test_OrderItemView_supplies_default_DispatchDate] failed: unexpected/missing resultset rows!
|_m_|ItemId|DispatchDate           |
+---+------+-----------------------+
|<  |2     |2222-02-22 22:22:22.000| |=  |1     |2012-10-27 10:00:37.757| |>  |2     |!NULL!                 |

+----------------------+
|Test Execution Summary|
+----------------------+

|No|Test Case Name                                                  |Result |
+--+----------------------------------------------------------------+-------+
|1 |[ViewTests].[test_OrderItemView_calculates_correct_value]     |Success|
|2 |[ViewTests].[test_OrderItemView_supplies_default_DispatchDate]|Failure|
-----------------------------------------------------------------------------
Msg 50000, Level 16, State 10, Line 1
Test Case Summary: 2 test case(s) executed, 1 succeeded, 1 failed, 0 errored.
-----------------------------------------------------------------------------

The view that passes this test looks like this:

CREATE VIEW [dbo].[OrderItemView]
AS
SELECT
      ItemId
    , OrderId
    , ProductId
    , Quantity
    , Price
    , COALESCE(DispatchDate, '22220222 22:22:22') AS [DispatchDate]
    , Price * Quantity AS [Value]
FROM
    dbo.OrderItem AS oi

Testing an Object that Depends on View

I said at the beginning of this post that tSQLt also allows us to fake views in the same way that we can fake tables. So let us assume that our OrderItemView now looks like this:

CREATE VIEW [dbo].[OrderItemView]
AS
SELECT
      oi.ItemId
    , oi.OrderId
    , oi.ProductId
    , p.DisplayName AS [Product]
    , typ.Name AS [ProductType]
    , oi.Quantity
    , oi.Price
    , COALESCE(oi.DispatchDate, '22220222 22:22:22') AS [DispatchDate]
    , oi.Price * oi.Quantity AS [Value]
-----------------------------------------------------------
FROM      dbo.OrderItem     AS oi
LEFT JOIN dbo.Product       AS p   ON p.ProductId = oi.ProductId
LEFT JOIN dbo.TypeOfProduct AS typ ON typ.ProductTypeId = P.ProductTypeId
-----------------------------------------------------------

… and that we also have another view that references OrderItemView that, at the moment, looks like this:

CREATE VIEW [dbo].[OrderDetailView]
AS
SELECT
      oh.OrderId
    , oh.OrderDateTime
    , c.LastName + ', ' + c.FirstName + COALESCE(' ' + c.MiddleInitial, '') AS [CustomerName]
    , e.LastName + ', ' + e.FirstName + COALESCE(' ' + LEFT(e.MiddleName, 1), '') AS [OrderTaker]
    , oi.ProductType + ' - ' + oi.Product AS [Product]
    , oi.Quantity
    , oi.Value AS [OrderValue]
    , oi.DispatchDate
    , oh.CustomerId
    , oh.EmployeeId
    , oi.ItemId
    , oi.ProductId
-----------------------------------------------------------
FROM       dbo.OrderHeader  AS oh
-----------------------------------------------------------
LEFT JOIN dbo.Employee      AS e  ON e.EmployeeId = oh.EmployeeId
LEFT JOIN dbo.Customer      AS c  ON c.CustomerId = oh.CustomerId
-----------------------------------------------------------
LEFT JOIN dbo.OrderItemView AS oi  ON oi.OrderId = oh.OrderId
-----------------------------------------------------------

You may have noticed that both of these views make use of LEFT joins – even though the join columns are NOT NULL meaning we could also use INNER joins. This is based on something I learnt from Rob Farley (blog | twitter) in an old post about query simplification and redundant joins. Basically, the principle is that if we use LEFT joins, the query optimizer can choose to leave out tables that are not required to fulfill the query. For example, in the query

SELECT SUM(OrderValue) AS [OrderValue] FROM dbo.OrderDetailView WHERE CustomerId = 1234;

Only the [OrderHeader] and [OrderItem] tables need to be accessed. Because we’re using LEFT joins, the [Customer], [Employee], [Product] and [TypeOfProduct] table are ignored – as you can see from this execution plan.

This also happens to make it much easier for us when writing tests since we can also choose to only populate the minimum number of tables – those that are actually relevant to the test in hand.  So if we want to test that the [ProductDescription] column in [OrderDetailView] is properly formatted, we do not need to populate the [Employee] or [Customer] tables.

However, we will need to populate [OrderHeader] plus [OrderItem], [Product] and [TypeOfProduct] or just the [OrderHeader] table and [OrderItemView].  As it’s less work, I’m going to go with faking one table and [OrderItemView].  Let’s take a first stab at this test

CREATE PROCEDURE [ViewTests].[test_OrderDetailView_formats_ProductDescription]
AS
BEGIN
    --! Assemble
    EXEC tSQLt.FakeTable @TableName = 'dbo.OrderHeader' ;
    EXEC tSQLt.FakeTable @TableName = 'dbo.OrderItemView' ;

    INSERT dbo.OrderHeader (OrderId) VALUES (1001) ;

    INSERT dbo.OrderItemView (OrderId, ItemId, Product, ProductType, ProductId)
    VALUES(1001, 1, 'Train set', 'Toys', 99)

    CREATE TABLE #expected (OrderId int NULL, ProductDescription varchar(1000) NULL, ProductId int) ;

    INSERT #expected (OrderId, ProductDescription, ProductId)
    VALUES (1001, 'Toys - Train set', 99)    

    --! Act
    SELECT
          OrderId
        , ProductDescription
        , ProductId
    INTO
        #actual
    FROM
        dbo.OrderDetailView

    --! Assert
    EXEC tSQLt.AssertEqualsTable #expected, #actual ;
END

In this test, we use tSQLt.FakeTable to mock both the [OrderHeader] table and [OrderItemView]. The syntax for faking a view is identical to faking a table (without the identity and constraint options etc). We then add one row to the header table and a row to the view so we have a ProductDescription to test.  If [OrderItemView] only referenced a single table, this would work but as there are multiple tables and there is no BEFORE trigger, we get the following error when we try to compile the procedure.

Msg 4405, Level 16, State 1, Procedure test_OrderDetailView_formats_ProductDescription_on_all_values_present, Line 11
View or function 'dbo.OrderItemView' is not updatable because the modification affects multiple base tables.

This is a SQL Server error (not tSQLt) and the solution is to use dynamic SQL to populate [OrderItemView]. This only gets compiled and validated at run time, by which point [FakeTable] will have created a table called [OrderItemView]. So our new INSERT statement might look something like this:

    DECLARE @cmd nvarchar(max);
    SET @cmd = 'INSERT dbo.OrderItemView (OrderId, ItemId, Product, ProductType, ProductId)'
            + ' VALUES(1001, 1, ''Train set'', ''Toys'', 99);' ;
    EXEC (@cmd) ;

This third test will now compile and when we run the tests using EXEC tSQLt.Run 'ViewTests';, all three tests pass:

+----------------------+
|Test Execution Summary|
+----------------------+

|No|Test Case Name                                                |Result |
+--+--------------------------------------------------------------+-------+
|1 |[ViewTests].[test_OrderDetailView_formats_ProductDescription] |Success|
|2 |[ViewTests].[test_OrderItemView_calculates_correct_value]     |Success|
|3 |[ViewTests].[test_OrderItemView_supplies_default_DispatchDate]|Success|
-----------------------------------------------------------------------------
Test Case Summary: 3 test case(s) executed, 3 succeeded, 0 failed, 0 errored.
-----------------------------------------------------------------------------

Interestingly, if you are testing CustomerName or OrderTaker you don’t need to fake [OrderItemView] at all, just add a row to [OrderHeader] and either [Customer] or [Employee] (depending on what you are testing). Because all our joins are redundant, the view (and indeed the tables it refers to) are not required to get the Customer or Employee name. For example:

CREATE PROCEDURE [ViewTests].[test_OrderDetailView_formats_CustomerName_on_all_values_present]
AS
BEGIN
    --! Assemble
    EXEC tSQLt.FakeTable @TableName = 'dbo.OrderHeader' ;
    EXEC tSQLt.FakeTable @TableName = 'dbo.Customer' ;

    INSERT dbo.OrderHeader (OrderId, CustomerId) VALUES (1001, 76) ;

    INSERT dbo.Customer (CustomerId, FirstName, MiddleInitial, LastName)
    VALUES (76, 'Theopholus', 'P', 'Wildebeeste')

    CREATE TABLE #expected (OrderId int NULL, CustomerName varchar(1000) NULL, CustomerId int) ;

    INSERT #expected (OrderId, CustomerName, CustomerId)
    VALUES (1001, 'Wildebeeste, Theopholus P', 76)    

    --! Act
    SELECT
          OrderId
        , CustomerName
        , CustomerId
    INTO
        #actual
    FROM
        dbo.OrderDetailView

    --! Assert
    EXEC tSQLt.AssertEqualsTable #expected, #actual ;
END

You can download the DDL for completed views here and all the tests here.

As the SQL Developer community grows to embrace the benefits of test-driven development for databases, so the importance of learning to do it properly increases. One way of learning effective TDD is by the use of code kata – short practice sessions that encourage test-first development in baby steps. Thanks to the guys at Red-Gate, I have a limited number of licences for SQL Test to give away free – just for practicing a bit of TDD and telling me about it.

All you need to do is to try out a code kata using T-SQL, either on your own or with a colleague, then tell me about it. This could be just a paragraph or two in a comment on this post or an entry on your own blog which links back to this post. Write about which kata you chose and why, how you wrote and tested the code and what you gained from the experience. I will pick the five best write-ups (from a mixture of blogs and comments) and send each of those contributors a single-user licence key for Red-Gate’s SQL Test. You can use any test framework as long as your tests are written in T-SQL.

You could even publish your experiences as an article on Simple-Talk. They pay for original content and provided you include a link back to this post, you could still win a SQL Test licence. Although you’ll have to be quick to allow for the editorial lead time between submitting an article and it being published.

SQL Test

Regular readers will be aware of my enthusiasm for tSQLt the open-source unit testing framework for SQL2005+. I’m not the only one that rates this library since Red-Gate liked it so much they have used it as the platform on which they built SQL Test. This graphical test manager is a plug-in for SQL Server Management Studio, originally developed during one of Red-Gate’s famous down-tools weeks and now an established part of their SQL Developer product suite.

What is a Code Kata?

According to Wikipedia, “kata is a Japanese word describing detailed choreographed patterns of movements practised either solo or in pairs… Kata originally were teaching/training methods by which successful combat techniques were preserved and passed on“. I believe the term “code kata” was first coined by Dave Thomas (blog) in his book The Pragmatic Programmer who writes:

Code Kata is an attempt to bring this element of practice to software development. A kata is an exercise in karate where you repeat a form many, many times, making little improvements in each. The intent behind code kata is similar. Each is a short exercise (perhaps 30 minutes to an hour long). Some involve programming, and can be coded in many different ways. Some are open ended, and involve thinking about the issues behind programming. These are unlikely to have a single correct answer.

What this means in test-driven programming terms is a simple, often deceptively so, logic problem which developers attempt to solve using strict test first development. People will often pair program on this task redoing the same exercise every day for a week or more pairing with different developers each day and trying to improve on the solution at each attempt. Typically, they will spend no more than 30 or 60 minutes per day on these exercises.

This practice is intended to help programmers learn to write code using TDD properly – not always a luxury we have in the real world. The aim is to write code in small baby steps each supported by tests to gain an understanding of the test-first paradigm. The code kata is a safe environment in which to experiment with ideas and best practice without any real consequences or deadlines. Peter Provost has a nice write-up on the importance of kata in learning effective TDD.

Why is SQL Different?

Although often language-neutral, many kata are written with object oriented languages in mind. Looking at how some online examples are developed and refactored, more complex kata would struggle to work in T-SQL in their current form. But whilst we don’t have many of the standard OO features in SQL Server programming, we do have a set-based language along with features like DRI, views and table-valued functions. The nature of the language will lead to different solutions and such features present different opportunities for refactoring.

Which Kata to Do?

The game of FizzBuzz can be a nice easy starting point and I have produced a SQL-friendly version too. The Potter kata might offer a little more challenge and there is also Toy Story which is a kata that I created specifically for SQL. If you’re feeling really brave there is always Uncle Bob’s Bowling Game

You could also choose from the list of kata on Coding Dojo or Dave Thomas also has a comprehensive list although these are not all programming tasks.

The Small Print

• This is just a bit of fun, these five licenses have been donated by Red-Gate and I get no reward from this other than the pleasure of spreading the TDD gospel and seeing how you get on with your kata.
• Participants agree to extracts from their comments or blog entry being quoted (with full credit and links) in a follow-up article which will be published on my blog and/or Simple-Talk.
• Comments and blog entries must be in by 19 October 2012 and winners will be announced and licenses sent by 30 November 2012.
• Only one licence per winning contribution and there is no cash alternative available.

So have some fun, try it and tell me how you got on for a chance to get one of five free licences for SQL Test worth around £175.00 each (donated by RedGate).

I noticed recently that my friends, Dennis Lloyd (blog | twitter) and Sebastian Meine (blog | twitter) are running some one day workshops on TDD and agile practices in SQL to coincide with Red-Gate’s US SQL In the City tour.

I’ve seen Dennis and Sebastian present, not to mention having had various deep technical discussions over beer or dinner and these guys really know their stuff. As the authors of tSQLt, the leading unit testing framework for SQL 2005+, they really understand both the patterns and practices of test-driven development and how to apply agile techniques, including TDD, to the database development paradigm.

The course covers test-driven development practices and how they can be applied to databases including how to isolate test dependencies and mock database objects. These last two are critical to being able to successfully implement TDD for databases without wasting time on unnecessary setup – the usual reason why people give up with TDD in the SQL space. The course also covers continuous integration and source control as they relate to database development. At the end of this one day program you should have a much better understanding of a range of agile patterns and practices and how to apply them in the real world.

Dennis and Sebastian are running this pre-con at just four of the six SQL In The City events and the price includes free or reduced-cost licences for selected products from the Red-Gate developer  suite. The venues are San Francisco, Chicago, Boston and Seattle and my advice is go here as soon as possible to book your place before they fill up. I should point out that I get no payment if you book on any of these days, I just think it’s a great opportunity to learn from two of the foremost experts in this field.  In fact, if I could get away from the UK I’d be going to one of them myself.

Practicing code kata is an established practice in agile shops but many kata are designed with object-oriented languages in mind and do not not always lend themselves to being reproduced in a declarative, set-based language like T-SQL.  So I have created this new kata specifically for SQL. Enjoy…

User Story

My name is Milt Pixney and I think I know a bit about SQL although the DBA won’t let me read to or write from any tables or even execute any stored procedures. I want to be able to write simple queries against a summary of recent toy sales aggregated by year, quarter and category so that I can provide a range of MI reports to the business.

Suggested solution: Create a view or user-defined function to summarise the value of toy sales by year/quarter/category against which Milt can run his queries. The code for the tables is at the bottom of this post but you should create your own test data according to the tests you write.

Business Rules

1. Recent is defined as the last two complete years – 2010 and 2011 if you’re reading this in 2012
2. Show data for all Year, Quarter and Toy Category combinations – even if there were no applicable sales
3. Any sales of toys with no category should rollup to an “unknown” category for each year and quarter

Possible Tests

• Only data for 2010 and 2011 is returned (consider what happens if there is no data for either year?)
• There is a value for every quarter in 2010 and 2011 regardless of whether there is any sales data
• Each quarter is listed for each year regardless of whether there are any applicable sales in that quarter
• Each category is listed for each quarter regardless of whether there are any applicable sales in that category
• Months are correctly translated or transposed to quarters
• Totals are aggregated correctly, consider how NULL SalesValue and NULL ToyCategoryId are handled
• Sales of toys with no sales category should appear as category = “Unknown”

Possible re-factoring or future enhancements

For any of the following changes, it is important that all the old functionality is still supported.  You should already have sufficient unit tests to guarantee this.

• Where a total is for a partial quarter this should be indicated somehow (1-3, 4-6, 7-9 and 10-12 are the standard quarters)
• Make the query dynamic so that it shows data from the last two years on a rolling basis (i.e. the last 24 months including partial years)
• Change the query so that only categories with at least one toy active in the quarter are visible (based on DateLaunched and DateDiscontinued)
• Roll up the aggregate values to show annual totals for each category
• Milt’s SQL skills aren’t as good as he thinks so you might want to include some indicator of how the totals are grouped.

The Code

IF OBJECTPROPERTY(OBJECT_ID(N'[dbo].[MonthlySalesSummary]'), N'IsUserTable') = 1 DROP TABLE [dbo].[MonthlySalesSummary]
IF OBJECTPROPERTY(OBJECT_ID(N'[dbo].[Toy]'), N'IsUserTable') = 1 DROP TABLE [dbo].[Toy]
IF OBJECTPROPERTY(OBJECT_ID(N'[dbo].[ToyCategory]'), N'IsUserTable') = 1 DROP TABLE [dbo].[ToyCategory]
GO

CREATE TABLE ToyCategory
(
  Id int NOT NULL
, Name varchar(32) NOT NULL UNIQUE
, CONSTRAINT PK_ToyCategory PRIMARY KEY (Id)
, CONSTRAINT AK_ToyCategory_Name UNIQUE (Name)
) ;

--! Add a default entry
INSERT ToyCategory VALUES (0, 'Unknown');

CREATE TABLE Toy
(
  Id int NOT NULL IDENTITY(1,1)
, Name varchar(16) NOT NULL
, ToyCategoryId int NULL
, DateLaunched datetime NOT NULL CONSTRAINT DF_Toy_DateLaunched DEFAULT GETDATE()
, DateDiscontinued datetime NULL
, CONSTRAINT PK_Toy PRIMARY KEY (Id)
, CONSTRAINT AK_To_Name UNIQUE (Name)
, CONSTRAINT FK_Toy_ToyCategory FOREIGN KEY (ToyCategoryId) REFERENCES ToyCategory (Id)
) ;

CREATE TABLE MonthlySalesSummary
(
  SalesYear smallint NOT NULL -- e.g. 2010, 2011
, SalesMonth tinyint NOT NULL
, ToyId int NOT NULL
, SalesValue decimal(18,2) NULL
, CONSTRAINT PK_MonthlySalesSummary PRIMARY KEY (SalesYear, SalesMonth, ToyId)
, CONSTRAINT FK_MonthlySalesSummary_Toy FOREIGN KEY (ToyId) REFERENCES Toy (Id)
, CONSTRAINT CK_MonthlySalesSummary_SalesMonth CHECK (SalesMonth IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12))
) ;

There has been some discussion recently over on the Google Groups discussion forum for tSQLt about practicing code kata in SQL.  One suggestion was to try the time-honored FizzBuzz game and I present here a slightly modified version adapted to work with a non-object oriented, set-based language like SQL.

What is a Code Kata?

In martial arts, where the term originates, a kata is a set of movements practiced many times over, either solo or in pairs in an attempt to improve execution a little each time. The term “code kata” was first coined by Dave Thomas – author of The Pragmatic Programmer.

The idea is that developers working alone, or more often pair programming, will spend no more than 30-60 minutes trying to solve a simple logic problem using strict TDD principles. They may work on one problem every day for a week or two, pairing with different colleagues and trying to improve on the solution each time. Code kata encourage programmers to work in small baby steps, writing unit tests at each stage. Test-first development forces developers to design and write code differently and kata are a way of practicing that approach.

Why is SQL Different?

The use of code kata is an established practice in agile application development, but as TDD is still relatively new for SQL, the notion may be less familiar to database developers. As SQL developers, we also have to deal with the fact that the set-based, declarative programming paradigm we use in the database world does not always lend itself to solving some of the katas more commonly attempted by object-oriented programmers. So with this in mind I thought I would produce a modified version of a simple, common kata making it more suitable for SQL.

About this Kata

Having played with this kata a few times, I have modified it slightly to suit how SQL Developers think.  For example, I suggest returning a result set rather than printing the results, although you could write and unit test either (I have).  Working with sets is what we do and as you work though this kata you will hopefully encounter some of the same challenges in writing good tests as you would encounter in testing sets  in the real world.

FizzBuzz is a popular children’s game, often played in schools where the teacher works his way round the class to each pupil in turn.  Starting at the number 1, each student calls out the next number in sequence except that if the number is divisible by 3 the student calls “fizz”, if divisible by 5 then “buzz” and if divisible by 3 and 5 the student shouts “FizzBuzz”.

One piece of advice I would give you for any kata is do not rush ahead, complete each section before moving on to the next.  The purpose of this exercise is to learn to code in small baby steps, building up a suite of passing tests that should continue to pass as the complexity of the solution increases.  This is not about writing an optimal solution for production use right at the start, it is about learning how to do test-driven development properly.

The FizzBuzz Kata

This is based on the version I found at codingdojo.org.

Create something in SQL that returns a result set containing the numbers from 1 to 100. But for multiples of three the value should be “Fizz” instead of the number and for the multiples of five, ”Buzz”.  For numbers which are multiples of both three and five the value should be “FizzBuzz”.  Your results should look like this:

Steps:

Lets divide this into different steps so, we can easily write and test this.

• Return a result set that contains numbers from 1 to 100
• Replace any number which is divisible by 3 with “Fizz”
• Replace any number which is divisible by 5 with “Buzz”
• Replace any number which is divisible by both 3 and 5 with “FizzBuzz”

Stage 2 – Refactoring

If you haven’t already done so, refactor the logic that defines the string value into a separate function that accepts a single number as an input.
Create a test to determine that the correct value is returned for a given input (thinking about minimum possible use cases) and what the behaviour is if the range is outside the range 1 – 100.
Refactor the production code to remove any loops or cursors (any loops in your unit tests are OK).

Stage 3 – Changing Requirements

• A number is Fizz if it is divisible by 3 or if it has a 3 in it
• A number is Buzz if it is divisible by 5 or if it has a 5 in it

(Previous rules take precendence over these new requirements)

Stage 4 – New Features

Create something in SQL that will return a list of only numbers in any given range that are Fizz, Buzz or FizzBuzz
Refactor as necessary to ensure that no looping will occur during actual execution regardless of result set size

Good Luck!

I hope you enjoy playing with this SQL-friendly version of the Fizz Buzz kata.  Do please add a comment on your experiences, what you learnt and how you went about designing your solution. You could also perhaps add a a link to your code.  At some point I will do a follow-up post with one of my solutions.

This month’s T-SQL Tuesday is hosted by Erin Stellato (blog | twitter) with a theme of “A Day in The Life”. Erin talks about the fact that our job title is often not a real reflection of everything we do, sometimes not even anything we do.  So for this month’s blog party, Erin asks us to describe a typical working day.

My Job Title

My job title “Development DBA” is what I call myself, although sometimes it still doesn’t cover everything I actually do.   The role description on my current contract refers to “Senior SQL Developer”.  I prefer the term “Development DBA” over “SQL Developer” because, in my mind, a SQL Developer is someone who just writes code. They may well write very good code, and do some database design, documentation maybe even a bit of query tuning – but their primary focus is writing SQL. As a Development DBA, I see my role as being broader than that. Yes I do all those things but I also expect to be the go-to guy in a wide variety of performance issues or when higher level design decisions regarding data stores need to be made. This could include archiving strategies, SDLC and change management processes or coaching and mentoring other members of the team.

So the question is how does a typical day shape up when compared to my perception of what I do.

The Start of My Day

My day typically starts earlier than it needs to so I can spend an hour working on a blog post or writing some other article before I leave for work. I find I’m at my most creative first thing in the morning – whether writing or coding.  I work for myself, and although I often work on long term projects with just one company, my online profile is important – especially when I need to start finding the next gig.  Today (Thur 12th July) is no different and I’m currently working on another post in my tSQLt Tutorial series which I haven’t added to in a while.  It’ll be nice to get this finished today as tomorrow’s SQL In The City is giving me some focus.  Red-Gate’s SQL Test uses tSQLt as its unit testing framework.

Yesterday, I started working on refactoring a business-critical piece of ETL code that needed some changes. My usual process when refactoring legacy code is to start adding unit tests to prove the existing code so that when I start refactoring I can quickly identify when I’ve broken something. Then, before making any actual logic changes I also like to run my version of the code against the original using a full set of live-like data as an integration test to compare the results of both versions of the code.  All was going well yesterday until I suddenly started getting some weird test failures.  The corollary of being at my best in the mornings is that I’m less insightful or intuitive later in the day.   Time to call it quits, I’ll look at this again tomorrow with fresh eyes (and brain).

A Tricky Problem

I’m lucky enough to live very close to the River Thames in South East London and I can get a fast catamaran service into work.  It’s only a half-hour journey, much nicer than train or tube and very relaxing – although this morning I spent the time thinking about why my tests were failing!

I don’t want to go in to specific details about the code but suffice to say that it was written based on some (perfectly valid at the time) assumptions about the cardinality of links between entities.  Imagine an UPDATE statement that is written in the belief that there is only ever a one-to-one relationship between entities but in a small number of perfectly valid cases the association turns out to be one-to-many. Under such circumstances, there is no guarantee of which of those one-to-many links will be the winner when updating a single destination row.  I know that the results in this scenario will be somewhat unpredictable but I would have expected any clustered indexes to play a part in the order of processing.

So after grabbing my morning espresso from the best coffee shop in Canary Wharf and getting to my desk around 08:30, I started to tackle the problem left over from last night.

After running several more tests I discovered that even if I ran the old code twice (with a DROP and CREATE in between) I was still getting different results.  First, this invalidated any attempt at comparing the old and new results and second, it disproved my assumption that any clustered indexes might give some kind of ordering. The only thing I could think of was that the recompile in between the two test runs was resulting in two slightly different execution plans so my next step was to compare those.

Learning Something New

A quick by-eye comparison of the graphical plans yielded no obvious differences, although as the plans were quite complex there was always the possibility that I was missing something. So the next thing I tried was to compare the plans as raw text (just by saving off the plans as xml).  Again, there were no significant differences in the overall plan but what I did notice is that there were differences in the number of rows processed by each thread of the parallel elements of the plan and also that the order in which the thread row counts were reported were different.

This makes perfect sense when thinking about parallel execution, in fact one of the reasons why parallel processing can generate lots of CX_PACKET waits is because the engine often has to wait for the last thread to finish before moving on to the next step. This isn’t necessarily a bad thing, CX_PACKET waits aren’t always an indicator of a problem.

Coming back to the problem in hand, if my UPDATE statement is trying to update one row from two possible source rows, and those rows are split between two threads then the outcome is going to depend on which thread finishes first – not how the clustered index is ordered.  To prove this, I modified the original version of the code to use MAXDOP 1. Several, somewhat slower runs later and I was at least getting consistent results each time.  This is no real help since it isn’t how it it works in production but at least I’d managed to prove I wasn’t going insane

It was nice to find the most likely reason for the inconsistent results and it felt great to learn something new.  That’s one of the things I love about what I do, I never stop learning and any day I discover something fresh is a brilliant day. Unfortunately it didn’t help my plans for creating a valid integration test, as it’s impossible to compare something that should be immutable with a moving target.

The Rest of My Day

The rest of my day was productive but typical. I continued working on that piece of code to make the required changes, supported by unit tests of course.  I also spent some time working on a database change management review document. This is a topic that I am passionate about and something I end up doing for most of my clients in some form or other. They don’t always accept all my recommendations or implement everything I suggest but I like to think that I normally leave a company’s database SDLC in a better state than when I arrived.  My day in the office finished around 6pm, followed by another relaxing boat ride home and an hour or so after dinner to finish off that blog post.  No work tomorrow as I’m off to SQL In The City.

In an earlier post in this series, I introduced tSQLt‘s [SpyProcedure] in a test to prove that one procedure calls another. In this article we delve deeper into mocking stored procedures and explore how to populate output parameters or add a row to a table without any of the intervening complex logic in the procedure being mocked.

[tSQLt.SpyProcedure] allows you to record that a call is made to a stored procedure and also what values were passed to that procedure without ever calling the real procedure. You can also define additional logic that will be completed when the mock procedure is called.

For example, imagine you have a procedure, [uspDoTheFirstThing] that contains a series of complex steps that ultimately result in a single row being inserted into a table called [Final]. Those steps depend on data in half a dozen other tables and you have already written tests to confirm that this procedure does everything it should.  Now you want to write a new procedure [uspAndAnotherThing] that, among other things, makes a call to [uspDoTheFirstThing] then has some other logic that builds on the row added to [Final]. Without the ability to mock [uspDoTheFirstThing], the only way to write tests for [uspAndAnotherThing] is to also pre-populate the tables used by [uspDoTheFirstThing] in addition to any setup required for the [uspAndAnotherThing] tests.  That gets tedious very quickly.   [SpyProcedure] allows you to capture the inputs and define the outcomes of [uspDoTheFirstThing] without all the extra set up..

This feature is well documented in the Official tSQLt User Guide but judging by the some of the questions I see cropping up in forums and on internet searches, I don’t think it is so well understood.

Syntax
tSQLt.SpyProcedure [@ProcedureName = ] ‘procedure name’
[, [@CommandToExecute = ] ‘command’ ]

Arguments
[@ProcedureName = ] ‘procedure name’ The name of an existing stored procedure. @ProcedureName is NVARCHAR(MAX) with no default. @ProcedureName should include the schema name of the stored procedure. For example: MySchema.MyProcedure
[@CommandToExecute = ] ‘command’ An optional command to execute when a call to Procedure Name is made. @CommandToExecute is NVARCHAR(MAX) with no default.

Let’s start by detailing the objects of interest for this post. As “copy & paste” from web pages doesn’t always work the way it should, you can download all the code samples for this post here.

The [Final] table gets populated by [uspDoTheFirstThing] using some complex but imaginary logic and the data from half a dozen or so tables.  This table is then used by [uspAndAnotherThing] later in our process.

CREATE TABLE [dbo].[Final]
(
  [Id] int NOT NULL IDENTITY(1,1)
, [ExtractCount] int NOT NULL
, [ValueCalculatedFromSixTables] decimal(28,6) NOT NULL
, [ProcessCount] int NULL
, CONSTRAINT PK_Final PRIMARY KEY CLUSTERED ([Id])
);

[uspDoTheFirstThing] is just a stub for the purposes of this post. Imagine that this actually looks up data from a number of other tables and then performs a number of complex logical operations before inserting a row into the [Final] table. We will assume for the purposes of this post that we already have a set of passing unit tests for this procedure.

CREATE PROCEDURE dbo.uspDoTheFirstThing
(
  @SourceSystem     varchar  ( 16 )
, @ProcessDate      datetime
, @BatchId          int               = NULL  OUT
)
AS
BEGIN
    --!
    --! Implementation ommitted (stub for unit testing demo purposes)
    --!
    INSERT dbo.Final
    (
      ExtractCount
    , ValueCalculatedFromSixTables
    )
    VALUES
    (
      101
    , 1.6377
    )

    SET @BatchId = SCOPE_IDENTITY();

    RETURN(0);
END

[uspAndAnotherThing] is the procedure we are now going to write some tests for.  Aside from validating inputs, this procedure calls [uspDoTheFirstThing] which supplies a Batch ID. This batch ID is then used throughout the subsequent processing (most of which has been omitted to help you stay awake).

CREATE PROCEDURE dbo.uspAndAnotherThing
(
  @SourceSystem     varchar  ( 16 )
, @ProcessDate      datetime
)
AS
BEGIN
    SET NOCOUNT ON

    --! Standard/ExceptionHandler variables
    DECLARE   @_Error         int
            , @_Step          nvarchar  (  128 )
            , @_Message       nvarchar  (  512 )
            , @_ErrorContext  nvarchar  (  256 )

    SET @_Error = 0;

    --! Working variables
    DECLARE @BatchId int, @ReturnValue int;

    BEGIN TRY
        SET @_Step = N'Validate inputs';

        IF @SourceSystem IS NULL
            BEGIN
                RAISERROR('Found invalid @SourceSystem: %s', 16, 1, @SourceSystem);
            END

        IF @ProcessDate IS NULL OR (CONVERT(char(8), @ProcessDate, 112) >  CONVERT(char(8), DATEADD(DAY, -1, GETDATE()), 112))
            BEGIN
                SET @_Message = 'Found invalid @ProcessDate: ' + COALESCE(CONVERT(char(11), @ProcessDate, 113), 'NULL');
                RAISERROR(@_Message, 16, 2);
            END

        SET @_Step = N'Get Batch Id';

        EXEC dbo.uspDoTheFirstThing @SourceSystem, @ProcessDate, @BatchId OUT

        --!
        --! TODO: Add more code
        --!
    END TRY
    BEGIN CATCH
        SET @_ErrorContext  = 'Failed to process trades at step: ' + COALESCE('[' + @_Step + ']', 'NULL')
                            + ' for Source System: ' + COALESCE(@SourceSystem, 'NULL')
                            + ' and Process Date: ' +  COALESCE(CONVERT(char(11), @ProcessDate, 113), 'NULL')

        --! Rollback any uncommitable transaction
        IF XACT_STATE() = -1
            BEGIN
                ROLLBACK TRAN;
                SET @_ErrorContext = @_ErrorContext + ' (Rolled back all changes)';
            END

        EXEC log4.ExceptionHandler
                  @ErrorContext  = @_ErrorContext
                , @ErrorNumber   = @_Error OUT
                , @ReturnMessage = @_Message OUT
    END CATCH

    IF @_Error > 0 RAISERROR(@_Message, 16, 99);

    SET NOCOUNT OFF;

    RETURN (@_Error);
END

One of the first tests we will write is to prove that [uspAndAnotherThing] does actually call [uspDoTheFirstThing] with the correct parameters. For this, we will use tSQLt.SpyProcedure as in the below example.

CREATE PROCEDURE [BlogDemoTests].[Test_AndAnotherThing_calls_DoTheFirstThing]
AS
BEGIN
    --!
    --! Assemble
    --!
    DECLARE @SourceSystem varchar(16); SET @SourceSystem = 'XYZ';
    DECLARE @ProcessDate  datetime; SET @ProcessDate = '20020202';

    --! Mock all the tables affected by this test
    EXEC tSQLt.FakeTable @TableName = N'dbo.Final';

    --!
    --! Replace the underlying stored procedure with a mock
    --! (This will just record the number of executions and inputs
    --! in a table called uspDoTheFirstThing_SpyProcedureLog)
    --!
    EXEC tSQLt.SpyProcedure @ProcedureName = N'dbo.uspDoTheFirstThing';

    --!
    --! Use temp tables for the expected and actual values as we don't
    --! then have to explicitly test for the number of times the sproc
    --! is called
    --!
    CREATE TABLE #expected (SourceSystem varchar(16) NULL, ProcessDate datetime NULL, BatchId int NULL)
    CREATE TABLE #actual (SourceSystem varchar(16) NULL, ProcessDate datetime NULL, BatchId int NULL)

    INSERT #expected VALUES(@SourceSystem, @ProcessDate, NULL);

    --!
    --! Act
    --!
    EXEC dbo.uspAndAnotherThing @SourceSystem, @ProcessDate;

    INSERT #actual SELECT SourceSystem, ProcessDate, BatchId FROM uspDoTheFirstThing_SpyProcedureLog;

    --!
    --! Assert
    --!
    EXEC tSQLt.AssertEqualsTable #expected, #actual
END

This test passes so we can move on to the next piece of logic. We said earlier that [uspAndAnotherThing] needs the @BatchId output from [uspDoTheFirstThing] for all its subsequent processing. And although we should already have tested that [uspDoTheFirstThing] does output a valid Batch ID, because this is such a critical value, I want to ensure that if something unforseen happens and the Batch ID is not valid, [uspAndAnotherThing] will throw an exception. So let’s add that check now…

BEGIN TRY
    SET @_Step = N'Validate inputs';

    IF @SourceSystem IS NULL
        BEGIN
            RAISERROR('Found invalid @SourceSystem: %s', 16, 1, @SourceSystem);
        END

    IF @ProcessDate IS NULL OR (CONVERT(char(8), @ProcessDate, 112) >  CONVERT(char(8), DATEADD(DAY, -1, GETDATE()), 112))
        BEGIN
            SET @_Message = 'Found invalid @ProcessDate: ' + COALESCE(CONVERT(char(11), @ProcessDate, 113), 'NULL');
            RAISERROR(@_Message, 16, 2);
        END

    SET @_Step = N'Get Batch Id';

    EXEC @ReturnValue = dbo.uspDoTheFirstThing @SourceSystem, @ProcessDate, @BatchId OUT

    --! Check that the Batch ID is valid
    IF ISNULL(@BatchId, -1) !> 0
        RAISERROR('Found invalid batch ID: %d', 16, 1, @BatchId);

    --!
    --! TODO: Add more code
    --!
END TRY

Adding this additional validation will cause [BlogDemoTests].[Test_AndAnotherThing_calls_DoTheFirstThing] to error like this:

Notice that this is actually an error, not a failure. The test itself passed but the extra validation on @BatchId in [uspAndAnotherThing] threw an exception because Batch ID was output (by the fake version of [uspDoTheFirstThing]) as NULL

So we need to ensure that the mock version of [uspDoTheFirstThing] created by [SpyProcedure] behaves as it would if it were real. Remember, we’ve isolated any table dependencies with FakeTable so the Batch Id can be any valid number. To do this we just need to change our SpyProcedure call in [Test_AndAnotherThing_calls_DoTheFirstThing] like this:

EXEC tSQLt.SpyProcedure @ProcedureName = N'dbo.uspDoTheFirstThing', @CommandToExecute = N'SET @BatchId = 1';

Now when we run the same test, the additional validation of @BatchId output value we added to [uspAndAnotherThing] succeeds and the test passes.

So what is actually happening under the hood within the mock version of [uspAndAnotherThing] created by SpyProcedure?  Well, after inserting a row into uspDoTheFirstThing_SpyProcedureLog with the input parameter values, it runs whatever SQL you specify in @CommandToExecute.

The ability to have mock procedures do additional SQL tasks is actually really powerful, although .Net developers will take this ability to control mock objects for granted. For the purposes of this post, we are going to skip whatever the implementation is for this procedure and jump to the final step, where the [Final] table gets updated. And, because we like to write robust code, we check that exactly one row was updated – otherwise an error will be thrown. The code in [uspAndAnotherThing] now looks like this:

BEGIN TRY
    SET @_Step = N'Validate inputs';

    IF @SourceSystem IS NULL
        BEGIN
            RAISERROR('Found invalid @SourceSystem: %s', 16, 1, @SourceSystem);
        END

    IF @ProcessDate IS NULL OR (CONVERT(char(8), @ProcessDate, 112) >  CONVERT(char(8), DATEADD(DAY, -1, GETDATE()), 112))
        BEGIN
            SET @_Message = 'Found invalid @ProcessDate: ' + COALESCE(CONVERT(char(11), @ProcessDate, 113), 'NULL');
            RAISERROR(@_Message, 16, 2);
        END

    SET @_Step = N'Get Batch Id';

    EXEC @ReturnValue = dbo.uspDoTheFirstThing @SourceSystem, @ProcessDate, @BatchId OUT

    --! Check that the Batch ID is valid
    IF ISNULL(@BatchId, -1) !> 0
        RAISERROR('Found invalid batch ID: %d', 16, 1, @BatchId);

    --!
    --! Trade processing logic omitted
    --!
    SET @_Step = N'Update [Final] Table with number of rows processed';

    UPDATE dbo.Final SET ProcessCount = 66 WHERE Id = @BatchId;

    SET @_RowCount = @@ROWCOUNT;

    IF @_RowCount != 1
        RAISERROR('Expected @@ROWCOUNT of 1 but found %d', 16, 1, @_RowCount);
END TRY

When we re-run the original test, we again hit an error like this:

This means we need to make a further modification to the SpyProcedure call in [Test_AndAnotherThing_calls_DoTheFirstThing].  This time, we need to have the mock version of [uspDoTheFirstThing] add a row to the faked [Final] table before setting the output parameter to the same value as the [Id] column (since that is what the UPDATE in [uspAndAnotherThing] is based on).  This is what the call to [SpyProcedure] now looks like:

DECLARE @command nvarchar(2000);
SET @command = 'INSERT dbo.Final ([Id], [ExtractCount], [ValueCalculatedFromSixTables], [ProcessCount])'
            + ' VALUES (12, 542, 17, NULL);'
            + ' SET @BatchId = 12;'

EXEC tSQLt.SpyProcedure @ProcedureName = N'dbo.uspDoTheFirstThing', @CommandToExecute = @command;

You can see that we are not actually testing any additional functionality with these progressive changes.  The test we have been writing only asserts that [uspDoTheFirstThing] gets called exactly once and is passed the correct parameters. The sequence of changes we have made as part of this post replicate the way we might have to refactor [Test_AndAnotherThing_calls_DoTheFirstThing] so that the mock version of [uspDoTheFirstThing] meets the minimum behaviours to allow us to write the rest of our tests. When testing [uspAndAnotherThing], we don’t actually care about the implementation of [uspDoTheFirstThing], all we are interested in is recording the inputs and controlling the outputs. This is the essence of mock objects.

In this post we looked at [SpyProcedure] in more detail and refactored one test to allow us better control over the outputs of a mocked procedure without having to implement any complicated, intervening logic. The next couple of posts in this series will focus on writing tests against views.

I left school more years ago than I care to count with minimal qualifications and more or less fell into a career in hairdressing.  I loved what I did for most of my time in that career but by 1999 had achieved everything I wanted to within that industry.  I decided it was time for a change of direction.  That new direction was IT, not the most obvious choice you might think and certainly not an easy change to make.  This post is about that change and tries to offer tips for others in a similar situation.

I was prompted to write this by a recent request for help on LinkedIn from a friend I haven’t met yet who is thinking of a career change similar to, although perhaps not quite as drastic to my own.

My Story

My last job in hairdressing was as a consultant travelling around the UK doing in-salon training, and presenting at seminars and hairdressing shows. For me, that was the pinnacle of my hairdressing career – I’d run my own salon previously and had no wish to go backwards so I started looking at my options. Computers had always been an interest of mine so after much research and careful thought, I decided to make the move into the IT sector.

Having decided on such a major career change, in 1999 I started studying for an MCSE juggling a long work week with evenings and weekends studying and using up all my annual leave attending courses.  I knew that I would need to be able to show potential employers that I was serious about such a dramatic change so passed my first two MCP exams before starting to look for my first IT job.

My first job in IT was on helpdesk for a small software house, a job where people skills were deemed more important than technical skills.  This worked well for me as they expected to have to train staff up on their own software anyway and helpdesk was very customer-facing which meant I could make the most of everything I’d learnt about dealing with people in my previous career as a hairdresser.

Helpdesk wasn’t what I wanted to do long term but I recognised that with such a drastic change of career, I would need to prove myself first and this type of role had more in common with my transferrable skills.  Within 3 months of starting my first job on helpdesk, I’d moved up to helpdesk management and systems administration. I later “discovered” T-SQL and the rest, as they say, is history.

Great Oaks from Little Acorns Grow

Or, start small, think big!

If you are planning a big career change, you have to be prepared to start on a lower rung of the ladder, possibly much lower, in order to gain experience that will lead to the role you do want.  The cold, harsh reality is that just because you want to be a Network Administrator and have read the books and passed a couple of exams, an employer is not going to let you loose on their network without you having proven experience.

There is no substitute for knowledge gained “in the trenches”.  When it all goes horribly wrong because some new virus is infecting networks all over the world, you won’t have time to get your books out trying to remember how to configure something you remember reading about whilst you were still a chef, hairdresser or milkman.  The higher up the technical ladder you aspire to, the more that real-world experience will count.

It is my experience that, the lower down that ladder you go, the less that technical experience matters and conversely, the more the right attitude and good soft skills can be enough to set you apart from the crowd.  So if you are serious about changing career, you have to be prepared to make a few short term sacrifices for the long term goal.

You need to develop a career plan, even write it down if that works for you.  Don’t plan further ahead than three years to begin with.  You will find that your plans will change once you are actually working in IT and get a better idea of what the different types of job entail.  My original plan was to complete my MCSE and become a Systems Administrator, possible specialising in Exchange.  Instead, after completing my MCSE, MCDBA and MCAD; then having worked through roles as a Production DBA, here I am now doing freelance database design, development and tuning along with some coaching and .Net development.

When you start applying for jobs in IT, look for roles where the skills the employer is looking for align with what you can already do.  Typically helpdesk, 1st line support or even junior development are the kind of roles where the soft skills you can pick up in other careers are more important than having a wealth of technical skills.

On the subject of development, if your chosen field is web development, have you created any websites that you can include links to in your CV?  These might be for local businesses or perhaps a charity that you’ve been involved with.  Words of caution here however, make sure it’s a good web site.  Four static pages written in FrontPage is not a good advert for your skills!

What Have You Got to Offer?

If you are at this point in your life, you need to try and think strategically.  If the only experience you have of a particular technology is passing an exam or two, this is probably not enough to get you the job you want.  You have to be prepared to start with the job you can do (at least in an employer’s eyes) rather than the job you want.

If you are considering a major career change, you have to think very carefully about the skills you have now and how they might relate to a potential employer in your new field.  To coin a phrase, it’s not about what the employer can do for you; it’s what you can do for them.  What aspects of the job you do now would be of interest or value in the IT field?  What skills do you have that would be readily transferrable?  This might be the ability to build diverse relationships working in sales, people skills gained in a call centre or the ability to quickly get to grips with new technologies in a technical (but non-IT) field.  You might need to think outside the box to answer these questions (see below).

In my case, I was short on IT experience but my years in front of the public meant that I was strong on soft skills.  Whilst I couldn’t hide the fact that I was in hairdressing, my CV focussed on transferrable skills like interpersonal skills, coaching, influencing and man-management.

Your CV is a Billboard

When was the last time you bought a burger from a fast food restaurant that actually looked like the one on the TV ads?  The burger you see in adverts or on hoardings is (allegedly) the same as the one they serve up in their restaurants just presented in the best possible way.  You are that burger.

Your resume is just an advertisement.  I am not suggesting you lie on your CV – that is a really bad idea – but you can still present yourself in the best possible light.

For example, my final role in hairdressing was as a technical consultant for a major hair product manufacturer.  This involved planning and delivery of training plans within larger salons, working on stage or behind the scenes at big hairdressing shows, managing a team of other trainers and assisting telephone helpline staff with product queries from hairdressers.  Also, as the only computer-literate member of the team I became the team’s power user, designing simple spread sheets (e.g. time and activity tracking, expenses etc.) or helping configure Lotus Notes.

So in addition to emphasising my people skills, my CV at that time included the following bullet points:

• Educational & Technical support to end users.
• First line PC & application support for field-based users.
• Design and implementation of computer-based performance monitoring system.
• Dealing with technical queries via telephone hot-line.
• Management, training and development of a team of field educators.
• Key role in wide variety of projects connected with IT, education, sales & marketing.

These were all aspects of the job I was doing at the time, I just chose to present them in a way that potential employers in the IT sector would be able to relate to.  If I’d talked about doing hair shows and visiting lots of salons and building relationships with hairdressing wholesalers it would have been much harder for a potential employer to see that I had suitable skills and my CV would have ended up in the NO pile.

Don’t Burn Your Bridges

Don’t assume that the only way you can achieve a dramatic career change is to change employer.  Getting the right job with the right company isn’t easy and if you have no experience in that role, a potential new employer is taking a bigger risk.  But if you have good standing in your current company, try and build relationships with the IT team – especially any IT managers that you deal with.  Managers often prefer to recruit people they or someone on their team knows but being able to get a reliable “off-the-record” reference is the next best thing.  It’s much easier for a hiring manager to get that reference if you already work for the company.  Obviously you wouldn’t be able to do this without the support of your current manager.  Larger companies especially may be more likely to be open to such sideways career moves as they may have better staff retention policies.

There may be a number of reasons why an internal move isn’t an option.  These could include lack of support from your current manager, or the company may be too small or the IT team located too far away to make the jump internally.  For me it was the latter, I was field-based and lived in Southampton whilst the IT department was based on the outskirts of London, over 100 miles away.

Should you tell your manager or even colleagues of your plans before you are ready to give notice?  In my experience, the answer is no.  As you are changing career, it might take longer than you think to get a new position and you don’t really know what could happen in the meantime.  For example, if the company need to start laying people off (not unheard of in these times) you may find that your name is at the top of the list, “as you’re planning to leave anyway”.

Another approach, depending on what you do now and what you want to do next, may be to try and get some experience in your chosen field within your current job.  This typically works better if what you do now and what you want to do are more closely related.  For example, I used to work with a guy who’s job was application support but he wanted to become a developer.  He started trying to extend his skills by actually looking at what the code was doing, later getting to the point where he was able to recommend then actually implement bug fixes in the code.  He is now doing what he want do to, development, full-time but it did take nearly a year to achieve that goal.

In a Nutshell

To summarise, these are the key points to remember:

• It is never too late to change, I was 38 when I started in IT.
• You can start a completely new career in IT, regardless of what you do now.  But it will take hard work, patience and a little humility.
• Be prepared to attend some courses and take an exam or two in your chosen field to prove to a potential employer you are serious.
• If you can, try for an internal company transfer as the door is already half-open.  Alternatively, can you start getting exposure to your chosen field within the job you’re doing right now.
• Regardless of whether you are going for an internal move or looking externally, re-write your CV using language that a potential employer can relate to and emphasising your relevant strengths and transferrable skills.  But do not lie on your resume.
• Be prepared to take a lower paid, more junior position if that starts you out on your new path.

One interesting point,  I thought that having all the hairdressing on my CV whilst trying to get IT jobs might cause problems but most people seemed to recognise that I had reached the top in one industry and were intrigued as to why I had made such a dramatic change to start again in another career.

Best of all, I have no regrets, in fact if I’m honest I almost wish I’d done it sooner – but then if I had I wouldn’t have had such great experiences as a hairdresser.  I think it was Victor Kiam (of Remington fame) who once wrote that those people lucky enough to find a job they enjoy should stick to it. I consider myself very lucky, I’ve worked in a total of three very different careers and have absolutely loved two of them.  I wish you the same luck in your careers.

This month’s T-SQL Tuesday is hosted by Aaron Nelson (blog | twitter) with a theme of “logging”.  Although Aaron’s definition of this topic is deliberately broad, I thought I would go with a more typical definition and talk about the approach I use for logging the actions of large multi-step stored procedures.

Some of the work I do revolves around supporting and/or refactoring some pretty large ETL routines where often much of the work is carried out or at least managed by a single stored procedure which can have many, many individual steps.  In the course of this type of work I’ve developed an approach to logging that has culminated in Log4TSql, an open source logging and exception handling library for SQL2005+.  I’ve blogged previously about the exception handling part of this framework for T-SQL Tuesday #024 and this month’s T-SQL Tuesday theme leads nicely into the other half of Log4TSql.

One of the challenges in logging the actions of a procedure that has so many steps is what to do with the information.  Storing each individual log entry in a permanent table means increasing the pressure on disk resource during what might already be a very I/O intensive process.  And if some or all of the load steps are wrapped in an explicit transaction and the code hits an error and needs to rollback, then you also lose the information that might help diagnose the very reason that the problem occurred in the first place.  One solution to the rollback problem lies in using a table variable to store the log items which can then be permanently stored after any transaction handling has completed but that approach can still have an I/O impact whilst the main code is processing.

So the approach I prefer is to use a varchar(max) variable within the procedure being logged and append each log entry to that, separated by some sort of line feed. This has the advantage of not being affected by any explicit transaction handling and also under normal circumstances has no disk I/O overhead.  I like to separate each line with a carriage return to make it easier to read later.  We will see how Log4TSql makes it simple to retrieve that information in an easy to read format later in this article.

Probably the best way to demonstrate the approach I am proposing is by example.  Most of the local variables used in the sample procedure below should be fairly self-explanatory but I’ve added a few thoughts about some of them here.

• @_Step – I always name each individual step in a multi-step procedure. This facilitates simpler logging using copy & paste code blocks and also makes it easy to identify the point of failure in case of error without having to use multiple TRY…CATCH blocks.
• @_Message – This is a succinct completion message indicating either that the process failed and why or that the process completed successfully.
• @_ErrorContext – This is used within the CATCH block to state what the code was doing at the time an exception occurred and is passed to Log4TSql’s ExceptionHandler procedure
• @_SprocStartTime – The time the stored procedure started executing (this is then used at the end of processing to record the total run time)
• @_StepStartTime – Set at the beginning of each individual step within the procedure and is used to record the run time of that step
• @_ProgressText – This is where we store the outcome of each individual step, number of rows, duration etc. Think of this like a log file, where each outcome is appended as a line of text.

Also, a quick style note: I really like the fact that in SQL2008 we gained the ability to declare and populate a variable in a single statement thus DECLARE @_Error int = 0; so now I tend to use a similar single-line style when writing backwards compatible code for SQL2005 which goes like this: DECLARE @_Error int; SET @_Error = 0;.

The aim of this type of logging is to record as much information as possible with minimal impact on performance. I like to include the number of rows affected and the duration of each step and will sometimes even include information from each iteration of a loop or cursor. This can help in identifying what was happening in the build up to any failure as well as being able to quickly tell historically which step is the bottleneck if performance suddenly falls off a cliff.

One thing to be really careful of when appending new lines to @_ProgressText is that you protect against NULL values as one NULL may wipe out all your logging information (depending on your “concat null yields null” setting).  Although, the ability to turn CONCAT_NULL_YIELDS_NULL off is more or less deprecated and in future versions trying to do so may generate an error so the sooner we learn to write defensively for NULLs, the better.

This might seem like a lot of extra code to write so whenever possible, I try to use the following format for each step, which means that I can copy & paste the same block as a starting point then just change the value of @_Step and write the actual code.

SET @_Step = 'Do something meaningful';
SET @_StepStartTime = GETDATE();

--!
--! Do the work here
--!

SET @_RowCount      = @@ROWCOUNT;
SET @_ProgressText  = @_ProgressText + @NEW_LINE
        + 'Step: [' + @_Step + '] processed ' + COALESCE(CAST(@_RowCount AS varchar), 'NULL')
        + ' row(s) in ' + log4.FormatElapsedTime(@_StepStartTime, NULL, 3)

If you want to actually run this code, you can download the code for the [TSqlTuesday031LoggingExample] procedure here.  You will also need to install the current version of Log4TSql into the database you are using for this demonstration.

CREATE PROCEDURE [dbo].[PointlessStoredProcedure]
(
  @SomeDate      datetime
, @RowsAffected  int = NULL  OUT
)
AS
BEGIN
    WAITFOR DELAY '00:00:00.636';
    SET @RowsAffected = 999;
    RETURN (0);
END
GO

IF OBJECTPROPERTY(OBJECT_ID(N'[dbo].[TSqlTuesday031LoggingExample]'), N'IsProcedure') = 1
    DROP PROCEDURE [dbo].[TSqlTuesday031LoggingExample];
GO

CREATE PROCEDURE [dbo].[TSqlTuesday031LoggingExample]
(
  @JobName                       nvarchar(128)  = NULL
, @SomeDate                      datetime       = NULL
, @NumberOfPointlessInsertsToDo  int            = 250
)

AS

BEGIN
    SET NOCOUNT ON

    --! Standard/ExceptionHandler variables (use SETs to keep it SQL2005 compatible)
    DECLARE @_Error int; SET @_Error = 0;
    DECLARE @_RowCount int;
    DECLARE @_Step varchar(128);
    DECLARE @_Message nvarchar(512);
    DECLARE @_ErrorContext nvarchar(512);

    --! JournalWriter variables (use SETs to keep it SQL2005 compatible)
    DECLARE @_FunctionName varchar(255); SET @_FunctionName = OBJECT_NAME(@@PROCID);
    DECLARE @_SprocStartTime datetime; SET @_SprocStartTime = GETDATE();
    DECLARE @_StepStartTime datetime;
    --!
    --! We will use @_ProgressText like a log file, writing lines of text
    --! separated by a line feed
    --!
    --! NB: INSTANTIATE @_ProgressText PROPERLY ON FIRST USE
    --! If this variable ends up as NULL at any point, you will lose all your
    --! logged info
    --!
    DECLARE @_ProgressText nvarchar(max); SET @_ProgressText = '';
    DECLARE @NEW_LINE char(1); SET @NEW_LINE = CHAR(10);

    --!
    --! If some of the input parameters for this procedure are time-sensitive,
    --! it can often be helpful to record what the inputs were at run time
    --!
    --! NOTE: With the exception of @_FunctionName and @_SprocStartTime which
    --! we've set within our code and have absolute control over, we use
    --! COALESCE() or ISNULL() for everything else.  We don't want one
    --! accidental NULL wiping out our carefully collected log detail.
    --!
    SET @_ProgressText = @_FunctionName + ' starting at ' + CONVERT(char(23), @_SprocStartTime, 121) + ' with inputs: '
            + @NEW_LINE + '    @JobName                      : ' + COALESCE(@JobName, 'NULL')
            + @NEW_LINE + '    @SomeDate                     : ' + COALESCE(CONVERT(char(11), @SomeDate, 113), 'NULL')
            + @NEW_LINE + '    @NumberOfPointlessInsertsToDo : ' + COALESCE(CAST(@NumberOfPointlessInsertsToDo AS varchar), 'NULL')
            + @NEW_LINE

    --!
    --! Within the TRY...CATCH... block below, we have a series of steps to
    --! simulate the multiple steps in a data load or other complex process.
    --!
    BEGIN TRY
        --! Use an explicit txn as we have multiple steps which must
        --! either all complete or all fail
        BEGIN TRAN;

        --!
        --! As we have multiple steps, we need to identify each step
        --! starting by checking that our inputs are valid. The value
        --! of @_Step can be used to simplify log collection and is
        --! also used in case of an error so we can record exactly
        --! where the exception occurred.
        --!
        SET @_Step = 'Validate inputs';

        IF @JobName IS NULL
            RAISERROR('Found invalid Job Name: %s', 16, 1, @JobName);

        IF @SomeDate IS NULL
            BEGIN
                SET @_Message    = 'Found some invalid date: ' + COALESCE(CONVERT(char(11), @SomeDate, 113), 'NULL');
                RAISERROR(@_Message, 16, 2);
            END

        IF ISNULL(@NumberOfPointlessInsertsToDo, 0) NOT BETWEEN 10 AND 250
            SET @NumberOfPointlessInsertsToDo = 250;

        SET @_Step = 'Define working values';

        DECLARE @pointlessDividend float; SET @pointlessDividend = 999.000999;

        --!
        --! By the time we get here we have validated our critical inputs and
        --! defined some working variables.  As these may be time-sensitive,
        --! we can record them here in the same way that we did for the inputs
        --!
        --! NOTE: That from this point on we must append new information to the
        --! existing value of @_ProgressText and always remember to use ISNULL()
        --! or COALESCE() against every variable to ensure we don't wipe out
        --! the log record with a NULL.
        --!
        SET @_ProgressText  = @_ProgressText
                            + @NEW_LINE + 'Working values: '
                            + @NEW_LINE + '    @NumberOfPointlessInsertsToDo : ' + COALESCE(CAST(@NumberOfPointlessInsertsToDo AS varchar), 'NULL')
                            + @NEW_LINE + '    @pointlessDividend            : ' + COALESCE(CAST(@pointlessDividend AS varchar), 'NULL')
                            + @NEW_LINE

        --!
        --! Name the step and also the time the step started
        --! so we can record the duration of this step later
        --!
        SET @_Step = 'Pointless Inserts Loop';
        SET @_StepStartTime = GETDATE();

        --! Put some dummy time wasting in here
        WAITFOR DELAY '00:00:04.045';

        DECLARE @PointlessTable table (CycleNumber int, Narrative varchar(64));
        DECLARE @CycleNumber int, @Narrative varchar(64);
        DECLARE @LoopCount int, @InsertCount int;

        SELECT @CycleNumber = 1, @LoopCount = 0, @InsertCount = 0;

        SET @_ProgressText  = @_ProgressText + @NEW_LINE + 'Starting Step: [' + @_Step + ']... ';

        WHILE @CycleNumber <= @NumberOfPointlessInsertsToDo
            BEGIN
                SET @LoopCount = @LoopCount + 1;

                IF @CycleNumber % 2 = 0
                    BEGIN
                        SET @Narrative = 'Now processing cycle number ' + CAST(@CycleNumber AS varchar(16)) + '...';

                        INSERT @PointlessTable VALUES(@CycleNumber, @Narrative);

                        SET @InsertCount = @InsertCount + 1;

                        --! We can even log each step with a loop or cursor
                        SET @_ProgressText  = @_ProgressText + @NEW_LINE
                                + ' - Inserting row: ' + COALESCE(@Narrative, 'NULL');
                    END

                SET @CycleNumber = @CycleNumber + 1;
            END

        --!
        --! Append more information about what we did as a new line, remembering to
        --! guard against any NULLs
        --!
        --! We use log4.FormatElapsedTime() to get a nicely formatted duration string
        --! Inputs are: start time and end time (which will default to GETDATE())
        --! plus the number of seconds duration at which to show whole seconds.
        --! So if we pass in 3 anything up to 3999 ms will appear as "1345 milliseconds"
        --! and if 4 secs or mor as "1 min(s) and 22 sec(s)"
        --!
        SET @_ProgressText  = @_ProgressText + @NEW_LINE + @NEW_LINE
                            + 'Inserted ' + COALESCE(CAST(@InsertCount AS varchar), 'NULL') + ' pointless row(s)'
                            + ' from ' + COALESCE(CAST(@LoopCount AS varchar), 'NULL') + ' even more pointless iterations'
                            + ' in ' + log4.FormatElapsedTime(@_StepStartTime, NULL, 3)
        --!
        --!
        --!
        SET @_Step = 'Copy pointless rows';
        SET @_StepStartTime = GETDATE();

        --! Put some more time wasting in here
        WAITFOR DELAY '00:00:02.450';

        SELECT CycleNumber, Narrative INTO #pointless FROM @PointlessTable

        SET @_RowCount = @@ROWCOUNT;

        --!
        --! This is just a simple recording of rowcount and duration
        --! If you had a lot of steps in turn, you could use this generic format
        --! over and over again. Because we define the value of @_Step for each
        --! step, the logging is just a copy and paste of this boiler plate code
        --!
        SET @_ProgressText  = @_ProgressText + @NEW_LINE
                            + 'Step: [' + @_Step + '] processed ' + COALESCE(CAST(@_RowCount AS varchar), 'NULL')
                            + ' row(s) in ' + log4.FormatElapsedTime(@_StepStartTime, NULL, 3)

        --!
        --! Use Itzik Ben-Gan's virtual numbers table to generate some activity
        --! http://www.sqlmag.com/article/sql-server/virtual-auxiliary-table-of-numbers
        --!
       SET @_Step = 'Even more pointless activity';
        SET @_StepStartTime = GETDATE();
        WITH
              N0 AS (SELECT 1 AS n UNION ALL SELECT 1)
            , N1 AS (SELECT 1 AS n FROM N0 t1 CROSS JOIN N0 t2)
            , N2 AS (SELECT 1 AS n FROM N1 t1 CROSS JOIN N1 t2)
            , N3 AS (SELECT 1 AS n FROM N2 t1 CROSS JOIN N2 t2)
            , N4 AS (SELECT 1 AS n FROM N3 t1 CROSS JOIN N3 t2)
            , N5 AS (SELECT 1 AS n FROM N4 t1 CROSS JOIN N4 t2)
            , N6 AS (SELECT 1 AS n FROM N5 t1 CROSS JOIN N5 t2)
            , nums AS (SELECT ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS num FROM N6)
        SELECT
            num AS [Number]
        INTO
            #morePointless
        FROM
            nums
        WHERE
            num <= 1000000

        SET @_RowCount        = @@ROWCOUNT;
        SET @_ProgressText  = @_ProgressText + @NEW_LINE
                            + 'Step: [' + @_Step + '] processed ' + COALESCE(CAST(@_RowCount AS varchar), 'NULL')
                            + ' row(s) in ' + log4.FormatElapsedTime(@_StepStartTime, NULL, 3)

        --!
        --! Final pointless step
        --! An example of how a procedure might be called to do something and
        --! report back the number of rows processed which can be used by the
        --! same boiler-plate block of code
        --!
        SET @_Step = 'Call a pointless sproc';
        SET @_StepStartTime = GETDATE();

        EXEC dbo.PointlessStoredProcedure @SomeDate, @_RowCount OUT;

        SET @_ProgressText  = @_ProgressText + @NEW_LINE
                            + 'Step: [' + @_Step + '] processed ' + COALESCE(CAST(@_RowCount AS varchar), 'NULL')
                            + ' row(s) in ' + log4.FormatElapsedTime(@_StepStartTime, NULL, 3)

        IF @@TRANCOUNT > 0 COMMIT TRAN;
        --!
        --! If we get here everything was successful so generate a useful
        --! success message.  We have to populate @_Message as a minimum
        --! for logging to be at all useful
        --!
        SET @_Message   = 'Successfully completed all steps, processing '
                        + COALESCE(CAST(@_RowCount AS varchar(16)), 'NULL') + ' pointless row(s)'
                        + ' for Some Date: ' + COALESCE(CONVERT(char(11), @SomeDate, 113), 'NULL');
    END TRY
    BEGIN CATCH
        --!
        --! Capture the value of @_Step at the point of failure and any other
        --! valuable info
        --! NB: Always use ISNULL() or COALESCE() to ensure that any unexpectedly
        --! missing values to do not prevent the rest of the info being collected
        --!
        SET @_ErrorContext    = 'Failed to demonstrate logging for t-SQL Tuesday #031 at step: ' + COALESCE('[' + @_Step + ']', 'NULL')
                                + ' for Some Date: ' + COALESCE(CONVERT(char(11), @SomeDate, 113), 'NULL');

        IF ABS(XACT_STATE()) = 1
            BEGIN
                ROLLBACK TRAN;
                SET @_ErrorContext = @_ErrorContext + ' (Rolled back all changes)';
            END

        --!
        --! Call ExceptionHandler supplying the context under which the error
        --! ocurred and collecting the error number and formatted error message.
        --!
        EXEC log4.ExceptionHandler
                  @ErrorContext  = @_ErrorContext
                , @ErrorNumber   = @_Error OUT
                , @ReturnMessage = @_Message OUT;
    END CATCH

--/////////////////////////////////////////////////////////////////////////////////////////////////
OnComplete:
--/////////////////////////////////////////////////////////////////////////////////////////////////

    --! Clean up
    IF OBJECT_ID(N'tempdb..#pointless') > 0 DROP TABLE #pointless;
    IF OBJECT_ID(N'tempdb..#morePointless') > 0 DROP TABLE #morePointless;

    --!
    --! Use log4.FormatElapsedTime() to get a nicely formatted complete run time
    --! (as opposed to step duration) in a nicely formatted string
    --!
    IF @_Error = 0
        BEGIN
            SET @_Step            = 'OnComplete'
            SET @_Message        = COALESCE(@_Message, @_Step)
                                + ' in a total run time of ' + log4.FormatElapsedTime(@_SprocStartTime, NULL, 3)
            SET @_ProgressText  = @_ProgressText + @NEW_LINE + @_Message;
        END
    ELSE
        BEGIN
            SET @_Step            = COALESCE(@_Step, 'OnError')
            SET @_Message        = COALESCE(@_Message, @_Step)
                                + ' after a total run time of ' + log4.FormatElapsedTime(@_SprocStartTime, NULL, 3)
            SET @_ProgressText  = @_ProgressText + @NEW_LINE + @_Message;
        END

    --!
    --! We call JournalWriter outside of the scope of any transaction handling
    --! to try and ensure that we persist any useful info (this is naturally
    --! dependant on whether the caller has wrapped us in it's own transaction)
    --!
    EXEC log4.JournalWriter
              @Task                = @JobName
            , @FunctionName        = @_FunctionName
            , @StepInFunction    = @_Step
            , @MessageText        = @_Message
            --! Supply all the progress info after we've gone to such trouble to collect it
            , @ExtraInfo        = @_ProgressText

    --! Finally, throw any exception that will be detected by the caller
    IF @_Error > 0 RAISERROR(@_Message, 16, 99);

    SET NOCOUNT OFF;

    --! Return the value of @@ERROR (which will be zero on success)
    RETURN (@_Error);
END

After running multiple steps and appending the outcome of each to @_ProgressText, we must always populate @_Message with some final completion state (be that success or failure) and optionally, the duration of the entire process. Once we have all that information we pass it to log4.JournalWriter where @_Message along with most of the other values get stored in a table called log4.Journal and the value of @_ProgressText gets stored in log4.JournalDetail.

If you’ve got as far as adding all this code into a test database, you can try running the procedure a couple of times like this.  The first execution will generate an error but both attempts will log detailed information.

EXEC [dbo].[TSqlTuesday031LoggingExample]
  @JobName   = 'TSQLTuesday #031'
, @SomeDate  = NULL
GO

DECLARE @date datetime; SET @date = '20020202 22:22:22'
EXEC [dbo].[TSqlTuesday031LoggingExample]
  @JobName                       = 'TSQLTuesday #031'
, @SomeDate                      = @date
, @NumberOfPointlessInsertsToDo  = 12
GO

You can use EXEC log4.JournalReader; to review the most recent Journal entries which might look like this…

This gives us an overview of the tasks that have been logged, the procedures involved in those tasks, the outcome and duration of each procedure call. If we want to look at a particular run in more detail we can query the [JournalDetail] table to access the progress information we captured at each step within the above procedure. For example…

SELECT * FROM log4.JournalDetail WHERE JournalId = 13;

If, like me, you normally view SSMS results in grid mode, you will end up with a result like this…

This is not very helpful as any carriage returns we added to make the log easy to read are not visible in grid mode.  Running the same query with “results to text” would solve this but then I have to remember to keep switching between grid and text mode. A cleaner alternative is to use another Log4TSql module: EXEC log4.JournalPrinter 13; which prints the results out correctly formatted like this:

We can now review the activity and run times for each step within our single procedure call in a nice, user-friendly format.

Incidentally [JournalPrinter] makes use of a stored procedure called [PrintString] which is part of Log4TSql and is designed to overcome the 8000 character limit in SSMS result sets and also the length limititions of the PRINT command in SQL Server. If you want to know more about this I’ve blogged about it here.

I know that this level of logging may be overkill in some situations but in others it can be a real time saver.

Given the choice, the way I prefer to write these monolithic procedures is to break out each logical unit of work into its own procedure – for example the DELETE from and INSERT into a single table.  I like to think of each of these as “workers” and then all the workers get called in sequence by a single “manager” procedure.  If using a pattern like this, you might ask why each worker isn’t responsible for its own logging.  There are two reasons why I don’t do it that way; 1) you again run the risk of losing log information if transactions are rolled back by the manager and 2) the simpler each worker procedure is, the easier it is to maintain and, if necessary, tune.

Just as a side note, if I am using the manager-worker pattern, each worker gets called like this.  Note that the worker sproc always provides the number of rows affected as an output parameter so that the manager sproc can utilise that information in the log record.

SET @_Step = 'Call a pointless sproc';
SET @_StepStartTime = GETDATE();

EXEC dbo.PointlessStoredProcedure @SomeDate, @_RowCount OUT;

SET @_ProgressText  = @_ProgressText + @NEW_LINE
        + 'Step: [' + @_Step + '] processed ' + COALESCE(CAST(@_RowCount AS varchar), 'NULL')
        + ' row(s) in ' + log4.FormatElapsedTime(@_StepStartTime, NULL, 3)

Log4TSql is an open source logging framework for SQL Server 2005+ written by Greg M Lucas on behalf of data-centric solutions ltd and licensed for public use under the GNU Lesser General Public License. You can download the latest version of this library along with all the source code from sourceforge. Please feel free to use and modify the library as you wish under the terms of the license. I am always interested in hearing how other people or organisations have put this framework to use.