Hi all,

Just a very, very quick post because I’m still at work

This very short script will loop through all of your (user created) tables and return the number of rows and space used. Very useful indeed.

SET NOCOUNT OFF

DECLARE @tblName VARCHAR(255)
DECLARE tblCursor CURSOR FOR
SELECT name FROM sysobjects WHERE type = ‘U’ ORDER BY name

OPEN tblCursor

FETCH NEXT FROM tblCursor
INTO @tblName

WHILE @@FETCH_STATUS = 0
BEGIN
  
   EXEC sp_spaceused @tblName

   FETCH NEXT FROM tblCursor
   INTO @tblName
END

CLOSE tblCursor
DEALLOCATE tblCursor

Enjoy

PD

Hi.

Have you ever set off a profiler trace and then realised that you’ve amassed tons and tons of data and now you need to start searching for that needle in a haystack?

If the answer to this question is yes then you might be interested in a pretty cool SQL function that lets you read the trace files and, even better, import them into a table.

Consider the following SQL statement:

FROM ::fn_trace_gettable(‘D:\DBAScripts\output\sql_trace_file.trc’, default);

This will import your SQL trace files into a table, after which you can add indexes to optimise the numerous queries you will no doubt have to run against the data.

Let me just break down the syntax of that statement to try and give you a better understanding of how you can best utilise this feature.

The SELECT * INTO is straight-forward stuff so I won’t bother with that bit, let’s just concentrate on the function syntax.

::fn_trace_gettable is the actual name of the function we are using. The two parameters within the parenthesis basically specify the full path and file name and the number of trace files you want to import. By specifying default you are effectively saying that you want all of the trace files that have been created under that name. Multiple trace files of the same name will be suffixed with an underscore followed by a number, for example, a trace file called sql_trace.trc may have rolled over a number of times to create sql_trace_1.trc, sql_trace_2.trc etc. If you only want to import the first 5 trace files then just specify the number 5 instead of ‘default’. Simple.

So as you can see, you have a pretty handy little function that can import all of that trace data into a SQL table which you can then slice and dice until your heart is content.

Enjoy

PD

Hi there.

Have you ever had that annoying problem where you’ve created a cluster resource of type “share” (typically a shared folder on one of your SAN drives) that just keeps failing? If I had to pick a scenario I would chose log shipping from one cluster (Production) to another (Disaster recovery). So, you have your shared folder that contains your log backups but the resource keeps failing. Even worse, you’ve forgotten to un-check that box that affects the cluster group so that when the share fails it brings your entire group down? Yup, been there, done that. And if it’s a share that SQL needs access to then the chances are that you’ve put this share in your SQL cluster group and you’ve got a yo-yo’ing SQL Server service all because of a failing share. This would probably seem like an appropriate time to update your cv

Well this is your “get out of jail free” card.

First off, clear that check-box that reads “Affect the cluster group”. If your cluster share fails it’s really not that big a deal. Certainly not big enough to fail-over your cluster.

Secondly, I have a fix for you. You basically need to add in a key to your registry (ON BOTH NODES!! ) and here it is.

The registry path is HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\lanmanserver\parameters and you basically need to create a key called IRPStackSize and set the value to HEX 12.

That’s it. Simple.

Reboot the node (at a convenient time) and hey presto, no more failing service.

Remember to repeat on both nodes.

Enjoy

PD

You’ve no doubt seen the security bulletins issued by Microsoft over the last few months and if, like me, you’ve had to apply them to your SQL Server estate you may well have had some cause for confusion.

I’ve been testing, re-testing and testing just a little bit more, all of the KB’s that I have to deploy and I’ve started rolling them out onto our production boxes. But how do you know which KB you need to apply to your SQL instance? Well it is actually explained on the Microsoft site but I’ve got a very quick “at a glance” version, which is the reason for this post.

This is my crib-sheet for security bulletin MS08-040

MS08-052 Security bulletin crib sheet to follow soon….

Enjoy

PD

This is a handy little function that will decrypt the owner_id from the msdb sysjobs table. Just wanted to post it very quickly because I’ve just had to search for it yet again.

SELECT name, suser_sname(owner_sid) as owner
FROM
sysjobs

Enjoy

PD

OK, here’s my first technical post and at the time of writing this, the issue is still not resolved, but I do have a hunch. Probably from spending too much time with a laptop perched on my knees but seriously, no really, I do have a theory on what the issue might be and that is exactly what I’m going to blog. If you are reading this and I haven’t posted an update and you have first hand experience of this issue, please drop me a line to either confirm or dispute my theory, I will be more than glad to hear from you. What ever happens, I am planning on applying the change on Saturday 4th October (yes I work in a tightly controlled organisation) and I will post my update after that date.

The Backgound

We have a windows cluster in an active/passive configuration (or whatever the new terminology is these days - I recently attended a course to prepare me for my MCITP where active/passive has now been replaced with what I considered to be an ambiguous term, so much so in fact that I have blanked it from memory – sorry digression is a major problem of mine) containing two resource groups, one for the Windows cluster resources and one for the SQL Server cluster resources. Straight-forward stuff. In the SQL group we have three disks, E:, F: and G: – they are LUN’s (SAN disks) that have been allocated to this cluster. I’m not going to divulge too much detail on the placement of what data file lives where, it’s largely irrelevant, it works and it’ll likely detract from the real issue.

The service (I’m using this term in the business sense) started to outgrow it’s home, rapidly, so extra LUN’s were allocated to the server to try and head-off a potential disaster. The disk letters were assigned, they were added as physical disks in the SQL cluster group and I added them as dependencies of the SQL Server cluster resource. Bingo, we were good to go on the new disks. I then set about adding some new datafiles to the filegroup which were physically placed on the new disks and we were off again, safe in the knowledge that I wasn’t going to get called out that night for a database that had hung because there was nowhere left for it to grow. Now that is not a nice wake-up call I can tell. Mind you, is there ever a nice wake up call when a server is involved? You don’t need to think about the answer, I can tell you now that the answer is no, there isn’t. If I’m getting called out by the command centre it’s not to wake me up and tell me that everything is rosey. Digression over, back to the post. Blah, blah, blah, new data files on new disks, great, everything was looking sweet. It didn’t last long.

The Errors, Oh The Errors

I’d never seen this type of failure before. In very basic terms, the SQL Server service crashes but doesn’t actually fail over to the second (passive) node, it simply dies and restarts on the active node and it all happens in the blink of any eye and without warning.

The Investigation

I trawled through the (Application) Event Viewer logs and sure enough there were a whole bunch of errors, all at the exact moment of failure and all having the value MSSQL$PC001 in the source column.

The Event ID is 17052, the category is (3) and the error text of the earliest error is as follows -

[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed

The detail of the failures are exactly the same, there are always 14 errors, always at exactly the same time, all for MSSQL$PC001 and they all start with the error message quoted above.

For completeness, these are the other error messages as they appear in chronological order. I say chronological order but in fact they all have the exact same timestamp -

[sqsrvres] printODBCError: sqlstate = 01000; native error = 2746; message = [Microsoft][ODBC SQL Server Driver][DBNETLIB]ConnectionWrite (send()).
[sqsrvres] printODBCError: sqlstate = 08S01; native error = b; message = [Microsoft][ODBC SQL Server Driver][DBNETLIB]General network error. Check your network documentation.
[sqsrvres] OnlineThread: QP is not online.
[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][ODBC SQL Server Driver]Communication link failure
[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][ODBC SQL Server Driver]Communication link failure
[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][ODBC SQL Server Driver]Communication link failure
[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][ODBC SQL Server Driver]Communication link failure
[sqsrvres] CheckQueryProcessorAlive: sqlexecdirect failed
[sqsrvres] printODBCError: sqlstate = 08S01; native error = 0; message = [Microsoft][ODBC SQL Server Driver]Communication link failure

The Action

The very first thing we did was to look at what had changed on this service recently and the two things that stuck out were the new disks and the new datafiles.

Nothing is ever straightforward and it’s not until you start scratching under the surface that you start to realised you should never take anything at face value. This point will become relevent later.

We updated the drivers for the HBA cards (SAN fabric cards) on the two nodes to the very latest versions, as they were way out of support and the errors were leading us down the “it just loses it’s connection” path. Initially it looked as if this was the solution but after a week of service stability we started to see a reoccurrence of the same issue.

As it happened, more by total accident than design,  we had to undertake a very similar exercise for another service, new disks and new data files on those new disks and sure enough we started seeing the exact same bunch (again, 14 in total) of error messages. Too much of a coincidence? I think not.

Now we’ve added tons of disks in the past and I’ve lost count of the number of times I’ve added new data files to filegroups and stuck them on new disks so what was different?

Well, this is where my theory comes in to play. It has neither been confirmed or rubbished by the storage team so I’m chucking it out there, into the wider community, for comment

The Theory

Well, the disks that were presented to our server were not of the same type as the previous ones, they weren’t even on the same SAN fabric. We’ve recently implemented a brand new SAN along with new disk arrays (from a completely different vendor) and have started rolling this out to all new production systems. Where this new SAN is attached to our SQL Servers, and this type of SAN alone, we are seeing no issues. Where the old SAN is attached to our SQL Servers along with the new SAN we are seeing no issues, it’s only when you have multiple data files in a filegroup and those files have been split over the two different SAN’s that you get the problem.

For details sake, I have listed the (from Disk Management) the details of the LUN’s below -

As you can plainly see, we have two types of disk and we have got datafiles (for the PRIMARY filegroup) spread over those disks and different SAN fabric.

So my theory, and the whole reason for this post is this – if you are sending I/O over two different types of SAN then you are likely to encounter these issues. I can’t ignore what I consider to be hard evidence so I’ve managed to convince myself that this is the root cause of the issue. Time will tell.

The Fix

I’ve got my change scheduled for Saturday 4th October, starting at 23:00 so whether or not this fixes the issue or not will only really become apparent after that date and even then I’m going to have to hold my breath for a few days. It’s so intermittent and unpredicatable it’s just not possible to pop open the champagne corks until, well, I don’t exactly know. I’d just better keep the bubbly on ice I guess.

So anyway, the proposed fix, well it’s actually very simple.

I’m going to move all of the data files (for the PRIMARY filegroup) from the RDAC virtual disks to the IBM disks and thus forcing the I/O down one SAN only. It turns out that we have plenty of spare capacity on the IBM disks to cater for all of the data files (and log files and tempdb files), it just means that, me being me, I can’t split them over multiple LUN’s as I normally like to do. Not that splitting them over multiple LUN’s in anyway guarantees that you’ve just put your data files, log files, tempdb files, other filegroup files on separate disks which is very, very nice indeed thankyou very much. Oh no. To have that level of reassurance you need to find out if your LUN’s are coming off separate disk arrays, if each LUN is in effect on a completely spindle to all of the other LUN’s and that’s where you need to have a good relationship with your storage team. But that’s a whole new post.

I’ll be posting an update to this, post-change, to let you know how it’s going. Hope you got through it ok, if you have any comments please let me have them, I’d be glad of some feedback.

Enjoy

PD

Hello.

How many chances do you get to make a good impression. One? Two? Well, this is my first ever attempt at writing a blog (ever, ever, ever!!) and I’m hoping that the first impression you have will be a good one. Thankfully, in CyberSpace, nobody can hear you scream.

This blog is mainly for my benefit because as I am getting older I forget the stuff that I need to remember and I remember the stuff that I need to forget.

So, with the pleasantries out of the way, let me just explain what I’m hoping to throw up. In short it’s going to be techie, nerdy stuff. More specifically, Microsoft SQL Server techie, nerdy stuff. Still there? Now please don’t shoot me down in flames for my affiliation with Microsoft, if you are a Microsoft hater then log-off. Thanks. I happen to think that MS SQL Server is an excellent product and this is coming from someone who started out working with Oracle. OK, that’s not quite true, my first database love affair was with Sybase System 10 back in the heady days of 1995, but I digress. Where was I? SQL Server, yes, great product and getting better with every major release. So now you know where I’m coming from, all that remains is for me to start brain dumping before the grey matter between my ears turns to total mush.

Enjoy.

PD