Message store tips and tricks.
Message store, storage groups and things related to them account for a large proportion of issues with exchange.
I want to suggest some tips and tricks and things that I do to cover the kind of problems I have seen and may help you, the busy Exchange administrator.
The most common problem is that the message store just un-mounts. Seemingly for no reason. The users start ringing you or come and tap you on the shoulder because outlook has frozen and they cant do any work. And suddenly you go from hero to zero in a heart beat.
So what kind of issues catch you out?
Out of LOG space
Mail floods
Hit the size limits of the Database
Out of Log space.
There are 3 main reasons that you can get into problems with not having enough space for the transaction logs. The primary reason is that the backups havent run for some reason. Exchange aware Backups do not just backup the files to tape. The other function of Exchange aware backups is that they flush the committed transaction logs after they have been archived to tape. Therefore, if the backups do not run, it is possible that the LOG LUN or LOG partition will fill. On Exchange servers that are very busy and have relatively small LOG LUNs, its critical that the Backup runs. Otherwise the LOG LUN runs out of space and the message stores that use that LUN will unmount.
Mail floods can be caused by your own environment. Such as looping messages. Trying to work out if you have lots of looping mail can be difficult. But a good way to check is to turn Diagnostic logging on. This is found in the ESM (Exchange System Manager). Turn on the Mailbox Rules and the Public folder Rules. This will tell you if some rules are running that shouldn't be.
The other thing that can fill the LOG LUN or LOG partition is public folders. If someone creates a new folder tree in Public Folders, this will replicate to all the other Public Folder partners. Again, Diagnostic Logging is your friend here.
Moving mailbox’s within the same server and to another server creates log files. There is a one to one relationship between moved mail and log file generation. What I mean by that is, say you are moving ten 1GB mailbox’s. This will generate 10GB of logs. You have to ensure that you have enough space left to move these mailbox’s within the LOG LUN or LOG partition before you start moving mail.
If you do not have the space within the LOG LUN or LOG partition, consider moving the location of the logs before starting the mailbox moves. Once this move is complete, run a full backup and move the log location back to its original location.
Log files are generated before transactions are committed to the database. The reason for this is that you should never lose any mail. In a disaster recovery situation, you would restore the database from the last full backup and all the transaction logs (from incremental backups) and either use Esutil to replay the logs or just mount the database and let ESE replay them. No part of Exchange uses these log files apart from when there is a recovery needed. During normal running, log files are generated in a one to one fashion. So for every 1024 KB (Exchange 2003, 2007) and 1024 bytes (Exchange 2010) mail transactions, a log file is written to store. That's it in a nutshell.
Hit the size limits of the Database.
Non enterprise versions of Exchange have limits set on how large the message store can grow. And it depends on the version of Exchange and Service Pack installed how big they can grow to. Further, message stores can be manually set to a lower limit to prevent the message store consuming all the Message Store partition and or LUN.
There is nothing to delete here. Moving logs wont help. What needs to be done here is an offline defrag of the database if you have a Message store limit.
There are 2 kinds of defrag for message stores. Online and offline. Exchange defrags the database every day. If you look at the Application eventlogs, you will see event ID 1221. This runs at around 3AM. This will tell you how much white space the database has free. Online defrag does not recover white space. White space as you may have guessed is space that is just empty. The Online defrag just juggles the space to compact the database and does not shrink it. So if the database is over size and it wont mount, you need to perform an offline defrag. And this takes time and space. At least 110% more space. It takes about an hour to defag between 1 - 4GB of database. So for a 75GB message store that's between 75 - 18 hours depending how quick your Exchange server is.
To check to see how much space you will gain by doing an offline defrag, you need to look at event ID 1221. This will tell you how much white space is available. This will give you an idea of how big the message store will be after the defrag.
The other thing you need to do is set the mail retention period down to zero (0). It usually sits around 14 days. If you don’t set this to zero, you may not have enough recoverable white space available to defrag the database as deleted items are not white space. Setting the retention period to zero (0) converts the deleted items into white space. And this may be just enough to allow you to mount the database once defraged.
But running out of LOG LUN and LOG folder space and exceeding the message store size issues are all preventable problems. Having too small a LOG LUN or LOG partition is related to poor sizing. This you should be able to do something about. Exceeding the message store size for non-enterprise versions of exchange is also down to allowing run away mail box sizes. Users should be limited to how much mail they are allowed to retain. I have seen individual mailbox’s that are 70GB in size. To even open outlook took 10 minutes. Let alone do any work with outlook. There should always be an upper limit to individual mailbox sizes.
When I have worked on Exchange systems that are non-enterprise, I have watched the database and event ID 1221 like a hawk. Keeping an eye on these two things will save the problem of them ever un mounting. Also keep an eye on the size of the LOG LUN or partition.
For the Esutil command and all the switches, go to here.
I want to suggest some tips and tricks and things that I do to cover the kind of problems I have seen and may help you, the busy Exchange administrator.
The most common problem is that the message store just un-mounts. Seemingly for no reason. The users start ringing you or come and tap you on the shoulder because outlook has frozen and they cant do any work. And suddenly you go from hero to zero in a heart beat.
So what kind of issues catch you out?
Out of LOG space
Mail floods
Hit the size limits of the Database
Out of Log space.
There are 3 main reasons that you can get into problems with not having enough space for the transaction logs. The primary reason is that the backups havent run for some reason. Exchange aware Backups do not just backup the files to tape. The other function of Exchange aware backups is that they flush the committed transaction logs after they have been archived to tape. Therefore, if the backups do not run, it is possible that the LOG LUN or LOG partition will fill. On Exchange servers that are very busy and have relatively small LOG LUNs, its critical that the Backup runs. Otherwise the LOG LUN runs out of space and the message stores that use that LUN will unmount.
Mail floods can be caused by your own environment. Such as looping messages. Trying to work out if you have lots of looping mail can be difficult. But a good way to check is to turn Diagnostic logging on. This is found in the ESM (Exchange System Manager). Turn on the Mailbox Rules and the Public folder Rules. This will tell you if some rules are running that shouldn't be.
The other thing that can fill the LOG LUN or LOG partition is public folders. If someone creates a new folder tree in Public Folders, this will replicate to all the other Public Folder partners. Again, Diagnostic Logging is your friend here.
Moving mailbox’s within the same server and to another server creates log files. There is a one to one relationship between moved mail and log file generation. What I mean by that is, say you are moving ten 1GB mailbox’s. This will generate 10GB of logs. You have to ensure that you have enough space left to move these mailbox’s within the LOG LUN or LOG partition before you start moving mail.
If you do not have the space within the LOG LUN or LOG partition, consider moving the location of the logs before starting the mailbox moves. Once this move is complete, run a full backup and move the log location back to its original location.
Log files are generated before transactions are committed to the database. The reason for this is that you should never lose any mail. In a disaster recovery situation, you would restore the database from the last full backup and all the transaction logs (from incremental backups) and either use Esutil to replay the logs or just mount the database and let ESE replay them. No part of Exchange uses these log files apart from when there is a recovery needed. During normal running, log files are generated in a one to one fashion. So for every 1024 KB (Exchange 2003, 2007) and 1024 bytes (Exchange 2010) mail transactions, a log file is written to store. That's it in a nutshell.
Hit the size limits of the Database.
Non enterprise versions of Exchange have limits set on how large the message store can grow. And it depends on the version of Exchange and Service Pack installed how big they can grow to. Further, message stores can be manually set to a lower limit to prevent the message store consuming all the Message Store partition and or LUN.
There is nothing to delete here. Moving logs wont help. What needs to be done here is an offline defrag of the database if you have a Message store limit.
There are 2 kinds of defrag for message stores. Online and offline. Exchange defrags the database every day. If you look at the Application eventlogs, you will see event ID 1221. This runs at around 3AM. This will tell you how much white space the database has free. Online defrag does not recover white space. White space as you may have guessed is space that is just empty. The Online defrag just juggles the space to compact the database and does not shrink it. So if the database is over size and it wont mount, you need to perform an offline defrag. And this takes time and space. At least 110% more space. It takes about an hour to defag between 1 - 4GB of database. So for a 75GB message store that's between 75 - 18 hours depending how quick your Exchange server is.
To check to see how much space you will gain by doing an offline defrag, you need to look at event ID 1221. This will tell you how much white space is available. This will give you an idea of how big the message store will be after the defrag.
The other thing you need to do is set the mail retention period down to zero (0). It usually sits around 14 days. If you don’t set this to zero, you may not have enough recoverable white space available to defrag the database as deleted items are not white space. Setting the retention period to zero (0) converts the deleted items into white space. And this may be just enough to allow you to mount the database once defraged.
But running out of LOG LUN and LOG folder space and exceeding the message store size issues are all preventable problems. Having too small a LOG LUN or LOG partition is related to poor sizing. This you should be able to do something about. Exceeding the message store size for non-enterprise versions of exchange is also down to allowing run away mail box sizes. Users should be limited to how much mail they are allowed to retain. I have seen individual mailbox’s that are 70GB in size. To even open outlook took 10 minutes. Let alone do any work with outlook. There should always be an upper limit to individual mailbox sizes.
When I have worked on Exchange systems that are non-enterprise, I have watched the database and event ID 1221 like a hawk. Keeping an eye on these two things will save the problem of them ever un mounting. Also keep an eye on the size of the LOG LUN or partition.
For the Esutil command and all the switches, go to here.
Comments
Post a Comment