Permissions, permissions, permissions

This blogpost is about disk permissions and asp.net applications like DotNetNuke. Although there are probably many posts like this I write this because permissions, or more precisely the lack of them, are the root cause of many support requests. And a little knowledge is all that would have been needed to avoid the situation.

Background: the worker process and its app pool

An asp.net application is just that: an application. It is a program running on your server calculating what HTML to spew out to the requesting browser. This is quite different from serving html in the classic way. There the server just streams files from your server’s hard disk over the wire. No processing as such. Just “serving”. But asp.net applications (and their predecessor ASP and alternatives such as PHP) require the server to process instructions of a program that will tell it what to send to the client. This has given us the data driven web site and the whole Web 2.0 revolution.

To do this processing the server starts up the so-called worker process. You can actually see this process in action if you’re on the server which is serving asp.net pages and bring up the Task Manager and flip to the “Processes” tab:

They’re called “w3wp.exe” (for WWW Worker Process) and in this screenshot you can see two of them. By the way: this is a really useful place to check the health of your site. It is probably always my first stop. You can quickly see if the worker process is clipping the CPU (i.e. it’s at 99% CPU) meaning it is consuming all processing power of your server, or if it is consuming a lot of memory (usually not a reason to panic).

The worker process encapsulates the code that is running for asp.net. Because the server is starting this up autonomously based on http requests, this process is boxed into an App Pool. The app pools are kept in a list in your IIS manager:

The app pools govern how much the worker process can consume from the server like in terms of memory for instance (lest they complete consume the server and bring it down). The app pool, therefore, protects the server from potentially hazardous code in the asp.net application. The app pool also gives the worker process its identity. It is the latter we are concerned with here but we’ll come back to this in a bit. What you need to know at this point is the existence of the worker process and the app pool, the relationship between the two and where you can find them.

Windows disk permissions

This should not be a big mystery. All items on your server’s hard disk (folders, files) have a set of permissions that commonly inherit down the folder/file tree. Right click on any folder on your server and select properties and you get a popup of that folder’s properties with a “Security” tab. Go to this tab and you’ll see something like this:

The concept is simple here. You can specify users and roles with the level of access to this resource. So giving someone Read but not Write access will allow them to see and download/open documents, but not change those documents or upload new ones. This paradigm is pervasive in computing. What you may not realize is that when the worker process (i.e. the asp.net code) accesses the hard disk it needs to have a “face” for the system to authenticate. It’s “identity” needs to have the right access level to what it wants to do on disk. The way that is implemented is through the identity of the aforementioned App Pool that is responsible for taming the worker process.

App Pool Identity

The app pool a worker process belongs to specifies which identity it will assume when it goes to Windows. Check out the advanced settings in IIS Manager for an App Pool:

So this app pool runs as “NetworkService” when accessing the disk. Note that the default for this depends on the IIS flavour. Commonly you’ll see “NETWORK SERVICE”. What is important to keep in mind though is that these default accounts are local to your server. I.e. NETWORK SERVICE on server SERV-A in your network is a different account than NETWORK SERVICE on SERV-B on your network. Even though they have exactly the same name. Well, actually their full names would be “SERV-A\NETWORK SERVICE” and “SERV-B\NETWORK SERVICE” but we don’t always see it like that in the UI. Keep this in mind!

The bottom line is that your app pool should have access to the files of your ASP.NET application. And write access as well in the case of an app like DotNetNuke as it is able to install extensions (modules, skins etc) which means writing files to the application’s file system. Let’s examine that in more detail.

DotNetNuke and its Installer

DotNetNuke’s most powerful feature is its ability to install and upgrade extensions which are uploaded as zip files. DNN’s installer unzips those and, based on the information presented in an XML manifest, it begins to write the contents of that zip file to the server’s hard disk. It will write files only inside it’s own application’s directory. Never beyond that. And that’s good. You want the server to encapsulate the application and not have the risk of some rogue module writing executable code to your Windows files. The installer typically writes ascx files to a folder it will create under DesktopModules. These files are template files that tell the server how to flow the HTML. Next to these ascx files it will also typically write one or more dll files to the bin folder. This folder is the most important folder of your asp.net’s application. It is where the compiled code sits. Waiting to be invoked by the worker process as the ascxs tell it to go to them and do some fancy processing.

The installer performs another very important function which will have some bearing on the problems we will describe below. That is the running of SQL scripts. The module programmer packages the zip file with a bunch of scripts that prepare the database so that the application may do its work. These scripts are incremental. That is: they are run in order and during an upgrade DNN’s installer works out which scripts it needs to run to upgrade the module. Keep this nugget of info in mind as we examine the common failure scenarios.

What you need to know by now is that the installer writes files to disk and very importantly to the bin folder of the DotNetNuke application.

Common Failure Scenarios

The focus of this post is on errors in disk permissions. These permissions can lead to serious malfunctions in your application.

After Server Migration

Servers die. Or you need a bigger/better etc server. There comes a moment in the life of your website that you’ll need to move it. So you (x)copy over the files of the asp.net app, register the app in IIS and away you go. But this is where the problem begins. The files with FULL permissions for “SERV-A\NETWORK SERVICE” are now running on SERV-B. And the asp.net app is now running as “SERV-B\NETWORK SERVICE” although all you saw was “NETWORK SERVICE”. If you’re lucky (or not depending on how you look at it) your site will still have read access for the new identity and the site will fire up correctly. But what will now happen when we try to install, or worse upgrade, a module? Check below.

After Restored Backup

Murphy’s law has it that a site will only die when we haven’t got a backup. But just in case you do have one please read this. Your backup might not have the full permission set of the files stored with it. It all depends a bit on your backup solution. What is also common is a combination of this and the above. I.e. a server has died and we restore to a new server from our last backup (pfew). Again what you need to keep in mind is that the permission set may not be right for the worker process’ identity.

Common Failure Effects, Symptoms and Solutions

These are common results from mishaps after the above.

Module Install Failure

Keeping in mind the application is unable to write to one or more places of the application’s file system it is easy to see that the installer would run scripts but not be able to write files. Although the installer is pretty smart this is still something that can happen and I’ve had customers report in with installation problems resulting from this. The biggest problem here is that DNN’s installer will have already run the SQL scripts to install the module but the actual installation failed. Now, when the user tries to install again the installer will re-run the scripts leading to all kinds of spectacular SQL failures as the SQL objects are already there that the installer should be creating. This is messy but ultimately easily solvable. Manually rip out all objects related to the module from SQL and correct the disk permissions then install again.

The main reason this scenario is not that dangerous is that (1) modules are usually (if well programmed) encapsulated so they will not damage other bits of the system and (2) there is no user data yet as we are just installing. So you can afford to horse around with the bits that came with the module provided you don’t touch the rest of the application.

Module Upgrade Failure

This is the real nightmare scenario. And it is not uncommon. This is also the main reason that I advocate making a backup of your installation before you upgrade a module. Why is this scenario so bad? Let me explain. The installer will begin to upgrade the database objects of the module first. At this time the data may well be transformed (columns being deleted, added and data being moved to other tables for instance). In short this is typically irreversible. DNN has no concept of uninstall largely due to the complexity of this bit. The scripts run first and then the file copying begins. So now the server has to begin adding files, overwriting files and possibly deleting files. Presuming this has become impossible due to a permissions issue, we are now left with module code (in the bin folder for instance) that thinks it is version A and the database tables thinking we are now version B. The installer will bomb out of the install process with “cannot overwrite file XYZ” or “cannot access ABC” and it won’t register in DNN that the module updated. Even though the scripts have run just fine.

The effects are generally that the module will fail spectacularly with messages of the kind “Column XYZ could not be found in table ABC” etc. The error is easy to spot but harder to solve. In fact you are going to have to do one of two things. You can mimic the DNN installer and continue where it left off. But for that you need a lot of DNN knowledge. In short you’ll need to (1) copy the files from the zip to their correct destination, (2) update the versions in Packages and DesktopModules tables and (3) update the module definitions if there are any changes. The latter can be quite elaborate work. And then we don’t know if upgrade code should be run from within the module. If it does then … well, we’re stuck. But this is relatively rare, as are changes to the module definitions, so just doing 1 and 2 above might do the trick.

But really the best way out of this is to roll back to the last backup. As you followed my instructions you have a backup ready to be restored. And then you just set the permissions correctly and try the upgrade again.

Internal Module Failures

Many modules use the file system. Certainly my Document Exchange does so. So having permissions on disk can be crucial to the operation of these modules. Keep this in mind. Typically you’ll see error messages starting with “Could not find a part of the path” or “Could not access”.

Setting Disk Permissions

So how to set these permissions correctly?

1. Look up the identity of the worker process in the app pool settings
2. Open up the folder of your DNN installation and select properties and go to the “Security” tab
3. Click “Edit …”
4. Click “Add …”
5. Type in the name you looked up under 1 (you may need to Google on alternatives at this point if you are still not having luck)
6. Make sure to transfer these new permissions to all children
7. Verify a couple of levels down on a random file. Are permissions as they should?

Final Note

First and foremost I am a programmer and not a Windows administrator. What is in this post is the result of me having to solve installation issues over the years. I will not claim this is the definitive guide to asp.net permissions debugging. But it is a good overview of the causes of a number of issues I’ve dealt with in DNN/DMX support. And it is not uncommon unfortunately. Though the above may at times be considered common knowledge, I know it is not. And I also know it is hard to explain in a paragraph or two. Hence this post so I can point you here.