We run a specific environment where all of our resources must be in Medium trust. We do have the ability to add things to the trust to make our own custom trust within reason though.

We've worked through all of the other items relating to trust levels with DMX and Medium trust except for search indexing. The one current production system we have in place using it is using the crazy process that is getting the MS Indexer working on the SQL server that is a different server than the IIS server. On our other production system we have been unable to get this setup working at all for unknown reasons so we decided to give Lucene a try again. From reading old threads I know it didn't work under Medium in the past but these were 3+ year old threads.

Mid 2009 the Lucene developers fixed the Medium trust issues associated with the application. This stackoverflow.com post sums it up best.
http://stackoverflow.com/...-net-in-medium-trust

However DMX is using Lucene 2.0 where a minimum of 2.3.2 is required (2.9.2 is the latest version as of this month).
http://www.inner-ninja.com/gallery/var/albums/miscalbums/Screen-Shots/DMX/Lucene_Versions.PNG

Will it be possible to upgrade DMX to use the newer versions to finally allow us a simple indexing process while running Medium trust levels?

Hi Leith, I asked this same question recently amongst other search related questions:

http://www.bring2mind.net...v/topic/Default.aspx

The answer at that point was still "no Medium Trust". I feel your pain.

So, it would be great if the Lucene Search provider worked in Medium Trust for DMX. It would give me more options as well.

Rob

We've got 2 licensed copies installed now with a planned 3rd coming soon. The system I'm working on now is required by Gov policy to be Medium or lower so it's not an options. I believe some of the other policy items are what's stopping that system from using the remote SQL server as they are not in a domain and have extremely strict policies between the machines.

We've worked through all other Medium trust issues, this is the last one stopping our systems from being fully operational.

Hi Leith,

I will check out the new Lucene. But there is a trust issue unlreated to Lucene. Lucene does "indexing". It takes bunches of text and meta-data and answers queries as fast as it can. It does, however, *not* know how to read a document. That is left to the programmer. In the case of a text document this is not really an issue. I (i.e. DMX) can read the text from the file and feed it to Lucene. It becomes more difficult with specific binary formats. These include MS Office and PDF (to name the most frequently used). Neither DMX, nor Lucene are able to read these. Instead, the code relies on the server's iFilters to "open" the document and get the textual content (this is essentially what an iFilter is all about).

In short: in medium trust you'll probably find that the iFilters cannot be called by the .net code. This effectively blocks content indexing. The reason Indexing Service works is that this runs outside .net and therefore at a higher trust level. What you can do to test is to upload a simple text document and give it a specific keyword (e.g. "foobar"). Then search on this document through the keyword (guaranteed to not pass through the iFilter and straight into Lucene) and on contents. What do you see? Now try it with a Word document. MS Office iFilters normally come with Windows pre-installed. What happens now? That test should give you an idea of what you're up against.

Peter

Thanks for that explanation, Peter. I had forgotten about the iFilter components in the mix to complicate matters.

Rob

Hmmm, iFilter reliant content is the bulk of our files. Is it then possible to run the portion that triggers the Lucene that uses the iFilters that does the indexing to be called from either a standalone executable or a separate website?

With our own .net apps all of your scheduled tasks actually run on a website that is independent of all of the other websites. This allows us to then lock that website down to only the machine for access allowing us to then run the website in High or Full because it is no longer externally exposed.

I know index content generated from one website can be then copied into the folder of another Lucene system and read back in from there. We're actually doing that already for a CAC/nonCAC enabled DNN we run. I would rather not setup an entire 3rd DNN website though to just do indexing from.

Hi Leith,

Yes, that might be possible in a custom search provider. You're free to do as you wish with that. All you need to do is to fill in the provider methods like AddDocument, Search, etc. I can't speculate on what might or might not work in this case unfortunately.

Peter

Hi Leith,

In your original post you stated " On our other production system we have been unable to get this setup working at all for unknown reasons..."

I also use MS Indexing running on a separate SQL Server from the Web Server. I am very curious as to the particulars of the production system where you are not able to get this to work. Would you be willing to share any of the details?

Thanks,

Rob Ralston
SilverBullet Technologies LLC

[quote]
Posted By Peter Donker on 06 Apr 2011 23:02:01
Hi Leith,

Yes, that might be possible in a custom search provider. You're free to do as you wish with that. All you need to do is to fill in the provider methods like AddDocument, Search, etc. I can't speculate on what might or might not work in this case unfortunately.

Peter
[/quote]

We do have Solr running which is based off of Lucene in our newer ColdFusion9 servers or I could also run it as a standalone. Where can I find documentation/examples on configuring the external indexer like this?

Also since it uses Lucene I looked at the index stored files and it uses the same layout as how DMX is storing its files as well. For giggles I'm going to drop the index files in the DMX index folder that get created by Solr and see what happens.


[quote]
Posted By Rob Ralston on 07 Apr 2011 13:41:46
Hi Leith,

In your original post you stated " On our other production system we have been unable to get this setup working at all for unknown reasons..."

I also use MS Indexing running on a separate SQL Server from the Web Server. I am very curious as to the particulars of the production system where you are not able to get this to work. Would you be willing to share any of the details?

Thanks,

Rob Ralston
SilverBullet Technologies LLC
[/quote]

I can't go into the details of our setup but I can point you to the documentation used to configure our setups. At a minimum the Windows Server, .NET, IIS, & SQL STIGs.

http://iase.disa.mil/stigs/checklist/

That's odd the quotes didn't translate correctly...

Hi Leith, that's quite a set of checklists. Thanks for that link.

What I really am most interested in are the versions of those items, nothing more. The checklists cover various versions. I am trying to anticpate where I might have problems as I migrate from W2K3 systems.

Windows OS for IIS Server
Windows OS for SQL Server
SQL Server version
.NET

If you cannot post here, perhaps you could post information in the "Contact Us" page on my site, which would make it pretty anonymous.

Thanks, if you can do that.

Rob

https://www.silverbullettech.com/

The systems in question are Win 2003 R2 for all systems, MS-SQL 2005 for the SQL instances, and we're running .NET 2/3.5 for the latest DNN installation.

Our system that does not require the crazy lock down is working with the same type of setup. However I can not get our newer Win 2008 R2 + MS-SQL 2005 or MS-SQL 2008 R2 to work cross indexing at all either. This as well is why I'm looking for an alternative indexing solution to solve both Medium Trust issues and the bizarre SQL server indexing config.

Leith, thanks for that information.

I take security very seriously, but I'm sure not to the extreme of the checklists you have to deal with, so I find this problem of great interest. I only use Medium Trust on web servers, with some very careful fine tuning of custom trust files when required.

Would you be willing to take this discussion off line, as I would like to learn more about the problematic configurations and any thoughts you have on why MS Indexing from the remote SQL Server does not work. If you cannot, I completely respect that.

If you can, you could contact me via rralston AT silverbullettech DOT com.

Thanks,

Rob Ralston

Peter,

So lets say I wanted to use Solr which is the official indexer built built on Lucene, how would I go about telling DMX to use the catalogs it creates?

For giggles I'm also trying to copy the Solr index files into DMX to see what happens, this might require the newer version of Lucene in DMX though.

Rob, I've been slammed with work right now but when I get a chance I'll send you some feedback.

While waiting/trying other solutions I've decided to go back and try to figure out why the SQL linked Index Service is not working.

From the SQL server I can successfully run the following command and it returns all of the documents I expect to see indexed from the linked catalog.
SELECT * FROM OpenQuery(MyDMX, 'Select Directory, FileName, Size FROM SCOPE()')

I then ran the following command and was able to get back the contents from one of the test documents. I ran several words and got back references to documents I expected.
SELECT * FROM OpenQuery(MyDMX, 'SELECT DocAuthor, size DocTitle, write FROM SCOPE() WHERE CONTAINS (Contents, ''test'')')

I have our setup as near to this as possible. The only real difference is that the files are on the same server as the IIS server and not a share and we're running Win 2008 R2 as the SQL Server.
https://www.silverbullettech.com/Tech_Blog/EntryId/5/So-you-want-DMX-to-use-MS-Index-Service-on-the-SQL-Server.aspx

The SQL connections are functioning with no issue at all from the DNN connection and is an owner of the DB as well as being a sysadmin for testing purposes. The system that is working actually uses a fairly locked down SQL user account.

I then proceeded to make a basic ASPX page that connects and executes the SQL commands above and I was successfully able to return data from the script that was running inside of the same DNN website as the DMX that wont run using the same web.config settings.

I was just informed our functioning system is running DMX 5.2.11 and the ones that are not are running 5.3.7, I'm going to try removing and installing the older version to one of the test sites to see if it fixes the issue.

Nope, it's not working in that version either it seems... I wiped out the index and started over and brought in new files. I can successfully get the data about those files from my test page within DNN still now that they've been indexed.

Of note the system that is workign is running 5.2.11 on DNN 5.5.0, the newer systems that are failing are running DNN 5.6.2 currently.

Hi Leith,

Contact me by email for the code of the Lucene provider. I'll be happy to share it with you.

Peter

Ok, more digging and digging and help from Peter I think we've finally resolved the issues with the remote MS-SQL Server and Windows Indexing Service working reliably. Using the following additional steps we were able to get DMX reliably returning data from the remote Index on 3 development websites and our 2 production websites.

* You must be running DMX 5.3.x at a minimum since it adds the remote folder path to the Index settings section

* Your remote file location must have a network share and not a full UNC file path (\\Server\DMX$ and not \\Server\e$\DNN\Portals\0\...) The Indexing Service has no issues with this but we found this stopped DMX/DNN from working correctly when passing the data to the MS-SQL server to limit returns.

* We found that even with these values correctly set and the other guides followed some installs would not return data. If this is the case we found running the following scripts fixed this. (This seemed to be an issue on portals that had DMX uninstalled at some point previously.)
- Prepare Portal
- Repair Permissions
- Repair Structure
- Reindex Installation