I was lucky enough to attend the latest IISYG (Independent Information Security Group) meeting hosted by Iain Sutherland, Managing Director of Information Security Solutions, this was my first attendance, up till now I had lacked either the experience or expertise to attend and now I see why

I am limited on what I can discuss as I signed Chatham House rules, but I did want to jot down my experiences.

Normally small discussion groups like this (about 20 – 25) have their fair share of people who just talk and don’t get anywhere or those with a lack of experience but don’t realise it, I found neither of these here, it was a gathering of experts and I found that I was too wrapped in following the discussions and the huge number of valid points raised, to venture any of my own publicly (although I did shuffle up to one the presenter and have a chat afterwards) , the presentation on the nature of risk transference as how it is pertains to companies and the attempt to mitigate the problem with insurance was particularly fascinating, now I know what you are thinking, but genuinely it was, it showed you the mind process of insurance underwriters and how you should present your security risk to them, what loopholes they will use in your policy and how to watch out for them. a true ‘how to’ guide.

The debate section of the meeting initially sounded quite dry but that only turned out to be due to its government title, it dealt with the opposing sides of the question of government security standards, these standards deal with the certification of the people who get to state if the IT systems we entrust our governmental data to are secure, bearing in mind that this goes from simple personal data such as voting registers all the way up the most secure of state secrets, it is something you really want to get right, opinion varied hugely and dealt with the problem from all sides, if the official speakers took away half as much as I did them it was well worth doing..

All in all a fascinating morning.

Now I try not to do too many ASW posts as the rest of LDC frown on them, so believe me when i say that this is not one of them, this is a genuine plea to IBM.

Please fix the NSF database

and by ‘Fix’ i mean:

1) Remove the obvious size limitations such as 32k view lookups, maximum field numbers and plain text field limits
2) Provide a good JDBC and .Net Drivers that enables access to the database from remote locations
3) Improve scaling (yes i know that it has huge theoretical size abilities but the performance degrades a lot when you add lots of documents or lots of views )
4) Externalise the View indexes
5) Provide Sharding in Clusters (optional)

The NSF has done us proud over the years and there have been lots of small improvements with each ODS structure update but it’s still not what it should be (too many size limitations and performance problems), and this is the right time to fix it!!

WHY?

1) NoSQL is a growing product platform, estimated to be worth on par with the relational market within about 5 years.
2) You already have a hell of a lot of experience in it.
3) You have a existing market for it (Notes apps), which it will help revitalise and it’s one of the final components in your ongoing “X” cause.
4) It will make money, if Oracle are busy building a proper NoSQL db platform then you know its a good idea. I know quite a few companies that would jump at the change of using a NoSQL db from a trusted vendor.
5) The market is still young, you have time!!

You don’t have to link it to Notes directly if you don’t want to, stick it on the XWork server and call it XNSF or something if you feel like it (as long as Notes apps can access it). Yes I know you have recently given us good relational access, but relational is starting to look a bit old hat for a lot of webscale apps, we NEED a good NoSQL., CouchDb has shown you the way, and MongoDb has taken the ball and is running with it giving developer and businesses what they want, these 2 are the leaders in the document based NoSQL world.

I’m going to pester the managers at LS2012, but if you see me stand up at the “meet the product managers” its because you have fobbed me off and I want you to see how much this would give your developers and community. you can find my ideaJam entry here.

Please!

Old Comments

Erik Brooks(04/11/2011 18:34:26 GMT)

As IBM has touted for the past couple of years, NSF is their strategic/NoSQL datastore.

Since ODS changes only occur on a major version, 9.0 is do-or-die time.

It’s time for IBM to step it up.

Mark Myers(04/11/2011 15:05:16 GMT)

@richard, good points and thanks for the vote, i agree its a monster change but if companies like Oracle are willing to put the effort in they there should be enough value for IBM to do it, or at the least provide an alternative NOSQL db format

Mark Myers(04/11/2011 16:29:35 GMT)

@Dave, you are not alone in this wish

Richard Moy(04/11/2011 14:54:06 GMT)

Mark,

I will add my vote to ideajam of yes. I would love to see these changes. But, what you are asking are major changes to the NSF format and I don’t see IBM investing the money especially with their big push towards Connections.

What you might consider is getting away from the entire approach of how data is store in an NSF document and treat it only as a data container. This reduces the number of views significantly and unifies your data which will increase the performance. However, if you are using XPages to access the data than you are out of luck.

Mark Myers(04/11/2011 21:14:58 GMT)

@Erik – well said that man

@Henning – you have a lot of good points (really good points), Cassandra has a slightly different use case than mongodb/couchdb it being classed as a bigdata NoSQL as opposed to DocStore NoSQL (the other major types being Graph and Key-Value) so i can understand IBM using it,

Stephan H. Wissel(04/11/2011 16:15:13 GMT)

… and it actually wouldn’t be too hard to create a data sourcve for {insert-your-favorite-noSQL-here} to be used with XPages

Henning Heinz(04/11/2011 19:19:29 GMT)

So IBM is implementing nsf into DB2, see { Link } ?
NSF is the datastore for Notes and Domino. Nothing more, nothing less but imho far from being anything strategic within IBM. IBM has been playing with Cassandra (and others) in IBM research and they currently use Apache Hadoop in their BigInsights products.
Still a renovation of nsf would be welcomed. Although when IBM is able to implement NoSQL into DB2 then why not embed a small SQL engine into nsf for easy querying / reporting!?
After the nsfdb2 experiment there even had been a promise to implement some of the lost functionality directly into nsf.

Dave Harris(04/11/2011 16:23:43 GMT)

Not a developer, but just one of those points would make my life as an admin better: increase the 32k lookup limit.

If you run Domino as a perimeter mail solution, running your own blacklists is pretty much impossible, simply because of this, unless you want to run RBLs on your internal DNS servers. And when you’re working on a global level, customers tend not to go for that.

Given the prevalence of 64bit OSes now, removing this limit must surely be possible, even if it needs to be caveated.

Mark Myers(04/11/2011 16:19:13 GMT)

@stephen, that is very very true, and it would be a solution that someone like you could write, or we could just install mongodb and use the java driver to do that, but im thinking more for the growth of the notes/domino/xwork platform rather than us doing our normal “how do we get round this problem”

Richard Moy(04/11/2011 17:22:32 GMT)

Stephen,

Yes it can be done in XPages but it would be contrary to why I would use XPages. In general, XPages follows the same approach of having the data container be a form and this integration is what makes it great for Domino developers. However, the approach adds the need for more views and indexes.

Stephan H. Wissel(04/11/2011 16:12:55 GMT)

You could go for PureXML in DB/2. There you have both SQL and NoSQL.

Mark Myers(07/11/2011 16:38:06 GMT)

@jerry – from what I can tell the feeling is “well IBM arn’t going to do anything so why vote” <shrug>

we just keep trying

Jerry Krak(07/11/2011 10:53:53 GMT)

Only 36 votes after three days?

Seriously this is all what the Lotus Community got?

This is probably the single most importat problem that will stop most of serious development on this plaftorm from happening!

What is going on here? Where all yellow developers went?

For those who only observe – Please vote!

John Foo(27/11/2011 07:55:43 GMT)

Although I have never gone to LotusSphere, I totally agree with moving the view indexes out of nsf (I was thinking of that years ago). Since the FT index has always been outside of nsf, it stands to reason that it should be done as well (and although tricky, it can be done without breaking any old apps).

Mark Myers(27/11/2011 21:47:33 GMT)

@John I agree that it would make a huge difference and would do a lot to help bring the nsf up-to-date

Adrian(14/12/2011 17:10:08 GMT)

At Sun & Son we have been working on these issues with the IBM CIO office for 3 years – Our Data Modeler & (Relational)Data Bridge now form the basis of a New architecture that requires no change to the NSF (or development work from Lotus) – We have a signed off (IBM CIO office 7 information management)specification for a Notes JDBC driver with embeded Derby (that exploits the data Bridge capability to map multi value fields to true relational structures – plus the new architecture also allows you to move notes indexing to DB2 to resolve key scalability issues – for more details please feel free to contact me.

Mark Myers(22/12/2011 14:39:04 GMT)

that sounds fab, i will certainly contact you about it