Now I try not to do too many ASW posts as the rest of LDC frown on them, so believe me when i say that this is not one of them, this is a genuine plea to IBM.

Please fix the NSF database

and by ‘Fix’ i mean:

1) Remove the obvious size limitations such as 32k view lookups, maximum field numbers and plain text field limits
2) Provide a good JDBC and .Net Drivers that enables access to the database from remote locations
3) Improve scaling (yes i know that it has huge theoretical size abilities but the performance degrades a lot when you add lots of documents or lots of views )
4) Externalise the View indexes
5) Provide Sharding in Clusters (optional)

The NSF has done us proud over the years and there have been lots of small improvements with each ODS structure update but it’s still not what it should be (too many size limitations and performance problems), and this is the right time to fix it!!

WHY?

1) NoSQL is a growing product platform, estimated to be worth on par with the relational market within about 5 years.
2) You already have a hell of a lot of experience in it.
3) You have a existing market for it (Notes apps), which it will help revitalise and it’s one of the final components in your ongoing “X” cause.
4) It will make money, if Oracle are busy building a proper NoSQL db platform then you know its a good idea. I know quite a few companies that would jump at the change of using a NoSQL db from a trusted vendor.
5) The market is still young, you have time!!

You don’t have to link it to Notes directly if you don’t want to, stick it on the XWork server and call it XNSF or something if you feel like it (as long as Notes apps can access it). Yes I know you have recently given us good relational access, but relational is starting to look a bit old hat for a lot of webscale apps, we NEED a good NoSQL., CouchDb has shown you the way, and MongoDb has taken the ball and is running with it giving developer and businesses what they want, these 2 are the leaders in the document based NoSQL world.

I’m going to pester the managers at LS2012, but if you see me stand up at the “meet the product managers” its because you have fobbed me off and I want you to see how much this would give your developers and community. you can find my ideaJam entry here.

Please!

Old Comments

Erik Brooks(04/11/2011 18:34:26 GMT)

As IBM has touted for the past couple of years, NSF is their strategic/NoSQL datastore.

Since ODS changes only occur on a major version, 9.0 is do-or-die time.

It’s time for IBM to step it up.

Mark Myers(04/11/2011 15:05:16 GMT)

@richard, good points and thanks for the vote, i agree its a monster change but if companies like Oracle are willing to put the effort in they there should be enough value for IBM to do it, or at the least provide an alternative NOSQL db format

Mark Myers(04/11/2011 16:29:35 GMT)

@Dave, you are not alone in this wish

Richard Moy(04/11/2011 14:54:06 GMT)

Mark,

I will add my vote to ideajam of yes. I would love to see these changes. But, what you are asking are major changes to the NSF format and I don’t see IBM investing the money especially with their big push towards Connections.

What you might consider is getting away from the entire approach of how data is store in an NSF document and treat it only as a data container. This reduces the number of views significantly and unifies your data which will increase the performance. However, if you are using XPages to access the data than you are out of luck.

Mark Myers(04/11/2011 21:14:58 GMT)

@Erik – well said that man

@Henning – you have a lot of good points (really good points), Cassandra has a slightly different use case than mongodb/couchdb it being classed as a bigdata NoSQL as opposed to DocStore NoSQL (the other major types being Graph and Key-Value) so i can understand IBM using it,

Stephan H. Wissel(04/11/2011 16:15:13 GMT)

… and it actually wouldn’t be too hard to create a data sourcve for {insert-your-favorite-noSQL-here} to be used with XPages

Henning Heinz(04/11/2011 19:19:29 GMT)

So IBM is implementing nsf into DB2, see { Link } ?
NSF is the datastore for Notes and Domino. Nothing more, nothing less but imho far from being anything strategic within IBM. IBM has been playing with Cassandra (and others) in IBM research and they currently use Apache Hadoop in their BigInsights products.
Still a renovation of nsf would be welcomed. Although when IBM is able to implement NoSQL into DB2 then why not embed a small SQL engine into nsf for easy querying / reporting!?
After the nsfdb2 experiment there even had been a promise to implement some of the lost functionality directly into nsf.

Dave Harris(04/11/2011 16:23:43 GMT)

Not a developer, but just one of those points would make my life as an admin better: increase the 32k lookup limit.

If you run Domino as a perimeter mail solution, running your own blacklists is pretty much impossible, simply because of this, unless you want to run RBLs on your internal DNS servers. And when you’re working on a global level, customers tend not to go for that.

Given the prevalence of 64bit OSes now, removing this limit must surely be possible, even if it needs to be caveated.

Mark Myers(04/11/2011 16:19:13 GMT)

@stephen, that is very very true, and it would be a solution that someone like you could write, or we could just install mongodb and use the java driver to do that, but im thinking more for the growth of the notes/domino/xwork platform rather than us doing our normal “how do we get round this problem”

Richard Moy(04/11/2011 17:22:32 GMT)

Stephen,

Yes it can be done in XPages but it would be contrary to why I would use XPages. In general, XPages follows the same approach of having the data container be a form and this integration is what makes it great for Domino developers. However, the approach adds the need for more views and indexes.

Stephan H. Wissel(04/11/2011 16:12:55 GMT)

You could go for PureXML in DB/2. There you have both SQL and NoSQL.

Mark Myers(07/11/2011 16:38:06 GMT)

@jerry – from what I can tell the feeling is “well IBM arn’t going to do anything so why vote” <shrug>

we just keep trying

Jerry Krak(07/11/2011 10:53:53 GMT)

Only 36 votes after three days?

Seriously this is all what the Lotus Community got?

This is probably the single most importat problem that will stop most of serious development on this plaftorm from happening!

What is going on here? Where all yellow developers went?

For those who only observe – Please vote!

John Foo(27/11/2011 07:55:43 GMT)

Although I have never gone to LotusSphere, I totally agree with moving the view indexes out of nsf (I was thinking of that years ago). Since the FT index has always been outside of nsf, it stands to reason that it should be done as well (and although tricky, it can be done without breaking any old apps).

Mark Myers(27/11/2011 21:47:33 GMT)

@John I agree that it would make a huge difference and would do a lot to help bring the nsf up-to-date

Adrian(14/12/2011 17:10:08 GMT)

At Sun & Son we have been working on these issues with the IBM CIO office for 3 years – Our Data Modeler & (Relational)Data Bridge now form the basis of a New architecture that requires no change to the NSF (or development work from Lotus) – We have a signed off (IBM CIO office 7 information management)specification for a Notes JDBC driver with embeded Derby (that exploits the data Bridge capability to map multi value fields to true relational structures – plus the new architecture also allows you to move notes indexing to DB2 to resolve key scalability issues – for more details please feel free to contact me.

Mark Myers(22/12/2011 14:39:04 GMT)

that sounds fab, i will certainly contact you about it

I recently found this in a Java tech spec, written by a developer who should have known better

“Note No Non-Functional or Quality of Service criterion has been established from the Phase 3 System Requirements in terms of facilitating the quantitative assessment and validation of the XXX Phase 3 Architecture”

Shoot me now!!!

Old Comments

Ben Poole(30/09/2011 07:44:57 GDT)

Marc: love it!

As for those developer notes Mr. Myers, we both know it’s not you who should be lined up and shot

Mark Myers(27/09/2011 14:08:51 GDT)

didn’t MBR do that with an Amazon voucher?

its a grand plan!

Mark Myers(27/09/2011 11:17:19 GDT)

Ohhhhhh good one, that’s an evil line

Mick Moignard(27/09/2011 11:16:02 GDT)

In my mainframe days as a programming project leader, I once got a system spec, all fully signed off, QA approved, the whole corporate works, where, under the heading Management Reports, was the one liner

“There will be Management Reports”.

I refused it. After the inevitable fight, the QA department sided with me (?!??!?!?) and the project got canned.

Marc Champoux(27/09/2011 13:04:55 GDT)

Back when I was coding a few years ago, I often had to come up with User Requirements and Design Specs documents. I’d create them and make sure they were complete (i.e. none of that “there will be a management report” one-liner).

However, I was sure nobody was reading what I wrote so I decided to have a bit of fun and for 2 years in a row, I would always add a line somewhere buried in a paragraph that said “If you read this line, let me know and I will buy you a coffee”. It took 2 years for QA to finally discover that line.

Even after QA discovered it, I kept doing it anyway because I wanted to see if the business owners were reading what I wrote for them… They weren’t because I never got a comment from them about it.

Marc

The 93rd First Tuesday club was a bit different from the previous ones. Not only was it set away from our normal haunt (Imperial City on Cornhill) in the grand London Capital Club, we also had a proper technical talk, instead of the usual small sales pitch (a hallmark of the club as it does not support hard sell) . Thisfs was given to us by Carric Dooley, and included such gems as “Fall off the security turnip truck” and “It’s like an encrypted onion”. This is new evolution of the First Tuesday club and as far as I was concerned it was a welcome change, showing how the club is growing. You could see from the eagerness of the questions from other listeners and the fact we all kept very quiet to listen that they enjoyed it as much as I.
I’m hoping for more of the same next time, although judging by the very high level technical staff that McAfee brought with them that’s not going to be a easy act to match. But it does stand out as one of the notable First Tuesday club meets for me. The conversations of the night included:

• Polymorphic virus attacks
• The difficulties of applying security policies to clever but reluctant users
• Identifying data centres by tell-tail external physical signs
• Changes George Lucas should make to the Empire Strikes Back re-release 🙂

Old Comments

Andrew Magerman(20/09/2011 09:09:58 GDT)

removing Jar-Jar Binks, surely.

cutting off all the yucky romance scences with Hayden Christiensen.

“I don’t like sand. It’s coarse and rough and irritating and it gets everywhere. Not like here. Here everything is soft and smooth.”

</sound of gunshot to my head>

Andrew Magerman(20/09/2011 09:20:29 GDT)

and midi-chlorians!

AAAAAAAAAAAH!

Mark Myers(27/09/2011 10:26:15 GDT)

Geeeeeek

Popping to the Skillsmatter freebie evening sessions is one of the best ways of getting an introduction in to any open source project that I know of (particularly if you are in London)

The one I visited last night was for MongoDB which LDC are looking at in our general search to see which of the newer generation of NoSQL dbs we like best (yes yes i know they all have different usages cases but it nice to have a favourite that you spend a bit more time with), this “do” consists of a quick 45 min to 1 hour session (the content was from a 1.5 hour to 2 hour session) ,question time afterwards, and then pub time after that (with sponsored drinkies), a nice thing to pack your head with at the end of the day.

We were given a introduction of what happens on traditional relational db applications when they reach the serious levels and their limitations. and then how to design a NoSQL db from scratch to scale to a complete monster, the room was full of people who REALLY wanted to know stuff, but Alvin Richards (Senior Director of Enterprise Engineering at 10gen) REALLY knew his stuff and answered all comers with nary a pause. (and gave out freebies)

items handled were :

• Sharding
• Replication
• Fail over and recovery (i love the arbiters)
• Scaling (indexing still seems to cause the same bloody problems)
• Safe writes (i was pleased to see that this is in mongodb as well as Cassandra)Thankfully Skillsmatter post the videos from all their talks online and you can see this one here (its well worth your time)

Last Tuesday was a great night at the 1st Tuesday club hosted by Information Security Solutions, as normal the conversations ranged from the intensely technical all the way through company purchasing patterns in the security market and ended up chatting about beautiful hikes around England, I was treated to an explanation on a super simple corporate CMS that minimises the chance for user errors resulting in unsecured documents (as we often see with share point) and got to catch up on all things security based. the sponsor for the night was HP who with their acquisition of Arcsight seemed to have really rounded off their portfolio of recently acquired security companies and had brought someone from each group along, all of them doing that tricky balancing act of being there to represent their company while not at any time doing any form of hard sell (Rob from Arksight was a particularly interesting person to talk to) it was a good thing that HP had not brought more people as it was packed with the waiters really earning their money keeping everyone’s glasses full and maintain a constant flow of food.

All in all a great night with good company that showed that HP is serious about the security arena.