Sean Cull

852 XPages runtime engine faced an OutOfMemoryError on EC2

Sean Cull — Fri, 27 Aug 2010 00:24:14 +0100

Update : Back on 851, just can't get 852 to run reliably on my ec2 setup ( note Ubuntu is unsupported )

Ever since I upgraded my EC2 Ubuntu 8.04 instance to 852 I have been having daily crashes due to Outofmemory Errors.

08.26.2010 23:19:48 HTTP JVM: >>>>
08.26.2010 23:19:48 HTTP JVM: The XPages runtime engine faced an OutOfMemoryError
08.26.2010 23:19:48 HTTP JVM: You can fix this by increasing the value of the HTTPJVMMaxHeapSize variable in notes.ini
08.26.2010 23:19:48 HTTP JVM: >>>>

A small EC2 instance only has 1.7Gb of memory so it is going to struggle in any case. I have tried HTTPJVMMaxHeapSize = 64M, 128M and 256M.
851 was set to 256Mb by default whereas 852 is set to 64Mb as default.

This technote http://www-01.ibm.com/support/docview.wss?uid=swg21377202 explains the background but it the only part I fully understand is the bit that reads

"XPages is an exciting new way to work with Domino applications. It brings Domino fully into Web 2.0 technologies as a world class leader. "

It basically says go to 64 Bit - is this windows, Linux or both. If Linux then I assume the 32bit installation file as there is no 64bit version ?
Will 64 Bit make any difference if it is memory limited in any case ?
If I am memory limited do I make the value higher or lower ?

I appreciate that this is not an issue to people running on proper production hardware but it is an issue for development servers and appliances.

It could also be affected by using Ubuntu but it has been very good in all other respects up until now.

If you can't read this its because my server has crashed !

Sean

Domino 852 - make sure that you test your Xpage applications first

Sean Cull — Wed, 25 Aug 2010 11:27:12 +0100

Just a quick tip for any other "traditional notes client developers". With traditional Notes applications it has been very very rare that an upgrade has caused problems with functionality, I have only ever seen it a couple of times over thousands of applications.

We upgraded to 8.5.2 yesterday and both of our Xpages applications currently under development stopped working. In both cases 8.5.2 was fussier than 8.5.1 about tolerating some dubious SSJS code.

as an example this code worked on 8.5.1

var Statusnumber : integer = Statustext

whereas in 8.5.2 it required the more correct

var Statusnumber : integer = parseInt(Statustext)

While this may indeed be the correct thing to happen the combination of steep learning curve, significant server changes in a point release and a complacency built up over years as a Notes client developer may also affect a few other people so heads up.

Sean

This blog is now running on 8.5.2 on Amazon EC2, S3 Firefox Organiser was useful

Sean Cull — Tue, 24 Aug 2010 11:25:19 +0100

Now running on 852, the whole upgrade took 5 or 6 minutes, in fact the hardest part was getting the install files to the server.

I used Amazon S3 storage and the wget command from Linux.

One useful tip is that although the S3 control panel has an upload limit of 600Mb the Firefox S3 Firefox Organiser does not and doesn't require flash either.

Well done IBM for getting the files up promptly. Stuart McIntyre has a list of part codes.

For some reason the Linux Traveler server is listed under Windows but it is there.

Sean

Xpages and why my tendency to Gold Plate has got harder to manage.

Sean Cull — Mon, 26 Jul 2010 22:06:03 +0100

When we are working on a new application we often talk about bronze, silver and gold plating.

I have a tendency to want to make things too good when in fact the customer is no better off for my efforts, we refer to this as Sean's Silver or Gold plating. Its a particular issue as we almost always work to a fixed price and experience has also shown me that every extra bell and whistle increases testing and support throughout the life of the application.

Xpages allows you to take this to whole new levels because the traditional constraints imposed by designing applications within the Notes client have been removed. Now you can have two equally functional Xpage applications that are wildly different in terms of the interface bells and whistles.

The new functionality is obviously much welcomed and as we develop more design patterns it gets easier but it does make me wonder if it is also a risk to Notes ?

I have heard the opinion expressed several times over the years that one Notes real strengths was that you could ( only ) produce cheap applications and it was therefore very competitive. With Xpages even a simple departmental app can now be very feature rich ( or Gold Plated ). Pricing ourselves out of jobs or failing to deliver value for money to our customers is now something that we have to work harder at because our toolbox is so much bigger. A good problem to have but one that needs thought.

Sean

Notes basic ( non eclipse ) is very useful now but will it hold Notes back in the future ?

Sean Cull — Mon, 26 Jul 2010 21:42:31 +0100

I had my "road to Damascus moment" recently when I saw the same app run in Xpages on the client and on the web with no extra coding, or more accurately not much extra coding.

I realised that Notes had passed a huge milestone and some of my long held beliefs were unceremoniously dumped. One of these was that customers were best with the Notes basic ( non-eclipse ) client. It is considerably faster and despite working on notes apps full time neither I nor my colleagues had yet had cause to use any of the eclipse functionality.

Now when we are looking at new apps our first inclination is to want to do it in Xpages if it is likely to be a repeatable solution. This is not just because we are geeks and it is fun but because with our business heads on we believe that we have a much better chance of selling that application to potential customers who do not, and unlikely to ever, run Notes clients.

Notes basic was good for getting people with older hardware ( often < 3 years old ) onto newer versions of Notes. That was a very good thing in the battle for hearts and minds. We needed to get them off R5 / 6 and onto something which really showed what Notes could do.

But hang on, is there not a danger that we will be repeating history with Xpages and Notes Basic. If developers mostly want to code in Xpages and as a consequence people with Notes basic will progressively get left behind and, even though they are on Notes 8.x or maybe 9.x. They will have that dangerously substandard experience that is analogous to users being on R5. I am stretching the point a little but I do think that it is a very valid concern.

So do I think Notes basic is a good thing ? Yes I do for now but I think it could quite possibly be a bad thing if it were continued on into Notes 9.x.

Sean

Receiving POP3 mail into a Linux Domino server using getmail and sendmail

Sean Cull — Mon, 26 Jul 2010 07:42:55 +0100

I have a need to receive mail from a POP3 account into a domino server running on Centos 5.5 as an appliance.

Some time ago on windows I used the Chimeras Email Forwarding System which worked very well but this was a Linux server.

The scheme I came up with uses a combination of Getmail version 4 and Sendmail.

Getmail is a Linux utility which will poll different types of mail accounts and then pass those messages on. It has an option to pipe the messages to an external mail delivery agent such as Sendmail. Sendmail can then send them on to the Domino server as SMTP on the localhost address.

These notes are for a Centos 5.5 installation with the Webmin web based management console

Install and configure Getmail

Create a new user called mailer or similar and a password. This is because Getmail will not call Sendmail if it is run as root.

As Root download Getmail

see http://pyropus.ca/software/getmail/documentation.html#installing

$ cd /tmp/
$ wget http://pyropus.ca/software/getmail/old-versions/getmail-4.20.0.tar.gz $ tar xzf getmail-version.tar.gz $ rpm -ihv getmail-version-release.noarch.rpm $ cd getmail-version $ python setup.py build
I then used Winscp to create the configuration file /home/mailer/.getmail/getmailrc
In this case the receiving account on the Domino server will be the Administrator account. Getmail has lots of other options

[retriever]
type = SimplePOP3Retriever
server = acme.com
username = peter pan
password =

[destination]
type = MDA_external
path = /usr/sbin/sendmail
arguments = ("-i","-bm", "administrator@localhost")

[options]
verbose = 0
read_all = false
delete = false
message_log = ~/.getmail/log

[default]
Configure Sendmail
Sendmail and Domino cannot normally run on the same box because they both need to bind to port 25. In this scenario Sendmail does not need to listen to port 25 so it can be configured to listen on port 26 ( or any other unused port ). I used the Webmin client to make this change under Servers > Sendmail > Network Ports

Testing the System
Log in as mailer enter getmail -v The -v gives a verbose output so that you can see what is happening.
Scheduling the getmail task
I used webmin to set up the cron job running as the mailer user
System > Scheduled Cron Jobs

Other things to remember
You may need to set Sendmail up to start on bootup You may need to allow pop3 mail through your firewall
As usual all feedback on better schemes welcomed ! Sean

Please read Jakes post and give your views in the comments

Sean Cull — Tue, 13 Jul 2010 11:20:29 +0100

Jake has apparently been asked to leave Planet Lotus. This is just daft.

I don't know Jake from Adam but he is certainly part of this community and his insights into both Domino and Sharepoint ( and PHP and RUBY etc ) are very much valued. Please express your opinions on Jakes blog.

http://www.codestore.net/store.nsf/unid/BLOG-20100713-0453

Sean

EC2, too clever for me, snapshots, multiple servers, instant servers, but "release" your IP address and it all falls over ( for days ).

Sean Cull — Fri, 2 Jul 2010 09:34:47 +0100

Dear reader, you will probably be reading this in a few days time because I have just pressed the wrong button in EC2.

It's been an interesting lesson. Part of moving to the cloud is all the extra redundancy that you get. You feel that the incredibly powerful controls that you are given mean that in some respects you haven't really given away control but rather gained it.

You might feel uncomfortable going for a small or new host organisation but you feel safer with the big people like Amazon EC2.

The problem is that humans are still involved and some of the services that you have been given are only on loan. In my case the fixed IP address from Amazon was only on loan and when I accidentally gave it back I couldn't get it back again. In the on premise paradigm people had redundant power supplies, redundant drives, redundant switches but did any one worry as much about loosing an IP address whilst still with the same connectivity provider ?

Its good that its Friday ( shame it wasn't Friday evening though ) because when you see this on Monday on Planet Lotus the link might actually work depending on your local ISP's DNS caching. Was it a good weekend for you ?

What happened

EC2 has a great feature called an "Elastic IP Address" that allows you to have a static IP address that you can switch between EC2 instances ( servers ).

If you would like to avoid some chaos then you SHOULD NOT EVER choose to "release" your Elastic IP address for once you do it disappears and is lost forever.

This means that you need to get a new one and re-do all of your DNS A records again. This means that your site will be down for days.

When moving the IP address between instances you should use the Disassociate option below. You will notice that the Release option is greyed out.

However, if you have just terminated the instance that had been associated with that IP address the console ( quite logically ) looks like this

Now I know that the logical thing to do ( in hind sight ) would be to click the Associate button but my brain has not twigged that the Dissociate action happened automatically when the instance was terminated so I clicked the "Release Address" button and got this.

I really want them to change it to say this

Amazon EC2 - good and bad

Sean Cull — Thu, 1 Jul 2010 15:49:44 +0100

Some of you may have noticed that the site has just been down for about 25 minutes.

I decided to make a new AMI ( basically a template ) of the running server now that it has FP3 and for some reason doing this took the server down - its probably in the small print somewhere.

I had to terminate the instance to get the data volume back under my control but I was then able to bring up a new server and switch the IP address within another 10 minutes. While I was at it I brought a bigger server up.

I had been updating the AMI so that I could create a temporary cluster to move from a small server to a larger server seamlessly - something I'll need to try another time !

Previous article on setting up a domino server on EC2

We didn’t sell Lotus Notes but we used Notes to make a customer 30% more efficient over £12M of work since 2002

Sean Cull — Thu, 1 Jul 2010 07:56:02 +0100

Following the suggestion that we should all have a theme on the value that users get from Notes I thought that I would mention this one.

It is a Notes application that we have been developing with ABB since 2002 and which Ed Brill has mentioned before. Our customer has now been good enough to quantify the benefits, hence the quote on the first page.

“Working with FoCul has delivered quality and efficiency improvement to our consultancy processes and services, whilst retaining the ability to customise work processes at short notice to meet specific customer needs

The benefit has been a 30% efficiency saving, allowing ABB to deliver over £12M of studies more competitively since 2002”

David Stanier, Principal Consultant, ABB

I should point out that the application has not made ABB £3M better off by increasing their margins because the efficiency savings have helped them to be more competitive and win more work - try putting an ROI on that.

No individual part of the application is rocket science ( although we are very proud of the whole piece ), it is just a good Notes Application, a business tool. The benefits come from enabling very highly skilled engineering consultants to collaborate with each other and with their clients, to use a shared best practice approach and to significantly reduce the time they spend on administration - its not hard to kill the competitiveness of a project by having experts compiling huge word documents.

The development costs have been 300K in total over the 8 years with the costs being split pretty evenly across that period as more functionality has been added. ABB have probably spent a similar amount of time internally in terms of specification workshops, testing and peer training / support.

It is worth noting that FoCul hardly ever sells Lotus Notes ( hoping to sell XPages though ) and really does very little in terms of renewals. IBM probably sells ABB 100 - 200k licences a year.

Am I happy with where Notes is from a technical standpoint ? Yes more than ever

Sean Cull — Fri, 18 Jun 2010 14:13:03 +0100

I have just recorded a podcast with Stuart McIntyre, Darren Duke, Sharon Bellamy, Matt White and Bruce Elgort and it got me thinking.

Although I have commented on some blogs over the last week I have not posted anything myself so here goes.

Am I happy with where Notes is from a technical standpoint - yes more than ever.

My company focuses on a "double niche" - organisations that are in the manufacturing sector and have a pre-existing Notes infrastructure. With XPages we can now think of it as less of a double niche because we can design once for the web and the Notes client at the same time. This is a huge advance and as someone pointed out on the podcast pretty unusual in terms of platforms. Notes has always been much more capable than the vast majority of its users know and now this is even more the case.

We are developing our first application which will run as an appliance at a customer site that is non notes. But it also runs as a local replica in the Notes client - how cool is that ? Think Google Gears but better ( and still supported ). I might regret this but here are the daily and weekly builds ( sorry Andrew ! ). Trust me it looks the same in the Notes Client.

I deliberately used the phrase "where Notes is technically" because I am not as happy around the non technical side. Its not that it has suddenly got worse rather that the gap between "what Notes can do" and what people "think Notes can do" has got much wider and so my frustration has got greater.

I will openly admit to having looked to see if we can offer solution on the MS platforms as well as Domino and I have always come away confident that for what we do Notes has very limited competition. So what are the issues ?

IBM has got to value retaining clients

Reps need to be incentivised for renewals - it will stop companies on < 8.x thinking the product is dated.

Its a virtuous circle that will ensure that IBM customers get better value from Notes through applications, DAOS, servers running faster on existing hardware etc. and IBM spends less time fighting a rear guard battle against a mail platform and a file server.

People often talk about life cycles - what is IBM's view on the lifecycle of a customer ?

IBM has got to be wholistic about Notes being a development platform

If I sell a 10K Xpage app to a company with <1000 employees the whole project is 12K ( 2k for the utility express server ). If I sell the same App to a company with 1001 employees the whole project costs maybe 25k. This means that I can't be competitive in large companies - that just seems daft.

At the same time pricing by individual users is not the answer either - for example if I write an app that manages brochures or safety data sheets the client will absolutely not pay £X per month for each customer just because they are listed in a NAB somewhere ( in reality it drives you to LAMP or to use a daft email based system with response parsing to avoid the restrictions ).

I honestly don't know the answer but the current scheme is not helpful. There may be ways and means around this but if so this should be open and transparent or people will just not bother.

IBM needs to lower the threshold for existing and "net new" developers to start using XPages

While I applaud the work by individuals in IBM the whole documentation piece is very very poor and the learning curve is much too steep.

The Wikis are a nice idea but they are too diverse - new users want focused examples and tutorials. For example follow this link to look at the wiki articles on XPages in the client - a strategically important area for Lotus ?. While again I applaud the work of the individuals who made the entries the formatting alone makes it unpalatable. If IBM is going to depend on Wikis they need to employ some editors.

IBM needs to bundle 2 or 3 really good Apps with Domino

so that businesses get more value and so that they can see how good Notes can be in terms of Business Applications.

I would caution that this is not an opportunity to deploy sexy Composite Apps because businesses need to be able to dissect the bundled apps and learn how to put new ones together - in most uses cases Composite apps are too expensive. If there was a good Holidays Management App wouldn't most customers use it ?

Some people will argue that OpenNtf does this. While I have huge respect for the IBM employed OpenNtf developers the Wiki as an example is not fit for production ( test being would you encourage the CXO to use it ). It is a marvellous thing for developers to use and dissect but it is too buggy and incomplete to give to a customer. I say this as someone who uses them a lot. I would say the same of the Blog template as well ( although Steve Castledine did recently seek feedback on this one ).

Just a couple more observations :

IBMs current methods to promote Notes do not work - I think that is a statement many people would respond "A strongly agree". Einstein said

"Insanity: doing the same thing over and over again and expecting different results"

IBM also wonder why LAMP, Ruby etc.. get such uptake when they don't have marketing either. To me the answer is in the barriers to adoption. If, for example, the on premises suite from IBM was available free for up to ten users would the Lotus community grow much ? Would the public perception of Notes improve ? I am not wedded to this idea, I don't think that it would directly create work for me, but it would require a shift in mindset on the part of IBM and I think that is what I am really looking for. It would also have negligible cost ( this licence would be unsupported) . Yes I know that there is a downside to giving away fee stuff, been there and had my fingers burnt, but it should be considered - think about Einstein's quote above.

The action list above is all "IBM to do". People will say two things - "what about you Mr Business Partner, ISV etc.. ?" and "look, just use another platform ?"

To the first I would say yes we all need to do more but our passion needs to be matched by IBM, not just Ed, IBM. There also needs to be a clearer roadmap about Notes as a design platform. People have rightly cautioned against building careers and businesses around a single vendor. I agree and it is a concern for me but to be good at what you do it is sometimes necessary. The thing that scares people away is not seeing a shared roadmap for the product. I don't understand the roadmap for Notes as a design platform and the lack of a sensible licensing structure sometimes makes me wonder if there is a roadmap.

To the second point I would respond that the platform is very very good and from a technical perspective is the best choice for me and my work. The hardest part is getting over the general lack of confidence in the platform that exists in the wider IT community.

Lastly ( I'm impressed your still here ! ) I think that some rebranding should be considered. I love Lotus Notes but "Lotus Notes" has negative connotations for too many people.

I personally would like to see "IBM Xpages" being pushed as a brand for non mail web delivered applications. It is easier to say to potential customers that the solution is based on IBM XPages rather than on Notes ( I think hardly anyone outside of this audience has heard of Domino ). Xpages could be positioned as an "Add-in" to Notes. I also like the idea of a better XPages specific development environment in addition to DDE. I feel like I've just publically disowned a child !

Anyhow, in conclusion I am hugely pleased with Notes, it is so much better than Exchange + sharepoint, I think the IBM developers are doing a fantastic job with the resources that they have but this misguided public impression of Notes needs to be addressed and the old ways are not working.

p.s.I know nothing about marketing so all of the above could be rubbish !

Fixing an Overheating Toshiba M700

Sean Cull — Thu, 17 Jun 2010 21:42:44 +0100

I have been quite frustrated with my Toshiba M700 for the last couple of months. It never seemed to run as well as it should with the CPU maxing out at 100%, the battery going very quickly and the casing getting hot.

I had deliberately bought the M700 as a desktop replacement but I was at the point where I was just about to but a desktop anyway because I thought it was beyond the M700 to run two external monitors. I even thought about a Mac Book Pro until I saw the price.

I noticed that while the CPU was maxed out the Windows 7 Resource monitor was showing that the CPU frequency was only around 40% of the maximum attainable. I downloaded coretemp which showed that one of the cores was running at 103 C and I noticed that there was a correlation between the temperature and the CPU frequency - the system was throttling the frequency at these extreme temperatures.

Being worried that it was a problem with Windows 7 I booted it to Linux from a USB drive. This was inconclusive so then I restored an old XP backup, Initially I thought that I had solved the problem but the machine was still running at 103 C although the throttling didn't seem so bad ( it is harder to measure in XP ). I still felt something was wrong.

Anyhow, being an Engineer I felt that there was something not quite right so I decided to take it apart, as you do. I found some good instructions here which helped, particularly with the keyboard connection.

Everything around the heat sync looked pretty clean and I almost gave up but when I blasted it with compressed air for good measure quite a lot of debris flew out. I rebuilt the machine - try and remember where the long bolts came from ! - and now it is running very very happily on windows 7 at 55 C and it feels like a powerful machine again.

If you are having similar problems It might be worth shooting some air into the fan vent. There is a hatch to give access but the access is not really that good.

Strangely I have just tried to re-create the problem by blocking the vent and cant get it to be bad enough to get past 90 C. It just shows how a small amount of dirt can clog the heat exchanger very effectively.

Sorted : Help ! Anyone know what has happened to the A Deep-dive on Building Lotus Domino Applications with XPages webcast ?

Sean Cull — Thu, 17 Jun 2010 16:03:04 +0100

Sorry folks, I seem to be attracting a lot of readers which was not my intention.

I was desperate to see the " Deep-dive on Building Lotus Domino Applications with XPages webcast " but the link in my confirmation email just went to a page that said "thank you".

When I followed the other link in the confirmation email the web cast was not listed either - the link was for the Rational site for some reason.

I got there in the end and it was an excellent webcast - the link below should show a replay by tomorrow - the start is a bit deep but it then goes on to show some good stuff and wets the appetite for 8.5.2

https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?lang=en_US&source=dw-c-wcsdp&S_PKG=062310A&S_CMP=dW&ca=ddc-

AppendRTItem - only appends the first 32 k - I never knew that

Sean Cull — Wed, 16 Jun 2010 09:32:32 +0100

A comment buried in the official example

Note The changes are not visible until the document (after the script has completed) is closed and reopened. Also note that the AppendRichTextItem method appends only the first 32K of a RichTextItem; the remainder is truncated.

Thanks to Jake's code store error list - I didn't even know the list existed.

update :
I have fed this back to IBM.

SNTT : Installing Traveler on an old 32 bit server is so much easier with linux than windows server 2008

Sean Cull — Thu, 10 Jun 2010 08:00:00 +0100

Note 1 : Notes 8.5.2 is beta and there are no guarantees that the features described here will be in the final product that IBM ships.

Note 2 : The description below is for a development server and may or may not be suitable for production depending on your circumstances.

Having moved to Amazon EC2 I I recently decommissioned an old Dell Poweredge 650 server and decided to put it to good use to test 8.5.2 CD5. This server has no DVD drive, is 32 bit and cannot boot from USB.

My original plan had been to get some more experience of Server 2008 R2 but I ended up going down the Linux route as it was much simpler.

Linux Install

Download CDs for Centos 5.5 ( yes I know it is not officially supported )
Do a minimal install - don't even choose the server options
Access the server via SSH using Putty
Upload the install files using Win SCP
Install Domino
Install Traveler ( using the silent install feature )
Install Daniel Nashed's startup scripts

Windows Install

Check Server 2008 R2 requirements - 64 bit - oops
Eventually find Server 2008 Standard 32 Bit
Can only be installed from DVD - oops
Create a boot usb instead using this very useful hint - easiest way to make a boot disk I have seen so far
Server won't boot from USB - oops
Create a Windows 7 Recovery Disk
Boot from the Recovery Disk
Navigate to the option for command prompt
Plug in the USB drive
Access the Server 2008 setup files on the USB drive
Install Windows Server 2008
Upload Domino Install Files
Install Traveler

The thoughts that came to mind were :

Wow - Linux Traveler

Is IBM missing a trick by not jumping on the green bandwagon and allowing people to recycle old 32 bit servers ?

The whole Linux dependency problems that were such a pain in R6 and R7 seem to have totally vanished

My colleague who uses an iPhone just continued to sync via Traveller against the IP address ( now a Linux server ) - he never noticed the change.

Wouldn't it be great if IBM supported a free distribution of Linux ( I run Domino on Ubuntu and CentOS, albeit on a small scale )

The 64 Bit install

To satisfy my original intention to try out Server 2008 I decided to install it on my 64 bit laptop.

I have previously installed 64 Bit Centos which allowed me to install it to and then boot from a USB drive leaving the original 32 bit Windows 7 install on the laptop intact.

No such luck with Windows. It will not install to a USB drive so you need to create a partition or swap out the disc - I guess I needed a disc upgrade anyway, maybe I should buy a 64 bit dev server after all. You would almost think that it was MS who sold hardware.

p.s. MS do some great software too !

SNTT: Creating a static HTML version ( e.g. for a CD ) of your Xpage Wiki

Sean Cull — Thu, 20 May 2010 07:00:00 +0100

On a number of occasions in the past I have been asked if I can make a Notes database into an interactive CD for non-notes users, for example at the end of a project.
I found that the best approach has been to make a simple html domino site and then use a tool like WinHTTrack to spider through it and create a HTML copy.
One of the restrictions of this approach is that you cannot use any client side JavaScript for menus etc..

I have tried this technique on the XPage Wiki and it works relatively well for simple sites. With the Javascript being server side the WinHTTrack is non the wiser and faithfully reproduces it in static HTML although you will obviously find that combobox filters don't work.

This is a live Xpage Wiki

This is a copy of same wiki as static HTML.

There are a couple of "features" that you may need to address.

1) Some CD burner software will not handle long URLs. In the past I have had to write code to parse the files and file names shortening these although with USB pens now being so cheap there is not so much call for CDs

2) Twisties / Sections that are closed by default need to be modified to be open as clicking on them to open them does not create a new url
In this example I had to modify the Xpage design to default the section / twisty open (prtContent)

SNTT: Creating a static HTML version ( e.g. for a CD ) of your Xpage application

Sean Cull — Thu, 20 May 2010 07:00:00 +0100

I'm speaking at the IBM Power of Business Applications conference on the 16th June at IBM Bedfont Lakes. This article is about a neat trick from the application that I will be presenting but won't have time to talk about on the day so I thought I would post it here.

On a number of occasions in the past I have been asked if I can make a Notes database into an interactive CD for non-notes users, for example at the end of a project.

I found that the best approach has been to make a simple html domino site and then use a tool like WinHTTrack to spider through it and create a HTML copy.
One of the restrictions of this approach is that you cannot use any client side JavaScript for menus etc..

I have tried this technique on the XPage Wiki and it works relatively well. With the Javascript being server side the WinHTTrack is non the wiser and faithfully reproduces it in static HTML although you will obviously find that combobox filters don't work.

This is a live Xpage Wiki

This is a copy of same wiki as static HTML.

There are a couple of "features" that you may need to address.

1) Some CD burner software will not handle long URLs. In the past I have had to write code to parse the files and file names shortening these although with USB pens now being so cheap there is not so much call for CDs

2) Twisties / Sections that are closed by default need to be modified to be open as clicking on them to open them does not create a new url
In this example I had to modify the Xpage design to default the section / twisty open (prtContent)

Amazon EC2 Running costs after 1 full month - 52 USD

Sean Cull — Thu, 13 May 2010 20:01:12 +0100

I have been running a Domino server in EC2 for a couple of months now and thought I would do a post about the costs. I wrote up the scheme here. Note that this is a Domino server built from scratch and not the new Domino development image. The server is based in Ireland.

The server is pretty small scale being used for the FoCul Web sites, this blog, as a development server and as a hub between 4 other servers. If anyone has any particular usage metrics they would like to see I would be happy to oblige. I have posted a screenshot of the usage view from log.nsf as a simple comparison. TLogging / DAOS is not being used.

The main point to note is that I am using a spot instance. This is less than 50% of the cost of a regular instance. As I mentioned before I am a little confused by this scheme as you are currently only able to bid up to USD 0.045 which is well below the usual USD 0.095 price. I am a little uncomfortable that I could be cut off by a price spike but the last spike was valentines day.

Using the spot instance is saving USD 484 per annum - possibly a bit less as I would probably switch to the standard rate at busy times. The prices I am achieving are similar to those from a reserved instance without paying the USD 227 pa or USD 116 pa over 3 years fixed fee - see EC2 pricing. On first inspection it would appear that VAT cannot be recovered on these costs. The costs include 2 backup snapshots of the 60Gb data drive. The scheme used means that for a true "development server" you can stop and start the instance at will with no loss of data. Starting the server easily takes less than 5 minutes.

In terms of the perfomance I have been pretty happy. Here is an example of a well populated wiki => CAM2 Project. Sometimes there is a short delay while Xpage apps load but once they are going they are fine.

I have taken out the Amazon Cloudwatch partial month and a little bit of S3. The Cloudwatch in its simplest form is not worth paying for.

At these kinds of costs the server is comparable with the cheapest physical servers I have seen ( GBP 80 2.33 Ghz 2 GB ) versus EC2 1.2 Ghz 1.7GB although if the server were being used to host systems under under an SLA you would really need to use the reserved instance pricing ( and annual fee ) to make sure that a price spike did not take your server down.

Overall I am very pleased with the whole setup. The flexibility and scalability of the EC2 system are much better than anything I have seen before.

SNTT : Accessing a Notes Server with a notes client via Port 22

Sean Cull — Thu, 13 May 2010 07:00:00 +0100

You can get to a Notes server from a Notes client via port 22 if you use the tunnelling feature in Putty.
Useful if you are looking at an appliance type deployment for browser apps.
Note that you will need a connection document listing the ip address as localhost as you cannot type localhost directly into the server name in the File >> Open >> Lotus Notes Application window as you can ( sometimes ) with an IP address.

SNTT: Creating a persistent EC2 Domino Instance with EBS boot volumes

Sean Cull — Thu, 1 Apr 2010 07:00:00 +0100

update : there is now also an official IBM Domino AMI and a wiki article although I think it will take some work to make it persistent.

After reading Michael Brown's excellent article about setting up an "Ephemeral" Domino server on EC2 I decided to look at using EC2 a bit harder. This blog is now running on EC2 and I have been very very impressed with the functionality that this service provides. I would obviously like to test the system more but early indications are that it has good potential for VSMB and SAAS Domino offerings.

When you follow Michael's guide you get a Domino server which uses the "Ephemeral" disk which is automatically deleted when the the instance is terminated. An alternative approach is to build the system using the Amazon Elastic Block Storage ( EBS ) system for storage. These EBS volumes have the potential ( note the word potential ) to remain intact after the instance is terminated. This means that should the instance fail your data will remain intact. It also means that you can take advantage of options to use "spot pricing" and to resize your server.

This article will take you through a method to set up such a server using Ubuntu 8.04. I am not an expert in Linux or EC2 so I would appreciate your comments on any improvements that I could make to the scheme. You should also note that as yet there is no licence for Domino on EC2 but the licences for DB2 usage suggest that a small EC2 instance is 50 PVU points. Ed Brill has posted of a forthcoming licence but this is intended for "Development and Proof of Concept" usage.

Michael covers setting up the Amazon account, it is really straight forward. The console is a great example what can be achieved with CSS.

The Scheme

Once you have an account you can set up an instance. These instances are usually set up from a images called AMIs. There are many images to choose from but I choose to use Ubuntu 8.04 because we already use that for other servers. This was probably a more complicated choice than Michael's Fedora but I wanted to be consistent with what we already do here.

A recent feature in EC2 is the idea of instances based on bootable EBS volumes which can ( but not always ) retain data after the instance is terminated. Although this seemed like a very straight forward concept there are some points of detail that I originally missed and which are important.

With the introduction of bootable EBS instances Amazon also introduced the ability to stop and start an instance as well as terminating an instance. You are not charged when your instance is stopped. The important point to note here is that the default EBS will retain its data while stopped but will loose its data when terminated. There is a way to stop this using the EC2 command line options but in this article I want to present a scheme that is straight forward for any admin to try ( including myself ! ).

The scheme I settled on ( any improvements welcomed ) was to create an EBS boot instance and have a second EBS volume which contained the Notes data. This allows me to terminate the instance and then restart it ( or a clone of it ) with no data loss. Termination is required if you want to take advantage of spot pricing or change the specification of server you are using in terms of CPU or memory. The production instance is based on an AMI which was created AFTER Domino was installed and configured. You could actually use this approach with an ephermeral boot instance and an EBS data volume.

Public ami-7fc6ed0b >> Instance >> Domino Installed >> Domino Configured >> Private ami -xxxxxxxxx >> Instance(s)

An "elastic IP" address was used to provide a static address for the server.

I got to this scheme after

1) trying a single volume approach and then realising that the EBS will reset and loose its data after a termination.

2) realising that although the EC2 snapshot facility is really powerful there is no easy way to restore a snapshot as an EBS boot instance whereas it is very easy to do this for a dedicated data volume.

Getting the Domino Install files

I found the easiest way to get the Domino install files onto the new instance was to fire up an off the shelf instance of a Linux Desktop so that I could download the installation files from the IBM Passport Advantage or Business Partner web portals. Having downloaded the files I then created the data volume that I would later use on the Domino server, mounted it on this temporary instance and saved the files for later use. You could use FTP or winSCP to upload them to the server instance but my upstream bandwidth is pretty limited.

I created the desktop instance using this AMI. ami-94446fe0 Note that the AMI codes are different for different regions - this one is eu-west-1.

login using putty and the username root and your existing key pair. Note that you will have to use puttygen to convert the Amazon keypair first.
Update the instance with any updates.
>>apt-get update

now set up a temporary user as the NXClient does not have the keypairs installed and I just want to use password authentication

>>user-setup

Now start the NXClient and login using the new account

Created a new volume in EC2 using the EC2 management console. I sized this to be my data drive for the domino server.

Now attach the volume using the EC2 management console Once it is attached in /dev/sdl ( or wherever ) you need to format it in a terminal window
>> sudo mkfs -t ext2 /dev/sdl

You can then make a directory and mount it

>>mkdir temp
>>sudo mount /dev/sdl /temp

I found that at this point the volume is owned by root so I had to chown it to my temp user acoount. There is probably a more graceful way of doing this.

>>sudo chown tempuser:tempuser temp

Now you can launch firefox and download your installation files from Passport Advantage onto this volume for use on the new Domino server later.

When you are finished unmount the volume >> sudo umount /dev/sdl ( note the spelling of umount )

Then go to the ec2 management console and detach this volume.

Then terminate your instance.

Creating the master Domino instance

I created a new bootable EBS image using ami-7fc6ed0b, my firewall rules and key pairs ( the username is ubuntu ) . Once the image had booted I attached the data volume to the instance as /dev/sdl ( note that this is sdL not sd1 ).
Don't forget to update

>> sudo apt-get update

I created a directory called local and mounted the data drive into this directory using the command

>>sudo mount /dev/sdl /local/

Then I changed the ownership to notes

>> sudo chown -R notes:notes /local

I installed Domino using my installation files as described by Michael but note that as Ubuntu does not have a root user you need to take some extra steps during the install.

Create the notes user and group

>>sudo useradd -m notes ( should create group at the same time )
>>sudo passwd notes

Now as per Michael's scheme you need to copy the keypairs from the existing user ( in this case ubuntu ) to /home/notes/.ssh and then assign ownership to Notes. Notes that in this image of Ubuntu you are copying from the /home/ubuntu/.ssh directory.

from the /home/ubuntu directory do a directory listing using

>>ls -a . This shows that there is a .ssh directory

now copy the files to /home/notes/.ssh

>>sudo cp /home/ubuntu/.ssh/authorized_keys /home/notes/.ssh
>> sudo chown -R notes:notes /home/notes

Now expand the Domino install files as shown below

>>cd /local
>>sudo tar -xvf XXXXXXX.tar

now go to the expanded folders until you find the install file

>>sudo su <- this is important on ubuntu
>>./install

I then used WinSCP to copy the server the server ID file to the local/notesdata directory while logged in as the user notes. This ensures that the id file belongs to the Notes account.

Now complete the Notes configuration using the remote setup console as described by Michael.

Next I installed Daniel Nasheds startup scripts. These make it much easier to start, stop and monitor a domino server on Linux. The readme has the latest instructuons.

copy rc_domino to /etc/init.d ( use sudo )
copy rc_domino_script to /opt/ibm/lotus

>>cd /opt/ibm/lotus
>>chmod 755 rc_domino_script
>> cd /etc/init.d
>> chmod 755 rc_domino

Then set run level

>> sudo update-rc.d rc_domino defaults

you can now start Domino using the command /etc/init.d/rc_domino start

/etc/init.d/rc_somino stop and /etc/init.d/rc_domino monitor are also very useful

You also need to tell the server to mount the data volume at startup. As Ubuntu doesn't give the option to use root access with Winscp the easiest way to do this is to use the vi editor

>> sudo vi /etc/fstab adding the line /dev/sdl /local etx2 defaults 0 0

Once the Domino server is just how you would like it ( don't forget to set the correct host name in the server document and on the ports ) you need to make a new private AMI of this ready to run server.

Shut down Domino and Linux and then stop the instance ( NOT Terminate ). Now create a snapshot of the 15GB EBS boot volume and convert that to a private AMI so that when you terminate the instance you can get back to this point easily.

Now terminate the instance and create a new one using your own private AMI. Because the instance will start before you get the chance to attach the data volume you will need to stop it and attach the volume via the EC2 control panel - make sure that you use the same designation as earlier e.g /dev/sdl. When you now restart the instance it will automatically mount the data drive and launch Domino successfully.

You will also have to re-assign the elastic IP address. Note that I also found that I had to unassign and then reassign the elastic IP address anytime I rebooted the server from the linux console - I am not sure if this is a feature or how it should be.

With the server running you can take snapshots of the data drive as often as you want to. These can be used to restore data as they can be mounted as an additional volume and the files accessed. You can stop the server at will ( and your charges will stop ) or you can terminate the server and then quickly create a new instance. You can also create other instances from the same image meaning that you could very quickly establish a cluster pair as an example.

Setting up inbound and outbound email

There are quite a few details to be understood here and I am not yet convinced that all of them are sorted satisfactorily.

1) You will need to open port 25 in your firewall rules. You can do this via the ec2 control panel and the change is instantaneous

2) Amazon add all of their IP addresses to the black list at Spamhaus - so any emails that you send out will be blacklisted. You need to ask them to remove it from the black list using the form on this page. They send you a copy as soon as you try sending mail

3) You will need to use the same form to set up a reverse DNS which also means that you need a valid A record in place for your elastic IP

4) You may need to add the physical IP address of the network connection to the internet site document for SMTP.

This is the one that I am not convinced about ( and any advise would be gratefully received ) . Initially I was getting "Authentication is not enabled in the SMTP Internet Site Document" even though I had one set up ( and a Global settings document ). The move to the EC2 server did also coincide with a move to using "Internet Site Documents" as we no longer run Sametime so it may be that I have configured something wrong. The work around seems to be to add the actual network address to the SMTP internet site document. You can get this by using >>ifconfig in the console. The behaviour does not seem terribly consistent so I need to explore further. This restriction will add another task to your short list of things to configure when you launch a new instance.

update
I recently switched servers and this was an issue again. I definitely had to insert the local IP address for the server.
This also seemed to require a full Server restart

Costs

The costs of running the server are made up of four components
1) you need an IBM licence for the server which I have assumed you already had
2) the EBS volumes have a cost
3) the CPU usage has a cost and
4) there are data transfer costs.

Although these are clearly set out by Amazon It was not clear how much a chatty Domino server would cost overall - this was part of my wanting to try out the system - time shall tell.

One interesting point is the spot market pricing for CPU usage. I must be missing a trick here because there seems to be a loop hole. Normally an hour of CPU usage in Europe would be 0.095 dollars on the flat rate ( not yet sure about the VAT component of this ).

If you pay a fixed up front sum for 1 or 3 years you can get a reduced tariff of 0.045 dollars per hour. On the spot market you bid a price and your server runs while the price is below your cap - it terminates ( not stops ) once the price exceeds the cap. Historically this price has been below 0.045 dollars and in fact the most you can bid is 0.045 dollars. So I am running this instance you are looking at for <= 0.045 dollars with no up front payment - how does that work ?

Performance

Firstly let me say that this is not an enterprise deployment - it runs www.focul,.net, www.deliverytoolkit.com, this blog and is a demonstration and development server. Having said that the performance so far is at least on a par with the physical server that it replaces which was a 4 year old single cpu 2.8hz 1.5gb machine on Windows server 2003 ( yes I know not exactly a speed demon either ).

I have not turned on Transaction Logging as I am not sure what impact it would have ( no real spindles ) so that is an experiment for another day. The Amazon cloudwatch system ( a paid for extra ) is actually very basic showing CPU utilisation and network traffic. CPU Utilisation is low and memory utilisation ( 1.7 GB ) is over 95% but that is how Linux works. I'll post the trends next week.

If anyone would like any more specific metrics please ask, I am interested in knowing what your impressions of the performance are.

Well that's it, a bit long and rambling but hopefully of use. Please let me have any ideas or corrections so that I can update the article.