Pentaho Musings

I'm Off!

2011-11-29T15:59:00.001Z

Its been fun. Pentaho Musings is now closed for business. But fear not the trout lives on.http://themagicaltrout.wordpress.com/Trout, out.

Saiku its and its licensing

2011-11-19T15:54:00.001Z

I've been asked to write a blog post for a while now, and whilst not Pentaho specific I'd like to write a post about the licenses that Saiku operates under.

The Saiku server is shipped under the GPL v2 license. This basically means that is you are prohibited from bundling up the Saiku server and shipping with your proprietary software, but if you just want to install and run the software, fill your boots. The Saiku UI is shipped under a far more permissive LGPL license, this allows the UI to be shipped with pretty much any thing and is far more business friendly. So why do we ship the 2 components under seperate licenses?

There is no easy answer. We find many of our more vocal advocates criticising our choice of licencing, which I can understand, we basically prevent companies from shipping our software with their products. But there is a reason for that. We are a startup, in the greenest of terms. We have no working capital to speak of, we can't afford a team of designers and engineers to work on the software and so 99% of the work on Saiku currently falls on Pauls shoulders after Mark got a real job :(

I can honestly say we would like to ship the whole Saiku stack under a more permissive license. But do I think it would help us as a company grow? Sadly, I do not. We have no EE features and we have no plans to close source any part of the stack to make an EE product. People turn to us and say, "Well you should look at the Pentaho stack they're open source and they are doing fine" True, they are a business that is thriving, sorta. It is no secret cash is tight in the Pentaho towers, and its hard to make a buck selling open source software. I am eternally grateful that Mondrian and OLAP4J are shipped under a very permissive license than allows us to ship the core of our product without us actually writing the code. That is the beauty of open source and I am a huge advoate. But I can also point to Pentaho Analyzer, Pentaho Interactive Reporting and other EE components within the stack. These are all closed source and are deemed to be the real "value add" to the product. Without Analyzer and PIR, Pentaho would be selling support, and that is hard, damn hard. I also know I am preaching double standards here, and it annoys me also :)

Maybe we have got the licensing the wrong way around, maybe we should have licensed the UI under the GPL, and said to people, feel free to sell our server in your software, but if you want to use the UI you'll either have to make sure your code is GPL compliant, or ask us for a commercial licence, or, write your own. Who knows. But telling us that Pentaho corp make their software open source for the good of the community isn't correct either, if Saiku were to follow a Pentaho model then we'd be shipping a core, with a Mondrian server running, then we'd close source the Saiku UI and Server and supply it as a plugin, which is completely out of the question.

It was also pointed out to us that we could LGPL the entire stack and sell hosting. I love that idea, we're working on the hosting platform right now, but again, do I think just selling support and hosting for Saiku would cover our food bill? Not yet at least. We have a number of companies asking about commercial licensing options, if we were LGPL would we get the same amount of contact? Maybe, maybe not, we already know Saiku is used in a number of commercial SaaS solutions. A lot of companies would take the software and we'd see nothing for it, because bug fixes and committing stuff back takes too much time, I am glad though that we commit features and fixes back to OLAP4J, we're doing our bit.

Also going back to Pentaho's software stack, back in the day, before it become part of Pentaho, Kettle used to be closed source. So I guess the message here is, things change, hopefully we'll get a chance to change our license in the future when we are happy that our business is secure, after all if we fold, chances are Saiku will go with us.

So I would like to thank all of you who over the years have made donations to the Saiku cause both in man hours, software and money, your contribution is greatly valued, and I issue this by way of apology to all the open source people who believe in true, fully permissive open source, one day we'll make that dream happen, but in the mean time, I need to go and spend some of the few pounds we have made over the years on Saiku on some dinner.

In the mean time our server is GPL for those of you who want to use our software, there is nothing in that license that stops you installing as many copies as you like, and having as many end users are you like. Just make sure you abide by the rules ;)

Trout, out.

Introducing.... Meteorite Hosting

2011-09-24T14:00:00.001+01:00

Its been a while since I had time to write a blog post and as I have exactly 38 minutes of battery life remaining as I fly over the Alps on my flight to Rome for the Pentaho Community Meet up I figured I might as well draft one.

Along with some fun Saiku demo bits and pieces we are demoing our new managed BI platform to people at the event. The bit where Doug said I was dreaming something up on the plane really wasn't true, just a bit of deception to keep you all on your toes ;)

Anyway, we have been working on this plan on and off for the last 12 months, but due to real life commitments it has taken this long to finally make it come to fruition. What Meteorite Hosting entails is a platform that allows you to spin up and shut down, highly flexible BI platforms on the fly. Instead of following in the footsteps of companies like Full360 where they install software on your EC2 instance, we allow you to spin up Pentaho CE, and Saiku instances within minutes, backed with MySQL, PostgreSQL or the fantastic Lucid database. This makes the platform great both for POC's when you don't want to hassle your customer with requests for hardware, or full installs where your client doesn't want to have the outlay money on new hardware and data centre costs.

Our unique approach to hosted BI platforms allows you to truly mix and match software to meet your clients needs. In the near future we shall also be adding Jaspersoft and Spago to our offering. We are working closely with a number of vendors to allow us to use EE versions of their products on our platform, bringing you added features for a small premium, much in a similar way to being able to spin up CentOS virtual boxes, or pay a small amount more for Redhat instances. Of course the good thing about cloud based platforms is the scability, and we can have you up and running with a fully configured BI server within 10 minutes, which I don't think is bad ;)

Meteorite hosting as it is known, allows you to interact with your server and solution repository using Git so if you break something it is always versioned. You can also interact with our server using either the web based admin interface or using a command line tool, keeping people like Pedro Alves happy ;)

The product will shortly be going from alpha into a closed beta, allowing us to put the business processes in place to make sure you guys as end users will end up with a faultless service. We envisage a public beta becoming available in November, with a full launch in December.

We hope you like it, and find it useful. Feel free to pitch in with suggestions and ideas, in a similar vein to Saiku, this is a community product written for community people, and your feedback and input helps us drive the product forward.

Cheers,

Trout, out.

Quick update on Guided CBF

2011-06-22T23:18:00.006+01:00

So a lot of you have tested the Guided CBF and as usual I've got virtually zero feedback, but it matters not I shall plow on regardless :)

If you've built and tested the guided CBF you may have noticed a few issues that I've been too lazy to fix, namely the admin console was broken out of the box, and that there are no startup scripts which can cause issues as the bi server is pretty path dependent and needing memory tweaks to get its bulk working properly.

So Slawo was bugging me for fixes, so I have duly obliged, the download now mirrors in structure the Pentaho CE download and so, the admin console works when you start it and there are the usual Pentaho startup scripts in place for tomcat.

I can't test the batch scripts so I'll have to just assume they work, but the shell scripts certainly do.

Next up, dynamic hibernate and quartz data source reconfiguration, and 4.0RC build!

So if you want a highly customized Pentaho CE server build, with the latest CTools and Saiku for good measure head over to http://ci.analytical-labs.com/jenkins/job/Guided_CBF/ and sign up!

UPDATE (2011-06-23)

So I got bored waiting for a very slow ETL today, so there are a couple of new features available to GuidedCBF users..

Firstly, the GuidedCBF build now patches the FF4 fix so you shouldn't get any annoying XUL errors.

Secondly, to celebrate the Pentaho BI Server 4 launch, I have added a GuidedCBF4 build for Pentaho 4 CE RC1: http://ci.analytical-labs.com/jenkins/job/Guided_CBF4/

This is clearly an RC and I haven't done much testing, but is seems to work along with Saiku and CTools.

Trout, out

Get out your party hats, Saiku reaches 2.0 GA!

2011-05-31T12:01:00.003+01:00

So after 3 years of hacking, countless rewrites, plenty of arguments and missed promises, Saiku reaches 2.0 GA.. Yes I know although we never got to 1.0 GA, I think the complete rewrite and lack of bugs merits a 2.0 over 1.0.

Firstly I would like to make it abundantly clear, we do not consider Saiku feature complete, but we have reached a point of relative stability and enough features to give end users enough functionality to get going. We are well aware there is and will continue to be a huge list of improvements people want. That said we're not far of reproducing JPivot in a nice UI so we're heading in the right direction.

Secondly I would like to send out my own personal thanks to Paul Stoellberger, Prashant Raju, and Mark Cahill with out these guys putting in the hard yards Saiku wouldn't exist and I wouldn't now be head of Marketing and Noise Making (do I put that on my CV?).

Anyway down to business what do you get for your $0?

So big feature in 2.0 is the Pentaho Plugin, Paul has been working on creating the Pentaho plugin which is a zero config plugin and you can just unzip it into pentaho-solutions/system.

The plugin also integrates with the Pentaho Solution repository so you can use that to save queries, reopen and edit them.

Also don't forget how easy it is to embed the Saiku UI in your own webapp (its only HTML and Javascript) or you can just write your own UI, the server just speaks JSON over a RESTful interface.

Anyway apart from that, we still have all the main functionality, selections, swap axis, excel/csv export and drill through.

As I said at the top of the post, we aren't feature complete, 2.1, fingers crossed, will contain, drilling, multilevel selection and a charting beta. So Paul and the gang are working very hard towards that.

In the mean time, if you are a Pentaho user, please check out the plugin, if you want to test Saiku standalone against your database, please do, although you may not think we have a great amount of features, you'll be surprised how much functionality that actually brings you.

Anyway, I have to return to kicking Paul so he gets 2.1 out of the door sooner rather than later.....

Trout, out

Rolling your own customized Pentaho BI Server

2011-05-23T22:04:00.009+01:00

Ever sat around reading up Prashant's excellent tutorials on customizing the Pentaho User Console and wishing you had an easy way to do that, instead of tweaking the latest BI Server package from Sourceforge?

Now you can. A few years ago Pedro Alves created the Community Build Framework, the idea behind the framework is that it allows you to build customized BI Servers on a repeatable basis using a set of patches to patch the original tag from SVN. But as often as we suggest this as a sensible suggestion for people to use, people are afraid or unwilling to create their own dev server with Ant and associated build tools and spend the couple of hours setting this thing up.

So you know us kind souls at Analytical Labs, we like to be able to simplify BI for people, be it Saiku, or any other platform to help make your lives easier... so without further ado, may I present to you.... the Guided CBF....

So it looks pretty weird huh? As you guys who know me, know, I am a huge fan of Continuous Integration namely Jenkins. Anyway the idea here is that we provide you with a set of options and the Analytical Labs CI server will build you a Pentaho BI Server package that you can then download unzip and run, just like you would with the Pentaho built package from Sourceforge. The only difference between our package and theirs is ours is customized to fit your needs.

So lets have a look at a few of the options available to us:
User Email: don't worry I'm not harvesting email addresses, all this is for is so when the build is finished and the package moved to our webserver we can email you to let you know the build is finished and what the package is called. For security reasons we prevented people listing the web directory and so you get the link in your inbox.

Mantle Small Logo: Ever wanted to remove the logo in the top right of PUC and replace it with your own corporate branding? Well now you can

Main Logo: Similarly with the big logo on the dashboard when you run in? Fancy your own logo? Upload your own image.

BASE URL: Forget to set this regularly? Forget no more.

PUBLISH PASSWORD: Forget this one too? Yeah I'm not surprised.

Include Saiku: Okay so the next few are special. Saiku for those of you who don't know is soon to be a close competitor to Analyzer, sadly though our plugin has only just become available so we figured why not give you guys the option of including it. Its not mandatory so if you don't want it just uncheck it, that said, it does rock so we invite you to test it.

Include CTools: Similarly, keeping the BI Server up to date with the CTools from Webdetails is an endless burden, so we've made it simple, not only do we update the CDF to the latest stable build, but we also bundle the latest CDA and CDE builds so you can create dashboards out of the box!

Include Custom login page: Lastly on the branding front, ever wanted to replace the Pentaho login page with something else? Well, whilst this isn't the finished solution, it does do a half decent job and I'll improve this feature over the next week or so.

Login Page Logo: If you check the above checkbox, make sure you also include a funky logo to show.

So there you go, this is far from complete, on the roadmap, are pre installed data sources, moving you away from the HSQL hibernate and quartz databases without having to hack around for hours working out what needs changing, also soon we shall bundle the build scripts and patches directory so you can then build this exactly same build on your local server.

So, "How exactly do I use this CI Server?" I hear you say, well that's very simple. Firstly visit http://ci.analytical-labs.com/jenkins/ and in the very top right you will see a sign up button, go through the process and login. Then click on the Guided CBF Project(or click here). Once in there click on the build now link on the left hand side and fill in the boxes and hit submit. The idea is every box is optional but feel free to play around. Once the build is complete it will be moved to the Analytical Labs webserver and you'll receive and email with a link to download the package.

I know there will be many features you guys want that aren't supported, initially I was just going for the easy branding and so more features will appear very soon, there will also be bugs, please file any issues you can find and I'll get them fixed. In the mean time I have setup a feature request tracker here, please file any ideas for features you have and we'll add them ASAP.

Trout, out

Saiku BI Server Plugin - Keeping Analysis, Open

2011-04-26T12:47:00.005+01:00

So a few weeks have passed since we shipped Saiku 2.0-RC1. In that time the guys have been doing a great job and closing the bugs that have been found and tweaking the UI to try and fix up various usability issues that have been flagged.

Whilst thats been going on, and in between revision(although I think its more likely revision in between coding) Paul has come up with the initial Pentaho BI Server plugin for Saiku, making sure you guys get to use a real piece of Business User targets analysis software, and not JPivot. This is a great step forward for end users as you guys no longer have to leave PUC to use the Saiku webapp.

As I said this is a first draft and it does have some rough edges, for example you have to run Saiku as a separate webapp current, although this is no necessarily a bad thing as you can then get rid of PUC when you don't want it.

Give it a whirl and let us know what you think. I think this a is a great achievement and will allow Pentaho Analyzer like analysis to return to Pentaho CE, the good thing is of course we shall soon surpass Analyzer in terms of functionality and especially flexibility(as predicted) so make sure you let us know what features you would like to see the most.

Install Information: http://projects.analytical-labs.com/knowledgebase/articles/10
Bug Tracker:http://projects.analytical-labs.com/projects/saiku/issues
Roadmap:http://projects.analytical-labs.com/projects/saiku/roadmap

Pentaho London User Group - May. The Column Store Bake Off

2011-03-24T20:44:00.002Z

So we've had 2 Pentaho London User Groups now, both kindly hosted by Skills Matter Exchange in Central London. The first was a bit of a tester as no one really knew what they were doing
but fun talks by Pedro Alves of Webdetails and John Durrant of a big Orange supermarket made the evening fun. I think about 15 people showed up and it was hailed a success, little did we
know what was to come.....

The 2nd PLUG was a big event, this time Pentaho send Matt Casters and Thomas Morgner down to showcase some of the newer stuff they have been working on namely, 3.8 reporting stuff and PDI Hadoop integration.
This went down a storm and got around 40 people down to the event which was a major surprise.

So what can we do to surpass the previous PLUG I hear you cry? Well, it will be hard work that is for sure. So we have reps from both Vectorwise and Infobright coming down and hopefully LucidDB too, to take
part in a fun and interactive column store with Pentaho, 'bake off'.

Whilst our time is limited I will try and arrange this session to be as informative as possible whilst still allowing all vendors and equal platform to demo their software. If anyone has any requests, hints,
tips, suggestions then please let me know.

Also everyone seemed in agreement that there was too much talking last time, so this time we will try and time it so there is about 75 minutes of talking/presentations and the rest of the evening can be for
group discussions and networking.

So put 6:30 May 19th in your diary and we look forward to seeing you there! :)

An apology.

2011-01-21T21:06:00.002Z

Okay, I'm tired of the fallout.

I would like to get a few things off my chest.

Firstly, the tweet that seems to have upset people with Pentaho Corporation so much. I apologise, it was spur of the moment. I may not agree with the finer points of your support process, but twitter was perhaps not the best medium to share those thoughts, you can't put it into 140 characters.

Secondly my post the other day also regarding support, again although this was more thought through clearly it has caused problems within the Pentaho corporation, so it has been removed.

From now on, this blog will serve the community as best it can. But I shall no longer post anything remotely controversial inflammatory as I am tired of fighting.

I would like to make it very clear, I have never set out to deliberately hurt the image or reputation of Pentaho, just aid and facilitate discussion. I seems a blog is not the correct medium to do this as I have never intended to hurt anyone's feelings or upset people in anyway.

I would also like to take this opportunity to make it perfectly clear that at no point did I ask my superiors to deliberately ignore the support chain and contact senior management directly, regardless of the rumours going around.

Thanks to the community for the support, and apologies to Pentaho Corp,

Tom

Maintaining your BI with CI

2011-01-21T07:20:00.007Z

Okay, so I have to write the documentation for work, so I may as well kill 2 birds with 1 stone as I've been promising this article/book for years.

Over the last 18 months I have made use of Hudson Continuous Integration Server to maintain multiple servers running different Pentaho BI servers and database versions and data. So in the interest of sharing and collaboration, I have put the source code for the book on Github and the builds from the book on the Analytical Labs Hudson.

This is an evolving project and currently contains nothing, but over time this will expand.

Current chapter ideas include:
Tools Used in the project
Hudson Pentaho CBF Configuration
Hudson Pentaho CBF EE Configuration
Hudson Installation and Configuration
Other Software Installation
Testing
Upgrading
Tips and Tricks

This list can change and increase but this is what I'm working on now, feel free to suggest other ideas.

As the project is controlled by git, I am running multiple branches, master, review and released. They are pretty self explanatory, and indicate different stages of the project. You can download a PDF version from Hudson at any point. I shall also push released versions of the HTML book to the analytical labs website.

If you want to help or have other ideas then get in touch.

Trout, out.

PGPS Day Two

2011-01-21T05:10:00.005Z

Sooo what fun stuff went through the Twit Stream whilst I was snoozing from Day 2 of the Pentaho Global Partner Summit.

So from what I can tell, the day was started with the Jos Van Dongen hour where first Pentaho plugged his books by giving people free copies (why are there not any photos of people holding them up with their tongue stuck out?) , then Jos started his keynote.

Few interesting tweets came out, Jos' survey says that 50% of clients aren't interested in advanced analytics, a stat I find most surprising but I'm sure will change in the next couple of years.

Then came the moment I would have paid to see, Jos talked about Hadoop being a fad, which validates my point from yesterdays blog post. Jos then went onto talk about the "firsts" (although for some these are disputed) from Pentaho, IPad support, Metadata, AgileBI and Hadoop support. Then the community got a plug(thanks Jos), I like that, Partner <-> Community Interaction is minimal at best in most cases.

In the midst of this another interesting tweet went past, Luc claims that support for a Mondrian pluggable cache is nearly complete, this will be an EE feature, but one I can see being very useful.

Next came the BI Score card
Red: Office Integration
Yellow:Business Query/Reporting, Information Delivery, Dashboards, Administration
Green: Production reporting, OLAP, Predictives, Architecture.

Good results.

Jos then went on to compare the Pentaho BI stack with others including Jaspersoft and Spago.

During clearly a less interestingn part of Jos's presentation , Nick, JAdP and Pedro decided to have an argument about flash, and nice looking charts.

Next up the ramp was Jared to plug EE benifits. Support mainly plus some extra features, expect these features to grow in number over 2011.

Also plugged was PDI Server. Never having heard a good thing about it, I can see why it needs plugging ;) The PDI server is a very good idea though, execution needs fine tuning.

After Dave and Will plugged EE a bit more, some lightning talks took place.

First up: Spatialytics.org explaining Spatially enabled PDI.
Next: Pedro Alves with all the C stuff.
I think Pedro will have to spice up his presentation, the view of the graveyard is more interesting. ;)

KSK Solutions from Japan next showed an OpenID plugin for Pentaho
Brainchild followed showing off Liferay and Pentaho integration.
Then SQLStream, trying to get some customers

Post lunch, the attendees got treated to Analyzer and Analysis with Benny and Julian.
Matt gave a PDI chat, usual stuff, plug Cookbook, KFF and visualizing WEKA analysis within Spoon.

Lastly: Nick Goodman posted this picture of the view, he said its fantastic. Nick did you not notice the graveyard? :)

Looks like it has been a massive success though guys, and I'm looking forward to 2011.

Trout, out

Pentaho Global Partner Summit 2011 - From the guy thats not there ;)

2011-01-20T08:11:00.004Z

So the start of 2011 is upon us and so that means the great and the good from the Pentaho Partner program have descended upon San Francisco to witness the 100's of Pentaho VP's tell them whats coming in 2011.

As you can see everyone is thrilled to be in San Francisco.

So far there's been presentations from James Dixon, presenting the Pentaho Hadoop initiative. Pentaho Corp are betting heavily on Hadoop taking off in the BI Sector, personally, I'm still unsure how likely it is for Big Data to take off, but we shall see.

Also on day 1 was Jared giving a product roadmap, which included I believe more advanced visualizations for the BI server in 2011. I'll be very interested to see what they have in store. Mike gave a presentation about his long awaited Guided Adhoc reporting package he demoed at a community webex some time ago. According to the twit stream he said he'd release this to partners... good one Mike, nice community spirit ;)

Lastly I see from the agenda Nick Goodman gave what looks to be a very interesting presentation, about PDI and Mondrian mistakes. Clearly this was a very engrossing talk as nothing seemed to go through twitter, oh well :)

Anyway interesting stuff going through twitter over the last 48 hours, especially an amusing tweet from Pedro saying that someone in the QA session asked when Saiku would be released. To whoever that is, I salute you. :)

I'm looking forward to 2011 from Pentaho Corp, it will be a very interesting time, I know I moan a lot but its only cause I care ;) . Good to see everyone enjoying themselves in San Francisco, I'll be there to cause trouble next time ;)

Trout, out

2010 Round up, and looking forward to 2011

2011-01-03T13:32:00.002Z

Okay so I got a little bored on the last day of my holiday so I figured I'd do a 2010 roundup and a look forward to 2011 for PA.. er Saiku and Analytical Labs.

So what happened in 2010:

January: I got a new job in London.

February: I managed to get through 2 months without being sacked

March: The PAT Team released 0.6(better late than never), oh and Pentaho released some "Silver support" for its CE products.

April: PAT Webex, went down a storm.

May: Got my hands on the first of 2 PDI books, and PAT 0.7 was released

June: Pentaho release CloudBI, and I piss off James Dixon, not once, but twice, fun times.

August: I went to the pub.

September: I went to the pub, in Portugal.... and had a epiphany

October: We started PAT, from scratch, again

November: Saiku was born, and Nick Goodman kindly got us a new logo, from a strangely familiar company

December: The SSB showed how lousy Infobright was.

Okay so that was 2010, what are the plans for 2011?

Analytical Labs will finally come of age, we have huge plans for Spring and Summer 2011 which will bring a stable Saiku, a proper, functional, website, more kick ass software supplementing what we are already doing and providing more services to you guys, and plenty more.

So stay tuned. Here's to a fun and worthwhile 2010 and an even more exciting 2011 (where most of my prophecies come true).

Trout, out

My very dodgy col store database benchmarking.....

2010-12-07T16:37:00.016Z

Introduction
So work tasked me with benchmarking a few databases. Fun I thought, after 2 weeks multiple tests, 2 different environments, 1 lost spreadsheet and a few late nights... here we go.

I have created this page as an overview, for more details please see the attached spreadsheet.

Here(XLS)

Here(ODS)

Benchmark files and queries were supplied by Percona, who have run a very similar benchmark(http://www.percona.com/docs/wiki/benchmark:ssb:start the queries I ran can be found on their website), and got very similar results over a larger data set it has to be said, but work like stuff done on their own hardware, and who am I to argue?.

The Star Schema Benchmark is designed to be a "relatively true to life" benchmark, where the schema is very similar to what would be used in a data warehouse and the queries likewise.

The dataset was 90GB, data warehousing terms this is not a huge amount. This of course included no aggregate tables are I was collecting information on the databases raw speed.

Databases in question

For this experiment I have picked 4 of the more commonly used and or known(on ##pentaho irc channel at least) column store databases on the market today.

Infobright, probably the most well known and widely installed col store MySQL engine.
InfiniDB, another similar database, MySQL drop in replacement.
Vectorwise, a row/column store hybrid, currently one of the hottest pieces of gossip around the DWH fraternity.
LucidDB, a completely open source database, a member of the Eigenbase Project(along with Mondrian).

Each of these databases has a range of different features, pros and cons of which I have listed a number below, I do not consider this to be anywhere near exhaustive, but I've got other things to be doing so please feel free to mention others.

I did not alter Infobright, InfiniDB or Vectorwise's configuration after the install as they self optimise after detecting the spec of your server, Lucid does not do this so I manually altered the settings to match what I thought would be best for this database.

Infobright

Version: 3.4.2 GA

Pros:
Fastest Bulkloader
Very good compression ratio

Cons:
Bulkloader swallows errors as a trade off for faster loading
Per TB pricing structure

Disk usage: 17GB

InfiniDB

Version: 2.0.Final

Pros:
Detect if not enough diskspace and rollback bulkload
No extra per TB licence cost

Cons:
Missing client alias post install
No compression
Issues with table locks preventing query execution and table manipulation
Lacking some SQL features (truncate table etc)

Disk usage 92GB

Vectorwise

Version: 1.0.0

Pros:
Stupidly quick

Cons:
Had to restart server after loading lineorder data
General stability issues reported by others whilst testing Vectorwise 1.0.0 using alternative data NOT the SSB dataset.
General stability issues

Disk usage 25GB

LucidDB

Version: 0.9.3

Pros:
Warehouse Labels
Hot backups
Very small footprint
Strong Mondrian and Pentaho integration
Extendible

Cons:
Small company

Disk usage 71GB

Load times

As the lineorder table is vastly larger than the rest I have made this scale logarithmic to make is representative of the various load times. Bulkloading is the main way we will insert data into a database, and as you can see Infobright clearly has the lowest value over all the tables and is therefore the quickest.

First Run

So once the data is loaded I started running the queries on the database. This is the result from the 1st run of the specific query, below you will find the average time over 3 runs. I did recorded this value because it is the databases first search of the data and therefore has no data in its caches from previous runs. A few things to note are, I did not record a parallel execution time for InfiniDB because the idea of parallel execution came after I had remove it, and reloading the data is quite a lengthy task, and I had to give the server back before 2020. Also you will notice Q3.1 does not have a result for Infobright, this is because the query ran for a looong and then threw an error. As with the other charts the smaller the bar the better.

Average Time

Lastly I recorded the data over 3 executions of the same queries to monitor caching performance gains. Another caveat here being, I did not re run Q4.x for Infobright because of the length of time this would have taken, and if you look at the figures I think it would be a safe bet you would see very little to no gain in performance.

Conclusion

This benchmark, is not supposed to be highly scientific, I'm sure there are a number of things that can be greatly improved upon, but I'd like to think it is a good indicator of database performance. I also know there is far more to databases than query performance, but considering these are column stores I think its fair enough to concentrate on this and leave the other bits to people with even less of a life than me ;)

Each database has issues, this is certainly the case. The faster two both has some stability issues, that I'm sure will be fixed soon and can certainly be worked around at the moment. Lucid, is very small, but the company is growing rapidly and the new offerings look very good, but it doesn't win in the speed stakes. Infobright, well what can you say? It loads data real quick, shame you can't get it back at quite the same pace.

Which would I choose? That I guess boils down to what your priorities are, cost and query execution speed are certainly factors, loading speed and extra features are others. If speed was the only factor in the equation I would pick Vectorwise every day of the week, it is so much quicker at virtually everything, and its query caching is excellent. InfiniDB wouldn't come much farther behind, and i suspect it would also be cheaper on a multicore machine(although as I haven't heard from their sales rep I can't be sure of this).

If features, and extendibility were the things top of your list, LucidDB would win hands down, Warehouse labels allowing you to load up the next load of data without people being able to see it, and Hot backups which enable you to backup your data warehouse even if ETLs are currently being run on your machine are certainly very appealing, as is Lucid's excellent integration with Mondrian and the Pentaho stack. Also the soon to be released Firewater will be a great extension to the product allowing you to run many Lucid instances, distributed over multiple servers, greatly speeding up concurrent query execution.

Which wouldn't I choose? Infobright, its come last convincingly at every single query being executed. I have to admit I didn't find anything particularly wrong with it, it is stable and does the job, but it does it oh so slowly, worryingly so to be honest, one thing I did notice is that it showed no real speed up once you'd run a query once, where as all the others seemed to make good use of their query caching. Im sure Infobright can be optimised upon, which is something you would have to work on and I didn't have the time, but still, I don't think it will come close to matching any of the others for the time being.

Let the flaming commence.....

Saiku Logo Compeition

2010-11-23T23:03:00.002Z

So 7 days ago we started a logo competition on 99designs.com with an aim to find a nice looking Saiku logo, we thought hey, we might get 60 or 80..... 350 designs later, we're left with a final 7. Now we're gonna see what the general community thinks.. please take some time to have a look at what we are left with, and select your favourite.

Thanks

The Trout

Saiku - a new dawn.....

2010-11-05T16:10:00.009Z

Okay so after much confusion, stress, amazement and more the PAT-Ui team have been going great guns with the new look and user interface.
It looks great, its uber responsive and is written in JQuery so anyone can help us extend it.

Whilst they've been working on things, me and Paul have been busy preparing the ground for the PAT 2.0 RESTFul backend, this has caused quite a few headaches but we're getting there.

As you can see burgers are an integral part of PAT2.0

But one other thing that is important and we'd been working on for months is the name.... so from this point on we are re-branding. Ladies and Gentlefolk I give you Saiku-Core and Saiku-UI. Following on from a discussion with Julian Hyde about the name, we worked on various ideas including Japanese knives(slice and dice, get it? ;) ) and this is what we came up with. Products will be developed by Analytical Labs, but what was PAT will now be far more modular and as you can see, be called Saiku. A big thank your to Nick Goodman who is sponsoring a logo competition on 99Designs for us :) .

So the future, our goals have still not changed, we're still working on a market leading Analyzer that will break new grounds in data analysis, just the way we go about it is slightly different. A very modular design with a rest back end that will speak to our(or your own) UI in XML or JSON.

New resources include, our Trac based Wiki and bug tracker (http://code.analytical-labs.com/wiki), content will arrive with code. For now Saiku-ui's code will reside in our googlecode project (http://pentahoanalysistool.googlecode.com/). Saiku-core code will be in our own git repository and svn mirror(details to follow shortly).

Anyway that pretty much sums up this week, the UI guys say they will update the Saiku-UI demo soonish so you can play around with it, and server code will be materialising shortly.

Trout, out.

So what happened last night?

2010-10-20T09:35:00.003+01:00

So last night Me, Paul, Mark, Harris and Prashant got on the phone to discuss where we were going with PAT and whats going to happen with PAT-UI.

First let me get the present out of the way....
PAT 1.0 will be tagged on or around the 2nd November. I can tell you now, it will still have a number of incomplete issues and some plain bugs, but we're tagging anyway.
The reason for this is, we'll be fixing bugs for ages and with Prashant's new JQuery UI around the corner I think calling a halt to our current progress would be the most benificial thing the
PAT team could do. So there, 1.0 will be at the start of November.

What does that mean for post PAT 1.0?
Okay so we're tagging so we can adjust slightly, Prashant has been working on this alternative REST/JSON based UI, its cool as hell even though its just a POC so thats what we're going to provide in the future.
This also means a slight departure from the norm for the PAT team, from 1.0 onwards Paul and me will be concentrating pretty much on the server side stuff and features we can implement for people wanting to write ui's, we'll also be doing a massive refactoring to make the PAT server properly RESTful. Our aim is now to give the UI design remit to Prashant and let him create some kick ass lightweight, easy to use and highly visual ui(the sorta stuff we forgot when creating PAT 1.0) and me and Paul will concentrate on creating a serverside platform that 3rd parties will be happy to interact with and integrate into their product.

So whats coming in the new PAT-UI?
Okay so I admit, me and Paul got a little hungup in trying to showcase every possible feature available to us from OLAP4J and that we lost focus on what we set out to create, which was a very lightweight and stupidly easy to use OLAP browser. So thats what we're now letting Prashant concentrate on, we want him to think like and end user, implent a ui that his end users would be happy to use, but still offer flexibility for people to extend the UI should they want to(think integrated CCC and CDF).

I create data driven web portals for client and I would like embedded OLAP analysis?
Ah cool, welcome to the PAT REST server, here we will create an extension point for PAT that allows a RESTful way to interact with PAT and OLAP in a programming language agnostic way, so you can integrate OLAP into any application you choose assuming you can make HTTP calls and manipulate JSON or XML. I know some people have asked why we'd be doing this, isn't this just XML/A? Well, no its not . What the PAT REST server will allow you to do is integrate OLAP into your application and then on top of that still have a mondrian server setup in seconds, still allow saving and loading of queries(we're gonna put in a proper respoitory) still manage security for you and offer more features that XML/A wont. Also Andy and Roland were asking why we don't just build a Javascript Query Model? The answer is simple I want the server to be able to serve a fridge if you wanted it to and still serve up all these extra features should you require them in your application. Our sole aim for >1.0 will be to make the api as easy to use as possible whilst still offering the developers as much flexibility as possible,
allowing you guys to create a UI as basic or as advanced as you like.

Oh..... and it will no longer be called PAT.......

PAT update

2010-09-29T18:04:00.003+01:00

Thought it was about time I collated some thoughts after the Pentaho Community Meet and chatting this afternoon on IRC.

We are very close to 1.0, in fact the feedback from Portugal was ship it, and get ready to ship 1.0.1 real soon after. So thats basically the plan. We have a few more things to tidy up, but expect PAT 1.0 soon.

There are a few things I would like to point out after chatting this afternoon. PAT 1.0 is not supposed to be a finished product, just a milestone and an indication as to where we think we are with PAT. Ideas, as ever, evolve over time and there are plenty of new things we want to be able to include in PAT but would require some major reworking, so that is exactly what we will do.

Some of these are...

Modularize the various PAT components to allow greater flexibility with deployment and integration.
Work on the rest interface for 3rd party ui integration, and try and make certain aspects of the server stack stateless
Write up a fat stack of use cases and create a more streamline query generation and resultset viewer
Then once we've created these use cases, strip out the current ui, update GWT to something not quite as archaic and start the ui again.(Improving the good stuff and reworking the crap stuff)

I had someone tell me on IRC today...

"Somehow I feel like I've been driving a Porsche, and now I have to critique a car under development."

Which firstly made me chuckle and secondly is very apt. PAT is being readied for 1.0, but its not supposed to have everything we aim and aspire to in there, else we'd never reach 1.0 and feedback (good or bad) will be arriving for a long time to come which means our code and workflow will be ever evolving.

So.. keep the feedback coming in especially once 1.0 comes out because thats what drives our roadmap for 2.0, but preferably the feedback that tells us what sucks, and don't worry I can take it, I wont call you a bunch of whiners I promise ;)

Good stuff and who's hosting our 1.0 launch party?

Trout, out.

OT - Maven Jetty Plugin and multiple webapps

2010-09-16T10:15:00.006+01:00

Here's an off topic post and something I ripped off Justin Simonelis's blog, but I feel it needs documents, mostly cause his blog is the only place I've found that shows you how to do it.

So... if you have a maven project that depends on a second webapp that need deploying in the same container add the following to your pom.xml
<plugin>
<groupId>org.mortbay.jetty</groupId>
<artifactId>maven-jetty-plugin</artifactId>
<version>6.1.7</version>
<configuration>
<scanIntervalSeconds>10</scanIntervalSeconds>
<contextHandlers>
<contextHandler implementation="org.mortbay.jetty.webapp.WebAppContext">
<war>../dependencyApp/target/dependencyApp.war</war>
<contextPath>/dependencyApp</contextPath>
</contextHandler>
</contextHandlers>
</configuration>
</plugin>

And it will deploy you 2nd webapp at http://server:port/dependencyApp
*Note* Keep an eye on the plugin version, at first I didn't specify one and with the plugin version in my repo it didn't work.

Hurrah for Justin.

Tom

Quick roundup of the last few weeks/months....

2010-08-18T14:21:00.002+01:00

Finally got round to blogging again, mostly cause I'm stuck in a hotel room in India waiting for someone to take me to dinner.... putting that to one side I thought I'd put a few thoughts down on paper...

PAT.... I have to partly apologise for not getting to PAT 1.0 yet, new jobs, new house, Paul taking an extended holiday, and me messing around with REST services have all got in the way of me finishing off the last few blockers for PAT 1.0. So erm, sorry about that community folk, I can assure you its not far off complete so we'll get there in the next month I'd hope.

The reason I've been tinkering away with PAT and RESTFul services is partly for a talk I plan to give in Portugal at the Community Meetup so if you haven't yet booked a flight or train or whatever for the end of September, make sure you do.

Speaking of the Pentaho community infact after Dougs interesting forum post the otherday, I noticed there was a total of 50 people in the IRC channel so great stuff guys! And if you don't know what I'm talking about then come check us out, we aren't that scary, ##Pentaho on the Freenode IRC channel.

Anyway for now....
IndianTrout, out

Pentaho’s Road To Profitability - Take 3 - The Retort

2010-06-30T09:59:00.007+01:00

Firstly, I'd like to say congratulations to Luc, both for getting a job at Pentaho, I can't think of anyone more qualified from working with him in the community, and also secondly, for actually writing down some thoughts after my blog post yesterday.

I would like to make it 100% clear that I didn't set out with the aim of upsetting anyone within the Pentaho Corp, and we're all big boys and girls anyway so I'm sure we'll have a beer in Portugal and forget about it. These latest blog posts aren't supposed to be negative, just raising some points that I think need airing. Anyway I'm sure I'll have something cool to blog about soon! Also contrary to Rolands comment I don't mean to sound like Pentaho has lost any battle, far from it.

My point wasn't necessarily that the corporation was employing too many sales execs whilst leaving the dev behind. It was more that Pentaho have to be careful as we have seen many times this year when it comes to walking the tightrope between being an "Enterprise Ready" company, whilst keeping the open source ethics that got them to where they are today.

Someone on IRC said to me yesterday

if you hire people that are used to the closed source sales strategies they will use them and that is what starts happenign at Pentaho

And I think this is true.

But digressing slightly from yesterdays points, innovation hasn't been lost within the community and has been mentioned to me by a number of people. Looking at the Pentaho offerings over the last 2 years whilst in some respects it has come on in leaps and bounds, JSP PUC UI to GWT PUC UI, PDI V2 to V4, PRD rewrites etc, PDI in particular is a world class piece of software and is absolutely fantastic at the job it does, but in other ways the Pentaho Suite has been very stagnant. A number of what appear to be proofs of concepts going to market as 'finished' products. Take WAQR for example, whilst I think the metadata concept is great for adhoc reporting, the UI to utilize such a powerful language is woefully inadequate.

Agile BI has been branded pretty much as the 'next big thing' but when it comes to it, how do you create a proper mondrian schema from it? Its stuck in single table mode. I really hope that the 'Big Data' sprint Pentaho are currently embarked on gets to a point of completion before it is pushed out. It has been pointed out to me that maybe the community should finish off WAQR and give it back to Pentaho, that is a point I don't disagree with, but with WAQR being like it is, maybe it would have been wiser to include it as an optional plugin?

Keeping with innovation what new visible aspects of the platform have appeared recently? Pentaho Analyzer, this is true but it was a product that already existed and was just adjusted a little to integrate into the Pentaho platform. When it comes to innovation sadly it looks like Pentaho rely a lot on companies like WebDetails to provide it and then have it integrated into the platform. I really hope this changes with the arrival of people like Luc.

But does it matter? Whilst innovation within Pentaho isn't lacking, polished products when they are announced, are. But, I don't really see other companies releasing groundbreaking products, so are they falling behind? No, not really. They also need to add value to the EE offerings to 'add value', which keeps the 'tightrope' interesting.

Also am I saying companies have to innovate all the time to keep ahead of the game? No of course not. But a little innovation never hurt anybody and as we have seen from all the hadoop noise made its certainly good for a companies visibility.

I know full well what I'm saying is not going to do me any favours within Pentaho, but we're big boys and girls and these posts are just purely some random notes of a mad man. But I do write them for a purpose, not to damage any one, or the Pentaho Corp, but to create discussion. Nothing more, nothing less. And I do it purely for the benefit of the Pentaho Community which I have spend the last 4 years in and am proud to play the role that I do.

Just to point out to anyone swinging by this blog, I would like to make it absolutely clear that I do not work for Pentaho or have any connections with the Company.

Can you now release something fun and innovative so I can review it and write something fantastic about the platform please? :)

Pentaho's Road To Profitability

2010-06-29T10:46:00.005+01:00

*Disclaimer*
Ok, so what I am writing today is purely observations for an interested outsider nothing more.

I was having a chat with a guy yesterday who had been told by a UK Sales Exec that this years Pentaho community meetup was in Barcelona. It isn't.

I tweeted this fact and one OSBI analyst replied with a very stark

starting to sound like any other BI company where the left hand doesn't know what the right one is doing :-(

Now the question is, is that really the case.

You can't help but feel given Pentaho Corps recent marketing explosion(anyone noticed the huge increase in spam?) and drive to reach profitability is certainly suffering from growing pains. I met with one of the new VPs just after he started working for Pentaho and he had very little idea what the Pentaho stack was and how it hung together, understandable as he had recently switched following the Sun/Oracle merger, but clearly Pentaho are hiring high flying sales execs who have yet to come into contact with the product.

Pentaho have recently hired a number of big hitters, this isn't necessarily a bad thing, but you have to hope they back up their no doubt sizable salaries with the sales figures to match, otherwise Pentaho will walk headfirst into that startup problem where you are top heavy with VP's and senior management without enough people on the ground beating the bushes and selling the goods, and that isn't the route you need to take to be a sustainable business.

Pentaho as a business are notorious for spreading crud to help boost sales, and in the community this is not really a big secret. For example Pentaho try tell you that the Community version is not production ready which is clearly untrue as the EE product is built from exactly the same codebase, whilst I have heard rumor that at the same time internally they are happier if you use the CE stack because then you can commit bug fixes back to the codebase.

Surely though if your project is good enough businesses will sell itself without the need to spread crud.

This isn't supposed to be a scathing attack on a platform that I have used for years and have no interest in switching away from, I don't honestly believe that any other BI software supplier would work in any other way, but at the same time Pentaho has to remember that it has grown up around a community of Open Source advocates which is must not forget, whilst it slowly transitions to a more 'Corporate' model. Will Pentaho be profitable in a few years time? Yeah I expect so. As a corporation will they remember their roots in the OSBI sector? I'm not so sure. I can't help but feel as Pentaho moves "forward" jobs like the Community Liaison officer will become less and less important to the company and eventually completely done away with.

At the end of the day as a business internally between senior management and the guys on the ground, what they seem to be missing is, communication. Is this because of the recent growth of the teams within Pentaho? Or is it because they are transitioning into "any other BI company?"

Pentaho Launch Cloud Service....

2010-06-08T13:42:00.003+01:00

So its finally happened, an OSBI vendor offering cloud based BI. Pentaho today launched a cloud based service which brings flexability and extendability, of course being cloud based it means if you have 1 user or a 1,000,000 it should scale accordingly. I think this is great for businesses because the flexibility this brings is immense.

Pentaho use a virtual image for each client and appear to allow you to scale and manage the systems yourself, via an admin dashboard I guess. This is a very good solution as it allows you to move the images around, in house/in the cloud etc.

Does it help developers quickly deploy a POC or fully functional solution? Sure it does. But is there a better way of going about doing it? I think so :)

Watch this space.

Trout, out.

ETA to CDA release

2010-06-04T16:53:00.002+01:00

30 minutes.

Enough said.

Kaapa said it wasn't enough, so here's a link

Whats going on with PAT at the moment?

2010-05-15T17:16:00.003+01:00

After a hearty lunch in Leeds City Centre and the Mrs going for a nap I thought it would be an opportune time to divulge some of the latest PAT developments and what we've got lined up for the future. So here goes....

As most of you will be aware PAT V1.0 is very near completion, for this version we have completely reworked the query design interface to hopefully allow more flexibility at design time and before you have actually executed your query. With a new Tree view to browse your dimensions and a preview panel to show you roughly what will be displayed hopefully you will all be happier creating your queries.

We have also reworked the top button panel area and swapped it our for a very dynamic toolbar which Paul has done an excellent job working on. This frees up valuable screen space and I believe makes the whole UI look a lot tidier.

No longer does the table take up the whole width of the window if you don't want it to which makes the display look tidier.

And Paul has done a great job implementing Drill Thru, it works pretty well and will hopefully be of use.

There are a few more things to be done for 1.0 but we are largely there and we hope you like the release. We also have to say a big thanks to Aaron Phillips and Doug Moran from Pentaho for helping us a) get some code in place for 3.6 that we needed for proper BI server integration, and also b) to actually get it bundled in the BI server, I know this must have been a very interesting decision to make, so thanks to you guys.

We will also be packaging the usual standalone mode for those of you who may just want to run PAT 1.0 by itself.

A glance in my crystal ball...
Looking ahead is the big thing we now have to work out and here are some of the ideas that have been formulated....

Hopefully we will have plenty of happy BI Server users that are glad to see the back of JPivot, because we are a Pentaho plugin we don't have to stick to the Pentaho release cycle, so as you guys get more used to it and pointing our defects and improvements to us we can hopefully do quite an agile and improvement cycle on the 1.0 codebase and get updates and fixes out to you guys quickly.

That aside we shall also be laying out the roadmap for PAT 2.0 and its new modular design over at analytical-labs.com, we have plenty of new stuff planned for this and this is where the majority of our time and effort will go. The PAT Core module will again be a standalone and BI server pluggable module, this will do nothing other than serve data requested from external applications and will largely be structured around our current RPC structure, cleaning it up a bit and removing some duplication, one big change will be our ability to server XML and JSON data to 3rd party clients.

The GUI we supply will still be a GWT based GUI but rewritten largely from the ground up with flexibility and visualization in the forefront of our minds. The current result display window is too rigid for my liking(not that JPivot is any different) and I think users would prefer a more customisable layout.

Mobile apps, because of our ability to serve different data we plan also a number of mobile applications possibly for Apple, Blackberry and Android, but we shall have to see how much time and resource we can devote to such a project.

There will also be an admin panel coming soon, so people not wanting to tie in with the BI server can administer users and connections in a more sensible fashion than currently possible.

We are also looking to tie in with the Lucid guys, so you can easily deploy a PAT and Lucid solution out of the box if you are looking for a high performance solution but without the extra requirements.

Thanks....
So I'd like to take this opportunity to thank a stack of folk for their help, support and guidance in making 1.0 possible when it is released as I'm pretty sure when that happens I'll be down the pub, and the sleeping a bit, so many thanks to....

Jos, Matt, Bart, Luc, Ingo, Doug, Aaron, Prashant, Will S, Nick, Gretchen, Kees, various folk who committed translations(we still need you to keep them up to date), Julian, Will G, Brandon and I'm sure countless others who have slipped my mind and will berate me later for not including them, thanks to you all and to everyone that has tested PAT from its infancy to what is very nearly usable.

Anyway, with the update and thanks distributed..
Over and trout.

PAT 0.7 Is Released!

2010-05-12T12:01:00.002+01:00

After 2 months of hacking and bug squashing we give you PAT 0.7! We spent the last two months on improving usabilty of PAT and stabilizing the code base.
Thanks to our generous donors we could celebrate a 10 day long PAT hackfest in the UK, where we discussed current and future features, planned out a roadmap for PAT 1.0 and of course spent lots of time on actual coding. Thanks for your support!

One of the major outcomes of the hackfest can already be seen in PAT 0.7 codebase: We have decided to re-design the menubar for simple space reasons and usability improvements.
The other one is still under heavy development and should be released in the next version: A completely revamped and enhanced query design workflow! This will hopefully make finer tuning your selections a lot easier.

We will have one more development release in the next few weeks with the query workflow included, but now is serious bug fixing time before we hit 1.0 and BI server integration, so please, if you want PAT to be as stable as possible test it with some of your own data, hack around and see where you can break it, so over the next 2 - 3 weeks we squash as many bugs as possible and make PAT a success.

For those of you interested in testing PAT 0.7, you can check out the demo at http://demo.analytical-labs.com and download it from our googlecode page: http://code.google.com/p/pentahoanalysistool/downloads/list. If you wan the latest and greatest we also have a CI server in operation at http://ci.analytical-labs.com.

An Overview Of Pentaho 3.2 Data Integration Beginners Guide

2010-05-07T10:01:00.005+01:00

As I agreed to review the PDI Beginners Guide by María Carina Roldán but only currently have the eBook I will give a brief overview of the structure and initial impressions before giving a more in-depth review next week.

Maria is a long time forum moderator and community gal and has far more willpower than me when it comes to writing. Anyway......

My initial impression is wow... its long. At topping 470 pages, it must certainly cover everything.
A couple of moans that I would like to get out of the way before I list out the positives I see, a) the PDF quality seems to be a bit poor b) The pictures look very amateur, arrows drawn in paint, arrows not really pointing to anything and dodgy font. Not hugely problematic as its a book for grown ups not children, but something I feel detracts from what is otherwise a very good publication.

Right initial criticism over lets dive in.
The writing quality from the very beginning is very good, informative, to the point and geared squarely at the newbs. First explaining how to install and taking you quickly through a MySQL installation routine, then the basics of creating jobs and transformations and what the steps do. There are also often Have a go hero sections which give the reader small PDI based tasks to do.

The main idea with the book pretty obviously is to go through each available step in PDI and give the reader an idea as to how this step works. We cover file manipulation, data flows, working with databases, and basic data warehouses.

Whilst I don't think this a book you can read from cover to cover it is certainly a very useful resource and a great addition to any Pentaho documentation library. When I get stuck now when writing an ETL I will certainly know where to turn.

So get out and buy a copy.
Over and trout.

Is the IPad, ushering a new era in BI interaction

2010-04-12T14:28:00.004+01:00

Sitting here with an IPad in front of me to play around with, and the plans we have for PAT V2, I can't help but ask the question, is the new era of touch screen and advanced visualizations heralding a new era for interactive BI. I've seen the fuss that Lyzasoft is causing with their Collaborative BI.

So is the future moving away from products similar to the BI in its current incarnation and towards highly visual, highly interactive OLAP style reporting, that incorporates something like Webdetails CDA allowing multiple data sources and resulting in a singular easy to use interface that allows end users to quickly visualise what they want to know, without the need for a BI developer to create the report for them? And then incorporating all of that with rich query metalayer that allows you to tag all cells and charts with comments and extended information?

I'll think about this a bit more and post a more expansive blog about it soon, but I think this may just be the future but will the big players pick up on this, or leave it to the small fish and then buy them out?

Trout, out.

PS: I don't think for a minute that fancy graphs/charts and interfaces will replace traditional BI reporting techniques, but will add value to a product and complement traditional reports in a more visual format.

PAT Community Webex feedback and aftermath

2010-04-06T00:58:00.002+01:00

Thanks to those who tuned into the PAT webex a couple of weeks ago, work has been busy so I haven't managed to write anything about it until now.

I'm very happy everyone enjoyed what we had to offer and although we haven't had masses of feedback it has become very apparent what is important to you guys because the issues are raised by plenty of people.

Conditional Formatting is raised time after time as is easier sorting and filtering and this is something we are working towards for 1.0. There are also the usual slew of bugs that need fixing before then but we are pretty much 'feature complete'. This week I have been working on member properties support which whilst not being quite complete is nearly there, I am aiming for better support than JPivot with the ability to select specific members and it has been requested that you can sort on them, I dunno if thats possible or not with the current Olap4J Query Model APPI but its something I will certainly investigate. I have also committed support for caption columns which was in desperate need of adding.

Elsewhere the feedback regarding modularization of the PAT into more of a platform that allows people to hook their own services onto it has received plenty of support and thanks to Aaron Phillips for help with future BI Server integration. The feedback regarding the visualizations has also been very positive and will be something I am very much looking forward to implementing in post 1.0.

If you want to have your say about the future of the PAT platform please fill in the questionnaire here.

Overall guys this a GREAT effort, and Ive been working with olap systems for 20 years.
Oracle Essbase, Oracle Express, Microsoft Analysis Services..and this one of the best Ive seen. I think your plans for visualizations in PAT V2 looked wonderful. Especially if this could all work with the same speed PAT V0.6 works at

Its feedback like that, that makes you come back from 8 hours in the office and continue working on it instead of going to the pub, so thank you unnamed user.

Anyway Paul is over in a couple of weeks for a weeklong hack-a-thon, so speak up and tell us what you'd like done!

Thanks a lot
Trout, out

Pentaho finally decide to do some Pentaho CE promotion...

2010-03-26T14:29:00.002Z

So, I've been using the Pentaho CE stack for around 4 years now, and have sadly yet to get my hand fully on a EE install, Doug Moran contacted me about the exciting new Agile BI launch that is coming next week.

What he also said was that the Marketing guys have finally come around to the idea that highlighting success stories with the CE product isn't actually a bad thing, maybe if there are a stack of happy CE customer success stories it will convert into EE conversions or Pentaho Silver support. Also check out their special offers at the moment.

So without futher ado, if you use ANY of the Pentaho CE products within your business, be it just the reporting libs, PDI, the Pentaho Server etc etc, get in contact with Doug and let him know how and why you use it.

Trout, out.

Better late than never..... PAT 0.6

2010-03-17T11:24:00.004Z

Soooo after 3 months of delays due to job swapping and going to the pub, me and Paul are proud to present PAT 0.6. In this release there have been a lot of changes under the hood and a few visible ones. Here's a quick run down:

I've reworked the query selection routine so you can now select individual measures or all of them depending on what you want to see. Getting the logic right for all permutations has been hard work, and probably still has some quirks, so let me know if you can break it.

Paul has put some beta excel export code in and we are now also testing the XML for Webdetails CDA framework.

The plugin has improved a million percent thanks to Paul and Aaron and we now support automatic authentication and data source detection. Along with some query saving with in the bi server.

The new and improved MDX panel has become pretty much stable and is a great addition for those who like writing their own MDX.

I've tidied up some of the layouting issues on the OLAP table and fixed a few bugs relating to that.

We now have our own CI server which has helped streamline building and testing over the last few weeks which you can access here: http://212.227.135.127:8080/hudson/ The server builds standalone wars and ready to run test servers, so feel free to download and test as many as you like.

You can find the 0.6 release bundles at our google code page here: http://code.google.com/p/pentahoanalysistool/

What has Google Analytics done to our beloved dashboards?

2010-03-13T10:21:00.003Z

I have been brought up during my years of learning reporting and dashboarding techniques to subscribe to the Stephen Few/Pedro Alves/Harris Ward school of dashboard design, which is let the data speak for itself, and I don't deny that isn't a very important part of our job.

But I keep being asked, "where's the flash based charts and fancy interraction?". Has it finally reached a point where the Google school of thinking has reached the world of the dashboard and do we have to sit up and listen or do we just carry on doing what we are doing?

Simon has plenty of examples of good design on his website, which accurately depict the data in an easy to understand manner, but the everytime I present something like this I find myself having explain that BI doesn't live in the past and whilst we do have the capability to display data like this we like to stick to the facts.

The same was true at my old place, sadly I could 100% guarantee that I would capture more MD's attention if I used an animated pie chart to the one that was just drawn on screen. For the right reasons? Probably not, but at least with that I had his attention.

So is it time we ditched the above and went with something like this:

Or do we go a different direction and utilise more crazy visualization toolkits like protovis?

Is (Commercial) Open Source adoption picking up pace in the UK?

2010-03-09T14:47:00.004Z

I have to say, after following various news articles over the last few months, it appears to me that the stigma that appeared to be attached to Open Source products in the UK seems finally to be wearing off, with a lot of big name clients setting a trend that means smaller companies will be likelier to follow. Obviously coming from a BI background I am mostly going to concentrate on this as a an example but I think the observation holds water across the entire IT market.

Until recently pushing Open Source operating systems, databases, or frameworks to managers and clients seemed like awfully hard work with the lack of understanding of the open source market place. The

"how can this be free?"

mentality prevailed and so did the

"well we'll just do what the big boys did, BO/SAP everyone?"

Now just coming out the other side of a recession I think it has caused IT managers and consultants to rethink their strategy when it comes to systems integration.

"Should we go for the most expensive option, because its got to be the best?"

"Can we get a bank loan to pay for such things?"

"Do we actually need to buy product X, if we aren't going to use all its features?"

the questions being asked when evaluating software solutions are changing, and peoples' awareness of Commercial Open Source is improving.

If you take my current employers, whilst the desktop PC's are still mostly Windows based, all our servers are now Linux based, running Centos, we use a mixture of Apache, JBoss and JBossweb to deploy websites to 1000's of users on a daily basis, but they aren't afraid to mix that with the commercial offerings from the Oracle behemoth.

Now if you get back to my initial point about Open Source and its uptake in the UK we can take a look at Pentaho's client base increase over the last few months, scanning through the Pentaho Customers page you can see listed in the UK we now have:

Camden council in London
NHS
Specsavers
Close Premium Finance

Also Pentaho has signed up one of the leading UK supermarket brands, but thats not on the list, so I'll keep quiet.

This shows the rate of expansion in the UK is certainly on the up, in a noticeable fashion. I also went to look at the jasper customers, but they fail to mention any customers that I recognise as being UK based, although I am sure they will have some clients based in the UK. This list also obviously doesn't count anyone based in the UK that just used the free Community Edition.

If you take all this and then add good news stories like Pentaho at the TWDI bakeoff, I think the future is pretty rosy in UK for both Pentaho and Commercial Open Source as a whole(and its proponents ;) ).

Pentaho Meet(s) 2010

2010-03-08T14:37:00.003Z

I got half way through a post musing why the Pentaho community is so much more visible than the Jasper/Birt crowd, and then realised we all knew that anyway... so instead, for those of you not on the forums, we are organising Pentaho Community Meet Europe 2010 you can find details about the idea so far here:
http://forums.pentaho.org/showthread.php?t=75088

The destination is so far a race between Vienna and Lisbon, so chime in with your preferred destination.

Also for those of you State-side there is a expression of interest forum post for a North America community meet, and to be honest, I like the holiday, so express your interest here:
http://forums.pentaho.org/showthread.php?t=75089

If last years meet in Barcelona is anything to go by it should be good, as long as Haza doesn't loose all his possessions again.

TTFN,
Trout, out.

First month using 3.5.2.....

2010-02-02T08:08:00.004Z

So I'm about a month into my new job.... well 2 days short, but as I've not used Pentaho BI Server 3.5.x until now I thought I'd pass comment on what I've seen.

3.5.0 was a bit of a mess for those of us who like MDX based reporting, suddenly there was a stack of stuff we couldn't do because the new PRD was too restrictive. Work arounds had been invented, but they were messy, who wants to have to nest subreports just to get parameters working?

Anyway I knew 3.5.2 was around the corner so I set about just finishing off our ETL and creating the agile build and deployment routine, for both ETL, CBF BI Server build and our solution directory. Details of which you can find in my previous post.

The tagging of 3.5.2 came at a very good time so I quickly changed the Hudson SVN path and within 8 minutes all my BI boxes had been upgraded to run the 3.5.2 tag, shiny.

So I grabbed the Pentaho Report Designer M1 build and started tinkering...

Lets not beat about the bush here, it had issues, but that's fully understandable with a milestone build. I had a bit of a rant on the Pentaho IRC channel and Doug Moran kindly went off and let the reporting wizard out of his cave. Thomas spent a few hours in the weird world of IRC and gave me and the other lurkers on that day a lot of hints and tips with how to utilize the power of the new reporting designer.

PRD is still quite a distance from being completely finished and a complete userfriendly experience, but it has come on a long way, inline subreports, post processing of parameters, MDX string replacement and more. Yesterday I had to open up PDS to create an XAction for the first time since before Christmas, and whilst it felt strange, I can't say I miss it. Publishing a PRPT straight to the Pentaho Server and running it without having to hack around in the resulting XAction is certainly one of the biggest steps forward in developer useablity for a while.

When the idea was put forward that everything would be contained within one file and you'd be very reliant on Report Designer, I have to admit I was one of the sceptics. But after using PRD CI for the last month I have to say whilst its still work in progress... its also a very nice way to generate reports.

Trout, out.

==Edit==

I noticed after creating the Email XAction, Dankins schedule alterations to allow automatic email attachment appear in the schedule box. So infact no PDS is required \o/

The New Job and how we're devloping BI Stuff

2010-01-19T20:41:00.005Z

So I'm now 2 1/2 weeks into my new job down in London, I work for a RBT Company called Muzicall. Cool guys and they know their stuff. So I took over a half implemented BI solution and we've been working hard to achieve an agile self deploying solution. This is what we've come up with so far:

As usual we have 3 systems development, testing and production, pretty self explanatory, but how do we make sure each system is running exactly what we want, when we want? Hudson and some ant magic mostly. We use the Pentaho CBF build system developed by Pedro to build Pentaho, so when we want to make a change I commit a patch to our SVN repository. For the development box I have told Hudson to poll the SCM so it automatically picks up committed changes, builds using our CBF build script and deploys them to the development server, Hudson has a webapp deploy system for wars so we use this to do a hot deploy on the running web server.

Ben, my system administrator showed me a cool tomcat alternative called Jbossweb, and so we have used that, I dunno really what the differences are but he already had it up and running and showed me some cool stats and charts, so that will do for me, from my point of view I've not really seen any differences, but it must exist for a reason.

To make sure that our datawarehouses remain in sync across dev, test and production we use another cool tool called DBDeploy. Instead of using kettle to update each warehouse we again use hudson, and create individual change scripts for each call to the database. DBDeploy also allows you to roll back if something goes awry, and knows what patches have so far been committed to which database. So once again we commit to hudson, and depending on whether its a trunk, tag or branch, hudson knows which database to deploy to. Pretty cool.

As far as databases go we currently use stock MySQL, as ever I'd advocate Lucid, but the general consensus is that we'll probably go for a MySQL replacement, InfiniDB seems the most likely.

So there we go using the power of CBF, Hudson, DBDeploy and a bit of time we now have a self compiling, self deploying BI Server and data warehouse, which will hopefully save a lot of time managing stuff in the future.

Short and sweet,

Lucid Date Manipulation

2009-12-23T08:41:00.002Z

Just a quick bit of knowledge for people unsure how to manipulate dates in Lucid's SQL syntax it goes a little something like this:

select * from myschema.mytable where date = current_date - interval '1' DAY

As you can tell this will subtract 1 day from today. Obviously there are many other operators and you can also use HOUR, MINUTE, MONTH, SECOND, YEAR.

What happened with Pentaho in 2009?

2009-12-21T16:11:00.004Z

I got bored doing housekeeping stuff as my time winds down in my current job, so I thought I'd look back over the year that was and come up with a quick time line over what I believe was quite a pivotal year in Pentaho's evolution so far.

Ok, so I'm gonna cheat a little in the interest of reminding people how far the stack has come, so first up we have :
December 2008:
Pentaho BI Server 2.0GA, thats right it was only 12 months ago Pentaho 2.0 and its Mantle goodness was thrust upon us to move us from the 18th century JSP's to the 21st Century GWT goodness. It brought the interactivity people dreamed of and the modding nightmare most of us hope never to have.

Jan 2009:
PAT 0.0 was shown off, I got bored and decided to take up the work of the Halogen demo that Bill wrote and the PAT project was born.

Feb 2009:
PAT 0.1 was released which wasn't much more than an updated and slightly more hackish Halogen, but the main thing was people started to become interested in a JPivot alternative.

March 2009:
Strange packages started appearing on my doorstep to help freak the Mrs out. (I no longer live in that house by the way).

April 2009:
PAT 0.2 was released after yet more major hacking and much help from Luc and Paul.

May 2009:
The PAT BI Server integration plugin was written and demoed by Gretchen as an excuse to do something before returning to the Pentaho fold.
PDI 3.2.0 was released, the latest and greatest PDI version was release to the masses, bringing a stack of new plugins and a load of nice new usability features.
Pentaho 3.0, after what seemed like no time at all, another BI Server major release arrived, bring with it, who knows what(something to do with plugins I believe), but it seemed to make people happy.

June 2009:
Completely unrelated to Pentaho, but Mimil release Gothic XML, possibly the most important tech event of the year.
PAT 0.3 was released with yet more fun stuff and more people got interested in it.
Oh, and everyones favourite BI people, Lucidera went bust.

July 2009:
PAT 0.4 was released(notice this recurring theme)
And WebDetails went CDF Commercial releasing a stack of well written CDF tutorials.

August 2009:
Finally someone wrote a book... After many months of writing and eating Jos and Roland release Pentaho Solutions, a highly recommended Pentaho book.

September 2009:
Not to be outdone, Will Gorman releases Pentaho Reporting for Java Developers. A book aimed mostly at people interested in integrating Pentaho Reporting into their own application.
Barcelona Community Meeting, how can we forget it. The 2nd Pentaho community event, organized this year by me, and attended by a load of Pentaho Devs, and a good number of Community folk. Much beer, talking and food was consumed.

October 2009:
Pentaho release Pentaho Analyzer, Pentaho bought ClearView from LucidEra, and release it as Pentaho Analyzer allowing EE customers to have the benefit of an OLAP Browser that isn't JPivot and is more geared up to the business user.
PAT 0.5 Released, told you, recurring trend.

November 2009:
I got a new job.... at last.

December 2009:
PAT 0.5.1... again.
Pentaho release 'Silver' Support for PDI CE allowing businesses to gain support for PDI Community Edition for a reduced rate.

There we go, quite a year and I'm sure I've not remembered or dug up half of it. Let me know what I've forgotten so I can add it to the list.

Trout, out.

Pentaho Provide Support to PDI CE

2009-12-10T19:36:00.003Z

Pentaho have launched a new PDI CE Support service which for my current position comes about 12 months too late... but its a fantastic plan, this gives businesses with less of a budget or pointy haired bosses who are unwilling to part with the $$ for a full support scheme and the EE software, to gain professional support for Pentaho's CE offerings.

This is currently limited to Pentaho Data Integration, but I am very much hoping to see this plan rolled out further assuming it proves successful.

Here's the deal, there are 2 CE supported schemes the Silver scheme, for which in return you recieve a number of incident support requests, guaranteed response times, web and email support and access to the very valuable knowledge base. You can also upgrade to the Gold scheme which gives you a lot more support requests and faster response times.

This is ideal for the small business/1 man BI department that could do with some additional support to help get them setup and troubleshooted. Should you require more support or some of the EE features, I am sure you will be able to migrate painlessly to an EE based support license.

So if you are in need of some PDI support, buy it now :)

The Pentaho Community and how we work...

2009-12-08T19:42:00.004Z

Jerome Pineau wrote a very long blog post yesterday about how support can sometimes be lacking when it comes to getting up and running, so I thought I post a bit of a rebuttal to go along with Doug Morans' response.

Whilst it is true there are a number of Pentaho employed developers very active on the Pentaho forums, an awful lot of questions are answered by the unpaid minions doing it.. well because we do. When I started out using Pentaho 2 and a bit years ago there was no book, an irc channel with about 4 people in it(for the first 2 days I was there no one said anything, until Haza said "who's this bugg guy?"), but, the forums were pretty active and had some of the answers, but not all of them.

Sadly my employer has deemed it unnecessary to pay for Pentaho support(because they are tight, but thats come back to bite them on the ass as I am their POC and I'm leaving), so as I am acutely aware of how much time and effort and money goes into creating such a complete platform, I took it upon myself to start answering as many forum posts as I could whilst twiddling my thumbs at work as it was a way I could at least give a little back.

A year or so ago, Doug kindly gave me moderation rights on the forums, so I could clean it up a little or move stuff into the right place, and whilst it says Forum Moderator under my name it does not mean but any stretch of the imagination that I secretly got brown paper bags with wads of cash in, and so for that reason whilst I have to admit on occasion my answers are a little terse, I feel that as I am not selling a product and merely providing a basic level of support, that is not really my problem. But at the same time I appreciate that I am to some degree part of the 'Community Face' of Pentaho and therefore have to conduct myself in a reasonably professional manner.

Forum posts you will generally fall into roughly 3 categories and here's pretty much how they get answered by community members, or more importantly me:

Repetitive easy stuff: There are a number of 'newb' questions that get asked daily and would take about 2 minutes to track down using the various tools at that persons disposal. These often either get a "use the search" type of response, or a one line link to somewhere else. I don't use the line "look it up yourself" because its amusing or rude, I do it because hopefully they will do that first the next time and save the time it takes me to look up the answer and they may learn a thing or two along the way
Well asked questions that community members may be able to answer: These questions go something along the lines of, "I've been trying x but it doesn't work and I need to do y,z to reach my goal", if one of these comes along and I have an idea as to how you'd do it, I will give as detailed response as I possibly can so it serves as a reference to that person and anyone else who finds that post in the future.
Questions that are impossibly tricky/impossible and take a developer's inside knowledge to answer, or questions that make absolutely no sense: These questions are invariably sadly the ones that go unanswered, and its not through the fault of the community, most of the communities knowledge only goes as far as their job or experimentation has taken them and for that reason there are boundaries, and these questions will get left, that being said, there isn't anything wrong with "bumping" a post if it goes unanswered over the course of a few days.

Also the likelyhood of you getting a response in a reasonable amount of time sadly depends on the area of your problem, reporting and PDI posts get a lot of help because they are the most commonly used areas of Pentaho, also BI Server support is pretty good, because everyone's seen the issues before. Sadly Mondrian support is lacking but that is mostly because a lot of the problem areas require a lot of in depth knowledge, that being said the Mondrian mailing list is very good for that.

There are also other options, it is an infinite amount easier, for example, helping someone in 'real time' over IRC than it is working on the forums, because you can extract all the information you need from the user, very quickly.

I am proud to be a member of a community of volunteers that is so supportive of others and whilst I wouldn't regard the community support as being perfect, as far as an Open Source solution goes, it aint that bad.

Also if you really need help buy support or the book!

Trout, out.

PAT 0.5.1

2009-12-01T20:57:00.005Z

Right, PAT 0.5.1 is out with a bunch of new half finished features ;)

I'm sick of showing off static screen shots so I put together a quick video showing off some of the new stuff(although I forgot to show excel export and I can't be bothered recording a second, sorry Paul)

You can see the real video here

Trout, out

My Dad Shaves Away Years Of Beard

2009-11-25T22:11:00.003Z

My Dad shaved his face today for the charity(Wilfward) that my mum works for:

Quite a bizarre sight! He's been a teacher forever and retired in September, so he has lots of spare time to fill ;)

Pentaho Reporting 3.5 for Java Developers..

2009-11-17T21:11:00.003Z

So ages ago I got sent Pentaho Reporting 3.5 for Java Developers by Will Gorman or Pentaho Dev fame, and was asked to review it. As everyone who follows this blog and irc will know, my life has been in a bit of a state of flux so I've been short of time, but whilst reading it I did write some notes which I will now translate into a bit of a review.

So we start with a well structured history of Pentaho reporting, coupled with some nice screenshots the obligatory terrible photos and a good way to get the read acclimatised with Pentaho reporting, what it does and where it comes from.

Next we move on to a very good introduction to reporting, in this chapter we run through Pentaho Report Designer basics and Will also explains in great detail how to create a basic swing application and access your report from within that application. This is a very good chapter for those people, like myself, who know Pentaho Reporting is Java based but have no idea how you would integrate your report into your application.

In Chapter 3 Will runs through some real world reporting scenarios, the chapter is very similar to the previous one but we expand on what we have done by adding basic charts, learning about groups and other features in Pentaho Report Designer and finally integration into a J2EE application.

Chapter 4 dives head first into Pentaho Report Designer design and layouting techniques. This chapter goes into great detail explaining a lot of the various properties and key features you will end up using.

The next chapter runs you through the various data sources available to you in Pentaho Reporting, this is a very important chapter if you're not planning to use basic SQL to retrieve your data, it runs through the various interfaces gives samples for each connector and hopefully allows you to connnect to your data in a sensible way.

I've run through the charting chapter in a previous post so I'll skip that, if you're interested in reading about the charts then check here.

Chapter 7 runs through the various parameters, functions, formula available to you in Pentaho Reporting, and there are a lot of them, there's a list of all the current functions available and how they work followed by an explanation of the formula Pentaho Reporting uses and the syntax involved. Lastly in this chapter Will writes a brief explanation as to how you can internationalize your reporting allowing different locales to see the labels etc in a different language.

Next we move on to subreports and cross tabs. Sub reports have been available for years in Pentaho Reports but 3.5 brings along some welcome improvements, notably side by side subreports. Will gives and example of this and a very detailed standard subreport example. Crosstabs are very new to Pentaho Reports but Pentaho Report Designer does have some hidden support for them, Will shows you how to access this support and how to create a basic SQL and MDX cross tab report, this is a very useful technique but at the moment a little complex, so be warned!

Chapter 9 introduces interactivity to your reports, using action listeners. There is a Swing based example and also interactive HTML. Obviously this is of great use to most people as we aren't just supplying static reports.

Chapters 10 and 11 are where it gets really hairy. Chapter 10 gives a very good explanation as to how you would create reports solely using Pentaho Reporting API. This isn't for the faint of heart but it could prove very useful to study to help you understand how Pentaho Reporting works. There is a handwritten XML definition, detailed information about every file in a PRPT bundle and how to create a bundle. Will also gives detail instruction on how to build a report using the API code. The next chapter goes on to explain how to expand Pentaho Reporting using various custom expressions and formula and an example as to how you can use Bean Shell Expressions with in your report.

We finish this brief insight into the book with the last chapter, chapter 12. I can't help but feel this is slightly unnecessary apart from to keep the Pentaho gods happy, which in itself isn't a bad thing if more of these books keep appearing. Anyway its a brief run through of the BI Server and Reporting integration into the BI Server. But then it does also go onto explain how to run the report to different output formats to pipe stuff off to POJO's etc.

All in all a great book and a very detailed introduction to Pentaho Reporting that will certainly serve as a great reference manual on my bookshelf for when we come to sort out some reporting functionality in PAT.

Good stuff Will and my apologies to Richard Dias at Packt Publishing for not having this online earlier. If you want a copy click here.

Trout, out.

Upgrade to 3.5

2009-11-12T16:17:00.004Z

So then, today I chose to upgrade the 3.0 BI Server at work to 3.5, great news, and it got off to a good start by 9am I had most of the stuff swapped over, authentication up and running and pretty much good to go. The rest of the day hasn't been so great, I thought this morning it would be a good idea to download Sugar CI, because I know people have been suggesting it due to reporting modifications made after 3.5 was released and things were complicated.

So I grabbed it downloaded, extracted, set it up, got it running, logged in, went to change some folder permissions... damn, the Properties window doesn't work, FF and IE throw a javascript error. Stuff it I thought, I'll play safe and use 3.5 GA, so I grabbed that and got it mostly running just before lunch, swapped over the config again, this time bothered to configure CAS hacking around my old ACEGI script, Spring Security btw is great, the script is so much cleaner.

Anyway got everything setup, then discovered the MDX parameters I needed to parse the date stuff only work in Sugar CI, balls. So I've grabbed a 3rd BI Server from CI downloaded and quickly tested it, also broken.
Since these modifications made post 3.5 GA were by a lot of people deemed very important, well important enough for people to go into production with a CI build, why has Pentaho not released a 3.5.1 or similar so we don't have to go around matching builds together and finding ones that work?

Enough moaning, I appreciate CI is exactly what it says on the tin. Anyway later tonight, I'll finally get around to posting a run down of Will Gormans book to keep Packt happy.

TTFN

Tom

Bowing out... for now

2009-10-30T20:58:00.005Z

The last 2 days have made me realise how much I need a change in workplace, not necessarily career direction but the 12 hours in 2 days commuting has brought it all home.

So until I get a new job whatever that may be(, to be honest at the moment I'd take a first line support job if it meant I wasn't getting home at 9pm), I will not be on IRC, or moderating the forums as I feel its too much of a distraction from what I really need to be doing which is hitting up recruiters and potential employers. I've spent the last 2 years sorta helping people and now its time I helped my self!

So cheers guys, I'll be back soonish, PAT guys, you can mail me or hit up the dev list, I'll still be working on it in my spare time, as usual ;) , Pentaho guys if you need anything you know where to find me.

See ya in a few weeks(hopefully),

Trout, out.

Lucid looking to the future

2009-10-21T06:41:00.003+01:00

Lucid was recently bought from the firesale at LucidEra by Nick Goodman. He's looking to take it forward and with the help of JVS bring new features and a commercial version. For those looking to help out taking LucidDB to the next level check out the mailing list, website and finally they've got around to sorting out an IRC channel ##luciddb on freenode.

Good stuff guys and I'm looking forward to the future.

TTFN
Tom

PAT and Paypal

2009-10-15T12:41:00.004+01:00

Just a quick note to all you generous souls out there, we have attached a paypal button to the PAT googlecode page here: http://code.google.com/p/pentahoanalysistool/

If anyone is feeling generous enough to help us maintain our Maven server and the public testing server we use, or just buy us beer... then you know what you can do ;)

A big thank you to anyone who donates!

Cheers

Tom

Google Analytics Traffic

2009-10-14T06:23:00.003+01:00

GA now reports our hits 35% above normal which is great stuff, and all this pretty much spawned from the Clear View buy out. I thought I'd show you the chart....

The area I colured in a faint yellow on the right is the are since clear view was announced to the world.... I'm quite impressed ;)

Cheers

Tom

Pentaho Analysis Tool 0.5.0-Snapshot

2009-10-13T21:07:00.004+01:00

Due to the fact we had the PAT Community Webex last week, and a stack of twitter and blog space this week, we have zipped up the latest 0.5.0 Snapshot and uploaded it to googlecode.

This isn't supposed to be a stable release just a snapshot of where we are and a good indication of the direction we're headed. We have some basic drilling in there, although the lack of true drilling support is proving problematic. We also have some basic OFC charts(Select the chart panel first before running a query, else you don't see the buttons allowing you to change chart type.) We also have a new dimension browser and other stuff for you to play with and break.

Coming soon we will have a new query builder that will basically allow drag to table, and also we will have a brand spanking new theme to bring us right up to date. Hopefully I will also find some time to crack query model saving and loading so we can soon bring that to you as well. I think once those features and whatever else we can add when we're bored and want a challenge we will look to consolidate, bug fix and ship a fuller release that will bring us closer to 1.0 sometime still targeted for around Christmas.

FOR THOSE WHO TEST IT, IT RUNS ON PORT 9999 SO YOU CAN TEST IT CONNECTING TO THE BI SERVER XML/A. LOGIN IS: ADMIN and PASSWORD: ADMIN

Let us know what you think!

The Differing Opinions Of The Clear View Buy Out

2009-10-13T06:06:00.005+01:00

Morning All,

A week or two ago Pentaho announced the purchase of Clear View from Lucidera and its been very interesting to see the range of opinion this has conjured up, which I find interesting because when it was announced that the Dashboard Designer was to be closed source, no one seemed to care, is it the right of every member of the Pentaho community to have enterprise level OLAP Analysis, or just those that cough up the money?

Seth Grimes has an interesting article asking whether Pentaho can really still call itself Commercial Open Source
James Dixon seems far from impressed with this blog post and has posted a retort pointing out that Pentaho already include the closed source Dashboard Designer and still offer far more open source functionality than the rivals.
Merv Adrian would like to point out that Pentaho is now technically 'Open Core'.
and lastly Jos Van Dongen covered the release using words that I don't even know the meaning of(doesn't take much)! ;) But goes on to say that there's nothing really wrong with including it in EE but make sure the community edition doesn't suffer in the process.

What I have noticed over the last week is:

I have yet to bump into anyone bitterly dissapointed that Clear View is EE only, although maybe thats because there isn't yet a demo available so people haven't seen it?
I thought it would be a kick in the pants for PAT, far from it, its provided us with plenty of interest with website traffic currently up 28% and many more people enquiring as to when its been ready.
Helpful suggestions and comments have increased, offers for help testing etc

So what does that say about the community, well I thought that JPivots flaws would see people desperate to buy EE and Clear View but that seems not the case. People seem far more interested in when we'll have a community version of PAT ready to replace JPivot. Now I know I speak for the community and not EE customers, and I have no knowledge of their thoughts and feelings behind Clear View.

My only real concern is that Pentaho will now be less inclined to include PAT within the server when we are stable as it would devalue the Clear View tool. I don't think I'm stepping out of line when I say that, it makes sense. But on the flip side I do think Pentaho need to include more stuff in their EE version because at the moment to companies like mine, support is not a big enough incentive to upgrade.

Anyway we shall see.

Thanks for all the support and we'll have a very unstable 0.5.0 out real soon for you all to break \o/

Cheers

Tom

The Open Source End User Mentality....

2009-10-09T07:26:00.002+01:00

I was pondering things this morning when I hit upon the problem, its the mentality of the open source end user.... When I sit in a presentation and ask nicely for end users to throw a bit of time in our direction, do I seriously expect an influx of willing volunteers? No, not really. Why?

Well I am as much to blame as anyone else when it comes to using Open Source software and when finding a bug or problem, actually attempting to fix that bug, if it involves me downloading the bi server source, and debugging in Eclipse, then you can pretty much forget it, I haven't go the time, I tell myself, I have better things to be getting on with, I tell myself, unless its a blocker in which case I may make half an attempt to solve it, then give up and get everyone to vote for it on jira, so why should I expect that mentality to be any different towards PAT? Well, I don't.

Apart from a few translations, Gretchens BI Server plugin and a few other bits and pieces, which I am very grateful for, we haven't had a single bug fix or patch come our way, as I presume everyone sits there and goes... oh, it will be done at some point, then we can start using it.....
And to be honest, I don't blame you ;)

Tom

PAT Webex

2009-10-08T06:03:00.003+01:00

Thanks to those guys and gal's who sung by for the PAT Webex, 30ish people turned out,o and from the feedback it proved largely successful even if it was rather hastily slapped together. Quick thanks to Doug and the Pentaho guys for organizing it, thanks to Jos for annoying me about it all week a making sure I don't have a life ;) And thanks to Benjamin from SQL Power for swinging by and checking out the half competition, and making me sure we do better than them, kidding, honest ;)

Anyway if you missed the delights of me droning on about where we are going check it out:
http://forums.pentaho.org/showthread.php?t=72655

Oh and also our PAT googlecode site and this blog's traffic is up 20%, so thanks for stopping by

Ta

Tom

Pentaho's New Addition To The Family

2009-10-05T13:54:00.003+01:00

As you will find out at some point today, Pentaho has bought Clear View from the now defunct LucidEra to bolster their Enterprise offerings and replace the old JPivot.

I've had a few people come to me today and ask where that leaves the PAT project, I would like to make clear in no uncertain terms, that whilst Clear View is an obvious JPivot upgrade, the fact it is only in the Enterprise version of the Pentaho Suite, and it is very closely tied in with Mondrian means that nothing has really changed for us in that regard. We are still looking to write what will become the best Analysis tool in the world (people think I'm joking when I say that..... I'm not), and whilst it can be tied into the Pentaho server, this will be available in the CE version and at the same time remain standalone so it can be shipped with databases and run without a Pentaho server installed.

So after all the screenshots you have seen and the webex on Wednesday, its most certainly full steam ahead for PAT and the project.

PAT Update

2009-10-02T07:21:00.004+01:00

Just a quickie to show whats happened in PAT Land in the last 48hrs:

Schone has returned from the wilderness to mock us up a new PAT Theme when he's not drunk.

And login page.

Paul is playing around with the dimension browser and mdx view

And I've been playing around with OFC and PAT:

We also have some drilling functionality on the way as well.

Good stuff, looking forward to the webex on Wednesday

TTFN

Tom

The Barcelona Gang

2009-09-25T11:24:00.001+01:00

IMG_1456
Originally uploaded by Gretchie Pentaho

Cheers to you guys for coming along, thanks to Gretchen for actually taking some photos, something I didn't do.

Back at work

2009-09-22T11:24:00.002+01:00

After a week and a half of training and meeting, I'm finally back at my desk :(
A big thank you to all that came, thanks to Doug, for paying for the refreshments that I was too broke to buy, thanks to Gretchen for photographing the event. Thanks to Daniel and Mozilla for flying out to show us some fun stuff. Thanks to everyone else who gave a presentation(Roland, Paul, Pedro, Rui, Matt, Julian, Thomas, and anyone else I've forgotten), but more importantly a big thank you to everyone else who turned up and made the community event a success for the 2nd year running.

As a slight aside, in the interests of not having to explain myself once again to people, I joined twitter, http://twitter.com/magicaltrout , there you go you can now stop looking at me like I'm weird.

Thanks again for making this weekend special, and its great to finally put some faces to names.

Until next year...!

Tom

Pentaho Reporting 3.5 for Java Developers..

2009-09-17T10:40:00.002+01:00

To pay homage to yet another free book, here's another cracking self portrait(sorry about the squinty eyes, can be bothered taking another picture, got a plane to catch). Review will appear shortly, when I've read it then sobered up from Barcelona.

When 50 places, er, wasn't

2009-09-17T09:29:00.002+01:00

Whilst preparing for this years community event I counted the heads in last years community photo and it came to about 35ish, great I thought, even if there are a few more people I should know early enough if its over 50, in which case I can find a bigger venue. So I put the eventbrite page up for people to sign up and request tickets with a limit of 50. About a week ago the page reached its limit and 50 had been sold. Great I thought, until the stack of emails in my inbox telling me that people were coming..... WELL IF YOU CAN'T GET IN TOMORROW.. DON'T BLAME ME!

Looking forward to seeing you all there.

Ta Ta till Tomorrow(TTTT)

Tom

Firebug, Console.log()

2009-09-15T20:34:00.001+01:00

Here's a tip that the guy I'm working with this week taught me about firebug, I'm sure the pro's already know, but when debugging CDF is you dump in console.log(var) it will print out that variable, so its great for query debugging when things don't work! Much easier than slapping breakpoints everywhere.

Pentaho Reporting 3.5 for Java Developers.. a preview.

2009-09-12T09:34:00.005+01:00

Packt Publishing got in contact with me last week asking if I'd be able to review Will Gormans' Pentaho Reporting 3.5 for Java Developers Book, which of course I'd be only too happy to. So Richard has sent me a sneak preview of Chapter 6, titled Including Charts and Graphics in Reports.

For those of you who don't know Will, he's been at Pentaho for ages and writes fun stuff, he's also an invaluable resource for hacks and patches on the ##pentaho IRC channel.

From the outset this looks like a well structured chapter with a screenshot of the Chart dialog and then various tables showing what the datasets should look like for different types of chart, which to be honest is a godsend as my technique is currently hack and hope.

Next Will has listed all the chart properties with a description of what they do. You can probably find this stuff on the wiki but I've not seen it, and having it to hand is certainly a very useful resource. Do you for instance know what dependencyLevel is?? No thought not, and I'm not telling ;)

On reflection with the amount of options, they may have served better in an appendix, but I didn't write it so its not my call :P

Next up will writes about the different graph types, giving some specific options and information for each chart. There's some cool stuff in here I didn't know was possible, admittedly I've not done a vast amount of graphing but I will be testing out some of the tips and tricks, that's for sure.

From what I can tell, Pentaho has implemented this new Charting Beans infrastructure that makes the charting engine pluggable, that's why we see what appears to be some Open Flash Charts and some JfreeCharts, but it highlights perfectly how interchangeable the two are.

Correction: Will Gorman informs me that its all JFree, its amazing, clearly I need to look at my chart beautification skills.

Lastly Will writes about how to add both static and dynamic images to your report, its gives the impression slightly its just been tacked on the end to make sure its in the book... but then I guess how much can you write about adding images to reports.

An excellent read if only one chapter. I'm looking forward to the real thing dropping on my doormat so I can review the rest of it.

Now if only I could find uses for bubble and radar charts...........

Mileage Record

2009-09-04T19:21:00.002+01:00

I've hit a new record for my yearly mileage on my commute to work:

Not bad for now, still a month and a little bit more to go!

Pentaho Solutions By Roland and Jos

2009-09-02T21:23:00.002+01:00

This is how you do book promotion, none of this Pentaho Dev garbage (see below)..

Being just an technical editor(don't blame me for the mistakes) I don't get to say anything, so I have a few thank you's, firstly thanks to Roland and Jos for inviting me to take part in a small way in such an interesting project, thanks to the ##pentaho guys for warning me about how its my fault when there are mistakes, thanks to Rebekah at Wiley for sending me two free copies, \o/, thanks to Sara at Wiley for putting up with late submissions and reminding me that I hadn't invoiced them to be paid. And ta to the devs for writing the software in the first place, good job.

TTFN

Tom

Pentaho Solutions hits the bookshelves

2009-08-21T21:26:00.004+01:00

This picture has nothing to do with Pentaho, don't blame them for it, if it upsets you, sorry.

TTFN

Tom
It would appear Pentaho Solutions has been released, go out and grab a copy.
The Pentaho Geeks appear to enjoy it:

Pentaho 3.5RC1, work in progress

2009-08-18T19:29:00.002+01:00

So I got around to downloaded and running the linux version of Pentaho 3.5RC1, and first impressions, it looks pretty much like the old one, apart from there are some new samples, cool, I thought, er, maybe not.

Chartbeans look cool, not 100% how that all hangs together but it has to be an improvement over vanilla JFreechart. The big improvement with 3.5 is supposed to be the new reporting features, too bad only 1 of the new sample reports currently works on my system.

Anyway, it starts and I'm sure there's plenty more so I will continue investigating and report back later.

TTFN

Tom

Selling Pentaho to a BO Shop

2009-08-18T19:09:00.003+01:00

So I have this presentation on Thusday to the head of BI for the parent company, who as I mentioned in a previous post, has never heard of Pentaho. So, as I've never even laid eyes on BO in real life, how do I sell Pentaho to a guy who uses BO on a daily basis?

Rumour has it he's most interested in the dasboarding capability, so CDF obviously comes top of the list, but what are the other advantages?

Pentaho 3.5 Beta

2009-08-13T05:55:00.003+01:00

Right, enough of my moaning, Pentaho 3.5 RC is out on sourceforge. Go grab a copy, and install it! The new server is compatible with the Citrus Report Designer, it makes me so happy I could cry......

Luck in Life

2009-08-12T22:07:00.004+01:00

I am a firm believer in you make your own luck in life, so can someone explain where I'm going wrong...

I work 7am till 5pm Monday to Friday, I don't take a proper lunch break, I replace my bosses work with proper solutions, I'm organising the Pentaho meet, I help with moderation/documentation etc, and 90% of my spare time is doing PAT...

So why in the last 3 days have I had 4 flat tyres, 1 broken chain, a failed job interview and now having to replace the rear cassette on my bike also, and have £60 to last me till pay day?

Oh and at a time when I most need it to cram PAT for Barcelona, HP have decided they need to take 12 days to fix a dodgy network port and a cpu fan on my laptop, 12 days, where they sending it, the Antarctic?
Whats up with life?

Commercial Open Source and its place in business

2009-08-06T10:50:00.002+01:00

My boss got a phone call on Monday from the head of Business Intelligence for our parent company, now this company even in the recession still made a sizable profit, and pre-recession made a profit of over £700m for the year.

So when this phone call came in I was quite surprised that the guy was interested in my dashboards that I'm creating for our dotProject setup and was interested in how they could be used in their company. Now our parent company are from what I have heard big SAP and BO users so I was mildly surprised that he was interested in such things anyway, but fair enough. But my boss then made the point of telling him the name of the product, Pentaho, and he said he'd never heard of it, and I recieved another email today, and he has already confessed to forgetting the name.

So what I am interested in, is what type of companies use Commercial Open Source, I don't mean just Pentaho, I'm interested across the board, databases, CRM, project tracking, BI etc and specifically with Pentaho, what needs to be done to raise awareness of such a product, specifically in the UK where uptake seems remarkably low, surely these BI managers should have at least heard of Pentaho or Jasper etc even if they choose to go the BO route.

Webdetails CDF Tutorials

2009-07-21T10:40:00.004+01:00

Webdetails have finally got their act together and released a series of CDF tutorials, these are available on their website for a meager sum.

If you purchase the tutorials you will

* have the source - so that you can see how everything is built
* get free updates made to the tutorial for 1 year
* get a refund guarantee if you're not happy with the result
* contribute with open-source development

A very worthwhile cause if you ask me.

PAT 0.4 Release Notes

2009-07-21T10:34:00.003+01:00

Right 0.4 is out in the wild, this release includes table model reworking so the majority of it is now server side, Pauls connection manager implementation and different usage modes, Live Table to dynamically buffer large tables so the browser doesn't bog down and more..... release notes below:

PAT 0.4 Release Notes

Support of plain MDX Queries - It is now possible to execute plain MDX queries. It is not possible to transform this query to the query wizard (use of the query model) and vice versa, but is planned in a future version.

Improve support for different integration modes - It is now possible to easily introduce new modes in which PAT can run in. These modes can be used for different permissions (e.g admin,power user, business user) or for creating lightweight versions of PAT for nice integration into other applications (e.g the pentaho biserver). The following modes are demonstrative implemented together with a panel in the startup dashboard that allows easy mode entering with a default connection to standard pentaho xmla servlet:

Lightweight PAT:
- OLAPTABLE: Executes a plain MDX query and just shows the resulting table
- ONECUBE: Connects to a specific cube in a defined connection Different Permissions:
- BUSINESSUSER: Users will just see the list of cubes and create queries only with drag and drop
- USER: This represents the power user who can connect to a specific existing connection, plus mdx querying enabled
- STANDALONE: This mode represents the PAT GUI an admin or management account would see (USER + add/delete connections)

Connection Manager GUI Code (only in mode demonstration STANDALONE / USER) - There is already an implementation GUI side for the connection manager, which will allow saving defined connections in the persistance layer. There is still missing some backend code but we've left it in there for demonstration purposes.

Refactoring / pruning of GUI classes - Most of the, previously very nasty, GUI code has been cleaned, refactored, pruned

Removal of Toolbar + Title Widget to gain more space - We decided to remove those widgets because we need this space for the important bits (query design, table, etc.)

Switch to Mosaic's Live Table + rewrite all Table related code - The new LiveTable supports lazy loading (which is perfect for large query results), looks much better and is easier to handle code-like. We ripped out all of the code that was related to the table and replaced it with something much nicer.

New Welcome Panel (Startup Dashboard) - Instead of the GoogleCode Page of the Project there is now a Welcome Panel, also known as Startup Dashboard, that allows quick and easy access to PAT functionality.

Bug Fixes - We also removed loads of bugs e.g rendering issues at the initial GUI startup and the connection popup

Next Steps / Plans for 0.5:

heavy UI design to make it as user friendly as possible
saving connections in the persistance layer (backend code) and improve multiple connection support
improve the table (column header, performance, ...)
introduce different users (admin, power user, user) with different permissions
improve the drag and drop query designer
introduce a rich text editor for MDX queries that allow syntax highlighting and auto completion

The following releases will all be a minor new version of 0.5 until
the UI redesign is finished

LRO Pictures, a look back in time

2009-07-20T07:39:00.002+01:00

Over the weekend the LRO returned some of its first images, and personally I find them amazing, its like looking back in time to a perfectly preserved time capsule of the late 60s and early 70s... but on another planet.

To see such detail of what happened way back before I was born fills me with hope and amazement about what happened back then and what will happen in the future.

Doug Moran and the Missing Apollo 11 footage

2009-07-15T12:53:00.001+01:00

As some of you may be aware its the 40th anniversary of the Apollo 11 moon landing, and it NASA appear to have conveniently found the much higher quality Slow Scan TV footage tapes from when Neil walks down the ladder, what they have also found is this... and its truly amazing!

Pentaho Community Meet 2009

2009-07-06T19:49:00.004+01:00

Right its booked and we're ready to go, get your tickets and holiday time sorted for 19th-20th September. We'll be finalizing the fun details over the next few weeks, but the main crux of the weekend is booked, Barcelona Universal Hotel, we have a meeting room with a capacity of 50ish, 10 minutes out of Barcelona City Centre opposite a subway station, maps, photo's etc to follow.

If everyone's happy we've got the room booked for a full day on the 19th and then in a similar fashion to last year those of us who stay for Sunday can go off and do fun stuff around Barcelona, ideas are currently being collected, and hopefully it wont involve me sprinting for a boat this year.

So there we are, roll on September!

TTFN

Tom

PAT Table Model 2.0

2009-07-06T06:02:00.002+01:00

Although I posted this on the PAT Dev list I thought it should probably also go here, so people have a vague idea as to what goes on in my head:

Time has come to rip out the main innards of PAT and start over, the table model. And whilst I'd never claim to be an expert in such matters I'll have a go at putting my ideas down on paper, after a good chat with George the Greek I think I've got the plan just about sorted.

First some issues with the current model:

* Hacked, yeah, I've hacked and hacked it to get it to display data in the current setup, its not pretty and certainly not optimal and its telling, we get javascript execution warnings on large queries, I've had to dump in a load of ___, it doesn't detect empty cells properly etc etc.

* Row Header writes in columns, whilst the column headers sort themselves out in a row based fashion, the row headers are processed in a columnar fashion, naughty table model.

* Like mentioned above, I've dumped in a load of visible ____ to make sure that it knows the cell is filled, not great :)

* Its all client side, all this processing is done client side, which doesn't matter for small tables, but for larger tables will end up being a large problem.

* The Table model was written with row and column spans in mind, GWT Grid doesn't support spanning.

Right so here's the plan:

I'm currently decoupling the client side hacking from the table model. Basically I want to be able to call
tablemodel.getRows() and it populate my table with data(obviously we have to iterate it for formatting attributes), it makes it clean and easier to customize. And whilst during testing the current layouting will remain client side it will become completely separate from the table rendering. Also the way that it works will change it will become a row based renderer and I would also like to remove its gwt bindings, so the finished result will return a list of objects containing their cell value and formatting attributes, this code could eventually be patched back into olap4j to make the whole experience cleaner, but anyway its a step in the right direction.

LiveTable: I've had a chat with George the Greek and he's written us the first implementation of a live table that buffers up 150 rows at a time and then loads and unloads as you scroll, this is a great step forward as it allows us to buffer up huge tables without the browser taking a huge hit, ok cause its still client side buffering the memory usage is still there, but its not rendered in the dom, so you don't get the annoying slow downs. He's also putting some thought into how we'd achieve a similar result with a column based setup, so we can buffer up columns and load when people scroll.

The live table can technically do server side rendering, which would further reduce overheads but from what I can tell, what we could do with this is limited and it doesn't allow the flexibility of generating GWT widgets.

Cell Span, whilst the GWT grid not supporting cell spans is a shame, its not the end of the world, we should be able to achieve a similar setup with some crafted css on the table model, and the column headers DO support column and row spanning so thats no problem.

So there we go....

Now its time to shoot me down in flames...

Tom

Kubuntu, Eclipse, Compiz, Nvidia, not the best combination in the world

2009-07-05T12:02:00.002+01:00

I work on this laptop I am writing this blog post from, and due to the nature of what I do, I use and IDE, and my personal preference is Eclipse. I also use Kubuntu, it's great cause it 'just works' compared to some distro's I've used over the years, and I'd rather spend my time coding rather than hacking config files to get things going. But I have one issue I can't seem to find a fix for...

Kubuntu comes with Compiz switched on by default, now I like Compiz, it makes the computer do fun stuff, most of which is a complete waste of time, but makes me happy, anyway I can be tap tap tapping away on eclipse and suddenly *kaboom* the screen freezes for a few seconds, and I'm left with this:

Not the best. Anyway I've now switched off compiz(I'm assuming thats the issue) and we'll see how we go, but I do miss my ridiculous effects.....

Whats in the pipeline for PAT?

2009-07-01T06:05:00.002+01:00

So with PAT 0.3 out of the way it gives us an ideal opportunity to reflect on what we've done and where we are heading.

So this release saw tabbed multiplee queries and a new table. All good steps forward but nothing amazing in their own right, just things that needed to be done to take us further in the right direction.

Whats coming up next then?

The table model. Quite frankly it sucks ass, mostly due to my hacking to crowbar it into place for what we have already. Luckily for us(oh you have no idea) though we have George the Greek from GWT-mosaic on our side, and he's doing us a client side lazy loading table, this is great because it will allow the user to theoretically buffer up 1000's of rows but only show whats in the browser window, with all the others being loaded and unloaded from the DOM. Good work Greek, like I pointed out to him, they will write stories about him, like they did Ancient Greece.

This will also mean when you run a large query the javascript execution warning will be banished *\o/*

Connection Manager. Pauls done some cracking work already on the Connection Manager, this will finally allow us to stop adding a new connection each time we log in, thank god.

MDX Code Support. Due to the fact we aren't god and can't foresee every requirement at some point we will need MDX support, so 0.4 will include basic support for MDX. There will be 2 modes, Query model(whats already in place), or MDX Query, the latter disabling all the query model stuff. Sadly there is no easy way of dismantling the query model for an mdx query, so and the moment its either/or I'm afraid.

BI Server Plugin, hopefully by the time 0.4/0.5 is ready CITRUS will be released, so we will get the BI Server plugin working properly so you can slot PAT easily into your existing BI Server setup.

Other Fun stuff, I'm sure there will be other stuff, when we get bored and decide to tinker.

The big one I'm looking forward to is when Julian gets Mondrian Writeback moved into Olap4J(come on Julian, you can do it) and then people will be able to adjust the data in the mondrian cells and watch the totals crank.

Anyway thats the plan, if you fancy helping feel free to jump on the mailing list and introduce yourselves.

TTFN

Tom

NASA.... how on earth do they get people into space?

2009-06-30T14:39:00.004+01:00

Here's some amusement for all you access loathers out there. I tried to get to one of their new websites today and got this:

So one of their ultra modern websites it backed by access.... since when did they put my boss in charge?

PAT 0.3 Released

2009-06-29T09:12:00.003+01:00

PAT 0.3 has arrived, whilst the design isn't really even started its a step forward in terms of usability as we now support multiple tabs and multiple queries in an alpha form. We also have a new GWT 'scrolltable' to help us add more usability to the table. Make sure you head over to the PAT download page and grab a copy to test out for us, and let us know what you think.

If you feel like helping out, we are still in need of bug squashers and designers, so if you feel like volunteering, jump on the dev list and speak up. EXPERIENCE NOT REQUIRED!

Thanks

Tom

PDI Lucid Bulkloader

2009-06-29T08:24:00.003+01:00

I finally became bothered enough to setup Kettle's Lucid Bulkloader and give it a whirl, I'm running it, postgresql, liferay, alfresco on a Quad Core machine with 4GB or ram, so not really conducive to super quick load times but I did manage to hit 37k r/s and I think that limit is because the table input from postgres is coming from a NFS NAS and tops out, so I'm not really complaining.

Nearly there....

2009-06-25T20:45:00.005+01:00

So after more delays and far too much 'real' work for the PAT devs liking the latest release of PAT is nearly upon us.. whats in this release?? I hear you cry... well,

3 main upgrades for the end user, the new Aegean theme from GWT Mosaic, a new interactive 'Scrolltable' which will allow us to add more interactive stuff much easier, and lastly, multiple tabs and multiple queries, whilst we don't yet allow for multiple connections we do allow for multiple queries on the same cube, or indeed different cubes on the same database at the same time.

Other things you may or may not notice, we've finally swapped out most of the standard gwt panels for mosaic panels, finished the upgrade to GWT 1.6 and the latest mosaic build, Kees has kindly provided us with a dutch translation, and various other usability 'tweaks'.

I'll post some more when I release PAT over the weekend, if you want to try it out beforehand feel free to grab a copy from the SVN

TTFN

Tom

LucidEra is dead, Long live LucidDB!

2009-06-25T04:40:00.002+01:00

As you may or may not have heard LucidEra have decided to close their doors, but don't let that worry you sports fans, LucidDB will still be under development by JVS and the Lucid Community(Nicely reassuring considering I've now switched over completely to Lucid apart from a staging database).

So if you use LucidDB and want to help out, jump on the mailing list and offer your help, if you are thinking about LucidDB but are unsure due to LucidEra don't let this put you off little one!

Till next time.

Tom

The amazing property that is a 2nd pair of eyes.....

2009-06-24T05:00:00.002+01:00

Its amazes me everytime, I've been strugging with a PAT issue for weeks with duplicate tabs and what is contained within them, to put it simply its been shit. Anyway I showed the code to Paul(embarrassing cause its full of a months worth of hacks and not really up to much) which he prodded away whilst I was sleeping and left me a message:

21:37 < pstoellberger_> hm, i have no idea why you would think that this code works
21:37 < pstoellberger_> :P
21:37 < pstoellberger_> scrolltable is null because you don't do a new()
21:38 < pstoellberger_> you are trying to clone the olapPanel not the olapTable right?
21:40 < pstoellberger_> and if you want a cloned olapPanel .. why: private static OlapPanel getNewOlapPanel() { ... static?
21:40 < pstoellberger_> eeh

Which wasn't exactly the nicest message to receive in the morning considering the amount of time and effort I'd put into trying to get this to work and failing miserably(honestly with this lot and work being manic it has crossed my mind to jack it all in, but annoyingly I find it impossible to drop a challenge, which is why I still work where I do).

So anyway I cracked open hosted mode again(at 4:30am) and thought I'd step through it once more to try and fathom it out.

Now he's wrong in the comments above, the code does work as proved when you opened 1 tab(it worked), so I knew he was talking rubbish, but he did make me think of something which was that when I was testing and 'closer' to the desired result the header table never updated, so using eclipses' object browser I paid close attention to what was being created, the key to this problem came to me whilst I was sleeping the other night, I'm not actually trying to clone, cloning involves duplicating widgets in their current state, and in GWT it's a right arse, but something I've been attempting for the last few weeks. Instead of cloning, I'm trying to create a brand new Olap Panel each time a cube is selected which is completely different and not cloning.

Anyway I ran the code and sure enough I noticed from new to new the header table wasn't being recreated. Check the code.... private static.... argh balls. Remove static definition, of course it starts working fine(ish) and we have multi tabbed querying.

So there we go folks although Paul's an idiot and wrong, he did make me think slightly differently about the problem and point me back in the right direction which for so many weeks I'd been an absolute mile from.... now where did I put that GPS?

Its Here!!!!!!

2009-06-18T09:23:00.002+01:00

A few days ago I told you to watch this space.....

WELL ITS FINALLY ARRIVED, the future of modern programming

GOTHIC XML

Check it out, Mimil's done a great job, and I can see its uses in every application ever written!

BBC Dashboard Design Flaws

2009-06-13T11:43:00.006+01:00

I was looking at a map based dashboard today of swine flu around the world, and whilst it conveys the information in a easy to understand fashion, it dawned on me, that I couldn't see Australia.
Now I think my image manipulation and resizing has actually done it a few favors and made it easier see, but regardless, if you're designing a dashboard, should you take into consideration all screen types and sizes, and make allowances for differences in Contrast and Brightness, so the end users don't have to adjust their monitors to see it?

Private Forum questions....

2009-06-13T06:41:00.003+01:00

Here's a quick question to ponder, why do some people on the Pentaho(and I would have to assume other forums) sign up, then straight away go looking for moderators to privately message their question to, instead of doing what the website specifies and participating in a forum?

Or, a better one from yesterday, using the report spam button to send their question to all moderators?

Do these people thing that they will get their questions answered more quickly? Clearly they must believe that. But I can tell you one thing if someone sends me a question and has a post count of 0 then unless its a blazing security alert, in which case it would be forwarded to the correct people, it will be heading for one place, and one place only, /dev/null.

Java Portals

2009-06-12T20:21:00.002+01:00

Over the last few days more interest in Java based Portals has grown as I'm looking into expanding my knowledge of Java based technologies and also implement a 21st Century Intranet at work that will interact nicely with the Pentaho stuff and Alfresco to make my end users experience far more pleasant.

A year or so ago I dabbled with Liferay and I can see how people love it, but I didn't mostly cause a year ago my Java sucked more than it does now, and I didn't really understand all the concepts behind it, anyway a year down the line and thanks to Pentaho and PAT I now understand enough to dabble again and although I think it has a couple of reasonably severe shortcomings for easy deployment in an business that has many remote offices and users, but nothing thats impossible to overcome. The biggest factor I've missed when compared to Pentaho is the community, now I'm sure there are plenty of people answering questions, but there are also many that go unanswered, and whilst we may be abrupt or cheeky with out responses on at the Pentaho forums, at least we try and respond to a majority of the posts(with answers or suggestions).

Today I hooked up Liferay to CAS and SPNEGO, it was hardish work for a newbie in the world of CAS(I started at 4:30pm last night) but its well worth the effort. Now my users are automatically logged into Liferay and shortly will also be automatically recognised on Alfresco and Pentaho(if I ever get time I'll try and write a dotProject connector also, although my project tracker my be written first) which will make our's and their lives far far easier.

On a completely different note, I've also been dabbling with Jetspeed for a lightweight portal model and it looks excellent, with a blend between being not to hard to setup, configure and administer for a fool, but retaining its flexablility and power at the same time. Anyway I'll be tinkering more with Jetspeed over this weekend and we'll see how it looks at the end of it all.

TTFN

Tom

Keep Watching This Space

2009-06-12T10:28:00.004+01:00

Keep an eye on this blog, I've been given a sneak preview of a project by JFreeReport hacker, and snail god, Mimil, which will without a shadow of doubt change the face of programming forever. It amazing, thats all I can say. So keep checking back over the next few weeks as development is finalized and the project released.

TTFN
Tom

Rock on Souceforge Finals

2009-06-12T05:58:00.002+01:00

Good work from the community has gotten Pentaho into the next round of Sourceforge Community Choice Awards.

It would appear Pentaho has been nominated for Best Commercial Open Source Project and as an alternative for Best Commercial Open Source Project, so what do the Pentaho guys have to do now?

Well they have to submit a video, and they want the community to help with the finer points... if you fancy helping out here's what they need from you,

The video should be interesting and fun so we are asking for anyone who wants to participate to email 2 pictures and some info:

A serious picture of yourself
A funny picture of yourself (optional but highly desired!)
Name or nickname
Company (Optional)
Country
What product you contribute to or use

This all needs sorting by Monday, so get your skates on, and email your stuff to
dmoran(blahaddtheatsignhere)pentaho.com

A Dark Night for the UK in European Politics

2009-06-08T06:10:00.003+01:00

Now, I'm not one to get involved with politics, while I think the country is badly run, but then I also think that other parties for all their showing off and I told you so's, would be in exactly the same position, so I just end up paying no interest in it... but then maybe I should.

Last night was the results of the European elections, and its a sad day why a bunch of racists win seats to the European parliament in your area, or at all for that matter. So would my one vote have made the difference? I very much doubt it, would the Mrs's have tipped the balance? I doubt that as well, but if the other 50 odd percent of the population who didn't bother to vote, had turned out at the polls, we might not be in this embarrassing situation.

So there you have it. A pretty dark day for the Government, Labour clearly aren't going to win another term in parliament so come on the Moster Raving Loony Party is what I say!

The Cost Of Computing..

2009-06-06T16:23:00.003+01:00

Here's a little conundrum that needs addressing...

The electrician came round yesterday and managed to destroy my mail/fileserver that I use at home, nothing special, just a bunch of disks and some ram, and now i have to rebuild it, not to worry though, I have an old junk PC in the attic.

A thought crossed my mind though as its not the first time I've had to rebuild it, is it worth it? Now I like having a mail server for my domains, it makes my life easier, and I certainly wouldn't think about giving it all up for gmail, even on a domain. That being said, I bought some server space in the mosso cloud yesterday and it got me thinking, how much does it actually cost to run a mail server vs hosting one in the cloud?

Now don't get me wrong, I have no scientific, or definitive answer but here's a bit of guess work and ideas...

A quick google seems to suggest a desktop PC would cost in the region of 2p P/H which sounds reasonable. So taking that and doing some simple maths brings me to the region of £15 to run a PC 24/7, now I don't have a monitor on it, but if you bang in 4 HDD's I suspect that it'll compensate it somewhat for the lack of monitors, so I think thats a reasonable answer (If you have a better answer let me know).

So how much does a virtual host cost?
Well at the moment, I run Zimbra, which has huge(and for my use, unneeded) overheads, so I'd have to downgrade, to usual imap stuff, but thats no big deal, lets take some boring figures into account, say I only need 256mb ram, which is about right, and I somehow use 10GB up and down the pipes in a month(i'd probably move Frankbot to the cloud), that costs me, $13.95 and on the current exchange rate, that equates to £8.80 per month.

So that would be a rough saving of £6.20 a month and it doesn't break down.

Worth it?

You bet.

I finally did it....

2009-06-05T06:00:00.003+01:00

Yes thats right folks, I missed my morning coffee cause I forgot to put the jug under the steam/coffee thing...

Anyway also I finally subscribed to the cloud stuff, which to be honest until has interested me about as much as twitter(I think I referred to some extreme act on my body rather than infect IRSSI with twitter yesterday). But to be honest I could do with a PAT maven repo so we don't depend on the Pentaho guys for dev upgrade testing and patches and also I wanted somewhere to host PAT so you guys and girls can get a look without dragging my home connection to its knees. So, its on its way, as soon as its all setup..

Oh and the coffee machine lived another day thank god.

Tom

OpenCommunityCamp

2009-05-28T19:48:00.002+01:00

Roland pointed out on his blog that OpenCommunityCamp is coming up at the end of July for those of you interested in some geek chat intermingled with many a beer and some grub. Anyway, I'll try and make it across for some if not all of it (depending on how much holiday I can blag) so I'll hopefully catch up with some of you there.

TTFN

Tom

Sly Change To Pentaho 3 and MDX reports

2009-05-28T17:03:00.002+01:00

For those of you who use MDX and JFree you might get a little concerned when upgrading to 3.0.0.stable and you see nothing but nulls... but never fear its just a slight change in the config,
as Pedro points out

Edit system/pentahoObjects.spring.xml and set the property useExtendedColumnNames.

Beerursday

2009-05-21T05:42:00.002+01:00

What day is it? Thats right its Beerursday!

Cheers Guys

Tom

Why MDX isn't as good as Pedro makes it out to be....

2009-05-20T16:35:00.004+01:00

A few weeks ago Pedro Alves mentioned on his blog that he thought MDX was created by God, and the Julian Hyde was Jesus.. or something of that nature, and I have to admit it is pretty good(if I didn't think that was the case I'd be wasting my time with PAT) that being said I do think some things are trickier than should be expected.

Last week I mentioned a query that Pedro had helped write for me that goes something like this:

select {([Date.Fiscal].[All Dates].[2009].[Q1].[1]:[Date.Fiscal].[All Dates].[2009].[Q4].[12])} ON COLUMNS,
(Generate({Descendants([Branch], [Branch].[Group], SELF_AND_AFTER)},
TopCount(([Branch].[Branch Code].CurrentMember * [Customer].[Customer].Members),
20.0, ([Measures].[Total Sales Value],
[Date.Fiscal].[All Dates].[2009].[Q3].[9]))) * [Measures].[Total Sales Value]) O
N ROWS
from [Sales]

In a nut shell what that does is iterate through my groups and branches and create a Top 20 Customer report for each one. Great, 90% of one of the month end reports written, I just need to create a PRD report for it and I'm sorted.. or am I? Now I'd like to expand it a little and add at the bottom of each iteration a sum of the Top 20 customers, a Pareto total for the other customers and also a grand total. But I've been working on this on and off now for a few days and its ridiculously tricky, I don't even know if its possible, I assume it is, why wouldn't it be, after all its a business language created by god isn't it?

Anyway for anyone that wants to show me how its done, here's a bit of a steelwheels mashup that you can use to show me the light:

with set [all] as '(Descendants([Markets], [Markets].[Territory], SELF_AND_AFTER) * [Customers].[Customer].Members)'

set [top20] as 'Generate({Descendants([Markets], [Markets].[Territory], SELF_AND_AFTER)}, TopCount(([Markets].[City].CurrentMember * [Customers].[Customer].Members), 2.0, ([Measures].[Sales], [Time].[All Years].[2004].[QTR3].[Jul])))'

select NON EMPTY {[Measures].[Quantity]} ON COLUMNS,
NON EMPTY ({[top20], [all]}) ON ROWS
from [SteelWheelsSales]

TTFN

Tom

PAT, Pentaho Integration

2009-05-16T08:32:00.002+01:00

Gretchen very kindly offered her professional services for a few weeks to get her back into the swing of programming and so we put her to work with Luc and Paul on Pentaho and Xaction integration, and here's the preliminary results, not bad for a weeks work..

A couple of mdx tricks

2009-05-14T14:12:00.002+01:00

Here's a couple of tips that I'd like to post as a bit of a reference for people. First up a quick properties hack

WITH member [Measures].[Market Sector Desc] as 'IIf([Market Sector].[Market Sector].CurrentMember.children.count=0,[Market Sector].[Market Sector].CurrentMember.Properties("Market Sector Desc"),"")'

If you use Calculated Members to show properties(I currently can't get DIMENSION PROPERTIES to work, so I don't have much choice), and you have an All Member level in your query it will return an error if it can't associate with a level property, so I found the above solution on the Pentaho forums. All it does is find the top level property cell and mark it null, not pretty, but it is effective.

Secondly just a quick query that replaces about 30 separate reports with a few lines of code courtesy of Pedro,

select {([Date.Fiscal].[All Dates].[2009].[Q1].[1]:[Date.Fiscal].[All Dates].[2009].[Q4].[12])} ON COLUMNS,
(Generate({Descendants([Branch], [Branch].[Group], SELF_AND_AFTER)}, TopCount(([Branch].[Branch Code].CurrentMember * [Customer].[Customer].Members), 20.0,
([Measures].[Total Sales Value], [Date.Fiscal].[All Dates].[2009].[Q3].[9]))) * [Measures].[Total Sales Value]) ON ROWS
from [Sales]

What this does for us is run through our warehouse and generate a Top 20 Customer sorted on the current month report, by Company then by all the Companies branches. It takes about 10 mins to run across all the group, but it saves me 2 hours of adjusting reports on a monthly basis...

now if only I could force people to accept our reports on a computer instead of me having to put them in envelopes.

Why Data Warehouses Exist, and why we love lucid.

2009-05-12T16:53:00.002+01:00

I finally plugged in my DWH yesterday after about 12 months of it working but couldn't be bothered getting it working daily. I've also had a mad dash to finish my mondrian cubes etc so I can prove to the head of IT that we currently do things wrong.

One of the things I am most happy about though is our slow moving stock report. Currently at month end I run this on a unused Progress copy, no one else uses the database and the computer (an old IBM RISC box) is only used as a print server, my kettle transformation takes the best part of 2 hours. I've turned that data extract over to Lucid now that the DWH is running.....5 Minutes and 11 Seconds, and my ETL techniques aren't exactly the best, so I'm sure this can be improved upon. Not bad for a Java based Column Store database eh?

TTFN

Tom

Wishlist

2009-05-09T17:28:00.002+01:00

Just to prove such things work, a quick thank you to Doug, who's very kindly sent me a couple of books that were on my wishlist. At least someone finally sent me gifts that will actually prove useful ;)

2009 SourceForge Community Choice Awards

2009-05-07T12:20:00.003+01:00

Time to get your voting boots on, vote for Pentaho in the Sourceforge Community Choice Awards.

Pentaho qualifies for three categories:

* Best Project for the Enterprise
* Best Commercial Open Source Project
* Best Project

Click here to enter