[Marketplace Spotlight] Stream Schema Merge

Stream Schema Merge

The contribution

mergeStreamMarketplace [Marketplace Spotlight] Stream Schema Merge

Our friends at Graphiq did it again; Not only they recently submitted a great contirbution for the Json Input plugin that we incorporated in 6.1, recently they added a new contribution, the Stream Schema Merge plugin.

Andrew did an amazing blog post describing it; I can’t possibly do better than what they did, so I’m really just highlighting their effort. But I’d like to just briefly describe what it does

How it works

I admit at first I was a bit confused; Why the hell do I need a step to do something that PDI does natively?

Then I saw this:

1*FvKu6VP301QnC6FA7Wv8ng [Marketplace Spotlight] Stream Schema Merge

And this:
1*Mlosfw5Mab4nJLPEyu3QRg [Marketplace Spotlight] Stream Schema Merge

Then it all made sense! It’s bloody genius! The trick here is that with this step there’s no longer the requirement of converting all streams to the same structure! Not only this simplifies the transformation a lot, it really improves performance: Every time we change the structure of a stream there’s a cost of building a new Object[][]

And with this step we don’t need that.

Amazing job guys! Honestly, after seeing this, the question I ask myself is why doesn’t PDI just behave like this by default?? And on that, we may talk again, Andrew and team :p



This entry passed through the Full-Text RSS service – if this is your content and you’re reading it on someone else’s site, please read the FAQ at fivefilters.org/content-only/faq.php#publishers.
Recommended article from FiveFilters.org: Most Labour MPs in the UK Are Revolting.

Pedro Alves on Business Intelligence