Category Archives: Data Mining
Why MDM is the First Step Toward Business Reinvention
Reading Time: 2 minutes
Managing critical data across the enterprise is an essential step toward business reinvention. That’s why organizations are increasingly turning to master data management (MDM) to solve their siloed data problem and provide a single, trusted view of their data. By providing one trusted place to go for information, MDM enables businesses to make better decisions and improve overall performance.
If you’re looking to adopt an MDM solution, the Constellation ShortList™ is a good place to start narrowing your search. This report is often helpful to organizations when evaluating solutions within technology market categories.
We’re excited to announce that TIBCO EBX™ has been added to the Constellation ShortList for Master Data Management Solutions for the third consecutive time. EBX™ is one of only seven solutions recognized for 2020. For TIBCO, we see the report as a further indication of EBX’s momentum in the MDM market. As a single solution to help organizations govern, manage, and consume all their shared data assets, EBX is proud to meet the threshold criteria outlined below.
For Master Data Management (MDM), this year’s threshold criteria included:
- Information integration
- Data quality
- Data governance
- Multidomain (i.e., customer, product, supplier, reference data, sites, location)
- Multiple deployment models: on-premises, in the cloud, hybrid or multi-cloud
- Industry-specific templates and accelerators
- Self-service business rules
- Support for data privacy rules and regulations
- Customer references
According to Doug Henschen, V.P. and Principal Analyst of Constellation Research, the threshold criteria are determined by capabilities that customers and partners deem critical for that technology space. The Constellation ShortList essentially indicates which solutions cover the capabilities most needed to meet current challenges and address top use cases in that category.
To learn more about how TIBCO EBX is leading in master data management, read Doug Henschen’s Constellation ShortList™ for Master Data Management for Q1 2020.
Constellation is not the only analyst firm that has recognized TIBCO’s master data management. Check out Gartner’s 2020 Magic Quadrant for Master Data Management Solutions to see why TIBCO EBX™ Software was named a Leader for the fourth consecutive time.
Solve Complex Problems with Data Science and ML with TIBCO
Reading Time: 3 minutes
The reach of data science is expanding across every industry. That means no matter what industry or line of business you’re in, you need to consider which data science platform can best help you leverage your data to make better business decisions.
We’re thrilled that Gartner named TIBCO Software a Leader in the 2020 Gartner Magic Quadrant for Data Science and Machine Learning Platforms for the second year in a row.
We believe this report shows how TIBCO helps customers solve their biggest business challenges with data science and machine learning. TIBCO supports this success by enabling the following imperatives:
Make Data Science a Team Sport
To democratize, collaborate, and operationalize data science and machine learning (ML) across your entire organization, you need to make data science a team sport. TIBCO can help empower everyone from data scientists and citizen data scientists to data engineers to business users to developers with flexible and extensible tools. By supporting the end-to-end analytics lifecycle, as well as security and governance, TIBCO allows teams to turn predictive findings into optimal business outcomes.
Enable Innovation with Open Source
For data science and ML, customers often look to open-source frameworks, ecosystems, algorithms, and programming languages to create innovative solutions for their most vexing problems. The challenge arises when customers need to work across cloud providers, different languages, and ecosystems like Amazon SageMaker, Google TensorFlow, and Microsoft AzureML. TIBCO solves this problem and enables customers to orchestrate the complexity and seamlessly work across these environments from a consolidated platform. Therefore, TIBCO’s platform allows customers to innovate faster and gain a competitive advantage.
Perform Analytics in Real Time
As data volumes continue to increase, just providing access to data and algorithms is no longer enough. Organizations cannot afford to wait to ship all data back to enterprise systems for analysis. Instead, they should employ the Internet of Things (IoT) and edge analytics to perform analyses in real time. TIBCO has a unique strength in the IoT space with capabilities for ML on the edge and real-time streaming analytics.
We believe our customers’ success in data science and machine learning speaks for itself. See how companies like Xactly, ZPower, and Hemlock Semiconductor have used TIBCO to innovate and solve complex problems faster—and how you can do the same.
Download a complimentary copy of the report here.
Learn more about how TIBCO® Data Science can help you promote collaboration, automation, and the reuse of analytic workflows to solve your most complex business problems—whatever your industry.
This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from TIBCO Software.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
(Gartner 2020 Magic Quadrant for Data Science and Machine Learning Platforms, February 13th, 2020, Peter Krensky, Pieter den Hamer, Erick Brethenoux, Jim Hare, Carlie Idoine, Alexander Linden, Svetlana Sicular and Farhan Choudhary)
Humanizing Data: How Data Can Actually Save Lives
Data can change the world, really. In Humanizing Data, we spotlight how organizations are using data to help build a better world. We explore the journey they took, the people they affected, and the change they drove.
Recent coverage of the Indiana Donor Network showcased the amazing story around how data and technology is helping in an unexpected way: it’s saving more lives. Located in Indianapolis, they advocate for patients in need of organ, tissue, and eye donation and transplantation. Nationwide, there are over 114,000 people waiting for donors and every 10 minutes someone is added to the US national transplant waiting list. Simply put, the demand exceeds the available supply, so the nationally-driven matching process and care approach are only getting more complex.
The Indiana Donor Network aims to use technology to modernize its business and leverages data in the form of demographics, locations, and all kinds of other elements to minimize the time it takes to market, match, and measure effectiveness. They work in a tough business that is both devastating and rewarding, supports a valiant cause, and works on not just matching but also the aftercare of patients — with a passionate staff that has its eyes wide open around its current challenges.
This past year, through using Sisense as it continued to digitally transform its business, it saved (or greatly affected) nearly 700 lives, an increase of about 30%. In the truest sense of the word, they are a model for other organizations that want to plan, execute, and measure their results more effectively. The following is a Q&A with some key members of their staff around their story and the journey they are on.
Sisense: Tell us a bit about your organization — what do you do and how do you measure success?
Steve Johnson (Chief Operating Officer): We are a non-profit organization. We are one of 58 Organ Procurement Organizations which exist to facilitate the donation process. But most importantly, we are here to save lives and measure our progress closely. Every number is a real person — so it’s truly meaningful work. Everything is focused on that singular goal. We cover the state of Indiana but work with other states to partner on cases that are across state lines.
Sisense: How do you differentiate and market yourself and who is affected by your work?
Steve Johnson: With limited resources, we lean on education for awareness in traditional media markets (in radio), to sports marketing (Indy 500, Colts), and more locally in schools. Like any other organization, we use data to drive decisions and these decisions are aimed at saving lives. All of us are passionate about it and this cause motivates us.
Sisense: What types of data do you measure?
Steve Johnson: We use data for our marketing and operations — so we’re aiming to understand the demographics of individuals to drive outreach and to help inform events. And often the way we communicate is still less electronic and more through phone calls for a more personal and memorable experience. But our events generally are more about education than specifically raising money. We use data to identify the start of a donation at a hospital (e.g. when a patient is taken off life support). That is the most crucial time to ensure it’s handled properly and notifications are sent out.
Sisense: How broad or expansive is that data?
Courtney Tillotta (Manager, Aftercare): Through data, we are able to better understand our population. Our aftercare relates to both the donor and recipient. As an example, we are caring for 3,500 next of kin because we can more easily track these families and provide better care (like grief support). The organization and relationship of data has made us more efficient.
Sisense: What other benefits do you provide to patients that might surprise people? What’s an example of a type of person you would typically help?
Courtney Tilotta: With data in hand, we can understand our populations much better. As an example, we’ve seen older females in their 60s affected after their husband passes, so with this information, we can tailor programs and better anticipate their needs. We also commonly have next of kin outside Indiana (48 states affected in all) so we work to connect those families with local organizations to support the grieving and support of any family member — no matter where they live.
Sisense: What is the problem you aimed to solve using data?
Bill Janczak (Report Writer): I joined 4 years ago and at that point, most of our data was managed in Excel. Like most organizations, we lacked a master reporting setup and were working manually in Excel, looking at reports 30 days after activity. The organization wanted to improve speed and accuracy.
Sisense: How has data changed the way you manage your business and how has that transformation occurred?
Bill Janczak: We aimed to better automate our process and use that to build internal trust with others so that we could be more efficient through technology. Today, every screen in the office uses dashboards which deliver insights to others across the organization.
Sisense: Why do you need a BI platform like Sisense?
Bill Jancazk: The amount and locations of data is only getting more complex, so Sisense helped at the collection and analysis phases. We have call centers, patient lists, and hospitals that we work with so just like any other business, we can use the power of technology to manage the way we work and where we spend our time. The analytic apps and dashboards that we build help inform what we do, they don’t drive it.
Sisense: What’s the end product look like? What’s the right balance between using data versus human decision making?
Bill Jancazk: We have a “magic board” that includes 20+ dashboards we created. This identifies trends and allows a human to drill down to affect an action, or the next step. Data has surely made these decisions quicker and easier. We rely on Sisense at this point. It helps us reach our cause of saving lives.
Brian Becker runs content marketing at Sisense. He’s previously held similar roles at Chase and Thomson Reuters and is a regular speaker at marketing conferences and quoted in media outlets such as Digiday, AdAge, Business Insider and the Content Strategist.
Green Light for TIBCO in the 2020 Formula One Season
Reading Time: 3 minutes
Attention all race fans! We are pleased to announce that we have renewed our partnership with the Mercedes-AMG Petronas Formula One team for the 2020 race season. This will be the fourth year we have partnered with the F1 powerhouse, providing them with Connected Intelligence-driven solutions to continue to lead with innovation, teamwork, and leadership. This year, Mercedes-AMG Petronas F1 will expand its TIBCO footprint with even more solutions to help it maintain its competitive advantage.
Quick hood check for those burgeoning fans
The team’s modern-day race dominance began 10 years ago, and this year, it is competing for its seventh Constructors’ Championship, consecutively winning the coveted title since 2014. Last season, drivers Lewis Hamilton and Valtteri Bottas won a combined 15 out of 21 races (over 70 percent), 10 pole positions, nine 1-2 finishes, 32 podiums, and nine fastest laps. When compared to the competition, Bottas (who finished second) and Red Bull’s Max Verstappen (who finished third) had a 48-point difference in the Drivers’ Championship, while the team won the Constructors’ Championship with a 235-point lead over Ferrari.
Rocking the new look
On February 14, the team launched its car for the 2020 season, the Mercedes-AMG F1 W11 EQ Performance. The “W 11” represents the eleventh Mercedes-made F1 car since the Silver Arrows returned as a constructor in 2010. “EQ Performance” alludes to the car’s hybrid system, bearing the same designation as all feature Mercedes-AMG performance hybrids.
Made for speed and strength
The car is comprised of an astounding 10,000+ individual parts. In terms of the design, regulations indicate that all of the cars need to have a ‘shark fin’ on the engine cover to house the drivers’ numbers, making it easier for fans to distinguish between each team’s two machines. Additionally, the first 50mm of front wing end plates have to be made purely of carbon fiber to avoid tire punctures in the event of a collision. The car will undergo preseason testing at Spain’s Circuit de Barcelona-Catalunya this week.
Fans will see the addition of two new races on the calendar: the Vietnamese Grand Prix and the Dutch Grand Prix. On the flip side, with a limited number of races, something had to give and unfortunately, race fans won’t see the teams battle it out in Hockenheim this year. This will be the first Grand Prix in Vietnam, and the first Grand Prix in the Netherlands since 1985.
Milestone celebration alert
This year also marks the 70th anniversary of the FIA Formula One World Championship™. Over the last 70 years, 33 drivers have been world champion, 108 have won Grands Prix, while 764 drivers and over 150 teams have taken part. But this year also marks the last year before the rules and regulations will get totally shaken up and the sport reimagined for the new decade.
Looking to 2020, Hamilton and Bottas are revved up and ready to take to the track. Hamilton rivals Michael Schumacher as Formula One’s most-decorated driver, and he has the chance to exceed Schumacher’s stats and championship wins. As for Bottas, he gained momentum in 2019, winning four Grand Prix and placing second behind Hamilton in the Drivers’ Championship. That said, everyone will be looking to Bottas (or any of the other drivers) as he goes head-to-head with Hamilton for that coveted championship.
TIBCO’s fourth year of the partnership will continue to support the team’s business reinvention and operational excellence with a number of Connected Intelligence solutions allowing it to connect seamlessly, unify intelligently, and predict confidently.
Learn more about how we help the Mercedes-AMG Petronas F1 team drive innovation through continuous innovation and collaboration to leverage cutting-edge technology, giving them a competitive advantage.
Find the Right Master Data Management Solution for You
Reading Time: 2 minutes
Master data management (MDM) is a large, fast-growing market, growing at 16 percent annually. And it’s no wonder: every business needs to first ensure that their data are consistent and accurate before it can be used efficiently.
Gartner recently published the 2020 Magic Quadrant for Master Data Management (MDM) solutions. According to Gartner, “Data and analytics leaders can use this Magic Quadrant to help them navigate through a wide choice of enterprise MDM solutions.” We believe, for any data and analytics leaders considering MDM, this report provides essential guidance on making the right choice for amongst the many options available today.
Why TIBCO has been named a Leader
For the fourth consecutive time*, Gartner has named TIBCO Software a Leader in the Magic Quadrant for Master Data Management Solutions.
Digital business requires a new approach to MDM, favoring a holistic solution with strong multidomain and multivector capabilities. That’s exactly what TIBCO EBX™ Software is all about. It’s a single pioneering solution to manage, govern, and consume shared master data assets across the broadest set of industries and domains.
Here are just a few reasons why TIBCO EBX™ should be your MDM solution of choice to gain accurate views across all business functions, generate trusted, insights, and mitigate risk:
- Model-driven: Implement with your data, workflows, and security models
- Multidomain: Manage any domain of data and their hierarchies
- All-in-one: Includes all the capabilities you need across the entire data management lifecycle
- Designed for business teams: Encourage adoption and collaborative data governance
TIBCO addresses every kind of data in your ecosystem in a connected way, including metadata, reference data, master data, and transactional data. A unified view and governance of master data is the key to ensuring your:
- Operational systems are using consistent information
- Analytics efforts are based on accurate dimensions and hierarchies
- Compliance efforts are driven by trustworthy data
Businesses that are able to successfully manage and share all of their data assets, thrive in today’s data-centric economy. TIBCO can help your organization avoid data silos and provide a single trusted view of the business for faster, smarter decisions.
For more on why TIBCO Software was named a Leader, read this complimentary copy of the 2020 Gartner Magic Quadrant for Master Data Management.
This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from [insert client name or reprint URL].
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
Gartner Magic Quadrant for Master Data Management Solutions, 13 January 2020, Simon Walker, Alan Dayley, Sally Parker, Malcolm Hawker
*Appeared as Orchestra Networks, Gartner Magic Quadrant for Master Data Management Solutions, 2017-2018
Fall in Love with Your Data All Over Again
Reading Time: 2 minutes
Show your data some extra love this Valentine’s Day. Recommit to solving your most complex business problems with data and develop an action plan to operationalize your data across your organization.
In this blog, find out why TIBCO is the one for you—from unifying intelligently for better access, trust, and control, to connecting any application, device, or data source seamlessly, to predicting confidently with real-time data-driven intelligence.
The assets below are part of the Love Your Data series, taking you through the comprehensive TIBCO ® Connected Intelligence platform that can help you fall in love with your data all over again. Check out the following helpful resources of this Valentine’s week series:
Unified in Love
Everyone today loves data, but not everyone is making the most of it. Only organizations that have unified their data intelligently can easily access, govern, and share it to deliver the greatest value. This Valentine’s day make sure not only your heart is in the right place, but so is your data. TIBCO’s data management solutions support 360-degree views of anything, both relationships in the master data and transactional details.
This whitepaper can help you gain a holistic view of all the information in your organization and remind you of why you fell in love with what you’re doing in the first place.
My Heart Belongs to Streaming BI
Data is at the heart of every business today. Therefore, data analytics can often feel like monitoring a pulse, tracking where the business is and where it could be going in the future. If you’re making decisions based only on historical data, you’re missing the real-time critical information (vital signs) needed to make faster, smarter decisions. You can’t treat current problems with old data.
Open your heart to TIBCO’s guide to streaming business intelligence (BI), diagnose issues in real time, and quickly outpace competitors.
Cupid’s Connected Cloud
While Cupid is flying the skies helping singles meet their mate, we’re hard at work making a different kind of connection. Watch this webinar to learn how the TIBCO Connected Intelligence platform can connect your enterprise’s disparate systems, allowing you to manage your entire digital business from one cloud-based interface. Now that’s what we call love.
I am a Sucker for Data
Don’t let your love of analytics become overly complex. Too many companies today have an “It’s complicated” relationship with their data, struggling both to understand the massive amounts of data coming into the business each day and to take action on the insights they uncover for better, faster results. Simplify your data analytics initiatives by first understanding the different analytics approaches and the unique challenges each is designed to solve, as well as key considerations, requirements, and example use cases for each.
Read this ebook to find the right type of analytics for you, perfectly matched based on your business needs.
I only have APIs for You
Your applications deserve to be together, and there’s no better way to do that than with a solid API-led integration strategy. Check out this eBook to learn how TIBCO’s API management solution will leave you—and your developers—on Cloud Nine!
Lay the foundation for a strong relationship with your data using TIBCO. Learn more about how TIBCO allows you to connect to any data source, unify your data across your business, and predict future outcomes with confidence. And what better time of the year than Valentine’s Day to show your data just how much you love it?
Sisense Hackathon 2020 Recap: Dreaming Big, Building Bigger with AWS
At Sisense, we are builders. Create This is your firsthand look at what that means to us. Read on for Hackathon stories, new product developments, moonshot ideas, and more.
We’re always working on something new at Sisense. Innovating and experimenting with new ideas are core parts of our DNA; our R&D and AI teams never stop building the future of data. With all that creativity in the air, it’s no wonder that our annual Hackathon takes the energy to a new level. This year, the entire effort was built from the ground up, allowing the organizers to better manage the event and encourage more participation.
“The Hackathon is a very human celebration,” said Lio Fleishman, one of the organizers and a Solutions Engineer in our Partnerships and Alliances team. “We rejoice in what makes us human and what keeps pushing us forward: problem solving, imagination, and curiosity.”
The response company-wide was amazing! A grand total of 39 teams, from 6 of our offices, across 4 countries formed, collaborating on a wide array of projects. A whopping 80% of teams took advantage of the industry-leading power of Amazon Web Services (AWS)! If you read our Hackathon announcement, you know that AWS’s partnership wasn’t just cosmetic. We even had visitors from AWS pop in at our Tel Aviv, New York, and San Francisco offices to offer support (and tasty treats) as the festivities got underway.
Amazonians on site!
Amazon and Sisense have very similar DNA, from our customer obsession and ceaseless self-improvement to our abilities to work hard and have fun at the same time. Nothing could exemplify our partnership better than a Hackathon. Here’s what our AWS visitors had to say, in their own words.
Bala and Gary from San Francisco
“[It was a] great experience supporting the Sisense Hackathon. Loved the customer sentiment analysis using the data from Support teams with AWS Comprehend to understand in real-time what your customers are saying,” said Bala Keelapudi, Startup Partner Solutions Architect. “It was great to see the developers explaining the ease of getting up and running with AWS Comprehend.”
“So impressive to see the level of customer-centric innovation possible in 24 hours or (much) less, using AWS! And, using the latest AWS managed services, such as Comprehend, Lambda, API Gateway, RedShift…”
Gary Stafford, AWS Emerging Partner Solutions Architect
Jon from New York City
“Within my first few minutes on-site we were already troubleshooting SSH connectivity from Sisense office to an ec2 instance in us-east-1. It was really educational to work directly with the team to walk them through the network configuration and to check and verify that everything was in place for connectivity,” reported Jon Myer, Partner Solution Architect. “I really enjoyed my visit and seeing how everyone was working hard but having fun during the hackathon.”
Oren and Eran from Tel Aviv
“During our time in Sisense offices we witnessed several interesting initiatives that leverage AWS services and abilities,” said Oren Reuveni, a Solutions Architect at AWS SaaS Factory. “Data ingestion and analytics mechanism using Kinesis-analytics and Athena, an end-to-end serverless architecture that enables users to consume insights about their social network profile data and more.”
“I was deeply impressed seeing the culture of innovation and collaboration at Sisense Hackathon. Looking at the team translating the user interface using Amazon Translate shows truly customer-centric thinking.”
Eran Sidi, AWS Account Manager
It’s always fun to have friends pop in for a visit, but let’s get down to the serious business of talking about the ground-breaking projects our enterprising teams put together. The overall winner was the Magnificent Unicorn team with live-streaming analytics. The Kyiv office winners called themselves Wasted Potential and used machine learning and an open camera to figure out when the cafeteria lines were shortest. And, maybe most vitally, Big BIC Energy used a Raspberry Pi computer to handle the critical task of tracking how much beer is in the NYC office keg.
Read on to explore these amazing ideas. (Other than the world winner, they’re in no particular order!)
World winners, Magnificent Unicorn: Make faster, smarter decisions
The Sisense platform handles data of every kind, from every conceivable source, even allowing you to mashup live and cached data. The Magnificent Unicorn team took this to the next level with their streaming analytics project. For users with rapidly-changing datasets who need to see changes to their data as they happen, this is life-altering. Real-time BI is possible! And of course, AWS is a key to that possibility: pulling data into a Kinesis Stream Producer, then into Kinesis itself, a Lambda function, and finally through Socket.io into Sisense. The power of AWS undergirding Sisense is a proven winner and could change the way users understand and interact with their fast-moving data streams. No wonder they took world prize!
Wasted Potential: No more lunch lines
No matter where you are, this group created an application that removes long lunch lines. Our Kyiv winners know this, so rather than keep suffering in silence whenever they go down to the cafeteria for food, they made tech work for them. Using the publicly-available camera in the lunchroom, some AWS horsepower, and a little Machine Learning (ML) they came up with an effortless solution that lets them know exactly when it’s the best time to grab a bite.
By teaching a computer to visually recognize where the line forms and when there are humans on it, their algorithm is able to understand approximately how long the wait might be and advise users whether they should wait, or hurry up and eat. Users access the functionality via Slack (who doesn’t love Slack?). Then the information from the camera is pulled into an AWS visual recognition neural network (who doesn’t love neural networks?) and an answer is produced. Every request (and the associated queue time) is stored in Sisense, so that users can look at historical data and patterns.
All this was made possible by Amazon EC2, Amazon Lambda, Amazon Rekognition, and the Amazon API Gateway (and Sisense, of course).
Big BIC Energy: Beer barometer
It’s no secret that after a hectic week, a lot of us here at Sisense in NYC love to unwind with a cold beer (or nonalcoholic drink), so the keg holds a treasured place in many of our hearts. Oh the wails of consternation when someone pulls that lever and receives only a gout of foam and sadness! Those days are gone now, thanks to the BBE team “Beer Mate” project (with some help from AWS and a Raspberry Pi).
By attaching a flowmeter to the beer keg, using some handy Python code, and piping the data into a MySQL database on AWS, they’ve enabled the Sisense platform to achieve its most significant use case yet: letting everyone know how much beer is left! The system also informs our intrepid office manager, Sabrina, through Sisense Pulse Alerts when it gets low so that she can order more. Sisensers can also use Beer Mate’s integration with
Untappd to rate the beer and leave their thoughts. This is a great demonstration of just how outside-the-box Hackathon thinking can be, and how fun and useful the results often are!
What will you build?
We hope this brief glimpse into the Sisense 2020 Hackathon has given you a few ideas on setting up and running your own hackathon. There’s a reason why tech companies of all shapes and sizes have been holding these events for decades (and show no sign of stopping). New ideas from Hackathons end up in production all the time, including Sisense BloX to name one notable example. Whatever form your company’s hackathon takes, remember, the most important thing is to have fun, connect with your teammates, and build what matters to you!
Jack Cieslak is a 10-year veteran of the tech world. He’s written for Amazon, CB Insights, and others, on topics ranging from ecommerce and VC investments to crazy product launches and top-secret startup projects.
How a Flexible BI Platform Drives Business Results
Reading Time: 2 minutes
Customers matter. And thus whatever an organization can do to make smarter customer decisions faster will surely drive a better customer experience. Easier said than done in a dynamic business such as mortgage fulfillment services where time is of the essence and the data points involved are immense.
This is where adopting a flexible BI platform can come in handy. And this is exactly what Promontory MortgagePath did to help its lender customers make more timely and accurate credit decisions for their end customers.
Promontory MortgagePath provides mortgage fulfillment services for third party lenders. Driven by innovative technology and a customer-first ethos, their goal was to reduce the cost and the time its customers required to originate a new mortgage. And to help its customers make more profitable loans, data-driven insights on crucial credit decisions along with regulatory compliance, were also a must.
The company turned to TIBCO Jaspersoft embedded BI software to create reports, embed visualizations, and add self-service capabilities. As a result, Promontory MortgagePath was able to bring over 5,000 pieces of data together from disparate sources, analyze it, and display it in an easy-to-understand way. Additionally, the delivery of the information to its lenders and their customers was 83 percent faster than before. Both of these actions led to a more efficient lending process and far higher customer satisfaction.
Promontory MortgagePath also saw a number of benefits from adopting a more flexible platform, including:
- Faster insights, enabling anyone providing mortgage services to ask questions and get the answers they need from complex data in a short timeframe
- Easy for lenders to perform self-serve reporting, allowing them to access the data they need and easily display it in an intuitive and easy-to-understand pre-built report
- Improving the competitive playing field, so lenders can offering new mortgage products across any spectrum, any size lender, and any type of end customer, while also maintaining compliance with new regulations.
See the full case study to learn more about how Promontory MortgagePath’s flexible BI platform drove better business results.
Startups: Focus your Energy on your Application, not your Infrastructure
Reading Time: 2 minutes
While smart infrastructure decisions are vital to application software companies, the most important aspect of intellectual property (IP) sits in the product capabilities expressed at the application feature and function level. The application layer is most likely where your company will realize its greatest IP value.
Newly formed application software companies typically have one very important job, which is to validate the market with a ‘minimum viable product’(MVP). Behind the scenes, the process to build the MVP is usually driven by a set of market requirements that are almost always closely tied to substantive feedback by different types of potential users. In simpler terms, the product development focus must have a good grasp on what the customer wants and align that to the features within the software solution. As a consequence of the above-stated approach, one can imagine, nearly 90% of important conversations for application product development at an early stage will center around application capability as opposed to application infrastructure.
An overweight focus on the application functional layer is generally a good thing, as your company’s priority should be to get the application features to align with early customer needs and drive adoption. That said, after going to market and maturing beyond the MVP stage, most software teams will need to start paying attention to the infrastructure capability, which generally sits behind the scenes and will touch the areas of: security, scalability, microservices, and integration, just to name a few.
Most startups face the difficult decision of how much and where to invest infrastructure capability. They must choose between building it in-house (organic) and using an (OEM) solution as a deeply embedded option. Let’s take a moment to look at the pros/cons and market value impact of the aforementioned options at a high level.
Building your infrastructure in-house vs OEM
| Infrastructure Options | Pros | Cons | Market Value Conclusions |
| Organic (In-House) |
Very specific to the app use cases Full IP ownership |
Expensive to hire Evolving requirements will require further investment $ $ $ Longer to go to market |
Medium-IP owned by the firm but not viewed as essential by the customer because it’s behind the scenes. So, bang for the buck is moderate at best and execution risk is high given low expertise |
| OEM (Deeply Embedded) | Fastest to market Provides road-tested capability to reduce risk Quicker to adapt to customer technology shifts quickly Less expensive to maintain than hiring, building, and growing in-house infrastructure team |
Generally broad capability hammer for a thumbtack IP is owned by OEM company Need to invest in learning and embedding OEM through series of POCs |
High-market value is driven by much faster time to market and less technology risk and debt that might accrue if product landscape shifts |
As you can see from the above chart, embedding your application will give you a much higher market value because you will get your application to market faster. You will also experience less technology risk that might accrue if the product landscape you are working in shifts.
Learn how TIBCO’s deeply embedded OEM solutions can help you get your tech start-up’s infrastructure right and allow you to focus your energy on your applications.
Pentaho 9.0 is available
Pentaho 9.0 is available
Without further ado: Get Enterprise Edition here, and get Community Edition here
PDI Multi Cluster Hadoop Integration
Capability
Pentaho connects to Cloudera Distribution for Hadoop (CDH), Hortonworks Data Platform (HDP), Amazon Elastic MapReduce (EMR). Pentaho also supports many related services such as HDFS, HBase, Oozie, Zookeeper, and Spark.
Before this release, Pentaho Server as well as PDI design time environment – Spoon, can work with only one Hadoop cluster at a time. It required multiple transformations, instances, and pipelines to execute multiple Hadoop clusters. With 9.0 release, major architecture changes have occurred to easily configure, connect and manage multiple Hadoop clusters.
· Users can access and process data from multiple Hadoop clusters from different distros and versions- all from single transformation and instance of Pentaho.
· Also, within Spoon, users can now set up three distinct cluster configs, all having reference to the specific cluster, without having to restart Spoon. There is also a new configuration UI to easily configure your Hadoop drivers for managing different clusters.
· Improved cluster configuration experience and secure connection with the new UI
· Supports following distros: Hortonworks HDP v3.0, 3.1; Cloudera CDH v6.1, 6.2; Amazon EMR v5.21, 5.24.
Existing single cluster/shim functionality will continue to work.
The following example shows the Multi-cluster implemented in the same data pipeline via connecting to both Hortonworks HDP and Cloudera CDH clusters.
Use Cases and Benefits
· Enables hybrid big data processing support (on-prem or cloud)- all within single pipeline
· Simplifies Pentaho’s integration with Hadoop clusters including enhanced UX of cluster configurations
Key Considerations
· Adaptive Execution Layer Spark isn’t validated to execute pipelines connecting to multiple Hadoop clusters.
· Pentaho Map Reduce isn’t validated to execute pipelines connecting to multiple Hadoop clusters.
Additional Resources
See Adding a new driver for how to add a driver. See Connecting to a Hadoop cluster with the PDI client for how to create a named connection.
· Follow the suggestions in the Big Data issues troubleshooting sections to help resolve common issues when working with Big Data and Hadoop, especially Legacy mode activated when named cluster configuration cannot be located.
PDI AEL-Spark Enhancements
Capability
The Pentaho Adaptive Execution Layer (AEL) is intended to provide flexible and transparent data processing with Spark, in addition to the native Kettle engine. The goal of AEL is to develop complex pipelines visually and then execute in Kettle or Spark based on data volume and SLA requirements. AEL allows PDI users to designate Spark as execution engine for their transformations apart from Kettle.
The v9.0.0 release includes the following performance and flexibility enhancements to AEL-Spark:
· Step level spark specific performance tuning options
· Enhanced logging configuration and information entered into PDI logs
· Added support for Spark 2.4, with existing 2.3 support
· Supports following distros: Hortonworks HDP v3.0, 3.1; Cloudera CDH v6.1, 6.2; Amazon EMR v5.21, 5.24.
The following example showcases Spark App and Spark Tuning on specific steps within a PDI transformation:
Use Cases and Benefits
· Eliminates black box feel with better visibility
· Enable advanced Spark users with tools to improve performance
Key Considerations
Users must be aware of the following additional items related to AEL v9.0.0:
· Spark v2.2 is not supported.
· Native HBase steps are only available for CDH and HDP distributions.
· Spark 2.4 is the highest Spark version currently supported.
Additional Resources
See the following documentation for more details: About Spark Tuning in PDI, Setup Spark Tuning, Configuring Application Tuning Parameters for Spark
Virtual File System (VFS) Enhancements
Capability
· The changes to the VFS are in two main areas:
· 1. We added Amazon S3 and Snowflake Staging as VFS providers to named VFS Connections and introduced the Pentaho VFS (pvfs) that can reference defined VFS Connections and their protocols. In the S3 protocol, we support S3A and Session Tokens in 9.0.
· The general format of a Pentaho VFS URL is:
pvfs://VFS_Connection/path (including a namespace, bucket or similar)
· 2. A new file browsing experience has been added. The enhanced VFS browser allows users to browse any preconfigured VFS locations using named connections, their local filesystem, configured clusters via HDFS, as well as a Pentaho repository, if connected.
Use Cases and Benefits
· Through the support of Pentaho VFS, you have an abstraction of the protocol. That means, when you want to change your provider in the future, all your jobs and transformations work seamless after this change in the VFS Connection. Today, you reference S3. Tomorrow, you want to reference another provider, for example HCP or Google Cloud. Using Pentaho VFS, your maintenance burden in these cases is much lower.
· VFS Connections also enables you to use different accounts and servers (including namespaces, buckets or similar) within one PDI transformation. Example: You want to process data within one transformation from S3 with different buckets and accounts.
· Combining named VFS connections with the new file browsing experience provides a convenient way to easily access remote locations and extend the reach of PDI. The new file browser also offers the ability to manage files across those remote locations. For example, a user can easily copy files from Google Cloud into an S3 bucket using the browser’s copy and paste capabilities. A user can then easily reference those files using their named connections, in supported steps and job entries.
A user can manage all files, whether they are local or remote in a central location. For example, there is no need to login to the Amazon S3 Management Console to create folders, rename, delete, move or copy files. Even a copy between the local filesystem and S3 is possible and you can upload/download files from within Spoon.
The new file browser also offers capabilities such as search, which allows a user to find filenames which match a specified search string. The file browser also remembers a user’s most recently accessed jobs and transformations for easy reference.
Key Considerations
As of PDI 9.0, the following protocols are supported: Amazon S3, Snowflake Staging (read only), HCP, Google CS
The following steps and job entries have been updated to use the new file open save dialog for 9.0: Avro input, Avro output, Bulk load into MSSQL, Bulk load into MySQL, Bulk load from MySQL, CSV File Input, De-serialize from file, Fixed File Input, Get data from XML, Get file names, Get files rows count, Get subfolder names, Google Analytics, GZip CSV input, Job (job entry), JSON Input, JSON Output, ORC input, ORC output, Parquet Input, Parquet output, Text file output, Transformation (job entry)
The File / Open dialog is still using the old browsing dialog. The new VFS browser for opening jobs and transformations can be reached through the File / Open URL menu entry.
Additional Resources
See Virtual File System connections, Apache Supported File Systems and Open a transformation for more information.
Cobol copybook steps
Capability
PDI now has two transformation steps that can be used to read mainframe records from a file and transform them into PDI rows.
· Copybook input: This step reads the mainframe binary data files that were originally created using the copybook definition file and outputs the converted data to the PDI stream for use in transformations.
· Read metadata from Copybook: This step reads the metadata of a copybook definition file to use with ETL Metadata Injection in PDI.
The Copybook steps also support metadata injection, extended error handling and can work with redefines. Extensive examples for these use cases are available in the PDI samples folder.
Use Cases and Benefits
Pentaho Data Integration supports simplified integration with fixed-length records in mainframe binary data files, so that more users can ingest, integrate, and blend mainframe data as part of their data integration pipelines. This capability is critical if your business relies on massive amounts of customer and transactional datasets generated in mainframes that you want to search and query to create reports.
Key Considerations
This step works with Fixed Length COBOL records only. Variable record types such as VB, VBS, OCCURS DEPENDING ON are not supported.
Additional Resources
For more information about using copybook steps in PDI, see Copybook steps in PDI
Additional Enhancements
New Pentaho Server Upgrade Installer
The Pentaho Server Upgrade Installer is an easy to use graphical user interface that automatically applies the new release version to your archive installation of the Pentaho Server. You can upgrade versions 7.1 and later of the Pentaho Server directly to version 9.0 using this simplified upgrade process via the user interface of the Pentaho Server Upgrade Installer.
See Upgradethe Pentaho Server for instructions.
Snowflake Bulk Loader improvement
The Snowflake Bulk Loader has added support for doing a table preview in PDI 9.0. When connected to Snowflake and on the Output tab, select a table in the drop-down menu. The preview window is populated, showing the columns and data types associated with that table. The user can see the expected column layout and data types to match up with the data file.
For more information, please see the job entry documentation of the Snowflake Bulk Loader.
Redshift IAM security support and Bulk load improvements
With this release, you have more Redshift Database Connection Authentication Choices, these are
· Standard credentials (default) – user password
· IAM credentials
· Profile located on local drive in AWS credentials file
Bulk load into Amazon Redshift enhancements: New Options tab and Columns option in the Output tab of the Bulk load into Amazon Redshift PDI entry. Use the settings on the Options tab to indicate if all the existing data in the database table should be removed before bulk loading. Use the Columns option to preview the column names and associated data types within your selected database table.
See Bulk load into Amazon Redshift for more information.
Improvements in AMQP and UX changes in Kinesis
The AMQP Consumer step provides Binary message support, for example allowing to process AVRO formatted data.
Within the Kinesis Consumer step, users can change the output field names and types.
See the documentation of the AMQP Consumer and Kinesis Consumer steps for more details.
Metadata Injection (MDI) Improvements
In PDI 9.0.0, we continue to enable more steps to support metadata injection (MDI):
· Split Field to Rows
· Delete
· String operations
In the Excel Writer step, the missing MDI step option “Start writing at cell”, has been added. This option can also be injected now.
Additionally, the metadata injection example is now available in the samples folder:
/samples/transformations/metadata-injection-example
See ETL metadata injection for more details.
Excel Writer: Performance improvement
The performance of the Excel Writer has been drastically improved when using templates. A sample test file with 40,000 rows needed about 90 seconds before 9.0 and now processes in about 5 seconds.
For further details, please see PDI-18422.
JMS Consumer changes
In PDI 9.0, we added the following fields to the JMS Consumer step: MessageID, JMS timestamp and JMS Redelivered.
This addition enables restartability and allows to omit duplicate messages.
For further details, please see PDI-18104 and the step documentation.
Text file output: Header support with AEL
You can set up the Text file input step to run on the Spark engine via AEL. The Header option of the Text file output step works now with AEL.
For further details, please see PDI-18083 and the Using the Text File Output step on the Spark engine documentation.
Transformation & Job Executor steps, Transformation & Job entries: UX improvement
Before 9.0, when passing parameters to transformations/jobs, the options “Stream column name” vs. “Value” (“Field to use” vs. “Static input value”) were ambiguous and led to hard to find issues.
In 9.0, we added behavior which prevents a user from entering values into both fields to avoid these situations.
For further details, please see PDI-17974.
Spoon.sh Exit code improvement
Spoon.sh (that gets called by kitchen.sh or pan.sh) sends the wrong exit status in certain situations.
In 9.0, we added a new environment variable FILTER_GTK_WARNINGS to control this behavior for warnings that effect the exit code. If the variable is set to anything, then a filter is applied to ignore any GTK warnings. If you don’t want to filter any warnings, then unset FILTER_GTK_WARNINGS.
For further details, please see PDI-17271.
Dashboard: Option for exporting analyzer report into CSV format.
Now it’s possible to export an analyzer report into a CSF format file even when embedded on a dashboard.
In the previous release the export option was available, but without the CSV format.
The CSV format was available only when using Analyzer outside dashboards, in this way we provide functional parity between Analyzer standalone charts and charts embedded in dashboards.
For further details, please see PDB-1327.
Analyzer: Use of date picker when selecting ranges for a Fiscal Date level relative filter.
Before 9.0 and for an AnalyzerFiscalDateFormat annotation on a level in a Time dimension, Analyzer did not show the “Select from date picker” link.
Now, relative dates can be looked up from the current date on the Date level, then the date picker can also be used to select the nearest fiscal time period.
For further details, please see ANALYZER-3149.
Mondrian: Option for setting the ‘cellBatchSize’ default value.
From a default installation the mondrian.properties does not include mondrian.rolap.cellBatchSize as a configurable property.
The purpose of this improvement is to include this property in the mondrian.properties by default in new builds so customers do not run into performance issues due to the default value for this property being set too low. The default value of the property should be clearly indicated in the properties file as well.
The default value has been updated to mondrian.rolap.cellBatchSize=1000000.
This value was chosen because this setting can run a very large 25M cell space report while keeping total server memory usage around 6.7 GB which is under the 8GB we list as the minimum memory required on a Pentaho server.