Source Archives - Business Intelligence Info

Tag Archives: Source

Why a Single Source of Truth Often Isn’t

November 22, 2019 CRM News and Info

This week, Salesforce used its annual Dreamforce mega-show to make a host of announcements, ranging from contact center partnership with Amazon to a smart speaker tool for sales.

For the most part, the keynote featured practical applications of Salesforce technology. Some Dreamforces focus on the highly aspirational; others focus on the ways to attain those aspirations. This year is the latter.

However, Salesforce did get very aspirational with the announcement of Customer 360 Truth. Last year’s event followed the Mulesoft acquisition, and
the big idea was the unification of data across various cloud-based systems in something it called “Customer 360.”

The cloud has enabled companies to become profligate in their use of on-demand, easy-to-deploy applications, which started to silo and separate information — the exact issue that CRM was introduced to combat way back when companies lacked a single source of sales data.

This year Salesforce took that idea a step further. Customer 360 Truth aims not simply to aggregate that data but to organize it across sales, marketing, service, commerce and more. The goal is to create a complete view that reflects the reality of the customer at this moment, enabling the company to deliver a trusted, personalized, customized relationship with each customer.

The Fourth Wave

Applying technology to customer data for a more accurate understanding is not a new concept. More than a decade ago, InsideView began gathering information about customers, examining it, and presenting the pieces of data that were the most current. Even back then, the term “single source of truth,” or SSoT, was bandied about with enthusiasm.

Mark Benioff took that enthusiasm to the next level Tuesday by declaring that SSoT was the “Fourth Wave of computing.” That is dramatically overstating things. We’ve spent the last 35 years using technology to collect, collate and employ data about our customers. Data was scarce 35 years ago. Now we can collect a lot of it — and sure enough, it needs to be sorted. However, the fundamentals have not changed.

What I worry about is the term “single source of the truth,” and what it implies. In the context of a business, it means the most accurate data set in its possession. That is never going to be the sole source of truth about someone, or even the best source of truth. It just happens to be the source of truth your analytic systems have decided on, based on what your data collection systems have gathered. It’s useful — but be careful.

Getting to Know You – or Not

Every day, I am exposed to evidence that the “truth” about me, as understood by the data systems of various companies, bears very little resemblance to the actual truth (no quotes) about me. I regularly receive “targeted” ads on social media that are wildly off the mark: emails touting products similar to one-off purchases of gifts, or Halloween costumes bought for my daughter years ago, or direct mail asking for my support for causes that I’m diametrically opposed to.

These ads would suggest that I’m a Republican gun owner who wears lots of T-shirts and is really into Tinker Bell. That’s about as wrong as you can get it, and businesses waste lots of money on me believing it’s a real truth.

I’m sophisticated enough to know why these false indications exist: flaws in data collection, half-baked inferences about customers, sloppy data entry, and so on. This is where the concept of a SSoT can become derailed. No matter how much you invest in technology to pull data together, when you use faulty data to determine what is true, what you think is the truth will be faulty.

The other mistake businesses make is to believe they own the SSoT about customers. In reality, we each own the best SSoT about ourselves. However, data about people is much easier to manage and deal with than data from the people themselves. So businesses spend ever more money making educated guesses about their customers, doing increasingly complex digital detective work, and kidding themselves that their SSoT is a perfect reflection of reality.

Separating specious data from accurate data is important, but unless your processes ruthlessly eradicate spotty or off-base data collection, routinely challenge inferences and assumptions about what data means, and check in with customers to see if their truth is close to your truth, your single source of the truth could be your single biggest weakness.

The opinions expressed in this article are those of the author and do not necessarily reflect the views of ECT News Network.

Chris%20Bucholtz Why a Single Source of Truth Often Isnt
Chris Bucholtz has been an ECT News Network columnist since 2009. His focus is on CRM and other topics surrounding buyer-seller relationships. He is director of content marketing for
NewVoiceMedia, and a speaker, writer and consultant. He also has written four books on World War II aviation.
Email Chris.

Let’s block ads! (Why?)

CRM Buyer

Multiple data source definitions on the On-premises data gateway for the same data source

July 6, 2019 Self-Service BI

Gateway management until recently, supported only one data source definition per data source on an on-premises data gateway. This posed limitations for gateway admins as they couldn’t create multiple data sources to the same data source each with different credentials based on the required security context for each one. The workaround would be to spin up a new gateway for every such new data source.

Note: The data source name still needs to be unique for every data source definition.

Below you can see that a gateway admin has created two data source definitions for the same database AdventureWorks2017 on the same server.

Once you publish a Power BI report, in the Datasets settings, you will be able to see the list of data sources you have access to as below. You can now select one of these data sources.

If you are a gateway admin, you could also additionally choose to add a new data source(as shown below) and that would take you to the Manage Gateways page to create a new data source.

If you have access to only one data source, you must explicitly select that data source.

Once all required data sources have been set up and selected, click on apply to bind this dataset to the selected data source on the gateway.

Note: This feature is currently available only for datasets. We plan to release the same feature for dataflows soon. Currently in dataflows, until this change is rolled out, the first available data source would be automatically selected.

Do try out this feature and provide your feedback. As always we would love to hear from you, learn from your experience and improve based on your feedback.

Thank you!

Let’s block ads! (Why?)

Microsoft Power BI Blog | Microsoft Power BI

Tier IV raises over $100 million to develop open source software for driverless cars

July 6, 2019 Big Data

Tier IV, a Japan-based driverless car software maintainer and provider, this week announced the closure of a round north of $ 100 million led by Sompo Japan Nipponkoa Insurance, with participation from Yamaha Motor, KDDI, JAFCO, and Aisan Technology. The fresh capital brings the company’s total raised to nearly $ 130 million following seed rounds totaling $ 28 million, and founder Shinpei Kato said it’ll fuel the global commercialization and expansion of Tier IV’s self-driving technology platform.

“Tier IV has a mission to embody disruptive creation and creative disruption with self-driving technology. We have derived a solid software platform and successfully integrated it with real vehicles,” said Kato. “It is time to step forward to real services, embracing functional safety and risk management.”

Tier IV, a University of Tokyo spinout founded in December 2015, spearheads the development of Autoware, which it describes as an “all-in-one” open source and BSD-licensed solution for autonomous vehicles. The platform supports things like 3D localization and mapping, 3D path planning, object and traffic signal detection, and lane recognition, plus tasks like sensor calibration and software simulation.

Tier IV funds this development in part by selling support equipment like remote controllers and logging devices, as well as desktops and laptops with Autoware preinstalled. Additionally, it offers subscription access to its data sets, labeling tools, and deep learning training services for $ 1,000 per year.

Tier IV’s stated mission is to “democratize” intelligent cars by enabling “any individual or organization” to contribute to their development. To this end, it and partner companies Apex.AI and Linaro 96Boards launched the nonprofit Autoware Foundation last December, which seeks to deploy Autoware in production products and services. The Foundation counts 30 companies among its membership, and Tier IV claims that Autoware has already been adopted by more than 200 organizations around the world to date, including Udacity (for its Nanodegree Program), the U.S. Department of Transportation Federal Highway Administration (in its CARMA platform), automotive manufacturers, and “many” self-driving startups.

Field tests of Autoware-powered cars have been conducted in over 60 regions in Japan and overseas “without incident,” according to Tier IV, and the company claims that vehicles running on its platform achieved level 4 autonomy (meaning they could operate safely without oversight in select conditions) as early as December 2017.

Tier IV competes to an extent with Baidu, which offers an open source driverless software stack of its own in Apollo. The Beijing-based tech giant claims that Apollo — which has grown to 400,000 lines of code, more than double the 165,000 lines of code the company announced in January 2018 — is now being tested, contributed to, or deployed by Intel, Nvidia, NXP, and over 156 global partners, including 60 auto brands. Notable Apollo collaborators include Chinese automobile manufacturers Chery, BYD Auto, and Great Wall, Hyundai Kia, Ford, and VM Motori.

Sign up for Funding Daily: Get the latest news in your inbox every weekday.

Let’s block ads! (Why?)

Big Data – VentureBeat

Oracle and Teradata data source support and hourly e-mail subscriptions for Paginated Reports now available

May 3, 2019 Self-Service BI

For the fifth week in a row, we’re pleased to announce two new features on “New Feature Friday” for Paginated Reports in Power BI. We’ve added support for two new data sources, Oracle and Teradata, and also added the ability to schedule your e-mail subscriptions on an hourly basis.

Oracle and Teradata Now Supported

We added Oracle and Teradata data source support to Power BI Report Builder in early April so report authors could connect directly and begin creating reports with these popular data sources. Now, you can publish these reports to your Premium workspaces in Power BI. These data sources require the on-prem data gateway to be setup, just as SQL Server and SQL Server Analysis Services data sources do today. All regions have support for these data sources fully rolled out, so you can take advantage of these providers right away in your Power BI tenant.

Hourly E-Mail Subscriptions

In addition, we’ve added the ability to set an hourly cadence for your e-mail subscriptions in Power BI. You’ll see it as an additional option in the frequency dropdown for your paginated reports within the next few days. You can setup multiple hourly subscriptions per report, allowing you to receive the latest data in your inbox as often as every 15 minutes. And with Paginated Reports always receiving the latest data from your underlying data source, you and your users can always have the latest report data delivered right to their inbox.

While the hourly frequency option is only available for paginated reports currently, we’re adding this option for all content types and all users in Power BI in the coming weeks, so watch out for that in a coming post!

We’re taking a bit of break the next couple weeks around new feature announcements, but there’s plenty more coming as we head towards GA of Paginated Reports.

Thanks and don’t forget to leave your feedback in the comments below!

Let’s block ads! (Why?)

Microsoft Power BI Blog | Microsoft Power BI

Power BI Dataset data source support for Paginated Reports is now available

April 29, 2019 Self-Service BI

As discussed in our blog post announcing support for Azure Analysis Services, we promised to unlock the critical capability of allowing Paginated Reports to be built on the same enterprise semantic model as your interactive reports. This week for “New Feature Friday”, we’re pleased to announce that Power BI Datasets in premium workspaces are now supported as data sources for your Paginated Reports. While we announced the ability for authors to create reports using Power BI datasets (via the XMLA endpoint) in Power BI Report Builder three weeks ago, we’ve now unlocked the capability to publish these reports to the service and be shared widely across your organization either via apps or e-mail subscriptions.

Please note the XMLA endpoint will be enabled again in production by April 28th.

Create paginated reports with Power BI Datasets

To create a paginated report using Power BI datasets with Power BI Report Builder, you can select SQL Server Analysis Services as your source and connect to your Power BI Dataset using the instructions discussed in the announcement for XMLA support. With paginated reports, you may choose one or more Power BI datasets to connect to in the same report, as there are no restrictions on the number of live data sources a report author may use. Both Power BI datasets using imported data or Direct Query can be used with your paginated reports. Be sure to read the documentation about creating reports with Report Builder to learn more about authoring paginated reports.

Upload to Power BI

After you have saved your paginated report file, you can then go to the Get Data screen within the service and upload your report to a Premium workspace. You’re not limited to the same workspace as your Power BI datasets being used for your paginated reports, so you may choose any workspace in your Premium capacity to publish your reports to. As always, when you consume the semantic model(s), the role-based security (RLS) defined in the model is honored for each user regardless of tool, and this is the same for Paginated reports.

Distribute your paginated reports across the organization

You can easily share your paginated reports you’ve created now against your Power BI datasets either via individual report sharing, apps, or e-mail subscriptions. You may subscribe yourself or other users to these reports with a full PDF of the report attached. Make sure you read the documentation to learn more about the specific paginated report capabilities around e-mail subscriptions.

This has been a top feature ask since we first launched paginated reports, and we’re excited to hear your feedback. We’ll have another post next week announcing more new features for Paginated Reports, including additional data source support. Have a wonderful weekend!

Let’s block ads! (Why?)

Microsoft Power BI Blog | Microsoft Power BI

Building a Better Tomorrow with Open Source Analytics Tools

April 17, 2019 Sisense

There’s nothing more satisfying than building something yourself. You know what you want your creation to accomplish, the unique challenges you want to tackle, and you pick the features you need. Then you find the raw materials or the base kit to start building from.

That’s the whole idea of open source software: the world should all be one giant Lego kit, where developers of all stripes can share skills, code, and create new and different programs to build the world they want to see. Linux is one of the most prolific operating systems on the planet precisely because it is open source, freely available, and infinitely customizable. Communities that spring up around individual distributions to discuss their nuances and to marvel at (and share) what others have done with the software testify to its lasting value as a phenomenon and an operating system.

So when it comes to building a database, going open source is a natural fit. While all databases have commonalities, every use case has its own hangups, and enterprising builders want to tackle these challenges in their own ways. Open source data solutions like the ones we’ll discuss here allow them to do just that: take the software and make it theirs. Whether the goal is to present data via simple visualizations, connect it to a robust BI tool, or anything else you want to do, having an open source option gives you the power and control you need to get the job done.

In this article, we’ll dig into Hadoop, PostgreSQL, Apache Cassandra, and Elasticsearch.

Hadoop

The world moves fast: tons of data is generated every minute and organizations of all kinds need a powerful system that can keep up with that dataflow. That’s something Hadoop excels at. Originally created in 2006, it’s one of the most popular open source BI tools. The secret to its speed and success in adapting to the current market could be its distributed cloud-storage model, which allows it to process extremely large datasets. The Hadoop Distributed File System (HDFS) transfers data rapidly between nodes and helps provide redundancy—even if one node fails, the rest of the system keeps running smoothly. This also protects the data itself, as multiple copies are stored in multiple locations. If you’re paranoid about data loss then this is a vital feature, protecting you and your data in the case of unexpected power failure or equipment damage.

Additionally, Hadoop uses a Java-based programming framework. Given the popularity of Java and the availability of Java developers, this consideration could make a difference when building a team or using existing developers within an organization. Between the language undergirding it and the power of its architecture, Hadoop has found a sizable following, tackling core BI tasks like statistical analytics and Big Data processing, including handling huge volumes of data from fleets of IoT sensors and more!

PostgreSQL

As the name implies, PostgreSQL is built on and extends the SQL language. The result is a robust, open source BI tool built for the toughest, most complicated workloads. It’s also got a lengthy pedigree, especially for software, tracing its origins all the way back to 1986. Over the years, the core platform has been continually worked on and refined and is now an ideal option to safely store and scale almost any project.

One amazing asset that PostgreSQL brings to the table is flexibility. Not only does it run on all major operating systems (Windows, all your favorite Linux distros, etc.), but you can also write code in other languages and add it to your Postgres database without recompiling your database. For a dev team tackling a unique data challenge, this could be a major game changer. It also connects easily to a variety of PostgreSQL reporting tools to add even more functionality to this robust open source analytics tool.

Apache Cassandra

Got tons of data distributed across commodity hardware? Cassandra is a distributed NoSQL database that’s built to handle Big Data with high availability via an architecture that leaves no single point of failure. Builders turn to Cassandra for continuous availability, performance scaling, and easy data distribution. Apple, Netflix, and eBay have turned to this system to create the relational databases to suit their needs.

The power to customize Cassandra is in your hands. Choose synchronous or asynchronous replication when updates come around, even optimize highly-available asynchronous operations. Additionally, Cassandra’s unique “ring” architecture, helps ensure amazing uptime, since every node communicates with every other node equally. This even ensures that there’s no downtime when adding new machines. If scaling data is in your future, then thinking of how new machines will come online is an important consideration.

Elasticsearch

With a name like Elasticsearch, you’d expect it to be flexible and functional, and you’d be right. Elasticsearch tackles data from any source and in any format, allowing you to build the open source analytics solution of your dreams, no matter what data you’re dealing with or where it’s coming from.

Originally compiled in Java, it uses a JSON-style query language that handles structured, unstructured, and time-series data. Elasticsearch’s flexibility also applies to hardware: run it on whatever machines suit your needs, communicating with one or one hundred nodes in the same, seamless style. And its distributed architecture keeps your data backed up in case of failure. Whatever you build, you know your data will be there when you need it and that Elasticsearch will help you perform the analysis you need.

Wrapping Up

Sometimes the hardest part of building the world you want is figuring out where to start. When your analytics dreams were small, it didn’t matter what platform you picked. Now that you have big goals, you have to pick the tools to make them a reality. Choosing an open-source OS like Linux and pairing it with an open-source data tool like the ones mentioned here is a powerful first step towards giving yourself a wide array of options and leveraging both the power of open-source software and of the vibrant community of developers and other professionals who’ve dedicated their lives to working with these programs.

Now that you have the background on each, the decision is in your hands. Choose wisely, build well.

Tags: Databases | linux

Let’s block ads! (Why?)

Blog – Sisense

Baidu unveils open source edge computing platform and AI boards

January 11, 2019 Big Data

Baidu’s had a busy week. Fresh off of yesterday’s unveiling of Apollo 3.5, the latest generation of its self-driving platform, the Beijing company announced OpenEdge, an open source computing platform that enables developers to build edge applications “with more flexibility.” It also announced two new AI hardware development platforms: the BIE-AI-Box, a kit for in-car video analysis designed in partnership with Intel, and the BIE-AI-Board, a chipboard codeveloped with NXP that’s optimized for object classification.

“The explosive growth of IoT devices and rapid adoption of AI is fueling great demand for edge computing,” Watson Yin, vice president and general manager of Baidu Cloud, said. “Edge computing is a critical component of Baidu’s ABC (AI, Big Data and Cloud Computing) strategy.”

OpenEdge

OpenEdge is the local package component of Baidu’s commercial Baidu Intelligent Edge (BIE), which the company claims is China’s first open source edge computing platform.

The BIE platform underpinning OpenEdge offers a cloud-based management suite to manage edge nodes, edge apps, and resources such as certification, password, and program code. It supports models trained on AI frameworks such as Google’s TensorFlow and Baidu’s own PaddlePaddle, meaning that developers can train AI models on BIE and deploy them locally. Moreover, devices deployed with BIE are afforded additional features, like the ability to cache data and perform on-device processing in the event of a flaky network connection.

These tools together, Baidu says, let developers build custom edge computing systems on a range of hardware that can collect data, distribute messages, perform AI inference, and synchronize with the cloud.

“By moving the compute closer to the source of the data, it greatly reduces the latency, lowers the bandwidth usage, and ultimately brings real-time and immersive experiences to end users,” Yin said. “And by providing an open source platform, we have also greatly simplified the process for developers to create their own edge computing applications.”

BIE-AI-Box and BIE-AI-Board

Baidu and Intel’s BIE-AI-Box is, as alluded to earlier, a hardware kit custom built for analyzing the frames captured by cockpit cameras. Toward that end, it incorporates BIE technologies “specially” engineered for the purpose, and connects with cameras for road recognition, car body monitoring, driver behavior recognition, and other tasks.

As for the BIE-AI-Board, which is designed for object recognition, it’s compact enough to be embedded into cameras, drones, robots, and other hardware. Early partners have integrated it with electric vehicles to assess the health of chargers and with agricultural drones to analyze crop spectral data, Baidu says. (In the latter case, it helped to reduce pesticide use by up to 50 percent.)

Baidu’s looking to the cloud for revenue growth. It recently partnered with Nvidia to bring the chipmaker’s Volta graphics platform to Baidu Cloud, and in July 2018, it unveiled two new chips for AI workloads: the Kunlun 818-300 for machine learning model training and the Kunlun 818-100 for inference.

According to Gartner, the cloud computing market is projected to be worth $ 441 billion by 2020.

Let’s block ads! (Why?)

Big Data – VentureBeat

Hyundai joins the Linux Foundation to embrace AGL’s open source connected car technologies

January 4, 2019 Big Data

Hyundai has become the latest car company to explore serious open source alternatives for developing its in-car services. Ahead of CES 2019, the South Korean automotive giant today announced that it has joined the Linux Foundation and the nonprofit’s seven-year-old Automotive Grade Linux (AGL) effort as it looks to contribute to — and reap benefit from — software developed by over 140 companies.

The announcement underscores the growing popularity of AGL, which has attracted dozens of car manufacturers and other companies in recent years. Members of AGL, which include Toyota, Ford, Honda, Suzuki, Intel, Nvidia, ARM, and LG, work in tandem to develop open source software for infotainment, telematics, and instrument cluster applications.

In a statement, Dan Cauchy, executive director of Automotive Grade Linux at the Linux Foundation, touted today’s announcement as a “significant milestone.” He added, ” … the rapid growth of AGL proves that automakers are realizing the business value that open source and shared software development can provide.” Hyundai is joining the Linux Foundation as a silver member and AGL at the bronze level, a spokesperson for the Linux Foundation told VentureBeat. (The tier of the membership is decided based on the amount a company donates to the Linux Foundation.)

For Hyundai, open collaboration is crucial as it pursues a “connected car vision,” Paul Choo, VP and head of Infotainment Technology Center at Hyundai, said in a statement. Car companies have traditionally taken three years or longer to develop in-vehicle services, such as infotainment systems.

The bottleneck usually lies in the quality of code their in-house programmers create. According to a case study published by AGL, a connected car uses some 100 million lines of code, which is about 11 times more than the number that went into the F-35 fighter jet. “Software has become more important than ever, but the traditional, proprietary software development model has created widespread fragmentation that is slowing the pace of innovation,” the organization said.

Getting on AGL’s bandwagon would also help Hyundai speed up development of its in-car technologies. Members of AGL get access to AGL Unified Code Base (UCB), which offers 70 percent of the starting point for a production project. It also includes an operating system, middleware, and application framework, according to the Linux Foundation.

The Linux Foundation, which will demo a number of its in-car projects at the trade show starting next week, is coming off a great year. In 2018, it attracted Uber, Sony Pictures Entertainment, Warner Bros., and dozens of other companies, as open source is finally getting the limelight.

Let’s block ads! (Why?)

Big Data – VentureBeat

Power BI solution templates moving to open source

December 9, 2018 Self-Service BI

January 1, we will be moving Power BI solution templates to open source.

This has the following implications:

The github repository will remain open and anyone can build solutions on the Power BI solution template code base.
The existing appsource entries for solution templates will be retired and links to appsource from Power BI will be removed.

One of the solution templates, Power BI Usage Metrics, is being hosted by its author, Neal Analytics, and will be available from appsource.

If you’ve implemented a Power BI solution template, this change has no impact on you.

Installed solution templates are unaffected. This only impacts you if you plan on implementing a new solution template in the future. If you want to do so, please do so this year.

We will continue to support solution templates until the end of the year. Beyond Jan 1, we’ll continue to respond to your questions .

If you’re interested in contributing to Power BI solution templates or have any questions about this change, please contact us at powerbisolutiontempaltes@microsoft.com

Let’s block ads! (Why?)

Microsoft Power BI Blog | Microsoft Power BI

From Source to Analytics: A Farm-to-Table Approach to Data Governance

September 4, 2018 Big Data

Tendu Yogurtcu

September 3, 2018

Editor’s note: This article on data governance written by Syncsort’s Tendü Yoğurtçu was originally published in Information Management.

Farm-to-table conjures up images of fresh produce delivered directly from the garden to the kitchen. A foundational part of this concept is knowing where the food is grown, where it’s traveled and if it’s safe to eat.

As it turns out, a similar approach can be applied to data governance. Farm-to-table is one way IT experts can frame how they’re governing data and evaluating the tools and technologies necessary for meeting compliance requirements and use cases like anti-money laundering or credit card fraud.

As data and IT professionals think about the “data lifecycle” – from origin of data to delivery in next generation analytics platforms – they must understand the importance of a strategic approach to data governance through security and data quality practices. This ensures data arrives fresh and clean for advanced analytics and can be easily tracked.

Perhaps the most important consideration for farm-to-table food is deciding if it’s safe to eat. So too is confirming the security of data for your organization.

From Source to Analytics: A Farm to Table Approach to Data Governance

The requirements for data availability and security are becoming more complex and broad thanks to the data explosion of the digital era. Organizations are forced to review their data availability and disaster recovery policies covering global data centers on-premise or in the cloud to plan for unexpected outages.

Companies must implement strategies and technologies to guarantee data is secure, available 24/7 and protected. For highly regulated industries such as banking or insurance, security is more important as data must satisfy specific, stringent regulatory compliance requirements. Yet for all industries, meeting privacy, confidentiality and compliance requirements on the growing sets of data is critical and challenging.

The General Data Protection Regulation (GDPR) and similar regulations require companies to implement best practices around data encryption, masking and anonymization to ensure their data lakes, data-as-a-service and data marketplace environments are as managed and secure as possible.

Securing data in the most optimized, scalable and efficient way possible while meeting SLA requirements and providing real-time insights is a key challenge. Organizations must prioritize selecting data products that are flexible and easy to integrate with security frameworks.

Just as consumers want to know if their food is fresh, so do organizations need to know their data is trusted and fit for purpose. As data flows in and out of the data lake, businesses must understand the sources of and relationships between data and set up business rules to continuously measure quality.

An effective data quality policy will take into consideration missing, outdated or redundant data and inconsistent formats to create consolidated, clean and verified data for analytics and reporting. Ultimately, bad data leads to incorrect insights.

The increased adoption of machine learning for mainstream use cases like fraud detection and more transformative use cases like analyzing social networks for customer insights must rely on trusted data sets. The impact of bad data is multiplied for next generation analytics, especially because the data is used to train algorithms to make recommendations based on new data sets.

For example, entity resolution – distinguishing matches across massive datasets that indicate a single specific entity – requires sophisticated multi-field matching algorithms and significant compute power. To meet these evolving use cases, data quality, cleansing and preparation routines must handle rich data formats with growing data volumes and be highly scalable.

Advanced business and operational analytics also require data to be kept in sync with the data source. Tracking and detection must happen rapidly, as current transactions need to be continuously added to combined datasets and prepared and presented to models as close to real-time as possible.

With the growing adoption of data-as-a-service and platform-as-a-service architectures including cross-platform, hybrid cloud environments and multi-cloud implementations, creating visibility into data becomes crucial. Knowing where the data originated, what transformations it has gone through and how many copies of the data set are replicated across multiple data stores is necessary both to serve compliance requirements and to optimize resources. For a use case like anti-money laundering, financial institutions require complete, detailed data lineage from origin to end point, making it a core data governance element.

In selecting data lineage tools, IT professionals should guarantee solutions offer a comprehensive and granular view of where the data has been and what’s been done to it along the way, regardless of whether data movement and transformation ran inside or outside of Hadoop, the data lake, on-premise or in the cloud.

As IT professionals create a strategic approach to data governance to deliver meaningful insights and meet compliance requirements, they’re placing greater emphasis on security, data quality and data lineage to guarantee trusted and current data is used for analytics. Current trends in analytics with machine learning and AI are becoming widespread, and these strategic considerations are integral as companies look to achieve greater insights and more value from their data.

Fore more, make sure to check out our webcast from Dr. Tendü Yoğurtçu on Data Quality and Lineage.

Let’s block ads! (Why?)

Syncsort Blog

Tag Archives: Source

Why a Single Source of Truth Often Isn’t

The Fourth Wave

Getting to Know You – or Not

Multiple data source definitions on the On-premises data gateway for the same data source

Tier IV raises over $100 million to develop open source software for driverless cars

Oracle and Teradata data source support and hourly e-mail subscriptions for Paginated Reports now available

Oracle and Teradata Now Supported

Hourly E-Mail Subscriptions

Power BI Dataset data source support for Paginated Reports is now available

Create paginated reports with Power BI Datasets

Upload to Power BI

Distribute your paginated reports across the organization

Building a Better Tomorrow with Open Source Analytics Tools

Hadoop

PostgreSQL

Apache Cassandra

Elasticsearch

Wrapping Up

Baidu unveils open source edge computing platform and AI boards

Hyundai joins the Linux Foundation to embrace AGL’s open source connected car technologies

Power BI solution templates moving to open source

From Source to Analytics: A Farm-to-Table Approach to Data Governance

Tendu Yogurtcu

Editor’s note: This article on data governance written by Syncsort’s Tendü Yoğurtçu was originally published in Information Management.

Recent Posts

Categories

Archives