data Archives - Business Intelligence Info

Tag Archives: data

MIT takes down 80 Million Tiny Images data set due to racist and offensive content

July 2, 2020 Big Data

Creators of the 80 Million Tiny Images data set from MIT and NYU took the collection offline this week, apologized, and asked other researchers to refrain from using the data set and delete any existing copies. The news was shared Monday in a letter by MIT professors Bill Freeman and Antonio Torralba and NYU professor Rob Fergus published on the MIT CSAIL website.

Introduced in 2006 and containing photos scraped from internet search engines, 80 Million Tiny Images was recently found to contain a range of racist, sexist, and otherwise offensive labels such as nearly 2,000 images labeled with the N-word, and labels like “rape suspect” and “child molester.” The data set also contained pornographic content like non-consensual photos taken up women’s skirts. Creators of the 79.3 million-image data set said it was too large and its 32 x 32 images too small, making visual inspection of the data set’s complete contents difficult. According to Google Scholar, 80 Million Tiny Images has been cited more 1,700 times.

MIT takes down 80 Million Tiny Images data set due to racist and offensive content

Above: Offensive labels found in the 80 Million Tiny Images data set

“Biases, offensive and prejudicial images, and derogatory terminology alienates an important part of our community — precisely those that we are making efforts to include,” the professors wrote in a joint letter. “It also contributes to harmful biases in AI systems trained on such data. Additionally, the presence of such prejudicial images hurts efforts to foster a culture of inclusivity in the computer vision community. This is extremely unfortunate and runs counter to the values that we strive to uphold.”

The trio of professors say the data set’s shortcoming were brought to their attention by an analysis and audit published late last month (PDF) by University of Dublin Ph.D. student Abeba Birhane and Carnegie Mellon University Ph.D. student Vinay Prabhu. The authors say their assessment is the first known critique of 80 Million Tiny Images.

VB Transform 2020 Online – July 15-17. Join leading AI executives: Register for the free livestream.

Both the paper authors and the 80 Million Tiny Images creators say part of the problem comes from automated data collection and nouns from the WordNet data set for semantic hierarchy. Before the data set was taken offline, the coauthors suggested the creators of 80 Million Tiny Images do like ImageNet creators did and assess labels used in the people category of the data set. The paper finds that large-scale image data sets erode privacy and can have a disproportionately negative impact on women, racial and ethnic minorities, and communities at the margin of society.

Birhane and Prabhu assert that the computer vision community must begin having more conversations about the ethical use of large-scale image data sets now in part due to the growing availability of image-scraping tools and reverse image search technology. Citing previous work like the Excavating AI analysis of ImageNet, the analysis of large-scale image data sets shows that it’s not just a matter of data, but a matter of a culture in academia and industry that finds it acceptable to create large-scale data sets without the consent of participants “under the guise of anonymization.”

“We posit that the deeper problems are rooted in the wider structural traditions, incentives, and discourse of a field that treats ethical issues as an afterthought. A field where in the wild is often a euphemism for without consent. We are up against a system that has veritably mastered ethics shopping, ethics bluewashing, ethics lobbying, ethics dumping, and ethics shirking,” the paper states.

To create more ethical large-scale image data sets, Birhane and Prabhu suggest:

Blur the faces of people in data sets
Do not use Creative Commons licensed material
Collect imagery with clear consent from data set participants
Include a data set audit card with large-scale image data sets, akin to the model cards Google AI uses and the datasheets for data sets Microsoft Research proposed

The work incorporates Birhane’s previous work on relational ethics, which suggests that the creators of machine learning systems should begin their work by speaking with the people most affected by machine learning systems, and that concepts of bias, fairness, and justice are moving targets.

ImageNet was introduced at CVPR in 2009 and is widely considered important to the advancement of computer vision and machine learning. Whereas previously some of the largest data sets could be counted in the tens of thousands, ImageNet contains more than 14 million images. The ImageNet Large Scale Visual Recognition Challenge ran from 2010 to 2017 and led to the launch of a variety of startups like Clarifai and MetaMind, a company Salesforce acquired in 2017. According to Google Scholar, ImageNet has been cited nearly 17,000 times.

As part of a series of changes detailed in December 2019, ImageNet creators including lead author Jia Deng and Dr. Fei-Fei Li found that 1,593 of the 2,832 people categories in the data set potentially contain offensive labels, which they said they plan to remove.

“We indeed celebrate ImageNet’s achievement and recognize the creators’ efforts to grapple with some ethical questions. Nonetheless, ImageNet as well as other large image datasets remain troublesome,” the Birhane and Prabhu paper reads.

Let’s block ads! (Why?)

Big Data – VentureBeat

Mind the Gap: 3 Ways to Fill Your Data Management Talent Shortage

June 29, 2020 TIBCO Spotfire

Reading Time: 3 minutes

As any London Underground rider can attest, “Mind the Gap” is a message reinforced many times on every trip. Yet when it comes to filling today’s pervasive data management talent shortage, minding the gap often seems like a journey without a destination.

But let’s face it, data management is a challenging arena covering a diverse data landscape that spans myriad technologies, multiple disciplines, and every business domain. The phrase, “It takes a village,” comes to mind.

Further, these challenges are growing exponentially as massive data trends, such as the ten I identified in a recent blog, combine to make data management more complex and difficult than ever. This puts a premium on finding and retaining the best data management talent, including data architects, data engineers, database administrators, data stewards, and so on.

As a provider of data management software solutions, we see how much our customers struggle to attract and retain the right data management talent. While recognizing there is no one size fits all approach, this blog seeks to help our customers “mind their ‘data management talent’ gap” in three important ways.

1. Radically simplify your complex data management processes

Data management solutions have evolved over many years resulting in a hodgepodge of sometimes adjacent/sometimes overlapping tools. These silos of data management functionality along with aging, inefficient processes conspire to consume extra resources without providing extra benefit. But what if you could free up a large share of those resources by radically simplifying your data management processes?

Take your data integration process for example. It is significantly simpler and faster to integrate data with TIBCO^® Data Virtualization than it is when trying to make traditional physical integration with all of its complex mappings, ETLs, staging schemas, and destination schemas work. In fact, data engineering staffing savings of 40 percent are typical.

Rather than using three or more distinct tools to support metadata management, catalog services, and data governance processes, what if you could use one product to do them all as you can with TIBCO Cloud^™ Metadata? This lightens your data management workload by reducing set up, administration, and execution efforts, while also speeding up new employee learning curves.

Rather than using three or more distinct tools to support metadata management, catalog services, and data governance processes, what if you could use one product to do them all? Click To Tweet

2. Team with your business to add value

When it comes to today’s complex data landscape, IT doesn’t have the resources to go it alone. Your business staff can add value in a number of ways. One way is by developing citizen data engineering talent on the business side to complement your overworked IT data engineers.

Let’s drill down on this citizen data engineering opportunity. As mentioned above, your IT data engineers can use TIBCO Data Virtualization to efficiently provision hundreds of reusable business views. From these views, your business analysts can quickly find the data they need, improving analytics productivity. But when just a bit more data engineering is required to further prepare and refine the business view for a unique need, you don’t need to add that to your IT data engineering queue. Instead, your business analysts can use specialized, citizen data engineer targeted functionality to finish the task.

Another way your business can address the data management talent shortage is by enlisting business domain experts, since who knows more about your data than your business users who create and use it every day. For example, at Panera Bread, store managers who know their store layouts better than anyone, use TIBCO EBX^™ to maintain their individual store configurations in the master data repository. Similarly, corporate chefs maintain menu, recipe, and ingredient master data.

With this new business-IT partnership, both sides work together to assess your needs and skills. They then collaborate to close your data management skills gap by proactively maximizing the value add each side can contribute.

3. Embed intelligence

Another way to address your data management talent shortage is to augment your people with artificial intelligence and machine learning (AI/ML). IDC analyst Stewart Bond outlines multiple such augmentation opportunities in his March 2020 report, The AI Automation Framework Applied to Data Integration and Data Intelligence Software.

By adopting modern data management solutions, such as TIBCO Unify, you can take advantage of embedded AI/ML capabilities to automate multiple manual activities across metadata discovery, data quality, model design, code generation, query optimization, resource allocation, and more.

Closing your data management talent gap

When it comes to managing your data if you are like most organizations, the skilled resources you require far exceed your current resources, which puts your ability to be a data-driven business at risk.

But you don’t have to settle for the status quo when you can help fill your data management skills gap with the three approaches outlined above. For more on how to effectively and intelligently manage your data and enable your teams with better access, trust, and control for all your data, check out these helpful resources.

Let’s block ads! (Why?)

The TIBCO Blog

Facebook claims wav2vec 2.0 tops speech recognition performance with 10 minutes of labeled data

June 23, 2020 Big Data

In a paper published on the preprint server Arxiv.org, researchers at Facebook describe wav2vec 2.0, an improved framework for self-supervised speech recognition. They claim it demonstrates for the first time that learning representations from speech, followed by fine-tuning on transcribed speech, can outperform the best semi-supervised methods while being conceptually simpler, achieving state-of-the-art results using just 10 minutes of labeled data and pretraining on 53,000 hours of unlabeled data.

AI models benefit from large amounts of labeled data — it’s how they learn to infer patterns and make predictions. However, as the coauthors of the paper note, labeled data is generally harder to come by than unlabeled data. Current speech recognition systems require thousands of hours of transcribed speech to reach acceptable performance, which isn’t available for the majority of the nearly 7,000 languages spoken worldwide. Facebook’s original wav2vec and other systems attempt to sidestep this with self-supervision, which generates labels automatically from data. But they’ve fallen short in terms of performance compared with semi-supervised methods that combine a small amount of labeled data with a large amount of unlabeled data during training.

Wav2vec 2.0 ostensibly closes the gap with an encoder module that takes raw audio and outputs speech representations, which are fed to a Transformer that ensures the representations capture whole-audio-sequence information. Created by Google researchers in 2017, the Transformer network architecture was initially intended as a way to improve machine translation. To this end, it uses attention functions instead of a recurrent neural network to predict what comes next in a sequence. This characteristic enables wav2vec 2.0 to build context representations over continuous speech representations and record statistical dependencies over audio sequences end-to-end.

Facebook claims wav2vec 2.0 tops speech recognition performance with 10 minutes of labeled data

Above: A diagram illustrating wav2vec 2.0’s architecture.

To pretrain wav2vec 2.0, the researchers masked portions of the speech representations (approximately 49% of all time steps with a mean span length of 299 milliseconds) and tasked the system with predicting them correctly. Then, to fine-tune it for speech recognition, they added a projection on top of wav2vec 2.0 representing vocabulary in the form of tokens for characters and word boundaries (e.g., word spaces of written English) before performing additional masking during training.

VB Transform 2020 Online – July 15-17. Join leading AI executives: Register for the free livestream.

The coauthors trained wav2vec 2.0 on several unlabeled and labeled data sources for up to 5.2 days at a time on 128 Nvidia V100 graphics cards to evaluate the system’s performance. Fine-tuning took place on between eight and 24 graphics cards.

According to the team, the largest trained wav2vec 2.0 model — which was fine-tuned on only 10 minutes of labeled data (48 recordings with an average length of 12.5 seconds) — achieved a word error rate of 5.7 on the open source Librispeech corpus. (Here, “word error rate” refers to the number of errors divided by total words.) On a 100-hour subset of Librispeech, the same model managed a word error rate of 2.3 — 45% lower than the previous state of the art trained with 100 times less labeled data — and 1.9 when fine-tuned on even more data, a result competitive with top semi-supervised methods that rely on more sophisticated architectures.

“[This] demonstrates that ultra-low resource speech recognition is possible with self-supervised learning on unlabeled data,” the researchers wrote. “We have shown that speech recognition models can be built with very small amounts of annotated data at very good accuracy. We hope our work will make speech recognition technology more broadly available to many more languages and dialects.”

Facebook used the original wav2vec to provide better audio data representations for keyword spotting and acoustic event detection and to improve its systems that proactively identify posts in violation of its community guidelines. It’s likely wav2vec 2.0 will be applied to the same tasks; beyond this, the company says it plans to make the models and code available as an extension to its fairseq modeling toolkit.

Let’s block ads! (Why?)

Big Data – VentureBeat

Microsoft acquires ADRM Software to beef up Azure with industry data models

June 18, 2020 Big Data

Microsoft today announced it has acquired ADRM Software, which provides large-scale industry data models companies use as “information blueprints.” The tech giant says it plans to combine these models with storage and compute from Azure to support the creation of data lakes where information from multiples lines of business is collated together at scale.

Industry data models enable companies to capture and define business concepts, refine and integrate processes, and build interoperability into their ecosystems. But despite the fact that they’re foundational for things like data quality, lineage, and governance, such models are often implemented on a fragmented and siloed basis.

That’s where ADRM comes in. It develops industry-specific enterprise, data warehouse, business area, and solution data models spanning 10 industry groups and 65 lines of business. For each organization, the company creates an enterprise data model containing a subset of entities from business area models for that organization’s industry. A slice of between 21 and 30 comprehensive business area models provides detail and expanded scope for each of the areas, offering a view of data for that business or specific subject area.

ADRM says its data models help customers in 18 firms around the world plan, architect, design, govern, report, and gather business intelligence and analytics. Complementing those models, the company provides industry best practices content and industry-specific naming and definitions.

VB Transform 2020 Online – July 15-17. Join leading AI executives: Register for the free livestream.

“Data and AI are the foundation of modern technological innovation, yet businesses today struggle to unlock the full value data has to offer as fragmented data estates hinder digital transformation,” Microsoft Azure CVP Ravi Krishnaswamy wrote in a press release. “Without a comprehensive and integrated view of their data, companies are at a competitive disadvantage, which hinders digital adoption and data-driven innovation … [Now] these capabilities will be delivered at scale, enabling our customers to accelerate digital progress and reduce risk in a variety of major initiatives.”

With the acquisition, ADRM’s team joined Microsoft’s Azure global engineering arm. Financial terms of the deal weren’t disclosed.

“As we worked closely with the Azure global engineering team during the past year, we became very enthusiastic about the tremendous additional value and acceleration we believe can be unlocked for large enterprises across many industries by combining ADRM’s comprehensive industry data models with the limitless storage and compute from Azure to power the next generation of intelligent data lakes — data lakes which aren’t just vast reservoirs of data but which are also metadata-rich foundations [that] can supercharge modern data warehouses, next-level analytics, and AI and ML,” ADRM wrote in a blog post.

Let’s block ads! (Why?)

Big Data – VentureBeat

Connecting Data is a Team Sport

June 11, 2020 TIBCO Spotfire

Reading Time: 3 minutes

I recently teamed up with Austrian customer Raiffeisen Bank, Dutch partner Connected Data Group, and German partner QuinScape to deliver a webinar entitled “Next-Generation Data Virtualization Has Arrived.” Combining our experiences and insights, we delivered an accessible roadmap of how TIBCO Data Virtualization can help enterprises connect their data to drive greater business value.

After the webinar, I spoke with Connected Data Group co-founder Erik Fransen, whom I first met at a data virtualization event in 2015. We talked about Erik’s latest insights on the European data and analytics market as well as his fast-growing business. Connected Data Group helps clients become more data-driven and was co-founded with Antoine Stelma. The pair realized early on that they would have to transition their clients from traditional warehousing and replication-based data architectures to more modern and agile solutions. Below, I share some highlights from our conversation.

What is driving the European data and analytics market today?

“Our clients understand that data is core to their business success. Data helps them innovate faster, operate more efficiently, and provide more valuable customer experiences. Their challenge is unlocking that data, connecting it via more modern adaptable data architecture, making it more consistent and consumable, and doing all of this in an agile, cost-effective way.

That’s where the Connected Data group comes in. We connect data with people and people with data. And together with our network of partners, we help our clients maximize the business value of their data.”

How do you connect people and data to realize data’s business value?

“We start by helping our clients identify their biggest data-driven business opportunities. With these objectives in mind, we then analyze their data management architecture and propose ways to modernize it.

This typically means new investments in TIBCO offerings such as TIBCO Data Virtualization and TIBCO EBX. TIBCO Data Virtualization makes their data architecture more flexible and responsive. TIBCO EBX helps organize their most important data assets, for example, their customer and product data. And when used together in their data architecture, they make our clients’ data available to the business in a simple, quick, consistent, and predictable way.

We also help our clients skill up their people to maximize the business value and gain insight from their modernized data architecture. e deliver an excellent training program that helps educate business and IT leaders, analysts, data scientists, data engineers, data modelers, and others.”

Tell me more about your network of partners?

“We are business value-driven, data management, and data analytics specialists with deep experience in business requirements, data modeling, data warehouse development, and automation, data architecture, analytics, and data science. But we alone cannot cover the entire data and analytics spectrum, nor the entire European market.

So we team with other specialist consultancies such as KVL and Systemation in The Netherlands, QuinScape and Statsoft Europe in Germany, Apgar Consulting in France, and Yael Group in Israel. And of course, we work with a selected set of technology providers including TIBCO. Our Connected Data Academy provides a broad range of Data & Analytics education programs focused on concept and skill development, IT tool training, and made-to-measure in-company programs. We partner with major training institutes in the Netherlands and we have a deep partnership with Genesee Academy in the USA focused on Data Vault Modeling Certifications worldwide.”

So the “Group” term in your Connected Data Group brand is intentional.

“Yes absolutely. Our clients’ data opportunities and challenges are massive. But when we connect our clients, our team, and our partner network, there is no challenge we cannot successfully address. Plus teaming with great people is a lot more fun and satisfying way to do business.”

Speaking of partners, can you tell us about your relationship with TIBCO?

“Our relationship began in 2014 driven by our client’s need for data virtualization. We expanded our partnership in 2018 when TIBCO added TIBCO EBX. We recently added the newest offering in TIBCO’s Unify data management portfolio, TIBCO Cloud Metadata, which provides a one-stop-shop for metadata management, data governance, and data catalog. We love the fact TIBCO now provides our clients with the complete, next-generation data management solution they require.

Once we put their consistent, complete, connected data architecture in place, we then help our clients gain insight from that data using additional solutions such as TIBCO Spotfire, TIBCO Streaming, and TIBCO Data Science.”

Can you give us a couple of examples of customers who have benefited from your TIBCO partnership and what kind of impact our partnership is delivering?

“We have teamed with TIBCO at over a dozen of Europe’s biggest healthcare, automotive, financial services, and retail companies. European candy maker Cloetta is one example. Raiffeisen Bank who spoke at the webinar is another. The impacts are significant, including top-line growth, lower costs, better compliance, and more.”

Erik, I enjoyed our conversation, and thank you for helping communicate why connecting data is a team sport. If someone would like to connect with you directly, how would one do that?

“I would enjoy speaking with your blog readers about their data-driven business opportunities, challenges, and how Connected Data Group and our partner network might help. Just reach out via LinkedIn. I look forward to the conversations.”

Let’s block ads! (Why?)

The TIBCO Blog

Announcing RapidMiner 9.7 — Making data science a team sport

June 10, 2020 BI News and Info

Today, we’re excited to announce the next step in RapidMiner’s growth—RapidMiner 9.7. With the launch of 9.7, we’re continuing our work to put people at the center of the AI journey by fostering better collaboration, all while improving the oversight and management that’s critical to creating successful machine learning projects.

Here’s a look at the biggest changes in 9.7.

RapidMiner Server is now RapidMiner AI Hub

As part of the 9.7 launch, we’ve rebranded RapidMiner Server as RapidMiner AI Hub. This new name communicates the AI Hub’s function of connecting machine learning to the people, processes, and systems that make it all work. Integrating all of this through the AI Hub will help tear down the silos that can stymie machine learning projects and help them succeed. You can read more about the name change here.

AI Hub includes a new projects framework

The change in name to AI Hub comes with the addition of a projects framework that allows for unprecedented central collaboration and governance of AI projects.

Projects help teams convert ideas into models easily and iteratively, so they can deliver value for the business. People with different tooling preferences and skillsets can collaborate in a single platform that supports automated, visual, and code-based authoring styles. Everyone involved in machine learning projects can work together in the same place, easing collaboration friction.

Projects offers precise version control

In addition to allowing everyone to work in the same place, projects also delivers a rare combination of agility and traceability by integrating fine-grained version control based on Git standards. This allows the tracking of all changes—including who executed them—which creates clear and distinct model lineages in case you need to backtrack or merge project code. What’s more, “Snapshots” functionality lets you easily roll back to earlier versions of projects, supporting iterative and agile AI development.

Supports large-scale collaboration

To support collaboration at large scale, we’ve introduced enterprise-grade identity & access management to RapidMiner’s products. This includes an enterprise-grade single sign-on experience, creating a seamless experience across the RapidMiner platform. It also includes precision access control that facilitates fine-grained user, group, and role management.

All of these new features make collaboration between coders, RapidMiner Studio users, and RapidMiner Go users even easier. If you’re interested in the nitty gritty of the release, you can check out the What’s New page for all the details.

Let’s block ads! (Why?)

RapidMiner

The Power of Data to Do Good #TIBCO4Good

June 7, 2020 TIBCO Spotfire

Reading Time: 2 minutes

Data is the transformative fuel that is changing the global economy. Partnering with TIBCO has allowed data-savvy businesses to lead in their respective industries and transform their strategies, cultures, and capabilities. But it’s not just businesses that are facing complex data challenges. Many of the world’s most pressing societal, educational, and scientific issues also depend on the ability to harness the power of data in order to gain insights, make smarter decisions, and implement the systems, applications, and processes that create change on a human scale.

TIBCO4Good applies the same technologies, strategies, and leadership deployed for business to the most pressing societal issues we face across the globe. TIBCO4Good is a way to come together as a community and harness data “for good.” Because nearly every complex challenge our planet and societies are facing is rooted in vast streams and reservoirs of data and confounded by the complexity of difficult-to-make decisions.

A great example of this initiative at work can be seen in the recent efforts by TIBCO’s data science team, led by Michael O’Connell, to create a Visual Analytics Hub to bring a data-centric approach to visualizing the deeper implications of the COVID-19 pandemic and build a fact-based understanding of reproduction, growth, and healthcare resources.

TIBCO + Conan Swag for Partners in Health

On a lighter note, as a part of #TIBCO4GOOD, TIBCO has kicked off a new project, inspired by the “Zoombombing” of our newest team member, Conan O’Brien. Conan crashed a recent TIBCO conference call and asked our CEO Dan Streetman, “Could I at least get a TIBCO sweatshirt out of this? We all know you have like fifty in the trunk of your car.”

Well, Conan—and everyone else eying Dan’s snazzy TIBCO jacket—now you too can purchase “TIBCO + Conan” branded apparel at www.tibcoswag.com. TIBCO will donate all proceeds from items purchased, up to a total of $ 50,000 to Partners in Health (PIH), a social justice organization supporting emergency response during the COVID-19 crisis, as highlighted by Conan.

Anyone around the world can shop the site and help support Partners in Health (PIH) as they mobilize high-impact programs to control the spread of COVID-19 and ensure that the most vulnerable receive access to dignified care. Learn more about how PIH is helping with COVID-19 emergency response here.

We’ll be back soon with more updates on new and upcoming TIBCO4Good Initiatives and how you can get involved. For more information on the program and the work that we’re doing, head to https://www.tibco.com/tibco4good.

Let’s block ads! (Why?)

The TIBCO Blog

The Need for Speed: Faster Data Access as Competitive Edge

June 5, 2020 Sisense

The cloud isn’t the future; it’s right now. In the Clouds is where we explore the ways cloud-native architecture, cloud data storage, and cloud analytics are changing key industries and business practices, with anecdotes from experts, how-to’s, and more to help your company excel in the cloud era.

The world of data is constantly changing and speeding up every day. Companies are storing more types of data from applications as well as the Internet of Things. This data flows into cloud-native warehouses where data teams manipulate it, allowing analysts to derive vital insights from it, and product teams embed those insights into products. Data is the bedrock on which the future of business is being built.

As the data that these businesses need to thrive continues to grow and its pace of change accelerates, it’s never been more important for employees at all levels of an organization to have fast access to actionable data in order to make strategic, operational, and tactical decisions. It’s both basic table stakes for success in the “new normal,” as well as a defining edge that companies can use to stay ahead of the competition.

packages CTA banners Cloud Data Teams The Need for Speed: Faster Data Access as Competitive Edge

Saving lives in real-time

Easier access to fast-updating datasets isn’t just about making better decisions or powering the next killer app. It also can also save lives and change the way the world works.

“There are a wide range of scenarios where having super-fast access to real-time data can make a huge difference,” said Christelle Scharff, a professor and computer scientist based at Pace University in New York. “Fast access to data captured by video surveillance systems, for example, can improve security… It’s also the driving force behind autonomous cars. Our biggest industrial firms can use it for preventative maintenance — saving potentially millions of dollars. And almost all organizations can use it to avoid potential threats from security breaches and malware attacks.”

The success of COVID-tracing efforts will depend on fast access to multiple data sources.

George Thiruvathukal, professor of computer science at Loyola University

During our current pandemic, access to real-time data can also save lives. “Health officials are investigating how contact-tracing apps can help manage the ‘reopening’ after we begin to reopen the country after the COVID-19 lockdown,” said George Thiruvathukal, professor of computer science at Loyola University in Chicago. “The success of this will depend on fast access to multiple data sources.”

There’s an impact on customer expectations too. According to recent research from IDC, consumers are embracing personalized real-time engagements and resetting their expectations for data delivery. As their digital world overlaps with their physical realities, they expect to access products and services wherever they are, over whatever connection they have, and on any device. They want data in the moment, on the go, and personalized. As a result, IDC predicts that nearly 30% of the global datasphere will be real-time by 2025.

Pressure on infrastructure builds

As enterprises demand data infrastructures that can meet this growth in real-time data — and ultimately assist with their product differentiation strategy — the pressure put on product teams is huge.

Product teams are already having to manage the growing complexities that come with modern data environments.

Chandana Gopal, Business Analytics Research Director, IDC

“Product teams are already having to manage the growing complexities that come with modern data environments,” said Chandana Gopal, research director for business analytics at IDC. “Not only do they have to deal with data that is distributed across on-premises, hybrid, and multi-cloud environments, but they have to contend with structured, semi-structured, and unstructured data types. Multiple technologies to manage data at rest and in motion have compounded the challenge of managing data and making it accessible to decision-makers in the right time, in the right format, and in the right context.”

Managing cloud data is a key challenge for data and product teams who are tasked with connecting to a wide array of datasets stored in cloud-native warehouses and other locations. In a large-enough company, there can even be multiple clouds being operated by different divisions and teams. BI and analytics providers have had to design their platforms to serve up fast insights no matter where the data being analyzed resides, even partnering with third-party companies to make sure that their platforms can handle data from oft-used services like AWS, Google Cloud, and others.

Which makes sense, as customers searching for an analytics solution are also often grappling with their recently-purchased or possible cloud options:

“When customers come to us for their BI and analytics needs, in the same sentence they’re often telling us that they’re considering their cloud options,” said Erin Winkler-McCue, Lead for Strategic Partnerships & Special Projects at Sisense. “These conversations are no longer siloed. Customers want to know that our platform will work seamlessly with their chosen cloud vendor, even if that just means something as basic-sounding as making sure queries between the vendor and Sisense are optimized.”

The Need for Speed: Faster Data Access as Competitive Edge

The challenges these teams face become even more daunting when one looks towards the future, as new technologies like the internet of things, machine learning, 5G, and augmented reality will add a new level of demand. Forbes Insights data shows that in order to benefit from emerging technologies like these, 92% of CIOs and CTOs say their business will require faster download and response time in the near future. What’s concerning is that, despite recognizing this, just 1% of data center engineers believe their data centers are updated ahead of current needs.

linux TCO CTA banner for white paper 770x250 The Need for Speed: Faster Data Access as Competitive Edge

Competing priorities within companies

All of this results in a lot of friction within data-driven organizations. “Multiple technologies are required for managing, integrating, and controlling the flow and consumption of data from the edge to the cloud and all points in between. That’s without mentioning outdated metadata—the data about data that provides data intelligence,” said Gopal.

An upcoming skills gap might compound the problem: According to the Forbes Insights research, 37% of engineers say they will likely retire in the next 10 years.

Adding to this hurdle is the fact that some firms are led by executives that don’t understand or champion the importance of having contextual and timely data embedded into applications. Recent research by Exasol has found that less than half of decision-makers believe that those working in senior management (40%) or mid-management roles (32%) are very effectively informed of their organization’s data strategy.

Creating a path to success

Gopal believes that future success requires that data teams take a structured approach focused on people, processes, and technology in order to make data available to all.

“Data teams should identify short- and long-term data and analytics use cases that will demonstrate business value with input from stakeholders at all levels—both business and IT,” she said. “They should also identify data-related assets that will be required for the project and be realistic about time constraints. They should then look to deliver measurable value with short term projects to build business cases for more expensive or longer projects.”

Teams should look to deliver measurable value with short term projects to build business cases for more expensive or longer projects.”

Chandana Gopal, Business Analytics Research Director, IDC

From a technology perspective, the introduction of new technologies, such as 5G-enabled edge computing, will have an impact on IT staffing. According to the Forbes Insight report, almost three-quarters (74%) of C-suite executives believe staffing will be reduced or handled by external cloud or edge service providers. The ability to implement new technologies like these in the data center will be a competitive differentiator, as will better security (according to 43% of respondents), and bandwidth (according to 27%).

Ultimately, it’s clear that organizations need to act quickly if they want to succeed. “Continuous efforts to update the data center will be integral to business success,” states the Forbes Insight report. “Partnering with external third parties is a central part of the data center journey in the age of hyper-connectivity.”

These collaborations are happening between companies and their cloud providers and between platforms like Sisense and companies like Amazon and Google:

“Partnerships with companies like AWS, Snowflake, Microsoft, and Google are only becoming more important as the modern data landscape evolves,” said Erin Winkler-McCue, Lead for Strategic Partnerships & Special Projects at Sisense. “We feel like every customer is either already in the cloud or it’s only a matter of time until they contemplate setting up a hybrid- or full-cloud model.”

Gopal agrees that adopting new technology through new partnerships is key: “A new class of intelligent data operations platforms are emerging that can reduce friction, improve efficiencies with automation, provide flexibility and openness with policy and metadata-driven processes that can accommodate the diversity and distribution of data in modern environments,” she said. Equipped with these, product teams will be much better prepared for a new and exciting future.

packages CTA banners Product Teams The Need for Speed: Faster Data Access as Competitive Edge

Lindsay James is a journalist and copywriter with over 20 years’ experience writing for enterprise business audiences. She has had the privilege of creating all sorts of copy for some of the world’s biggest companies and is a regular contributor to The Record, Compass, and IT Pro.

Let’s block ads! (Why?)

Blog – Sisense

Facebook creates shopping data sets to make more humanlike chatbots

June 4, 2020 Big Data

Facebook researchers this week introduced Situated Interactive MultiModal Conversations (SIMMC), a novel research direction aimed at training AI chatbots that take actions like showing an object and explaining what it’s made of in response to images, memories of previous interactions, and individual requests. In a technical paper, they detail new data sets designed for this purpose containing around 13,000 human-to-human dialogs across two domains — furniture and fashion — along with several tasks framed as objective evaluation protocols.

Facebook would appear to be working toward an assistant capable of processing data a user and the assistant co-observe, and then outputting replies beyond just plain text based on this data. The hope is that this assistant emulates human chat partners by responding to images, messages, and messages about images as naturally as a person might. For example, given the prompt “I want to buy some chairs — show me brown ones and tell me about the materials,” the assistant might reply with an image of brown chairs and the text “How do you like these? They have a solid brown color with a foam fitting.”

SIMMC supports the development of such an assistant with the aforementioned data sets and new technical tasks, which address task-oriented dialogs encompassing multimodal user contexts in the form of a co-observed image or a virtual reality environment. The tasks get updated dynamically based on the dialog flow and the assistant’s actions.

VB Transform 2020 Online – July 15-17. Join leading AI executives: Register for the free livestream.

In SIMMC-Furniture, the furniture-focused data set, a user interacts with a conversational assistant to get recommendations for items like couches and side tables. To create it, the Facebook researchers built a virtual environment within Unity where volunteers were connected randomly with humans posing as a virtual, full-featured assistant. The users could ask to see a particular type of furniture, and the assistant could filter a catalog of 3D Wayfair assets by price, color, material, and more while navigating through the filtered results to share their view in focused (i.e., zoomed-in) or carousel (three slots containing three different items) presentations.

Meanwhile, in the SIMMC-Fashion data set, users asked humans posing as virtual assistants for jacket, dress, and other clothing and accessory suggestions. Within the same Unity environment, assistants could sort by price, brand, color, and more as the users browsed and explored options informed by preferences and visual scenes, memories, and assistant-recommended items.

For both corpora, the researchers noted which items appeared in each view. They also developed an ontology to capture the multimodal interactions within dialog flows and provide semantics for assistant and user utterances, consisting of four primary components: objects, activities (e.g., “add to cart”), attributes (“brands”), and dialog acts (“ask”). To complement this, they derived a labeling language for annotation that allowed for the representation of dialog exchanges, such that the SIMMC annotations capture the relations of objects with their corresponding dialog annotations.

Building on these data sets, the Facebook researchers built a basic assistant consisting of four components: an utterance and history encoder, multimodal fusion, an action predictor, and a response generator.

The utterance and history encoder creates encodings (numerical representations) from user replies and the dialog history.
The multimodal fusion step combines information from the text and multimodal context into a mathematical object called a tensor.
The action predictor predicts actions to be taken by the assistant by transforming the tensor into another object called a vector, and then by predicting an API the assistant might need to call.
The response generator generates an assistant response that’s semantically relevant to users’ requests. For example, given the request “Show me black couches less than $ 500,” the generator might reply “Here are some” or “Sorry, we do not have any black couches cheaper than $ 500” based on available inventory.

After training the models on both SIMMC-Fashion and SIMMC-Furniture, the researchers found that they outperformed two baseline AI systems across a number of metrics. Despite not leveraging the fine-grained annotations, the best-performing action predictor chose the right API 79.6% of the time for the SIMMC-Furniture corpus and 85.1% of the time for SIMCC-Fashion. Facebook says that it will publicly release the data, annotations, code, and models in the future.

The research follows Facebook’s detailing of the AI systems behind its shopping experiences, which continue to evolve across Instagram, WhatsApp, and Facebook proper. The company says its goal is to one day combine its approaches into a system that can serve up product recommendations on the fly, matched to individual tastes and styles — a sort of hardware-free take on the recently discontinued Echo Look, Amazon’s AI-powered camera that told customers how their outfits looked and kept track of their wardrobe.

Let’s block ads! (Why?)

Big Data – VentureBeat

Understanding Some of the Key Facets of Your Master Data

June 2, 2020 TIBCO Spotfire

Reading Time: 2 minutes

Master data is at the center of every critical business activity today. Operations rely on master data for a consistent flow of information. Analytics relies on master data for accurate dimensions and hierarchies to find trends and make predictions. And compliance efforts rely on master data to ensure they are driven by trustworthy data. Despite this importance, there are several key facets that many organizations overlook in their master data management (MDM) programs. In this blog and the following series, we will examine these facets to help your organization improve its overall MDM strategy and therefore improve its business performance.

Let’s start off with a few key definitions. Master data is defined as data shared across business teams and IT applications, that defines key information of a company such as products, organization, customers, suppliers, financial hierarchies, assets, locations, or reference codes.

Master data is a complex, multifaceted object that:

Has associated definitions, attributes, connections, metadata, and taxonomies
Has relationships with other master data (and itself)
Can live in the past, present, and future
Needs to be adapted to multiple contexts
Has various owners across the organization
Is based on a specific life cycle
And integrates with applications using many patterns

A Master Data Management platform will help define, manage, share, and govern all those shared data in a central solution where business users can collaborate and consume it in a secure, consistent way.

Not only is master data complex, but all your master data have different characteristics

So, before you launch a project or shop for MDM software, we think it is important to understand those various facets. It will have an impact not only on your vendor selection but also in the way you will conduct the initiative and ultimately get adoption—and success—of MDM across the enterprise.

Discussions around MDM cover some but not all of the characteristics that one should look for in an MDM platform. In this series, we’ll focus on the gaps in those discussions, the areas that can be the tipping point between project success or failure.

Keep a lookout for the rest of this blog series where we will examine the following factors that we believe are overlooked by many when evaluating an MDM platform:

Modeling: Documenting and defining your master data
Relationships: Managing the intra- and inter-domain relationships in your data
Time: Going beyond simple versioning
Points of entry: How does you master data get in?
Business context: Sharing master data across groups, divisions, and functional areas
Governance/Lifecycle: Managing different scopes
Integration: How does your master data get out?

And for more on why your organization needs master data management to thrive, check out this infographic.

Let’s block ads! (Why?)

The TIBCO Blog

Tag Archives: data

MIT takes down 80 Million Tiny Images data set due to racist and offensive content

Mind the Gap: 3 Ways to Fill Your Data Management Talent Shortage

1. Radically simplify your complex data management processes

2. Team with your business to add value

3. Embed intelligence

Closing your data management talent gap

Facebook claims wav2vec 2.0 tops speech recognition performance with 10 minutes of labeled data

Microsoft acquires ADRM Software to beef up Azure with industry data models

Connecting Data is a Team Sport

Announcing RapidMiner 9.7 — Making data science a team sport

RapidMiner Server is now RapidMiner AI Hub

AI Hub includes a new projects framework

Projects offers precise version control

Supports large-scale collaboration

The Power of Data to Do Good #TIBCO4Good

TIBCO + Conan Swag for Partners in Health

The Need for Speed: Faster Data Access as Competitive Edge

Saving lives in real-time

Pressure on infrastructure builds

Competing priorities within companies

Creating a path to success

Facebook creates shopping data sets to make more humanlike chatbots

Understanding Some of the Key Facets of Your Master Data

Recent Posts

Categories

Archives