Category Archives: Sisense
Quantitative and Qualitative Data: A Vital Combination
We live in a world of data: There’s more of it than ever before, in a ceaselessly expanding array of forms and locations. Dealing with Data is your window into the ways organizations and data teams tackle the challenges of this new world to help their companies and their customers thrive.
Almost every modern organization is now a data-generating machine. As soon as a company’s systems are computerized, the data-generation engine starts up. When these systems connect with external groups — customers, subscribers, shareholders, stakeholders — even more data is generated, collected, and exchanged. And, as industrial, business, domestic, and personal Internet of Things devices become increasingly intelligent, they communicate with each other and share data to help calibrate performance and maximize efficiency. The result, as Sisense CEO Amir Orad wrote, is that every company is now a data company.
Smart use of your data can be the key to optimizing processes, identifying new opportunities, and gaining or keeping a competitive edge. But are you paying attention to all of your data? Do you have the means to handle every kind of data? And can you take full advantage of the insights it can reveal? Your answers will depend on whether you can gather and analyze both quantitative and qualitative data. Let’s consider the differences between the two, and why they’re both important to the success of data-driven organizations.
Digging into quantitative data
Most commonly, we think of data as numbers that show information such as sales figures, marketing data, payroll totals, financial statistics, and other data that can be counted and measured objectively.
This is quantitative data. It’s “hard,” structured data that answers questions such as “how many?” or “how often?” It’s in an organized format, usually rows and columns, and is stored as a relational database, in which there are relationships among these rows and columns.
All descriptive statistics can be calculated using quantitative data. It’s analyzed through numerical comparisons and statistical inferences and is reported through statistical analyses.
As quantitative data is always numeric, it’s relatively straightforward to put it in order, manage it, analyze it, visualize it, and do calculations with it. Spreadsheet software like Excel, Google Sheets, or traditional database management systems all mainly deal with quantitative data. These programs and systems are great at generating basic visualizations like graphs and charts from static data. The challenge comes when the data becomes huge and fast-changing.
Why is quantitative data important?
Quantitative data is often viewed as the bedrock of your business intelligence and analytics program because it can reveal valuable insights for your organization. These numbers show performance, efficiency, reach, market share, revenue in, and expenses out. This is fundamental information when it comes to understanding how your organization is doing.
Additionally, quantitative data forms the basis on which you can confidently infer, estimate, and project future performance, using techniques such as regression analysis, hypothesis testing, and Monte Carlo simulations.
These techniques allow you to:
- See trends and relationships among factors so you can identify operational areas that can be optimized
- Compare your data against hypotheses and assumptions to show how decisions might affect your organization
- Anticipate risk and uncertainty via mathematically modeling
Consequently, using quantitative data, you can make strategic and tactical decisions that will benefit your organization and drive growth.
What are the problems with quantitative data?
Despite its many uses, quantitative data presents two main challenges for a data-driven organization.
First, data isn’t created in a uniform, consistent format. It’s generated by a host of sources in different ways. To access it effectively, you need to organize it and clean it to avoid mistakes, oversights, and repetitions that could compromise the integrity of your data and, by extension, the accuracy of your insights.
The solution to this problem is to ensure that your BI and analytics platform can handle as many sources and forms of data as possible, both on-premises and on the cloud, so that you don’t neglect or miss any of it.
Second, and more challenging, is the fact that not all the information you generate and collect is structured, quantitative data. As data is produced from a growing array of sources, quantitative data may be as little as 20% of all the data available to most organizations. If you focus your efforts entirely on quantitative data, you’ll overlook a huge amount of valuable information, and your insights and decision-making could become distorted as a result. You need a solution that can access and analyze the other 80%: qualitative data.
Exploring qualitative data
Qualitative data is unstructured, meaning it’s not in a predefined format. In its raw state, it comes in a variety of forms: text, social media comments, phone call transcripts, various logs files, images, audio, video, and more.
Often, the information in qualitative data is a categorical variable. This means it describes the characteristics or qualities of data units, such as “what type,” “which category,” “who” (or “which persona”). These create a picture of the context and environment in which the data can help build an understanding of feelings, opinions, and intentions. That’s because qualitative data is concerned with understanding the perspective of customers, users, or stakeholders. This type of data is often collected through less rigid, measurable means than quantitative data. Examples include comments, recordings, interviews, focus group reports, and more, typically received in the natural language of the informants.
Qualitative data is much more subjective, “soft” information than quantitative data. However, advanced analytics can now identify and classify this information and transform it into findings that lead to game-changing insights for organizations.
As the sources of data continue to proliferate, an increasing proportion of it is unstructured and qualitative. It’s more complex, and it requires more storage. Traditional methods of gathering and organizing data can’t organize, filter, and analyze this kind of data effectively. Advanced technology and new approaches are needed. What seem at first to be very random, disparate forms of qualitative data require the capacity of data warehouses, data lakes, and NoSQL databases to store and manage them.
Qualitative data benefits: Unlocking understanding
Qualitative data can go where quantitative data can’t. For example, it’s the gateway to sentiment analysis — understanding how users, customers, and stakeholders think and feel, as well as what they do. Techniques that focus on qualitative data, such as content analysis and narrative analysis, enable you to:
- Analyze text to find the most common themes in open-ended data such as user feedback, interview transcripts, and surveys, so you can pinpoint the most important focus areas
- Better interpret feelings or perceptions about your organization, product, services, processes, or brand, to help identify what changes or innovations would be most effective
Having access to this information gives you a new dimension of insights on which to base decisions and determine the tactical and strategic direction of your organization. With qualitative data, you can understand intention as well as behavior, thereby making predictive analytics more accurate and giving you fuller insights. You can analyze and learn from the large volume of unstructured data to ensure that your data-driven decisions are as solid as possible.
For example, Skullcandy, the manufacturer of headphones, earbuds, and other audio and wireless products, uses predictive and sentiment analysis to understand customers better. This enables the company’s product development team to explore opportunities for reducing returns and warranty requests for new products before they’re even released and get reactions from customers about current products. Skullcandy uses these insights to hone its new products based on what customers say they like and dislike, so that it can offer more attractive products to its target market.
Getting the most from qualitative data
Qualitative data eludes the rigid categorization and the storage limitations of traditional databases. It provides the raw material for information that is more varied and harder to organize than structured, qualitative data. Making sense of and deriving patterns from it calls for newer, more advanced technology.
Natural language processing (NLP), involving machine learning, is how your BI and analytics platform can understand the meaning of unstructured data such as emails, comments, feedback, and instant messages. It enables search-driven analytics that allow you to process complex NLP, because you can ask the system questions using natural, everyday language. Plus, you can receive visualizations and embed insights into external apps, even if what you’re asking for is relatively complex from an analytical point of view.
Furthermore, systems powered by AI and augmented analytics continuously learn what people choose to do with data. Algorithms identify patterns in the data, and these make search results faster, more accurate, and more complete. You can integrate AI analytics tools with other interfaces like Amazon Alexa and chatbots, drawing on these technologies’ impressive developments in NLP, too.
With all this potential to find new insights, it’s no surprise that 97% of business leaders say their businesses are investing in big data and AI initiatives.
Better together: Working with qualitative data and quantitative data
Quantitative data is the bedrock of your BI and analytics. It provides a solid foundation on which you can build data visualizations and insights that are fundamental to your organization.
With that foundation in place, qualitative data enables you to take your analytics further, get more understanding from a fast-growing array of sources, and power your decision-making with data from even more perspectives.
You have to be able to work with both kinds of data in order to unlock the most comprehensive insights. Together, they can help set the stage for game-changing pivots and innovations that can keep your organization at the head of the pack.
Adam Murray began his career in corporate communications and public relations in London and New York before moving to Tel Aviv. He’s spent the last ten years working with tech companies like Amdocs, Gilat Satellite Systems, and Allot Communications. He holds a Ph.D. in English literature. When he’s not spending time with his wife and son, he’s preoccupied with his beloved football team, Tottenham Hotspur.
How to Build a Flexible Developer Documentation Portal
Developing analytic apps is a bold new direction for product teams. The Toolbox is where we talk development best practices, tips, tricks, and success stories to help you build the future of analytics and empower your users with the insights and actions they need.
When creating a resource and community to help developers get the most out of your product, it’s important to empower them to contribute to developer documentation and not just have all your content coming from product or tech writers.
If you haven’t read how we overhauled our developer portal recently, check out our prior conversation with Moti Granovsky, Sisense’s Head of Developer Relations. In this second part, we will take a deeper dive and share how we built a new system from the ground up to both deliver great information and be flexible enough to allow for continued evolution and growth.
Let’s kick off our journey into the rebuild by understanding what our requirements were and how we went about meeting them.
Building instead of buying
Shruthi Panicker: Why did we choose to build the portal on our own from scratch and not just use one of the many great products out there?
Moti Granovsky : There are great products out there focused on developer documentation, which are used by many companies successfully. When we set about rebuilding the portal, we knew we wanted engineers to contribute to our developer documentation. We also wanted to auto-generate some of the content, especially API references (which are very dry; there’s no “writing” necessary). Our product has so many APIs (rich and large) that writing and maintaining them is very time-consuming. We didn’t want tech writers to waste time repeatedly updating tables and tables of parameters.
We wanted to invest our time in generating content that is more engaging and valuable, like tutorials, open-source demos, cool features, and so on.
Make your requirements specific
SP: What were you asked to do?
MG: Our requirements were, in essence:
- The platform should allow us to separate styling from content. We wanted to make sure that unlike the old website, it looks great, matches the Sisense brand, and keeps on looking great even as more content is added. This means not only did we want to white-label and design the content, we wanted to separate the layout and design from the content. (Remember, the people who are writing documentation are not necessarily experts at visual design.)
- The format should be accessible to developers (aka Markdown-based). Besides an UI-based editor, some content management systems have their own specific language to programmatically format content. However, oftentimes nobody knows enough about these languages, and they’re not very intuitive either. Instead, we looked at the most common way for developers to write content — Markdown. It is used by Git itself and all Git providers. Even readme files for projects are written in Markdown. Every developer knows this syntax, and it has this benefit of separating design from content. Markdown content will also allow us to auto-generate content. If you look around, most API tools like Swagger, for example, have the ability to generate Markdown content.
- Our customers should be able to contribute to docs (eventually). If a customer finds a typo or a mistake or they want to add some detail, we want to make that possible. For that, we should be able to store the content somewhere public, such as a GitHub repository. Markdown makes this possible since it’s text format and can be easily shared in GitHub. But we need the content to be separate from the website itself (something that is publicly available). If we were to write HTML pages and put them on a server, then a customer can’t contribute to that. But if those pages are generated from something that is publicly accessible like a GitHub repository, then we can solve this challenge.
- We should go beyond just the documentation. We wanted the new website to be a true developer portal — a hub of knowledge and information for developers with which they can find all the resources they need. We want to make sure that, in the future, it includes a blog, playground, links to community, etc. that are not necessarily documentation. We want to create a home for developers where they can come and find anything that they need to do their job.
We looked at products in the market, and none of them met all of these requirements combined. Additionally, they were relatively costly because they include hosting, but documentation is generally static content so it is cheap and easy to host. There was no reason we couldn’t build it ourselves since we have the technical know-how to do it in-house.
Choose the right framework
SP: So what tools did you decide to use to build the new portal?
MG: We found out some companies (and a huge shoutout to the developer relations team at Okta whose approach to developer docs inspired us) have taken this different approach and built their own websites using different tools. There are frameworks that can generate websites from Markdown. We looked at different ones and picked one that fit the Sisense technology stack (in terms of languages that developers are familiar with. Since most of our developers are JavaScript developers, we wanted one that used that and not Python, for example).
We found one called VuePress that did exactly what we needed. It has all the flexibility in the world. It is an open-source framework built and used by the Vue.js project. It allowed us to build exactly the type of website we had in mind in terms of look and feel while avoiding a lot of the effort involved compared to starting from scratch.
We invested effort in creating a great design and refactoring a lot of the existing content we had into Markdown from the old format. However, this process let us build the new portal from the ground up exactly the way we want without any compromises.
While it took a lot of effort initially, in the long term it will all pay off. First, the hosting is very cheap. Second, if we have to add new content, we don’t have to design anything. It will only be as expensive as writing that text. Anyone can just write text in a notepad and have beautiful content up on the website.
New portal, new options
SP: What can users expect to find in the new portal that they couldn’t in the old one?
MG: As a result of the new design, there are a few big changes in the new Sisense developer portal:
- Separation of API reference and documentation: Developers can find a dedicated API reference section that is well organized and structured. They can quickly access the exact information they need. It is a lot more detailed and a lot more standardized than what we previously had, and they don’t have to scroll a bunch of text to access what they need.
- More code samples right within the documentation: Code can be easily copied with a copy code button. It is nicely formatted, and the syntax is highlighted according to the language, making it much easier to read.
- A lot more cross-referencing and links than we used to have: When something is mentioned, it is always linked right there, or you can find the relevant link at the bottom of your page. You can navigate through and gain all the knowledge to accomplish a task without breaks in the thought process.
- Improved search: You can quickly see search pages and headers within pages. You can quickly jump into specific areas within a page.
- Responsive mobile experience: We also optimized all pages for mobile consumption.
- New and refactored content: Much of what’s been added to the portal is entirely new, plus we refactored a lot of existing content, revalidating and updating it.
- Broad range of content types: We added a lot of external references to our GitHub account, webinars hosted on Vimeo, and a playground built in conjunction with the developer portal.
Overall, the content now is not just a block of text but part of an ecosystem of knowledge that developers can utilize. Everyone has a different learning process. Some prefer to read, some prefer to see a demo, some like to watch a video while others prefer to just dive right in and learn through experimentation. We are trying to provide different ways to help no matter what a person’s approach to learning is.
In addition, we also have features that cover:
- Developer release notes and release notes for the website itself so you can track what has changed
- More about the DevX team, why we exist, and and how to get in touch with us
- Ability to reach the playground, blog, and forums from the portal directly
In the future, we are looking at adding a developer-focused blog within the portal directly. We are also continuously working on more video content and documentation for some of the APIs that aren’t covered yet.
How you can improve your own documentation
SP: Lastly, how can an organization decide if building its own developer documentation site ground up is the right approach for them?
MG: Take into consideration these points to decide if this approach is good for you:
- You need engineering capability to build the website.
- You need to make sure that the people who are writing documentation are comfortable with the format. For example, if the people writing documentation in your organization are technical writers who are comfortable with Markdown, then great, use this approach. Or maybe they want to learn, then that is great as well. If they do not want to learn, they may need another platform that is what-you-see-is-what-you-get.
- Make sure you have the ability in terms of IT to host and maintain the site. If you can’t assure that it will be up globally and available 24/7, then you are going to have problems. Since this is the largest interface with the largest surface area, fault or downtime will have a huge negative impact.
It all boils down to capacity and need. If your current documentation website works and customers are happy with it, and all you need to do is to enhance and update the content, then do that. Don’t waste your time and money on building something new. But if you have a lot of room for improvement like we did, and you want to use the opportunity to upgrade and overhaul the user experience, then this is a viable solution that should be considered.
Shruthi Panicker is a Sr. Technical Product Marketing Manager with Sisense. She focuses on how Sisense can be leveraged to build successful embedded analytics solutions covering Sisense’s embedding and customization capabilities, developer experience initiative and cloud-native architecture. She holds a BS in Computer Science as well as an MBA and has over a decade of experience in the technology world.
The Cloud Goes On-Premises: AWS Outposts
We’re stronger when we work together. In our Partner Showcase, we highlight the amazing integrations, joint projects, and new functionalities developed with our technology partners at companies like AWS and others.
You’ve heard again and again that the future is in the cloud: Cloud computing and storage gives businesses of all kinds unparalleled agility to adapt to changing conditions and build amazing new products and services. The COVID-19 pandemic has only increased the demand for companies to turn on a dime and for multiple workers across the globe to be able to remotely access data. However, not every industry and organization has decided to go all-in on the cloud.
Most applications can be easily migrated to the cloud. However, in some cases, there are special latency or local data processing requirements that necessitate keeping the data on-premises. This used to effectively limit the ability of these companies to make full use of the many benefits that the cloud offers.
In the past, this was the end of the conversation; you were out of options. But what if you could bring your BI tools closer to your on-premises data, increasing the speed at which you could access that data and get insights? A truly hybrid experience, bringing modern cloud capabilities to on-premises systems, allows such companies to still get the benefits of cloud systems: the latest data querying methods, development processes, and cloud-IT practices.
AWS Outposts is ideal for workloads that require low latency access to on-premises systems, local data processing, or local data storage. Now companies with on-premises data can enjoy cloud-style benefits in a truly hybrid setup. In this article, we’ll explore what that looks like and how users can connect AWS Outposts to Sisense to get more out of their data.
Analytics & BI users: Better queries, greater efficiency, different data types
As front-line users, analytics and BI teams just need their BI tools to work, easily serving up the right insights from the right data, though where that data is stored isn’t usually up to them. Even when the company is required to store some of its data on-premises, a hybrid experience like AWS Outposts can allow these teams to take advantage of many of the benefits traditionally only offered by being fully in the cloud, such as low storage and compute costs and access to fast-moving data and unstructured data.
A truly hybrid setup treats all your data like it’s in the cloud, making it easier to connect disparate data sources to one BI tool (like Sisense) and pull insights from it. With AWS Outposts plus Sisense, users can perform complex analyses and drill down into the key performance indicators that matter most to them, all while preserving the requirements that kept their data on-premises in the first place.
“AWS Outposts is a godsend for teams who wish to dabble in the cloud but are afraid to lean in too much,” said Guy Levy-Yurista, Chief Strategy Officer at Sisense. “Now they can enjoy both worlds in a fully hybrid way. Place an AWS Outposts rack with AWS and Sisense servers in your data center, and connect it to datasets that are located in the cloud. The on-prem Sisense platform can easily mash up data no matter where it is sitting, in the cloud or locally, and still deliver against mission-critical data workflows which must exist on-prem.”
Product teams: Modern development practices
The future of app building/development is on the cloud. There are entire applications and businesses being built on the cloud as we speak, where the code is only ever deployed to cloud servers. Teams building cloud-native apps like this need cloud-like environments to take advantage of the latest DevOps practices.
This is another place where a truly hybrid deployment like AWS Outposts can help empower organizations that still have to keep some of their information in on-premises systems. A joint system of Sisense and AWS Outposts gives product teams access to cloud development best practices and saves them from having to duplicate development efforts (i.e., if your data is currently split between being in the cloud and on-premises, your team has to build for both setups). They get the cloud services’ ease of use, within their own data center, leveraging the best cloud tech in the world delivered by Sisense and AWS.
Modern apps demand modern cloud technologies and practices. AWS has set the standard for cloud technology since its inception. With a hybrid deployment, product teams can utilize the same deployment methodologies both on AWS Outposts and on their other AWS databases, pulling the data from wherever it lives to power the next generation of apps with the data and insights that users demand. The Sisense platform is a great fit for AWS services and applications of all kinds.
Data teams: The latest IT practices
Connecting to cloud data sources is a vital aspect of every company’s data strategy. When some data must remain in on-premises systems, a hybrid model like Sisense with AWS Outposts allows organizations to embrace the benefits of cloud systems without relocating their data.
Modern data teams increasingly deal with cloud data sources more than on-premises ones. AWS Outposts plus the Sisense analytics platform allows data engineering teams to apply one set of data preparation best practices to all of their data. These teams can also manage their data with the most modern and easy-to-use tools and methods.
Instead of having to handle multiple setups for your different datasets (on-premises and cloud), AWS Outposts empowers data teams to use the same AWS hardware infrastructure, services, APIs, and tools they’re familiar with to build and run applications on-premises and in the cloud. It’s all one consistent hybrid experience. Your AWS compute, storage, database, and other services run locally on Outposts, while also putting AWS services at your fingertips to build, manage, and scale your on-premises applications with industry-leading AWS services and tools.
New frontiers for builders
Cloud is the future (and the present). Many companies today are being created entirely on the cloud, but some still need on-premises data for a variety of reasons. That doesn’t mean these builders should be stuck with outdated tools. AWS Outposts bridges the gaps between the on-premises and the cloud worlds, then Sisense takes the data to the next level by seamlessly connecting and empowering builders to pull insights out of it.
“Builders have always been on the frontier of their worlds, building new things that no one could have imagined but them,” said Guy. “Whatever you’re building with your data, wherever it lives, AWS Outposts and Sisense give you more flexibility to do more with it.”
Ahuva Hazan-Fuchs is a Product Director, Platform & Cloud at Sisense, specifically focused on OEM and enterprise customers. With 19+ years of experience in product management and development, Ahuva is passionate about ease of use, simplicity, and ceramic art.
Building Bridges — Best Practices for BI Teams Working with Data Teams
Implementing analytics at your company is a multi-team job. In Building Bridges, we focus on helping end-users, app builders, and data experts select and roll out analytics platforms easily and efficiently.
Selecting and implementing a new BI and analytics platform is a big decision and can be a vital part of an organization’s digital transformation. Rolling out a new platform involves everyone who’ll implement, maintain, and most heavily use such a platform.
Advanced analytics and BI democratize access to data, empowering more business users to develop insights, with less reliance on data professionals who have previously been gatekeepers of this information. As business teams become more involved with data in their day-to-day work, it’s natural that they should play a role in choosing the right platform and determining how it will benefit their organization.
Making these decisions — which platform to choose and how to put it into operation— requires buy-in from both the analytics and BI team (the probable end-users/frontline users) and the data team (who will prepare the data, build the models, and connect datasets).
Importantly, to make the best decision for your organization, each team must understand, acknowledge, and address the needs and concerns of the other.
Working together: Understanding priorities
Mutual understanding can only come about via dialogue between teams, so that they can understand the priorities and needs that their BI and analytics platform should meet. It’s important that the analytics and BI team clearly indicate their needs and that the data team understand what the BI platform will be used for and how they can build the right data model(s) to suit the analytics and BI team’s requirements.
To help achieve this, let’s look at some considerations that data teams and analytics & BI teams should discuss in the vital conversation about selecting and implementing a new platform, so that they both can get the most out of the process.
The five big questions
First, when considering a BI and analytics platform for your organization, there are five big questions that everyone should ask, irrespective of their function. A simple yes or no answer in each case, will help you determine quite simply some fundamental requirements that a platform should fulfil. These five big questions are:
The conversation
Armed with the answers to the five big questions, you’re then in a position to drill down into some details about what a BI and analytics platform should do for you. This is where the really interesting conversation starts, as youestablish where your requirements, questions, and concerns both converge and diverge. We envisage that the conversation could go something like this:
Usability
BI & Analytics Team: Can we find a BI and analytics platform that’s really user-friendly? Can you involve us in choosing something that offers self-service analytics, so we don’t have to hassle you all the time to help us crunch complex data? We’d really like to be part of the selection process.
Data Team: Sure. Let’s work together to find a platform that can meets your needs and ours. From our perspective, we need to be confident that a platform is robust and that it can handle on-premise and cloud data from any source.
While the data team is concerned with storing, connecting, and preparing data for analysis, the BI and analytics team is concerned with examining the data and creating relationships and comparisons between datasets, in order to surface insights and visualize the data. The former are data experts. The latter is more focused on the business needs and usability is a priority for them.
“When we were using a different BI platform, I wouldn’t let frontline business users touch it,” says Jennah Crotts, data analytics manager at Jukin Media. “They’d click one thing, everything would break, and I’d have to rebuild it from scratch. Now, with Sisense, if somebody is up to speed on their data and has gone through some basic training, I can copy a dashboard and give them ownership.”
Governance
BI & Analytics Team: It’s really important that we make our data as accessible as possible to as many users as we can, without unnecessary impediments. Can we make this happen?
Data Team: Certainly, but let’s not forget governance too. It’s also important for us to be able to control access to our data and ensure that proper policies are in place
Data management, including security, is a priority for the data team. Accessibility and the democratization of data are key focuses for the BI and analytics team. To maximize the utility of an analytics platform and the value of data to an organization, data for decision-making should be as accessible and as easy to understand for as many end-users as possible.
Trust and accuracy
BI & Analytics Team: For data, and ultimately insights, to be as accessible as possible, we need analysis and visualizations to be as simple as possible to find and use. Can we ensure that this is possible?
Data Team: We want to make this happen for you, while at the same time ensuring that data isn’t inadvertently mishandled and that problems don’t arise when setting up new KPIs or dashboards. We need to protect the data, the methods of accessing, and the ways it can be analyzed and visualized, while making it as accessible as possible
Both teams are concerned about the quality of the data they’re handling. The BI and analytics team, and end users, want to be sure that they’re receiving and analyzing the most accurate data as possible. However, it’s the data team’s job to manage and prepare the data, and optimize its accuracy, by ensuring it’s “clean”, without errors, duplications and without it being available to unauthorized users. The ability to set access programmatically, by group, team, department, or individual is an important feature that Data teams will look for in a BI platform.
“The benefit to using a BI system,” says Jennah, is that: “Even if you have all of your data mapped beautifully, is: Can you make sense of it? Can you notice trends? Can you identify what’s going on? Having the right BI tool and visualizations can allow you to do a lot of things.”
Scalability, agility, and capacity
BI & Analytics Team: It’s really important that we get the most value we can from a BI and analytics platform. So it shouldn’t just be suitable for now. It must be able to scale with the growth of our business. And scaling up shouldn’t be costly. Positive ROI is essential.
Data Team: Agreed. The platform we choose must be scalable, and agile, to respond to the growth of our business, the increase in the volume of data we’re handling, and changes in our market that require us to pivot. It needs to have the capacity and capability to grow with us.
In the interests of the business, the BI and analytics team seeks to maximize ROI, which is why ensuring that a BI and analytics platform is future-proofed so that it can scale up with the growth of the organization and the data it generates. The data team is solely focused on the capability of the platform to scale up and its capacity to handle increased volumes of data.
“To test the agility of the platform, I was able to connect the technical person at Sisense with our guy in engineering,” Jennah says. “This was helpful because if you try to reword what your tech guy is saying, it just won’t make sense. So when the right technical people can speak directly to each other, it helps the flow of the setup enormously.”
Cost-effectiveness
BI & Analytics Team: On the subject of ROI, a BI and analytics platform has to be cost-effective. It needs to save users time and resources while improving performance, and decision-making. We need to be sure we can achieve these aims, so we should conduct a proof of concept before making a final decision.
Data Team: Indeed. We appreciate that we can’t implement a complex platform just for complexity’s sake. It has to serve our business, our needs, and our objectives. And it has to address the particular challenges that any business has. A POC is essential.
When it comes to cost-effectiveness, the teams align. Both need to ensure that their choice of BI and analytics platform can deliver what it promises, what each of them requires and the maximum benefits to their organization, with no hidden or unexpected costs.
Versatility
BI & Analytics team: Please keep people — the users — in mind. We really need to put end-users first vs focusing mainly on IT needs. Any platform that we choose must be able to solve a range of different data-related issues, each of which affects different people in different functions within the business.
Data team: We actually think that our objectives meet here. Our platform of choice must be able to handle the widest variety of data from the widest variety of sources, precisely to maximize accessibility and usability. More and more data is unstructured (text, video, audio, graphical data and the like) and comes from non-technical sources. It requires technology like NLP to deliver usable findings for all. We want to reduce the time it takes to clean, integrate, and maintain data.
So, for instance, the C-Level wants to see high-level performance reports across departments. Mid-level executives want to make better decisions in their line of business. All data, and all types of data, must be accessible to all, subject to governance and policies. This necessitates that a BI and analytics platform be as versatile as possible.
“We’re starting to get more and more positive feedback from our C-level executives,” Jennah reports. “They’re saying things like, ‘This is a lot cleaner looking.’ ‘You were able to make those changes a lot quicker than I’m used to.’ ‘You’re able to get this to me faster.’ and stuff like that.”
Speed
BI & Analytics team: We’re missing opportunities when we can’t get insights quickly. We need data and visualizations in real time so we can make decisions and pivot fast.
Data team: We get it. Speed and ease of use are critical, but so is thoroughness. We must be sure that we choose a platform that handles and manages all our data properly so you can be confident that you’re getting the best and most accurate insights.
In a world where data is generated by the second, quickly getting your BI and analytics up and running could give you a competitive advantage. More importantly, making data-driven real-time decisions can mean the difference between success and failure, so from a business perspective, speed and ease of use are imperative. They should not be impeded by the need for accurate data management and preparation, which advanced BI and analytics should be capable of achieving equally quickly.
“The head of our content acquisition team, a major data consumer, was unsure about our switch. She was worried about transition costs and maintaining access to her particular data requirements.” Jennah says. “But when I showed her the first Sisense dashboard on a remote screen-share and was able to edit the dashboard to her specifications in real-time, she lost her mind over the speed and ease of use of the new platform.”
Implementation
BI & Analytics Team: How are we going to implement a new platform? We want results quickly so we can make decisions fast and get changes under way.
Data Team: Let’s not rush this. We’ll do our due diligence with a POC and ensure the right platform meets our needs, then consider how best to implement a new platform within the organization. If we get it right the first time, it’ll be easier, quicker, and more cost-effective in the long term. Tell us what you need and that will dictate timing.
Implementation can happen incrementally by department, company-wide in one sweep, limited to certain levels within an organization, and it’s important to decide whether to include embedded analytics and white-labeling for new revenue opportunities. All of these considerations will influence how quickly a new platform can be implemented.
“There’s been solid social interaction with the right kind of people who understand what they’re doing. Our sales guy did his homework and came back and hit every single request with full beautiful explanations,” says Jennah. “That allowed me to go to leadership and say they’re definitely getting a POC because they hit all the boxes. Because when the VP of engineering gets on the phone and starts asking the questions, they need to be able to answer them.”
Cloud, on-premises, or hybrid?
BI & Analytics Team: We need to have the fastest, most flexible, and most scalable access to data of any kind and in any form, to benefit as much as possible from the information we generate and handle. The Cloud has the capacity and flexibility that legacy on-premises solutions lack. Plus, cloud solutions include support, maintenance, and upgrades, which is cost-effective.
Data Team: Sure. We just need to consider whether we put all of our BI and analytics on the cloud, run it on-premises, or have a hybrid of the two? And how do we handle legacy systems on-premises? Is it more cost-effective and efficient to migrate them to the cloud, or maintain them on-premises alongside cloud analytics for newer data?
This is a key consideration when seeking to maximize the capabilities of BI and analytics and future-proofing them. The future of data and analytics is in the cloud. Its capacity is almost unlimited. You only pay for the capacity and processing power you need. There’s no need to invest in your own IT servers or additional data center facilities or hire a team to manage and maintain the application. The cloud services vendor will do all of that for you. And implementation is immediate, with no lead time need for ordering, installation, or deployment.
“I was really interested in a cloud-based interface.” Jennah reports. “I didn’t want a situation I had with another platform where I was constantly downloading and uploading data to communicate with internal business users. Other platforms that are so reliant on Internet speed won’t cut it in our current situation. It’s faster for me to build reports from scratch in Sisense than it would be to edit them in another platform.”
The first step to data-driven success
Choosing and implementing the right BI and analytics platform is a major project that can be hugely valuable for your organization. It will enable you to maximize the value you get from your data and will optimize the insights you can take from it. Plus, done right, it will empower many more of your colleagues to engage with data, and make vital decisions. Asking these questions and having this conversation, can be the first, essential step in establishing a successful, data-driven future that can supercharge your organization.
Adam Murray began his career in corporate communications and PR in London and New York before moving to Tel Aviv. He’s spent the last ten years working with tech companies like Amdocs, Gilat Satellite Systems, and Allot Communications. He holds a Ph.D. in English Literature. When he’s not spending time with his wife and son, he’s preoccupied with his beloved football team, Tottenham Hotspur.
Better Analytics Through AI: Our Take on Gartner’s AI Trends
AI and machine learning are the future of every industry, especially data and analytics. In Growing Up with AI, we help you keep up with all the ways these pioneering technologies are changing the world.
Reading through the Gartner Top 10 Trends in Data and Analytics for 2020, I was struck by how different terms mean different things to different audiences under different contexts. We hear a lot about AI and analytics not only in internal conversations, but also from our customers and prospects. But what do we really mean when we talk about these issues?
Seeing as how they will only become more important to our world, I thought it would be worthwhile, as Sisense’s head of AI research (AIR), to dive into 7 of the 10 trends on the list and give my views on each.
The article starts with a big statement about AI starting to operationalize, moving the requirements for data and analytics infrastructure to accelerate the development and adoption phase:
“By the end of 2024, 75% of enterprises will shift from piloting to operationalizing AI, driving a 5X increase in streaming data and analytics infrastructures.”
This is a major change in the way AI has been used in the past alongside data and analytics, making both more powerful and effective. Let’s dive into these trends and see what else is on the horizon.
Trend 1: Smarter, faster, more responsible AI
Gartner:
“Within the current pandemic context, AI techniques such as machine learning (ML), optimization and natural language processing (NLP) are providing vital insights and predictions about the spread of the virus and the effectiveness and impact of countermeasures.
“Significant investments made in new chip architectures such as neuromorphic hardware that can be deployed on edge devices are accelerating AI and ML computations and workloads and reducing reliance on centralized systems that require high bandwidths. Eventually, this could lead to more scalable AI solutions that have higher business impact.”
My take:
Augmentation and reinforcement learning are much more powerful than out-of-the-box solutions, and this is what’s guiding us along the way. Planning for every feature starts with questions about how the user will be able to play around with and modify the input to see how it affects the result. It was only natural for us here at Sisense to put significant investment into knowledge graphs, NLP, and automated machine learning. Together, they enable users to actively engage with the system, enjoying recommendations along with analysis. These features also facilitate a positive feedback loop, using engagement to strengthen what works and get rid of what doesn’t.
One result is that systems become much more intuitive: Users can take advantage of the “Simply Ask” feature to check “what are my sales next two months” and receive chatbot messages with projected visualizations and suggestions for further exploration routes. In a similar way, the forthcoming “Explanations” feature provides users with possible drivers of the movements in the data automatically, using knowledge graphs to go beyond the boundaries of their charts. This can turn the problem definition environment to multidimensional and learn from the user interaction with the system to personalize and match the results.
From Forecast to Trends to natural language querying, we are completely transparent about the technology behind and the statistical characteristics of the output. Whatever you’re seeing when you use Sisense, you can easily dig into the systems behind it.
Trend 2: Decline of the dashboard
Gartner:
“Dynamic data stories with more automated and consumerized experiences will replace visual, point-and-click authoring and exploration.”
My take:
At Amazon, everyone in a meeting sits down at the beginning and reads a full write-up, and then the discussion begins, rather than sitting through an endless PowerPoint presentation during the whole meeting. They focus on real storytelling rather than bullet points. We expect something similar to happen with dashboards: fetching insights-driven digests just in time, but also accompanying the daily routines with an “agent” supporting business flows in various tools.
Do you like to see what you missed first thing in the morning? Be alerted on significant movements? Is an executive summary enough to start the ball rolling, knowing you can always do a deep dive and ask for more? Using your favorite task management solution? The world is moving from the static, rigid experience to the data-, insight-, and personalization-driven assistant that knows how you want specific analytics to be served.
In order to make that work, a number of moving parts need to come together as one well-oiled machine: embedded interfaces (on-the-go via your device, in your email, chat, or in-app), pretrained analytics services and training pipeline, the vehicle to facilitate the data model creation, and the right visualization and narration to make the results digestible, trustable, and learning.
This is what keeps Sisense AIR busy: dashboard automation research and our knowledge graph, which has incorporated the behavior of thousands of past users.
Trend 3: Decision intelligence
Gartner:
“By 2023, more than 33% of large organizations will have analysts practicing decision intelligence, including decision modeling.”
“It provides a framework to help data and analytics leaders design, model, align, execute, monitor, and tune decision models and processes in the context of business outcomes and behavior.”
My take:
Decision-making automation requires a lot of steps: First you document the process, then configure it based on the result, then automate the possible parts. My take on it is that if you can automate the loop from data to analysis to decision back to data, it is not analytics, it’s robotic process automation. There’s an argument to be made that once decision-making on a use case becomes predictable, it should be moved from BI to a part of the back office.
But that kind of thinking comes from the world we used to know, a world that was less volatile and more manageable, more influenced by the proximity ecosystem than by world events and climate. Today, the world changes at a speed that’s hard to fathom, so decision-making needs to be adjusted based on insights coming from data, accompanied by recommended actions. “Survival of the fastest” is the rule today.
Trend 4: X analytics
Gartner:
“Gartner coined the term ‘X analytics’ to be an umbrella term, where X is the data variable for a range of different structured and unstructured content such as text analytics, video analytics, audio analytics, etc.”
My take:
The world is wider than the traditional BI tabular data. It’s visual, it’s spoken, it’s audible. Why use just one of the senses and limit your perspective?
Sisense recently used our ecosystem of ML service providers to help scan and surface the medical crowd wisdom of COVID treatments from piles of textual data from a site called G-Med. There was no point in reinventing the wheel to build our own video, image, speech, and text analysis tools — there are plenty of those on the market already.
How exactly is all that data going to talk to each other and come together to provide the end-to-end analysis? Knowledge graphs will be the base of how the data models and data stories are created, first as relatively stable creatures and, in the future, as on-demand, per each question.
Trend 5: Augmented data management
Gartner:
“Augmented data management uses ML and AI techniques to optimize and improve operations. It also converts metadata from being used in auditing, lineage and reporting to powering dynamic systems.”
My take:
The Gartner article doesn’t go beyond lineage or workload automation. That’s important, but that’s only what’s going on today. Fetching calculation results ahead of the question improves performance, but it’s still limited to the data model or dimensional paradigm of the single individual in the organization. Do they have the required perspective to include hurricane data for the supply chain dashboard for East Asia? Domain experts would likely decide to include that information after reading about losses in the news. What if the relevant data could be added to the context to tell the data story without humans needing to take action themselves? Data exchanges will play a more significant role in the future, extending their offerings to data modeling.
Trend 6: Cloud is a given
Gartner:
“By 2022, public cloud services will be essential for 90% of data and analytics innovation. As data and analytics moves to the cloud, data and analytics leaders still struggle to align the right services to the right use cases, which leads to unnecessarily increased governance and integration overhead.”
My take:
Cloud is here to stay. I witnessed the mainframe/PC/cloud/personal graphics processing unit evolution. To me, the tipping point of cloud analytics will be in the “context as a service” combination of data and logic components served based on user questions. With offerings like AWS Outposts, it couldn’t be easier to start the cloud journey.
In the analytics world, it’s crucial to stay up to date, implementing “continuous integration/continuous delivery” systems and A/B testing for better performance and experience. This is only possible with cloud services. Cloud combined with compliance with General Data Protection Regulation and SOC are vital to gain customers’ trust. Data-hungry calculations will be costly to perform in the cloud if data is on-premises due to data gravity and latency. Adjusting a system’s architecture can make all the difference quickly, meaning you can easily pull insights from large datasets.
Trend 7: Data and analytics worlds collide
Gartner:
“Data and analytics capabilities have traditionally been considered distinct entities and managed accordingly. Vendors offering end-to-end workflows enabled by augmented analytics blur the distinction between the two markets.
The collision of data and analytics will increase interaction and collaboration between historically separate data and analytics roles. This impacts not only the technologies and capabilities provided, but also the people and processes that support and use them. The spectrum of roles will extend from traditional data and analytics roles in IT to information explorer, consumer, and citizen developer as an example.”
My take:
I agree that new roles are required. As new data and analytics products are built and every product begins to have data and analytics elements in it, data/knowledge product managers will emerge. These specialists will understand data and be able to run and create queries and transformations but will also be knowledgeable about the applications running on top of those data streams.
Regarding data and tools, “extract, transform, and load” (ETL) will become ETLT. The “T” stands for the “transformation pipelines” either bringing data from the exchanges or pre-trained ML services or training pipelines for both structured and unstructured data. Software developers and data scientists can use these same pipelines to deploy their parts of the application, and analytics workflows can be automated to the point where business users can even trigger them without outside help.
AI and analytics: Building the future together
If you have data, odds are you have a lot of it. You’ve probably got more than you can handle. Alone, that is. Only AI will be able to help humans make sense of the huge datasets being generated every day by countless individuals and devices. AI systems will play greater and greater roles in our personal and business worlds, so whatever you’re building, start thinking about the ways AI can help your product, service, colleagues, and customers be better. And whatever you’re working on, build boldly.
Inna Tokarev-Sela, Sisense’s Head of AI Research, has over 15 years’ experience in the tech industry. She spent the last decade at SAP, driving innovations in cloud architecture, in-memory products, and machine learning video analytics. A frequent speaker at industry events like IBC, NAB, Wonderland AI, and Media Festival, Inna holds a BS in physics and computer science, an MBA, and an MS in information systems, having written her thesis on neural networks.
Must-Have AI Features for Your App
In Augmented Apps, we examine how product teams are exploring AI and Machine Learning to make their products more intuitive and enhance the user experience.
Artificial intelligence is transforming products in surprising and ingenious ways. Whether it’s core to the product, as with a stock market forecasting algorithm in Quants, or a peripheral component, such as a healthcare domain chatbot that diagnoses diseases via dialog with a patient, building reliable AI components into products is now part of the learning curve that product teams have to manage.
Certain proven methods and use cases are invaluable in helping product teams understand how to implement AI in apps to reproduce and enhance existing successes. In the case of a stock trading AI, for example, product managers are now aware that the data required for the AI algorithm must include human emotion training data for sentiment analysis. It turns out that emotional reaction is an important variable in stock market behavior!
Developing the right data model for your in-app AI is now a critical branch of programming, because correctly prepared data is vital to the algorithm model. It follows then that data scientists are suddenly integral to building embedded AI components.
A few AI components that have already achieved a high level of performance include X-ray and symptom diagnosis in healthcare, human emotion pattern recognition in stock trading apps, and innovative virus and malware detection in software systems. We’ll explore many of these use cases to build an overview of the state of the art in AI components today, starting with innovation in antivirus software.
Safeguarding software and users: AI in cybersecurity apps
One of the most exciting AI use cases today is happening in the field of cybersecurity, where AI methods are now commonly used to detect malware and malicious virus code embedded in files before they can do harm to systems. In fact, a breakthrough in AI security research occurred last year when researchers successfully used self-organizing incremental neural networks (SOINN) to identify malware and virus code.
The SOINN method converts binary files to visual representations and in doing so has achieved over 94% accuracy in detecting files infected with viruses! SOINN could become a standard use of AI for detecting viruses in security apps. By using a visual representation of code, the virus code can be detected without running the code and endangering the test system. Previously, machine learning-based malware detection algorithms were problematic because, according to Baptista and Shiaeles:
“…they rely on a virtual environment to analyze samples; this not only affects their runtime performance, but also endangers the whole system since the samples need to be executed. In addition, the ability of malware to adapt its behavior to the execution environment, the excessive number of features that need to be extracted per sample, and the high volume of malware instances being reported, reduce the chances of accurate detection.”
Breakthroughs like this — in which the fundamental representation of the subject is transformed — will certainly reshape and expand our concept of AI while providing new benefits.
Natural Language Processing: Teaching AI to talk
Natural language processing (NLP) unifies many of the other branches of AI through a human-oriented, AI-based interface. NLP performance improves significantly when the app is trained in a specific expert knowledge domain, such as symptom diagnosis. This is because a specialized context narrows the scope of the training data used in training the dialog algorithm. An immediately related branch of data science, called natural language querying (NLQ), focuses on using NLP to achieve particular goals.
Credit where it’s due: Understanding AI credit agents
A special use case of NLP and NLQ includes helping lending agents evaluate the creditworthiness of potential borrowers who have no prior credit history — the traditional basis of lending decisions. NLP can now mine vast and diverse data stores using trained patterns on alternative data sources that can help indicate the probability of reliable loan repayment.
For example, most credit applicants use smartphones. An AI can check their internet browsing patterns and see how closely they align with those of reliable clients. This type of supervised learning requires substantial data preparation: Important factors for correlation algorithms include a potential borrower’s social media activities, geolocation data, blogging contributions, peer networks, and relationship strength and duration. In fact, training metrics for these creditworthiness algorithms may bank on thousands of variables to generate an alternative credit score and also predict its own accuracy.
Predictive analytics AI boosts web app performance
Extraordinary advances in diverse applications such as visitor and log analytics owe their success to AI- and machine learning (ML)-based pattern recognition. Let’s look at a few such apps and their AI components and discover how we can benefit from their use cases:
- Web app visitor analytics: BI insights are derived from ML analysis of visitor stats and metrics and displayed via data visualizations
- Log analytics: The effectiveness of traditional error logs is enhanced by detecting app performance issues and by intelligently alerting targeted endpoints via email and Slack to critical user concerns
- Automated testing: Discovers bugs in new app releases using computer vision to determine page-rendering completeness
The potential uses of app behavior and visitor activity data stores are bounded only by the ingenuity of the data engineer. Analytics and BI platforms like Sisense can instantly provide data visualizations and alerts on visitor activity issues. Loggly and Timber likewise elevate traditional error logging by recognizing router issues as well as deployment and release failures. SmartBear and Tricentis enhance Selenium to provide AI-driven app testing automation. There is truly no limit to what can be achieved with a creative idea applied to data!
In all areas of predictive analytics, AI components perform best when the AI model can update its neural network during real-time transactions. This means feeding the client’s current preferences to the training data on the fly. AI web apps such as recommenders are the most obvious examples of real-time supervised training models that update themselves with each new transaction so that users can actually see the system learning in progress. And AI-based recommenders that update their own neural nets per transaction offer a more alive feeling in the user experience with such apps as:
- Swype personalized spelling suggestions (NLP)
- Ride-hailing apps’ suggesting destinations and best routes (geolocation)
- “Similar to favorite” music recommenders like Spotify
Healthcare benefits from AI diagnostics
As mentioned, NLP-powered chatbots with domain-specific expert knowledge can diagnose illness after a Q&A dialog with a patient. The next level in diagnostics is an AI-powered app that also takes a tissue sample from the patient and uses the lab output in conjunction with medical records and dialog to diagnose diseases, including cancer.
This amazing biotech application of AI uses genome sequencing in conjunction with neural nets to diagnose disease, outstripping the capabilities of traditional diagnosis methods. The app takes a patient’s tissue sample as input and then compares it with a database of every known pathogen (over 37,000 sequenced genomes!). The outcome is a comprehensive diagnostic report of possible disease agents.
Securing aviation hubs with AI
For those of us who are frustrated with long security lines at airport gates, new AI applications that automate the recognition of prohibited items in carry-on luggage will come as a refreshing change to speed up a sluggish human procedure. AI is also expanding rapidly on the frontier of emotion recognition and sentiment analysis. This will change the scope of preventive security to include biometrics of passengers who pose potential threats.
AI in apps: The future is today
AI apps are the future. On every horizon, we see automated systems ready to be dedicated to our safety, security, and comfort. At the moment, these systems are insulated from each other because data preparation for AI apps is a wild frontier, and there are no mutually intelligible standards for AI systems to share outcomes with each other.
However, just as the transmission control data protocol evolved toward a standard, so will data-wrangling eventually evolve standards for AI systems to learn from each other instead of exclusive datasets. When this happens, we will witness an exponential enhancement in the capability of AI now in its infancy. It’s an exciting frontier promising to deliver breathtaking benefits for everyone. Keep checking in as we bring you more insights from the emerging world of AI.
Inna Tokarev-Sela, Sisense’s Head of AI Research, has over 15 years’ experience in the tech industry. She spent the last decade at SAP, driving innovations in cloud architecture, in-memory products, and machine learning video analytics. A frequent speaker at industry events like IBC, NAB, Wonderland AI, and Media Festival, Inna holds a BS in physics and computer science, an MBA, and an MS in information systems, having written her thesis on neural networks.
Adding AI to Products: A High-Level Guide for Product Managers
In Augmented Apps, we examine how product teams are exploring AI and ML to make their products more intuitive and enhance the user experience.
AI and machine learning (ML) are not just catchy buzzwords; they’re vital to the future of our planet and your business. Datasets have quickly grown too huge, complex, and fast-moving for humans to grapple with. Integrating AI and ML into your product or service is becoming basic table stakes for staying in the market. Doing it right can mean the difference between thriving in the new world of data and disappearing from it.
So what are the high-level steps to incorporate AI and machine learning into new and existing products? AI can add value to your product/service in many ways, including:
- Improved business performance
- Reduced costs
- Increased customer satisfaction
- Improved brand value
- Risk reduction (reduced human error, fraud reduction, spam reduction)
- Improved convenience and accessibility of products
In this article, we’ll dig into the ways AI can help you accomplish these goals, allowing you and your team to envision the future of your product or service.
Improving performance with AI
Improving customer experience and reducing cost in a single step sounds impossible, but this is exactly what correctly implemented AI can achieve. Usually, we have to spend more money to achieve a better customer experience, but AI simultaneously delivers greater accuracy and focus and reduces human capital cost.
One aspect of AI that makes this outcome possible is training using expert domain knowledge. An AI that is trained with expert knowledge in a specific field can perform much better than a chatbot having a conversation on random topics. Visit the Mitsuku chatbot, for example, and ask the simple question, “How does a toaster work?” The reply, “You could spend years researching it,” is not useful. By contrast, have a look at the discussion with the stain removal chatbot in the diagram below.
Notice how the chatbot in this case guides the discussion with questions on a specific topic. This is where expert knowledge AI products can add enormous value to a brand. This is also an important takeaway for teams seeking to implement AI successfully: Start with the key performance indicators (KPIs) you want to measure your AI app’s success with, and see where that dovetails with your expert domain knowledge. Then tailor your approach to leverage your unique data and expertise to excel in those KPI areas.
Example: Products with recommender components
Suppose you’re working with a recommender engine that suggests products to a site visitor. What is the process to improve recommender engines? An obvious mechanical answer is: use relevance as a metric. Look at implicit and explicit feedback. One proven method for success is to refine preferences in real time by inserting new customer choices into the neural net during transactions. This means that the intelligence of the recommender actually improves while a visitor is browsing, even before a transaction happens!
Another important method is to benchmark existing metrics. Know the limitations of your existing dataset and answer these questions: What categories of data are there? What is the specific problem or obstacle to creating accurate recommenders now?
Next, collect root causes to address these questions. If possible in your marketing scope, talk to existing customers and improve their experience by comparing their expected outcomes with current outcomes. Learn the customer’s expectations. Specifically, use an affinity diagram to prioritize and plan the minimum viable product (MVP).
Points to remember:
- Understand the customer problems you want to solve: What are the social and emotional aspects? Can a chatbot help improve relations? What are the business consequences?
- Inclusiveness: Seek to discover and eliminate bias around race, age, and gender. Be sure test cases represent the diversity of app users.
- Recognize user emotions: Emotion is crucial to recommenders. Natural language processing can be used to detect tone and intent in written conversations.
- Build a product that continuously learns: AI systems should adapt to and learn from feedback.
Better data, better AI, better products
Having the right data (and enough of it) to train your AI is vital. Take Grammarly as an example: This popular program checks the grammar, tone, and style of documents. It groups content into one of three styles: professional, moderate, or casual, based on natural language processing. Getting this AI properly trained required a huge learning dataset with countless documents that were tagged according to specific criteria. Accurately prepared data is the base of AI.
As an AI product manager, here are some important data-related questions you should ask yourself:
- What is the problem you’re trying to solve?
- What are the right KPIs and outputs for your product?
- What data transformations are needed from your data scientists to prepare the data?
- What will it take to build your MVP?
Today, we’re developing AI in an era where data is treated as code, or at least as an extension of code, because the code alone cannot achieve deep learning without the data. Moreover, training data must conform to the app design to such an extent that data preparation resembles coding. There are several important steps to take to ensure data integrity for your AI project:
- Correct: Common data errors include Fahrenheit/Celsius, UTC, and a 12/24-hour clock
- Compliant: Data must comply with legal consent and privacy rules
- Current: Data must be relevant in the time scope and not expired
- Consistent: You should validate consistent data types with test cases
- Consolidated: You should determine the output desired, for example, data for regression vs. classification should be distinguished by the ML model in use
The perfect fit
When building your data model, it’s vital to avoid both underfitting and overfitting. This is done by an ML method called validation. Although AI product managers may not be involved at the level of algorithm development, they can benefit from recognizing the symptoms of overfitting in the behavior of the model under development.
An underfitted model is inflexible and doesn’t learn well. An overfitted model learns from all the data, including noise in the data.
The underfitted model looks as if it ignores the data altogether, while the overfitted model at right models the noise in the data! Without getting too technical, we want to mention an example of validation that can be used to resolve these problems: k-fold cross-validation.
Understanding k-fold cross-validation
K-fold cross-validation works by dividing the dataset into multiple subsets called “folds.” This method has the advantage of improving the model without requiring additional datasets. The model with the lowest cross-validation score will model the training data best while achieving a balance between underfitting and overfitting. Here we can begin to see why it is beneficial for AI product managers to have a data science background. The ability to understand ML models at a high level improves our ability to work directly with our team of data scientists to create accurate models.
Sampling the mechanics of AI algorithms as we have just done leads us into the related topics of model selection and model evaluation. As a product manager, you’ll want to do your own research to learn about the best modeling methods for your AI product domain and scenario. Learning this kind of information will aid you in interacting meaningfully with your team of data scientists.
Building smarter products, boldly
The future will be intelligent. Data continues to proliferate, and only AI assistance can help humans make sense of it. These are huge, existential issues but have simple, material applications when it comes to building the next wave of apps, products, and services. Whatever you’re building, data needs to be a part of it. Adding an AI element will help your users get the most out of that data, improve their experience, and keep your company in the market as AI becomes the norm for every aspect of daily life. Envision a better world and a better product, then build it boldly with the right data and AI.
Inna Tokarev-Sela, Sisense’s Head of AI Research, has over 15 years’ experience in the tech industry. She spent the last decade at SAP, driving innovations in cloud architecture, in-memory products, and Machine Learning video analytics. A frequent speaker at industry events like IBC, NAB, WonderlandAI, and Media Festival, Inna holds a BS in Physics and Computer Science, an MBA, and an MS in Information Systems, having written her thesis on Neural Networks.
Artificial Intelligence, Real Enhancements, Better Insights
Sisense News is your home for corporate announcements, new Sisense features, product innovation, and everything we roll out to empower our users to get the most out of their data.
Ashley Kramer is our new chief product and marketing officer. She began her career as a software engineering manager at NASA, worked at Oracle, Amazon, and elsewhere before moving to Alteryx, serving as senior vice president of product. She recently spoke with SearchBusinessAnalytics about her vision for the Sisense platform and what it’s like being one of the few women helping shape software development at a major business intelligence and analytics vendor.
SearchBusinessAnalytics: What led you to the decision to leave Alteryx and move to Sisense to head the development of its analytics platform?
Ashley Kramer: What was really interesting to me about Sisense was the approach they took from the beginning, which was to give agility to customers. That means being able to do analytics the way that they need — the Periscope acquisition means it can be code first or drag-and-drop — and being able to allow people to leverage analytics wherever they need, whether that’s embedded deeply within a product, whether that’s in an internal portal, or whether that’s a dashboard that you deploy.
In addition, I have a lot of cloud experience, and I understand what works and what doesn’t, and Sisense took a really unique approach here a year ago to say, “Let’s rewrite our products to be more of a Linux microservices-based architecture,” so that as companies scale in the cloud and as they leverage different complex data scenarios, we can grow with them. We can meet them where they are today, and we can grow with them as they go into the future, and that was truly unique to me and super exciting.
SBA: What is the dynamic between you as chief product officer and chief technology officer Guy Boyangu?
AK: He deeply understands the architecture, both where it was and where it’s come, and what we collaborate on is around the AI space. We have something called the Knowledge Graph that gathers all kinds of intelligence so we can give our customers smartness out of the box when we deliver our analytics, so what Guy and I collaborate on is on the under-the-hood side of where Sisense goes next.
I’m the product strategist and visionary working with the entire team, and Guy is able to take that and say, “OK, technology-wise, this can be our strategy.” It’s a really good partnership that we have together.
SBA: As you come to Sisense, what is your vision for where the analytics platform will be in two or three years?
AK: There are a lot of places we can go, and we need to focus those efforts so we can make sure we’re delivering what customers need. But I can really break this down into further investments in three different areas.
First, how will we continue to scale across clouds? If you think about [many] of the other vendors in this space, they’re mostly part of a cloud ecosystem — Looker and Google, Tableau and Salesforce, Power BI always being part of Microsoft — and we’re a cross-cloud platform, we can work with data analytics cross-cloud, so we want to continue innovating there and scaling to make sure we work seamlessly with all of the different services that all of the different clouds provide. The second is more intelligence. How do we continue to do deeper smartness within the platform, so automated recommendations and automated decisions, alerting, those types of things. And of course we have natural language querying so you type a sentence and get your results. There are many more things we can do to make analytics easier for everybody involved and get more people involved in analytics.
And that brings me to the last one, which is if you look at statistics on the different usage and adoption of analytics it tends to be pretty low in general within organizations, and I truly believe that’s for one reason — most BI platforms assume people will unnaturally leave their everyday workflows and processes and look at a dashboard and then come back. With our API-driven platform and approach, we can bring analytics to the salesperson that spends their entire day in whatever sales platform or CRM (customer relationship management) platform they use, and for someone like me that’s always on the go, send it to my cellphone. By continuing to innovate in that area, I think we can start to see adoption go up and really start seeing organizations make data-driven decisions, and then of course as an end result have better business outcomes.
SBA: What are some trends you see in BI and analytics that Sisense is responding to and are helping drive Sisense’s roadmap?
AK: The first trend is we’re finally starting to see people migrate their data to the cloud. It’s been something people have been talking about, but we’re really seeing it come to life. The second is adoption — different statistics say that only about a third of an organization are actually using the analytics to make decisions. And the next one is a huge trend around AI and augmented analytics, so how do we not just allow people to make a decision about what’s happening right now, build a dashboard and see sales are down in the west, but how do we actually start to help them predict what to do next and prescribe how to take the action. A lot of the things we’re doing with our Knowledge Graph and a lot of the features we’ve released and have planned as part of our roadmap will start getting people out of just descriptive and taking them to those next two steps of predictive and prescriptive analytics.
SBA: How did your time at Oracle, Tableau, Alteryx, and other organizations prepare you for your new role at Sisense?
AK: My early career at Oracle and NASA was valuable because I was a programmer, so I understand technology pretty deeply, and I understand how sometimes a product team can dream up something but it might actually be a hard engineering challenge. That was helpful for me to understand how to be a better product leader. My time at Amazon helped me understand the crazy importance of the cloud. They were the early innovators in that area. Tableau was really interesting because Tableau started coming into the market with cloud analytics right at the cusp of when it was happening, so I got to learn a lot about the early market and then see how it evolved. And of course when you’re one of the earliest to arrive you don’t get everything right, and that’s okay. You learn lessons and figure out how to do it better next time. And at Alteryx it was trying to understand what the more data-science part of the world needs.
As a product leader it’s super important to understand what is possible for engineering and what is not, understanding the cloud is important because we’re cross-cloud here at Sisense, and then understanding where people want to go next with analytics by actually scaling it across the organization and taking that next step to be predictive. So, this really was the perfect role.
SBA: As chief product officer at Sisense you are now one of the few women helping shape the product development of a leading BI vendor. Did you face barriers on your path to Sisense?
AK: I was always interested in technology growing up — even in the early days, the Oregon Trail days. Fast-forward to high school and I was in the very first computer science class taught by my physical education teacher, which is crazy when you think about it. It doesn’t make any sense, but at least my school was thinking about offering computer science. I was a senior and the only woman in a class of 10 people. I was very fortunate to have people to encourage me to continue on this path. Move on to college, and in my graduating class there were three to five women in the class. Since then, I think what the world has done a better job of is have things like Girls Who Code, different programs to push the importance of STEM and make girls and women be a part of that. The world has changed and evolved, but it wasn’t really there when I was coming to my professional stage of my career.
SBA: What have you found following school in your professional life?
AK: As far as the glass ceiling, I feel like I’ve had a lot of great mentors along the way, some of them were women and some of them were men, and I think sometimes we undervalue the importance of what we can do for each other. Women can get together and I can go to every women leadership CPO dinner that exists, but we need men there too supporting us. We need them to be part of the conversation. Something that’s super attractive to me about Sisense is that not only is 50% of our core operational team reporting to the CEO women, but also [CEO] Amir Orad’s wife is a CEO [of LimitScreen Inc.]. He definitely understands the value of women in these leadership roles. Everyone has an equal voice, and I think that’s really powerful.
It’s been interesting. I’ve seen a trend with my generation and there are other women CPOs — there aren’t many, but even at my last company there was a woman chief strategy officer and we were really big supporters of each other. There was a historic thing in the past that there could only be one woman in the room, but that didn’t exist at any of the companies I’ve been. I feel fortunate to not have to have lived in that world. The more women in the room at leadership levels, the better, so I don’t feel like I have to speak up or be the alpha woman because that doesn’t exist anymore.
Editor’s note: This Q&A has been edited for clarity and conciseness.
The Data Journey: From Raw Data to Insights
We live in a world of data: there’s more of it than ever before, in a ceaselessly expanding array of forms and locations. Dealing with Data is your window into the ways organizations tackle the challenges of this new world to help their companies and their customers thrive.
In a world of proliferating data, every company is becoming a data company. The route to future success is increasingly dependent on effectively gathering, managing, and analyzing your data to reveal insights that you’ll use to make smarter decisions. Doing this will require rethinking how you handle data, learn from it, and how data fits in your digital transformation.
Simplifying digital transformation
The growing amount and increasingly varied sources of data that every organization generates make digital transformation a daunting prospect. But it doesn’t need to be. At Sisense, we’re dedicated to making this complex task simple, putting power in the hands of the builders of business data and strategy, and providing insights for everyone. The launch of the Google Sheets analytics template illustrates this.
Understanding how data becomes insights
A big barrier to analytics success has been that typically only experts in the data field (data engineers, scientists, analysts and developers) understood this complex topic. As access to and use of data has now expanded to business team members and others, it’s more important than ever that everyone can appreciate what happens to data as it goes through the BI and analytics process.
Your definitive guide to data and analytics processes
This flowchart infographic shows how raw data becomes actionable insights. It will navigate you through every consideration you might need to make about what BI and analytics capabilities you need, and every step of the way that leads to potentially game-changing decisions for you and your company.
1. Generating and storing data in its raw state
Every organization generates and gathers data, both internally and from external sources. The data takes many formats and covers all areas of the organization’s business (sales, marketing, payroll, production, logistics, etc.) External data sources include partners, customers, potential leads, etc.
Traditionally all this data was stored on-premises, in servers, using databases that many of us will be familiar with, such as SAP, Microsoft Excel, Oracle, Microsoft SQL Server, IBM DB2, PostgreSQL, MySQL, Teradata.
However, cloud computing has grown rapidly because it offers more flexible, agile, and cost-effective storage solutions. The trend has been towards using cloud-based applications and tools for different functions, such as Salesforce for sales, Marketo for marketing automation, and large-scale data storage like AWS or data lakes such as Amazon S3, Hadoop and Microsoft Azure.
An effective, modern BI and analytics platform must be capable of working with all of these means of storing and generating data.
2. Extract, Transform, and Load: Prepare data, create staging environment and transform data, ready for analytics
For data to be properly accessed and analyzed, it must be taken from raw storage databases and in some cases transformed. In all cases the data will eventually be loaded into a different place, so it can be managed, and organized, using a package such as Sisense for Cloud Data Teams. Using data pipelines and data integration between data storage tools, engineers perform ETL (Extract, transform and load). They extract the data from its sources, transform it into a uniform format that enables it all to be integrated. Then they load it into the repository they have prepared for their databases.
In the age of the Cloud, the most effective repositories are cloud-based storage solutions like Amazon RedShift, Google BigQuery, Snowflake, Amazon S3, Hadoop, Microsoft Azure. These huge, powerful repositories have the flexibility to scale storage capabilities on demand with no need for extra hardware, making them more agile and cost-effective, as well as less labor-intensive than on-premises solutions. They hold structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs), and binary data (images, audio, video). Sisense provides instant access to your cloud data warehouses.
3. Data modeling: Create relationships between data. Connect tables
Once the data is stored, data engineers can pull from the data warehouse or data lake to create tables and objects that are organized in more easily accessible and usable ways. They create relationships between data and connect tables, modeling data in a way that sets relationships, which will later be translated into query paths for joins, when a dashboard designer initiates a query in the front end. Then, users, in this case, BI and business analysts, can examine it, create relationships between data, connect and compare different tables and develop analytics from the data.
The combination of a powerful storage repository and a powerful BI and analytics platform enables such analysts to transform live Big Data from cloud data warehouses into interactive dashboards in minutes. They use an array of tools to help achieve this. Dimension tables include information that can be sliced and diced as required for customer analysis ( date, location, name, etc.). Fact tables include transactional information, which we aggregate. The Sisense ElastiCube enables analysts to mashup any data from anywhere. The result: highly effective data modeling that maps out all the different places that a software or application stores information, and works out how these sources of data will fit together, flow into one another and interact.
After this, the process follows one of two paths:
4. Building dashboards and widgets
Now, developers pick up the baton and they create dashboards so that business users can easily visualize data and discover insights specific to their needs. They also build actionable analytics apps, thereby integrating data insights into workflows by taking data-driven actions through analytic apps. And they define exploration layers, using an enhanced gallery of relationships between widgets. A package such as Sisense for BI and Analytics Teams is designed to achieve easy to visualize analytics and form game-changing insights.
Advanced tools that help deliver insights include universal knowledge graphs and augmented analytics that use machine learning (ML)/artificial intelligence (AI) techniques to automate data preparation, insight discovery, and sharing. These drive automatic recommendations arising from data analysis and predictive analytics respectively. Natural language querying puts the power of analytics in the hands of even untechnical users by enabling them to ask questions of their datasets without needing code, and to tailor visualizations to their own needs.
5. Embed analytics into customers’ products and services
Extending analytics capabilities even further, developers can create applications that they embed directly into customers’ products and services, so that they become instantly actionable. This means that at the end of the BI and analytics process, when you have extracted insights, you can immediately apply what you’ve learned in real time at the point of insight, without needing to leave your analytics platform and use alternative tools. As a result, you can create value for your clients by enabling data-driven decision-making and self-service analysis.
With a package like Sisense for Product Teams, product teams can build and scale custom actionable analytic apps and seamlessly integrate them into other applications, opening up new revenue streams and providing a powerful competitive advantage.
Find out more about how you can benefit from AI-powered self-service BI and build actionable analytic apps with all your data, simplify complexity and access insights that can drive your business forward.
Adam Murray began his career in corporate communications and PR in London and New York before moving to Tel Aviv. He’s spent the last ten years working with tech companies like Amdocs, Gilat Satellite Systems, and Allot Communications. He holds a Ph.D. in English Literature. When he’s not spending time with his wife and son, he’s preoccupied with his beloved football team, Tottenham Hotspur.
People-First Leadership: Shock, Transition, Transformation
In Navigating Change in Crisis, we explore how individuals and companies are adapting to a “new normal” in order to keep essential services functioning. We provide actionable advice around how organizations, and ultimately the builders of data and analytic apps, are adapting to meet these changes. These insights aim to help you and your team navigate these unprecedented times.
There’s more on the line today than ever before — for people, businesses, and their leaders. A steady hand navigating uncharted waters is a must. Leaders who prioritize people first can mean the difference between success and failure.
In the COVID era, human connections have become even more vital. People-first leadership means bringing positivity to a world where we are all experiencing countless challenges. This leadership style can lead to increased productivity, performance, employee retention, and an overall positive work culture.
In our recent Sisense webinar, People-First Leadership During a Crisis: Shock, Transition and Transformation, Nick Mehta, CEO of Gainsight, joined Amir Orad, Chief Executive Officer at Sisense, and Nurit Shiber, Chief People Officer at Sisense, to discuss the keys to building a robust people-first leadership strategy when companies are faced with new and unexpected challenges.
Managing during crisis: Focusing on the human
People-first leadership means acknowledging that we are all human and experiencing a generation-defining trauma right now, regardless of position. Being honest and transparent about what’s happening throughout the business is essential.
Successful leaders understand that during a crisis it’s crucial to be:
- Human
- Optimistic yet Realistic
- Vulnerable
- Empathetic
- Decisive
“As leaders, we can share that we are going through these challenges as well. For example, everyone has experienced a bad wifi connection and it’s easy to feel embarrassed. But it’s important to remember that it’s ok because it’s showing a vulnerability that we all have and are experiencing.”
Nick Mehta, CEO of Gainsight
Become the “Chief Empathy Officer”
As a people-first leader, your job isn’t to be the “Chief Cheerleader” but to understand that everyone has unique situations and perspectives. Being empathetic about where your people are at right now is key to your success as a leader. You have to be conscious of how you deliver information and how people perceive it. Emphasizing the need for your workers to engage in self-care and leading by example is necessary for the future of the business.
“We owe it to our customers and employees to give our best, so we need to be successful at balancing extreme work/not extreme work. For our sanity, we implemented ‘Self-Care’ days across the company to balance the extreme work efforts that Sisensers naturally obtain, so we don’t burn ourselves out.”
Amir Orad, CEO at Sisense
Balance your decisions and consider key trade-offs
With people’s wellbeings and livelihoods at stake, it’s important to understand the “trade-off” of being empathetic, while still setting your company up for business success with extra preparation. Having open discussions with your management team on how to push their teams to perform at their highest potential, while being cognizant of the personal issues that everyone might be dealing with is important.
“The trade-offs are always challenging because what’s at stake is people’s livelihoods, long-term and short-term success, and balancing people’s well being and health without being overburned with what’s going on.”
Amir Orad, Chief Executive Officer at Sisense
Maximizing performance during crisis
Understanding that there are things you can and can’t control is crucial in times of crisis. While you can control the quality of presentations, the amount of outreach, and the research and preparation going into demos, you can’t control your customers’ and prospects’ new decision processes post-COVID-19.
“We’re doing more account reviews, more celebration of the pipeline, and more all-around around preparation to set ourselves up for success post-COVID-19.”
Nick Mehta, CEO of Gainsight
Shifting focus: Finding new ways to give back
When approaching new initiatives in the COVID-era, shifting energy to focus on ways you can help your customers survive, and new ways to connect with them in this new reality is vital. With more companies using analytics to pivot during the pandemic, Sisense has taken the initiative to offer a COVID-19 relief package for customers.
“During the COVID-era, we have shifted more energy into thinking about the community and helping institutions fight COVID with free products and offerings.”
Amir Orad, Chief Executive Officer at Sisense
Focusing on the future and innovation, while adapting to the present, starts with people-first leadership. To strive and thrive in times of crisis you need to balance your company’s need for financial durability with a focus on the health and well being of your employees and maintaining customer-obsession.
For more tips on people-first leadership during crisis, watch the full on-demand webinar.
Nurit Shiber leads our organizational strategy, operations and developing our people and culture. Previously, Nurit was the VP of Human Resources at NICE Actimize where she helped double the employee base worldwide while building strong leadership and organizational foundations. Nurit is best known for cultivating strong company culture, building human-centric people functions and building teams to help companies scale.